Univariate feature scoring for feature selection.
Each function scores every column of X independently against the target and returns one number per feature, so the columns can be ranked and the weakest dropped before a model ever sees them. Because the scoring is univariate, these metrics are fast but blind to interactions: a feature that is useless alone yet informative in combination will score low.
Which one to reach for:
-
chi2-- non-negative / count features against a class label.
-
f_classif-- continuous features against a class label (ANOVA F). -
f_regression-- continuous features against a continuous target. -
correlation-- absolute Pearson correlation; linear relationships only. -
single_rule_score-- accuracy of a one-rule classifier built on the feature
-
relief_f-- nearest-neighbour based; the only one here that is
The statistical tests return a (scores, pvalues) pair; the ranking-style scorers return scores alone.
python
>>> from tuiml.datasets import load_iris
>>> from tuiml.evaluation.metrics import f_classif
>>> X, y = load_iris()
>>> scores, pvalues = f_classif(X, y)
>>> int(scores.argmax()) # petal length is the most discriminative
2Functions
chi2(X: np.ndarray, y: np.ndarray) -> Tuple[np.ndarray, np.ndarray]
Compute chi-squared statistics between each feature and the class.
Parameters
X
ndarray of shape (n_samples, n_features)
Feature matrix (must be non-negative).
y
ndarray of shape (n_samples,)
Target values.
Returns
chi2_scores
ndarray of shape (n_features,)
Chi-squared statistic for each feature.
pvalues
ndarray of shape (n_features,)
p-values corresponding to the chi-squared statistics.
Raises
ValueError
If any value in
X is negative.
python
>>> from tuiml.datasets import load_iris
>>> from tuiml.evaluation.metrics import chi2
>>> X, y = load_iris()
>>> scores, pvalues = chi2(X, y)
>>> [round(float(v), 1) for v in scores]
[134.4, 78.0, 244.0, 241.7]
>>> int(scores.argmax())
2f_classif(X: np.ndarray, y: np.ndarray) -> Tuple[np.ndarray, np.ndarray]
Compute ANOVA F-value between each feature and the class.
Parameters
X
ndarray of shape (n_samples, n_features)
Feature matrix.
y
ndarray of shape (n_samples,)
Target values (class labels).
Returns
f_scores
ndarray of shape (n_features,)
F-statistic for each feature.
pvalues
ndarray of shape (n_features,)
p-values associated with the F-statistic.
python
>>> from tuiml.datasets import load_iris
>>> from tuiml.evaluation.metrics import f_classif
>>> X, y = load_iris()
>>> scores, pvalues = f_classif(X, y)
>>> [round(float(v)) for v in scores]
[119, 47, 1179, 959]f_regression(X: np.ndarray, y: np.ndarray) -> Tuple[np.ndarray, np.ndarray]
Compute F-statistic and p-value for regression on each feature.
Parameters
X
ndarray of shape (n_samples, n_features)
Feature matrix.
y
ndarray of shape (n_samples,)
Target values.
Returns
f_scores
ndarray of shape (n_features,)
F-statistic for each feature.
pvalues
ndarray of shape (n_features,)
p-values associated with the F-statistic.
Notes
Use this when the target is continuous. For a class label use
f_classif instead. The statistic tests a LINEAR relationship, so a strong non-linear dependence can still score near zero.
python
>>> from tuiml.datasets import load_iris
>>> from tuiml.evaluation.metrics import f_regression
>>> X, y = load_iris()
>>> scores, pvalues = f_regression(X, y.astype(float))
>>> [round(float(v)) for v in scores]
[234, 32, 1342, 1590]correlation(X: np.ndarray, y: np.ndarray) -> np.ndarray
Compute Pearson correlation coefficient between each feature and the target.
Parameters
X
ndarray of shape (n_samples, n_features)
Feature matrix.
y
ndarray of shape (n_samples,)
Target values.
Returns
scores
ndarray of shape (n_features,)
Absolute Pearson correlation coefficient for each feature, in [0, 1]. The sign is discarded, so a strong negative correlation ranks as highly as a strong positive one.
Notes
Measures LINEAR association only: a feature related to the target through a curve can score near zero.
relief_f and single_rule_score pick up relationships this misses.
python
>>> from tuiml.datasets import load_iris
>>> from tuiml.evaluation.metrics import correlation
>>> X, y = load_iris()
>>> scores = correlation(X, y.astype(float))
>>> [round(float(v), 2) for v in scores]
[0.78, 0.42, 0.95, 0.96]single_rule_score(X: np.ndarray, y: np.ndarray, n_bins: int=10) -> np.ndarray
Evaluate features by the accuracy of a one-rule classifier built on each.
Parameters
X
ndarray of shape (n_samples, n_features)
Feature matrix.
y
ndarray of shape (n_samples,)
Target values.
n_bins
int
= 10
Number of bins for discretizing continuous features.
Returns
scores
ndarray of shape (n_features,)
One-rule accuracy for each feature, between 0 and 1.
Notes
Builds a one-rule classifier on each feature alone -- discretize, then predict the majority class of the bin a sample falls in -- and reports its training accuracy. Unlike
correlation this captures non-linear structure, provided it is axis-aligned. Scores are not comparable across datasets with different class balance, since the floor is the majority-class rate.
python
>>> from tuiml.datasets import load_iris
>>> from tuiml.evaluation.metrics import single_rule_score
>>> X, y = load_iris()
>>> scores = single_rule_score(X, y)
>>> [round(float(v), 2) for v in scores]
[0.73, 0.59, 0.91, 0.9]relief_f(X: np.ndarray, y: np.ndarray, n_neighbors: int=10, n_samples: int=..., random_state: Optional[int]=None) -> np.ndarray
Compute ReliefF scores for each feature.
Parameters
X
ndarray of shape (n_samples, n_features)
Feature matrix.
y
ndarray of shape (n_samples,)
Target values.
n_neighbors
int
= 10
Number of nearest neighbors to consider.
n_samples
int
= -1
Number of instances to sample (-1 for all).
random_state
int
Random seed for reproducibility.
Returns
scores
ndarray of shape (n_features,)
ReliefF score for each feature. Higher is better; a score can be negative for a feature that separates samples of the same class.
Notes
For each sampled instance ReliefF looks at its nearest neighbours of the same class (near hits) and of other classes (near misses), rewarding features that differ on misses and agree on hits. Because it works from neighbourhoods rather than one column at a time, it is the only scorer in this module sensitive to feature INTERACTIONS -- at the cost of a distance computation over the sampled instances.
python
>>> from tuiml.datasets import load_iris
>>> from tuiml.evaluation.metrics import relief_f
>>> X, y = load_iris()
>>> scores = relief_f(X, y, random_state=0)
>>> len(scores)
4
>>> int(scores.argmax())
3