API Reference / evaluation / metrics /

classification.py

Classification evaluation metrics.

Scoring functions for models that predict a discrete class label. The module is organised around four groups:

Contingency summaries — confusion_matrix, classification_report and the raw cell counts (num_true_positives and friends) describe how a model is wrong, not just how often. Threshold-free scores — accuracy_score, balanced_accuracy_score, precision_score, recall_score, f1_score, matthews_corrcoef and cohen_kappa_score summarise a single set of hard predictions. Ranking scores — roc_curve, roc_auc_score, precision_recall_curve and average_precision_score work on continuous scores or probabilities and evaluate a model across all decision thresholds at once. Losses — log_loss, hamming_loss and zero_one_loss are lower-is-better quantities suitable for model selection.

On imbalanced data prefer balanced_accuracy_score, f1_score, matthews_corrcoef or average_precision_score over plain accuracy, which a majority-class predictor can trivially inflate.

Multiclass averaging is controlled by the average keyword shared by precision_score, recall_score and f1_score: 'binary' (default) scores only pos_label, 'macro' gives every class equal weight, 'weighted' weights by class support, and None returns the per-class array.

Call signatures follow the conventional metric(y_true, y_pred) ordering, so the functions drop into any evaluation loop unchanged.

Functions

Func

confusion_matrix

Line 48
confusion_matrix(y_true: np.ndarray, y_pred: np.ndarray, labels: Optional[np.ndarray]=None, normalize: Optional[str]=None) -> np.ndarray

Compute the confusion matrix of a classification result.

Entry C_{ij} counts the samples whose true class is labels[i] and whose predicted class is labels[j]. The diagonal therefore holds the correct predictions and every off-diagonal cell names a specific confusion.

For the binary case with labels sorted ascending, the layout is: 

                predicted 0    predicted 1
actual 0            TN             FP
actual 1            FN             TP

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
labels
np.ndarray of shape (n_classes,)
Label values, in the order they should index the rows and columns. When None the sorted union of the labels appearing in y_true and y_pred is used. Samples whose true or predicted label is absent from labels are skipped.
normalize
{'true', 'pred', 'all'}
Normalisation applied to the counts: 'true' divides each row by its sum (per-class recall), 'pred' divides each column by its sum (per-class precision), 'all' divides by the grand total. None (default) returns raw integer counts.

Returns

cm
np.ndarray of shape (n_classes, n_classes)
Confusion matrix. dtype is int64 when normalize is None and float64 otherwise.

Notes

Complexity: O(n) time, O(k^2) memory for k classes.

When to use: whenever a single accuracy number is not enough — the matrix is the only view that tells you which classes a model trades off against each other.

See Also

classification_report

Per-class precision/recall/F1 built on this matrix.
accuracy_score

Scalar summary of the diagonal.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import confusion_matrix
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> confusion_matrix(y_true, y_pred).tolist()
[[2, 0], [1, 1]]
>>> confusion_matrix(y_true, y_pred, normalize='all').tolist()
[[0.5, 0.0], [0.25, 0.25]]
Func

accuracy_score

Line 146
accuracy_score(y_true: np.ndarray, y_pred: np.ndarray, normalize: bool=True, sample_weight: Optional[np.ndarray]=None) -> float

Compute the accuracy of a classification result.

Accuracy is the fraction (or, with normalize=False, the count) of samples whose predicted label matches the true label.

\text{accuracy} = \frac{1}{n} \sum_{i=1}^{n} \mathbb{1}[y_i = \hat{y}_i]

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
normalize
bool = True
If True return the fraction of correct predictions. If False return the (possibly weighted) number of correct predictions.
sample_weight
np.ndarray of shape (n_samples,)
Per-sample weights. When given, correct predictions are summed with these weights and, if normalize=True, divided by the total weight.

Returns

score
float
Accuracy in :math:`[0, 1]` when normalize=True, otherwise a count.

Notes

Complexity: O(n) time, O(n) memory.

When to use: only when the classes are roughly balanced and every kind of error costs the same. On skewed data a constant majority-class predictor can score very high accuracy while being useless — reach for balanced_accuracy_score, f1_score or matthews_corrcoef instead.

See Also

balanced_accuracy_score

Accuracy that gives every class equal weight.
zero_one_loss

1 - accuracy.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import accuracy_score
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> accuracy_score(y_true, y_pred)
0.75
>>> accuracy_score(y_true, y_pred, normalize=False)
3.0
Func

balanced_accuracy_score

Line 222
balanced_accuracy_score(y_true: np.ndarray, y_pred: np.ndarray, adjusted: bool=False) -> float

Compute the balanced accuracy: the mean per-class recall.

Each class contributes equally regardless of how many samples it has, so a majority-class predictor scores 1/k rather than the majority frequency.

\text{balanced accuracy} = \frac{1}{k} \sum_{c=1}^{k} \frac{\text{TP}_c}{\text{TP}_c + \text{FN}_c}

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
adjusted
bool = False
If True, rescale the result so that random guessing scores 0.0 by applying :math:`(b - 1/k) / (1 - 1/k)`. Perfect prediction still scores 1.0 and the adjusted score can go negative.

Returns

score
float
Balanced accuracy in :math:`[0, 1]`, or in :math:`[-1/(k-1), 1]` when adjusted=True.

Notes

Complexity: O(kn) time for k classes, O(n) memory.

When to use: the default replacement for accuracy_score on imbalanced problems, and the natural choice when every class matters equally regardless of its frequency.

See Also

accuracy_score

Unweighted, support-dominated accuracy.
recall_score

Per-class recall, which this averages.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import balanced_accuracy_score
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> balanced_accuracy_score(y_true, y_pred)
0.75
>>> balanced_accuracy_score(y_true, y_pred, adjusted=True)
0.5
Func

precision_score

Line 378
precision_score(y_true: np.ndarray, y_pred: np.ndarray, **kwargs) -> Union[float, np.ndarray]

Compute the precision: how many predicted positives are correct.

\text{precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}

Precision is the metric to optimise when a false positive is expensive — flagging a legitimate transaction as fraud, or a healthy patient as sick.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
**kwargs
dict
Forwarded to _precision_recall_fscore_support. Recognised keys: average : {'binary', 'macro', 'weighted', 'micro'} or None, default='binary' How to reduce the per-class scores. 'binary' returns the score of pos_label only; 'macro' averages classes equally; 'weighted' averages by support; None returns the per-class array. pos_label : int, default=1 Class treated as positive when average='binary'. labels : np.ndarray, optional Label values and their order. zero_division : float, default=0.0 Value used when TP + FP == 0.

Returns

score
float or np.ndarray of shape (n_classes,)
Precision in :math:`[0, 1]`; an array when average=None.

Notes

Complexity: O(kn) time for k classes.

When to use: pair it with recall_score — precision alone is trivial to maximise by predicting the positive class almost never.

See Also

recall_score

The complementary error type.
f1_score

Harmonic mean of the two.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import precision_score
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> precision_score(y_true, y_pred)
1.0
>>> round(precision_score(y_true, y_pred, average='macro'), 4)
0.8333
Func

recall_score

Line 442
recall_score(y_true: np.ndarray, y_pred: np.ndarray, **kwargs) -> Union[float, np.ndarray]

Compute the recall: how many actual positives were found.

\text{recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}

Recall is the metric to optimise when a missed positive is expensive — an undiagnosed disease, or an undetected intrusion.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
**kwargs
dict
Forwarded to _precision_recall_fscore_support. Recognised keys are average, pos_label, labels and zero_division; see precision_score for their meaning.

Returns

score
float or np.ndarray of shape (n_classes,)
Recall in :math:`[0, 1]`; an array when average=None.

Notes

Complexity: O(kn) time for k classes.

When to use: alongside precision_score. Recall alone is trivial to maximise by predicting everything positive.

See Also

precision_score

The complementary error type.
true_positive_rate

Same quantity, ROC naming.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import recall_score
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> recall_score(y_true, y_pred)
0.5
Func

f1_score

Line 494
f1_score(y_true: np.ndarray, y_pred: np.ndarray, **kwargs) -> Union[float, np.ndarray]

Compute the F1 score: the harmonic mean of precision and recall.

F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}} {\text{precision} + \text{recall}}

The harmonic mean is deliberately unforgiving: it stays near the smaller of the two, so a model cannot score well by sacrificing one for the other.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
**kwargs
dict
Forwarded to _precision_recall_fscore_support. Recognised keys are average, pos_label, labels, beta and zero_division; see precision_score for their meaning.

Returns

score
float or np.ndarray of shape (n_classes,)
F1 score in :math:`[0, 1]`; an array when average=None.

Notes

Complexity: O(kn) time for k classes.

When to use: the standard single number for an imbalanced binary problem where the positive class is the one you care about. It ignores true negatives entirely — if those matter, use matthews_corrcoef.

See Also

precision_score

One half of the mean.
recall_score

The other half.
matthews_corrcoef

Balanced alternative that uses all four cells.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import f1_score
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> round(f1_score(y_true, y_pred), 4)
0.6667
>>> round(f1_score(y_true, y_pred, average='weighted'), 4)
0.7333
Func

true_positive_rate

Line 591
true_positive_rate(y_true, y_pred, pos_label=1)

Compute the true positive rate (TPR, sensitivity, recall).

The fraction of actual positives that the model correctly flags; it is the y-axis of an ROC curve.

\text{TPR} = \frac{\text{TP}}{\text{TP} + \text{FN}}

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
pos_label
int = 1
Label treated as the positive class.

Returns

rate
float
True positive rate in :math:`[0, 1]`.

See Also

false_positive_rate

ROC x-axis.
recall_score

Identical quantity, precision/recall naming.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import true_positive_rate
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> true_positive_rate(y_true, y_pred)
0.5
Func

false_positive_rate

Line 630
false_positive_rate(y_true, y_pred, pos_label=1)

Compute the false positive rate (FPR, fall-out).

The fraction of actual negatives that the model wrongly flags as positive; it is the x-axis of an ROC curve.

\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}} = 1 - \text{TNR}

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
pos_label
int = 1
Label treated as the positive class.

Returns

rate
float
False positive rate in :math:`[0, 1]`; 0.0 when there are no negatives.

See Also

true_positive_rate

ROC y-axis.
roc_curve

Traces both rates over all thresholds.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import false_positive_rate
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> false_positive_rate(y_true, y_pred)
0.0
Func

matthews_corrcoef

Line 672
matthews_corrcoef(y_true, y_pred)

Compute the Matthews correlation coefficient (MCC).

MCC is the correlation between the true and predicted binary labels. Unlike F1 it uses all four cells of the confusion matrix, so it cannot be inflated by a model that simply predicts the majority class.

\text{MCC} = \frac{\text{TP} \cdot \text{TN} - \text{FP} \cdot \text{FN}} {\sqrt{(\text{TP}+\text{FP})(\text{TP}+\text{FN}) (\text{TN}+\text{FP})(\text{TN}+\text{FN})}}

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.

Returns

score
float
MCC in :math:`[-1, 1]`: 1.0 perfect agreement, 0.0 no better than chance, -1.0 total disagreement. Returns 0.0 when the denominator vanishes.

Notes

Complexity: O(n) time.

Only the binary case is implemented; for three or more classes this function currently returns 0.0 rather than the multiclass generalisation.

When to use: the most informative single number for an imbalanced binary problem, and the safest default when you are unsure which errors matter.

References

Matthews1975
Matthews, B. W. (1975). "Comparison of the predicted and observed secondary structure of T4 phage lysozyme." Biochimica et Biophysica Acta, 405(2), 442-451. :doi:`10.1016/0005-2795(75)90109-9`

See Also

f1_score

Ignores true negatives.
cohen_kappa_score

Another chance-corrected agreement score.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import matthews_corrcoef
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> round(matthews_corrcoef(y_true, y_pred), 4)
0.5774
Func

cohen_kappa_score

Line 740
cohen_kappa_score(y_true, y_pred)

Compute Cohen's kappa: accuracy corrected for chance agreement.

Kappa compares the observed agreement p_o against the agreement p_e you would expect if the two labellings were independent with the same marginals.

\kappa = \frac{p_o - p_e}{1 - p_e}

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels (or the first rater's labels).
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels (or the second rater's labels).

Returns

score
float
Kappa, at most 1.0. 1.0 is perfect agreement, 0.0 is chance-level agreement, and negative values mean worse than chance. Returns 0.0 when :math:`p_e = 1`.

Notes

Complexity: O(n + k^2) time for k classes.

When to use: whenever a high raw accuracy might just reflect a skewed class distribution, and for inter-annotator agreement — kappa is symmetric in its two arguments. It supports multiclass input, unlike matthews_corrcoef.

References

Cohen1960
Cohen, J. (1960). "A Coefficient of Agreement for Nominal Scales." Educational and Psychological Measurement, 20(1), 37-46. :doi:`10.1177/001316446002000104`

See Also

accuracy_score

The uncorrected :math:`p_o`.
matthews_corrcoef

Binary-only correlation alternative.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import cohen_kappa_score
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> cohen_kappa_score(y_true, y_pred)
0.5
Func

roc_auc_score

Line 866
roc_auc_score(y_true, y_score, average='macro', labels=None)

Compute the area under the ROC curve (AUC).

AUC is the probability that a randomly chosen positive sample is ranked above a randomly chosen negative one, so it evaluates a model's ranking across every decision threshold at once rather than at one fixed cut-off.

\text{AUC} = P(\hat{s}(x^{+}) > \hat{s}(x^{-}))

Binary input accepts a 1-D score vector for the positive class. Multiclass input accepts a probability/score matrix of shape (n_samples, n_classes) and computes one-vs-rest AUC for each class.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_score
np.ndarray of shape (n_samples,) or (n_samples, n_classes)
Continuous scores or probabilities. A 1-D vector is interpreted as the score of the positive class; a 2-D matrix must have one column per entry of labels.
average
{'macro', 'weighted'} or None = 'macro'
How to reduce the per-class one-vs-rest AUCs in the multiclass case. None returns the per-class array. Ignored for 1-D y_score.
labels
np.ndarray of shape (n_classes,)
Label values and the column order of a 2-D y_score. Defaults to np.unique(y_true).

Returns

score
float or np.ndarray of shape (n_classes,)
AUC in :math:`[0, 1]`. 0.5 is random ranking and 1.0 is perfect separation. An array is returned when average=None.

Raises

ValueError
If a 1-D y_score is given for more than two classes, if y_score is neither 1-D nor 2-D, or if its shape disagrees with y_true or labels.

Notes

Complexity: O(n \log n) time, dominated by the sort inside _binary_roc_curve.

When to use: for comparing rankers and probabilistic models independently of the operating threshold. On heavily imbalanced data AUC can look optimistic because the false positive rate has a very large denominator — prefer average_precision_score there.

See Also

roc_curve

The curve this integrates.
auc

Generic trapezoidal integration.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import roc_auc_score
>>> y_true = np.array([0, 0, 1, 1])
>>> y_score = np.array([0.1, 0.4, 0.35, 0.8])
>>> roc_auc_score(y_true, y_score)
0.75
Func

precision_recall_fscore_support

Line 975
precision_recall_fscore_support(y_true, y_pred, beta=1.0, labels=None, pos_label=1, average=None, zero_division=0.0)

Compute precision, recall, F-measure and support in a single pass.

Returns all four quantities from one traversal of the data, which is both cheaper and more consistent than calling precision_score, recall_score and f1_score separately.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
beta
float = 1.0
Weight of recall relative to precision in the F-score. beta > 1 favours recall, beta < 1 favours precision.
labels
np.ndarray of shape (n_classes,)
Label values and their order. Defaults to the sorted union of the labels present in y_true and y_pred.
pos_label
int = 1
Class scored when average='binary'.
average
{'binary', 'macro', 'weighted', 'micro'} or None = None
How to reduce the per-class scores. None (default) keeps the per-class arrays.
zero_division
float = 0.0
Value substituted when a denominator is zero.

Returns

precision
float or np.ndarray of shape (n_classes,)
Precision, reduced according to average.
recall
float or np.ndarray of shape (n_classes,)
Recall, reduced according to average.
fscore
float or np.ndarray of shape (n_classes,)
F-beta score, reduced according to average.
support
float or np.ndarray of shape (n_classes,)
Number of true instances per class, or their total when average is not None.

Notes

Complexity: O(kn) time for k classes.

When to use: building a report, or logging several related metrics at once.

See Also

classification_report

Formats these numbers as text.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import precision_recall_fscore_support
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> p, r, f, s = precision_recall_fscore_support(y_true, y_pred)
>>> np.round(p, 4).tolist()
[0.6667, 1.0]
>>> np.round(r, 4).tolist()
[1.0, 0.5]
>>> s.tolist()
[2.0, 2.0]
Func

classification_report

Line 1046
classification_report(y_true, y_pred, labels=None, target_names=None)

Build a text report of the main per-class classification metrics.

Produces a fixed-width table with one row per class (precision, recall, F1 and support) followed by the overall accuracy and the macro average.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
labels
np.ndarray of shape (n_classes,)
Label values and the row order. Defaults to the sorted union of the labels present in y_true and y_pred.
target_names
list of str
Display names for the classes, in the same order as labels. Defaults to str(label).

Returns

report
str
Multi-line, newline-terminated report ready to print.

Notes

Complexity: O(kn) time for k classes.

When to use: for a human-readable summary. Parse precision_recall_fscore_support instead if you need the numbers programmatically.

See Also

precision_recall_fscore_support

The underlying numbers.
confusion_matrix

Shows which classes are confused.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import classification_report
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> print(classification_report(y_true, y_pred).rstrip())  # doctest: +NORMALIZE_WHITESPACE
          class  precision     recall   f1-score    support
              0     0.6667     1.0000     0.8000          2
              1     1.0000     0.5000     0.6667          2
------------------------------------------------------------
       accuracy                           0.7500          4
      macro avg     0.8333     0.7500     0.7333          4
Func

fbeta_score

Line 1112
fbeta_score(y_true, y_pred, beta=1.0, **kwargs)

Compute the F-beta score, a precision/recall trade-off.

The F-beta score is the weighted harmonic mean of precision and recall, with beta controlling how much more recall counts than precision.

F_\beta = (1 + \beta^2) \cdot \frac{\text{precision} \cdot \text{recall}} {\beta^2 \cdot \text{precision} + \text{recall}}

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
beta
float = 1.0
Weight of recall relative to precision. Currently ignored: this function delegates to f1_score, so it always behaves as beta=1. Pass beta to precision_recall_fscore_support for a true F-beta score.
**kwargs
dict
Forwarded to f1_score; see precision_score for the recognised keys.

Returns

score
float or np.ndarray of shape (n_classes,)
F-beta score in :math:`[0, 1]`; an array when average=None.

See Also

f1_score

The beta=1 special case.
precision_recall_fscore_support

Honours beta.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import fbeta_score
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> round(fbeta_score(y_true, y_pred), 4)
0.6667
Func

true_negative_rate

Line 1159
true_negative_rate(y_true, y_pred, pos_label=1)

Compute the true negative rate (TNR, specificity).

The fraction of actual negatives that the model correctly leaves unflagged — the mirror image of recall.

\text{TNR} = \frac{\text{TN}}{\text{TN} + \text{FP}}

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
pos_label
int = 1
Label treated as the positive class; everything else is negative.

Returns

rate
float
True negative rate in :math:`[0, 1]`; 0.0 when there are no negatives.

See Also

specificity_score

Alias for this function.
true_positive_rate

The positive-class counterpart.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import true_negative_rate
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> true_negative_rate(y_true, y_pred)
1.0
Func

false_negative_rate

Line 1202
false_negative_rate(y_true, y_pred, pos_label=1)

Compute the false negative rate (FNR, miss rate).

The fraction of actual positives the model misses, i.e. 1 - TPR.

\text{FNR} = \frac{\text{FN}}{\text{TP} + \text{FN}} = 1 - \text{TPR}

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
pos_label
int = 1
Label treated as the positive class.

Returns

rate
float
False negative rate in :math:`[0, 1]`.

See Also

true_positive_rate

The complement of this rate.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import false_negative_rate
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> false_negative_rate(y_true, y_pred)
0.5
Func

sensitivity_score

Line 1239
sensitivity_score(y_true, y_pred, pos_label=1)

Compute the sensitivity, the clinical name for the true positive rate.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
pos_label
int = 1
Label treated as the positive class.

Returns

score
float
Sensitivity in :math:`[0, 1]`.

See Also

true_positive_rate

The function this delegates to.
specificity_score

Reported alongside sensitivity in diagnostics.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import sensitivity_score
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> sensitivity_score(y_true, y_pred)
0.5
Func

specificity_score

Line 1272
specificity_score(y_true, y_pred, pos_label=1)

Compute the specificity, the clinical name for the true negative rate.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
pos_label
int = 1
Label treated as the positive class.

Returns

score
float
Specificity in :math:`[0, 1]`.

See Also

true_negative_rate

The function this delegates to.
sensitivity_score

Reported alongside specificity in diagnostics.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import specificity_score
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> specificity_score(y_true, y_pred)
1.0
Func

num_true_positives

Line 1305
num_true_positives(y_true, y_pred, pos_label=1)

Count the true positives: positives correctly predicted as positive.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
pos_label
int = 1
Label treated as the positive class.

Returns

count
int
Number of true positives.

See Also

confusion_matrix

All four counts at once.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import num_true_positives
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> num_true_positives(y_true, y_pred)
1
Func

num_true_negatives

Line 1337
num_true_negatives(y_true, y_pred, pos_label=1)

Count the true negatives: negatives correctly predicted as negative.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
pos_label
int = 1
Label treated as the positive class; everything else is negative.

Returns

count
int
Number of true negatives.

See Also

confusion_matrix

All four counts at once.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import num_true_negatives
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> num_true_negatives(y_true, y_pred)
2
Func

num_false_positives

Line 1369
num_false_positives(y_true, y_pred, pos_label=1)

Count the false positives: negatives wrongly predicted as positive.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
pos_label
int = 1
Label treated as the positive class; everything else is negative.

Returns

count
int
Number of false positives (type I errors).

See Also

false_positive_rate

The same errors as a rate.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import num_false_positives
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> num_false_positives(y_true, y_pred)
0
Func

num_false_negatives

Line 1401
num_false_negatives(y_true, y_pred, pos_label=1)

Count the false negatives: positives wrongly predicted as negative.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
pos_label
int = 1
Label treated as the positive class.

Returns

count
int
Number of false negatives (type II errors).

See Also

false_negative_rate

The same errors as a rate.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import num_false_negatives
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> num_false_negatives(y_true, y_pred)
1
Func

roc_curve

Line 1433
roc_curve(y_true, y_score, pos_label=1)

Compute the receiver operating characteristic (ROC) curve.

Sweeps the decision threshold from +inf down through every distinct score and records the resulting (FPR, TPR) operating points. Plotting TPR against FPR shows the full trade-off a model offers; the diagonal is random guessing.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_score
np.ndarray of shape (n_samples,)
Continuous score or probability for the positive class.
pos_label
int = 1
Label treated as the positive class.

Returns

fpr
np.ndarray of shape (n_thresholds,)
Increasing false positive rates.
tpr
np.ndarray of shape (n_thresholds,)
Increasing true positive rates.
thresholds
np.ndarray of shape (n_thresholds,)
Decreasing thresholds, the first being inf.

Raises

ValueError
If y_score is not 1-D, the inputs differ in length, or y_true contains only one class.

Notes

Complexity: O(n \log n) time.

When to use: to pick an operating threshold, not just to score a model — the curve tells you what recall costs in false alarms.

See Also

roc_auc_score

Scalar summary of this curve.
precision_recall_curve

Better view on imbalanced data.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import roc_curve
>>> y_true = np.array([0, 0, 1, 1])
>>> y_score = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, tpr, thresholds = roc_curve(y_true, y_score)
>>> fpr.tolist()
[0.0, 0.0, 0.5, 0.5, 1.0]
>>> tpr.tolist()
[0.0, 0.5, 0.5, 1.0, 1.0]
>>> thresholds.tolist()
[inf, 0.8, 0.4, 0.35, 0.1]
Func

auc

Line 1493
auc(x, y)

Compute the area under a curve by the trapezoidal rule.

A generic integrator: given the x and y coordinates of a curve it returns \int y \, dx approximated by trapezoids.

Parameters

x
np.ndarray of shape (n_points,)
x-coordinates, monotonically increasing (e.g. the false positive rates returned by roc_curve).
y
np.ndarray of shape (n_points,)
y-coordinates (e.g. the corresponding true positive rates).

Returns

area
float
Area under the curve. Negative if x is decreasing.

Notes

Complexity: O(n) time.

When to use: to integrate any curve produced elsewhere in this module. roc_auc_score is the convenience wrapper for the ROC case.

See Also

roc_curve

Produces suitable x/y arrays.
roc_auc_score

ROC area in one call.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import auc, roc_curve
>>> y_true = np.array([0, 0, 1, 1])
>>> y_score = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, tpr, _ = roc_curve(y_true, y_score)
>>> auc(fpr, tpr)
0.75
Func

precision_recall_curve

Line 1536
precision_recall_curve(y_true, y_score, pos_label=1)

Compute the precision-recall curve.

Walks the samples in order of decreasing score and records the precision and recall obtained by treating each prefix as the set of positive predictions. Unlike an ROC curve it ignores true negatives, which makes it the informative view when positives are rare.

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_score
np.ndarray of shape (n_samples,)
Continuous score or probability for the positive class.
pos_label
int = 1
Label treated as the positive class.

Returns

precision
np.ndarray of shape (n_samples,)
Precision at each threshold.
recall
np.ndarray of shape (n_samples,)
Recall at each threshold, increasing.
thresholds
np.ndarray of shape (n_samples,)
The scores, sorted in decreasing order, that define the thresholds.

Notes

Complexity: O(n \log n) time, dominated by the sort.

When to use: on imbalanced problems, where a large true-negative pool makes an ROC curve look better than the model really is.

See Also

roc_curve

The balanced-data counterpart.
average_precision_score

Scalar summary of this curve.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import precision_recall_curve
>>> y_true = np.array([0, 0, 1, 1])
>>> y_score = np.array([0.1, 0.4, 0.35, 0.8])
>>> precision, recall, thresholds = precision_recall_curve(y_true, y_score)
>>> np.round(precision, 4).tolist()
[1.0, 0.5, 0.6667, 0.5]
>>> recall.tolist()
[0.5, 0.5, 1.0, 1.0]
Func

average_precision_score

Line 1595
average_precision_score(y_true, y_score, pos_label=1)

Compute the average precision (AP), the area under the PR curve.

AP summarises precision_recall_curve as the precision achieved at each threshold, weighted by the gain in recall it produces.

\text{AP} = \sum_{k} (R_k - R_{k-1}) \, P_k

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_score
np.ndarray of shape (n_samples,)
Continuous score or probability for the positive class.
pos_label
int = 1
Label treated as the positive class.

Returns

score
float
Average precision. The baseline for a random ranker is the positive class prevalence, not 0.5 as it is for ROC AUC.

Notes

Complexity: O(n \log n) time.

When to use: as the headline ranking metric on heavily imbalanced data, where roc_auc_score is dominated by the abundant negatives.

Warnings

The current implementation negates the recall increments, so it returns the negation of the value defined above.

See Also

precision_recall_curve

The curve being summarised.
roc_auc_score

Threshold-free alternative for balanced data.
Func

log_loss

Line 1639
log_loss(y_true, y_pred_proba, eps=1e-15)

Compute the logistic loss (cross-entropy) of predicted probabilities.

Log loss scores a probabilistic classifier: it rewards assigning high probability to the correct class and punishes confident mistakes severely, because the penalty grows without bound as the predicted probability of the true class approaches zero.

L = -\frac{1}{n} \sum_{i=1}^{n} \log p_{i, y_i}

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels, used directly as column indices into y_pred_proba — so they must be integers 0 … n_classes - 1.
y_pred_proba
np.ndarray of shape (n_samples,) or (n_samples, n_classes)
Predicted class probabilities. A 1-D vector is treated as the probability of class 1 and expanded to two columns.
eps
float = 1e-15
Probabilities are clipped to [eps, 1 - eps] so that :math:`\log 0` never occurs.

Returns

loss
float
Mean negative log-likelihood, in nats. Non-negative; lower is better, and 0.0 means every true class was predicted with probability 1.

Notes

Complexity: O(n) time.

When to use: whenever calibrated probabilities matter — ranking or thresholding metrics such as AUC cannot tell a well-calibrated model from an over-confident one with the same ordering.

See Also

zero_one_loss

Ignores confidence entirely.
roc_auc_score

Scores ranking rather than calibration.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import log_loss
>>> y_true = np.array([0, 0, 1, 1])
>>> proba = np.array([[0.9, 0.1], [0.6, 0.4], [0.35, 0.65], [0.2, 0.8]])
>>> round(log_loss(y_true, proba), 4)
0.3175
Func

hamming_loss

Line 1700
hamming_loss(y_true, y_pred)

Compute the Hamming loss: the fraction of mismatched labels.

For single-label input this is simply 1 - accuracy. For multi-label indicator arrays it is the fraction of individual label positions that disagree, which makes it more forgiving than an exact-match criterion.

L_H = \frac{1}{n} \sum_{i=1}^{n} \mathbb{1}[y_i \neq \hat{y}_i]

Parameters

y_true
np.ndarray
Ground-truth labels, of any shape.
y_pred
np.ndarray
Predicted labels, broadcastable to the shape of y_true.

Returns

loss
float
Fraction of positions that disagree, in :math:`[0, 1]`; lower is better.

Notes

Complexity: O(n) time.

When to use: for multi-label problems, where partial credit for getting most labels right is the behaviour you want.

See Also

zero_one_loss

Single-label equivalent.
accuracy_score

1 - hamming_loss for single-label input.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import hamming_loss
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> hamming_loss(y_true, y_pred)
0.25
Func

zero_one_loss

Line 1746
zero_one_loss(y_true, y_pred, normalize=True)

Compute the 0-1 loss: the fraction (or count) of misclassifications.

The direct complement of accuracy_score; it charges 1 for every wrong prediction regardless of how wrong it was.

L_{01} = \frac{1}{n} \sum_{i=1}^{n} \mathbb{1}[y_i \neq \hat{y}_i]

Parameters

y_true
np.ndarray of shape (n_samples,)
Ground-truth class labels.
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
normalize
bool = True
If True return the fraction of misclassifications; if False return their count.

Returns

loss
float
Misclassification rate in :math:`[0, 1]`, or a count when normalize=False. Lower is better.

Notes

Complexity: O(n) time.

When to use: when every error costs the same. If errors have different costs, weight them via confusion_matrix instead.

See Also

accuracy_score

1 - zero_one_loss.
log_loss

Confidence-aware alternative.
python 
>>> import numpy as np
>>> from tuiml.evaluation.metrics import zero_one_loss
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> zero_one_loss(y_true, y_pred)
0.25
>>> zero_one_loss(y_true, y_pred, normalize=False)
1.0