Start with the decisions and their consequences
Classification models assign inputs to categories: spam or legitimate email, defective or acceptable products, relevant or irrelevant documents. Evaluating them starts with a practical question: which mistakes matter? Blocking a legitimate message and allowing spam into an inbox are both errors, but they have different consequences.
For binary classification, designate one class as positive. A true positive is a positive case correctly identified; a false positive is a negative case incorrectly flagged. A true negative is correctly rejected, while a false negative is a positive case the model misses. These four counts form a confusion matrix.
Suppose a test set contains 1,000 messages, including 100 spam messages. A filter catches 80 spam messages, misses 20, and incorrectly flags 40 legitimate messages. That gives 80 true positives, 20 false negatives, 40 false positives, and 860 true negatives. Each metric summarizes a different aspect of these results.
Accuracy: useful when overall correctness is the goal
Accuracy is the proportion of all predictions that are correct: accuracy = (true positives + true negatives) / total predictions. In the spam example, 940 of 1,000 predictions are correct, so accuracy is 94%. It is intuitive and useful when the evaluation data represents actual usage and mistakes have broadly similar consequences.
However, accuracy can conceal weak performance on a rare class. A filter that labels every message legitimate would achieve 90% accuracy on this dataset while catching no spam. The model’s 94% accuracy therefore needs context, including comparison with that simple baseline.
Class imbalance does not make accuracy mathematically wrong; it can make accuracy insufficient for the task. Always examine class frequencies and the confusion matrix before treating a high score as evidence of a useful model.
Precision and recall answer different questions
Precision asks: of the cases the model labeled positive, how many were actually positive? The formula is precision = true positives / (true positives + false positives). Our filter flags 120 messages, of which 80 are spam, giving approximately 66.7% precision. Precision matters when false alarms create disruption or consume limited review capacity.
Recall asks: of all actual positive cases, how many did the model find? The formula is recall = true positives / (true positives + false negatives). The filter catches 80 of 100 spam messages, giving 80% recall. Recall matters when overlooking positive cases undermines the system’s purpose, such as missing defective items during screening.
Neither metric is sufficient alone. A model can achieve high precision by flagging only a few obvious cases, leaving many positives undetected. It can achieve perfect recall by labeling everything positive, generating many false alarms. Also, precision depends on how common positives are in the evaluated population, so test-set composition matters.
F1 combines precision and recall, with limitations
F1 is the harmonic mean of precision and recall: F1 = 2 × precision × recall / (precision + recall). For the spam filter, F1 is approximately 72.7%. The harmonic mean penalizes a large gap between the two metrics, so high performance on one cannot easily compensate for poor performance on the other.
F1 is useful when both false positives and false negatives matter and you want one summary of positive-class performance. But it ignores true negatives and does not encode the actual operational cost of mistakes. Two models with similar F1 scores can create very different workloads.
When recall deserves more emphasis, an F-beta score with beta greater than 1 is an option; beta below 1 emphasizes precision. These are still summaries, not substitutes for explicit requirements. Document how undefined cases are handled, such as precision when a model predicts no positives.
Choose a threshold that matches the workflow
Many classifiers produce scores that become labels through a decision threshold. Lowering that threshold generally flags more cases, increasing recall while potentially reducing precision. Raising it usually reduces the number of alerts and may improve precision, at the cost of missed positives.
Instead of automatically using a threshold of 0.5, define a practical constraint. For example, maximize recall while maintaining an acceptable precision level, or find the threshold that fits the review team’s capacity. A precision-recall curve helps reveal the available trade-offs, especially when positives are uncommon.
Select thresholds using validation data, then evaluate the final choice on a separate test set. Repeatedly adjusting thresholds against test results weakens the independence of that evaluation. If scores are used as probabilities, assess calibration separately: strong classification metrics do not guarantee reliable probability estimates.
Report a small set of metrics, not one winning number
For multiclass classification, inspect precision, recall, and F1 for each class. Macro averaging gives every class equal weight; weighted averaging accounts for class frequency. Micro averaging pools prediction counts and, in standard single-label multiclass classification across all classes, micro F1 equals accuracy. State the averaging method explicitly.
A useful report combines the primary metric with complementary measures, class counts, the confusion matrix, and the chosen threshold. Check important subgroups and realistic operating conditions, especially where small samples make results uncertain. The best metric is the one that supports the intended decision, not the one that produces the most impressive number.
