Md. Asif Uddin

Proposition 2III.7.P0272 of 76 in the corpus

A model does not have a sensitivity — a threshold does.

The score is continuous and the decision is binary, so every operating point trades misses against false alarms. ROC shows the whole trade; a deployed system occupies exactly one point on it.

The ROC curve is a sweep, not a scoreA receiver operating characteristic curve, traced from the strict end to the permissive end. Each point on it is one decision threshold with its own sensitivity and specificity. AUROC summarises the whole curve; a deployed system occupies a single point on it.sensitivity against 1 − specificitychancestrict — few alarmspermissive — few missesAUROCis thearea underall of itScreening wants the permissive end and pays in false alarms. Confirmation wants the strict end.
Fig. 2 — The ROC curve traced from strict to permissive. Every point is a threshold; a deployed system occupies exactly one of them.

Demonstration

The definitions, because they are constantly swapped:

sensitivity (recall) = TP / (TP + FN)    of the diseased, how many did we catch
specificity          = TN / (TN + FP)    of the healthy, how many did we clear
precision (PPV)      = TP / (TP + FP)    of our positives, how many were right
F1                   = harmonic mean of precision and recall

Sensitivity and specificity are properties of the model at a threshold. Lower the threshold and sensitivity rises while specificity falls, always. There is no setting that improves both; that would be a better model, not a better threshold.

The ROC curve is that sweep drawn out: sensitivity against 1 − specificity across every threshold. AUROC is the area under it, and it has a clean interpretation — the probability that a randomly chosen diseased case is ranked above a randomly chosen healthy one. It is a summary of ranking, independent of any threshold.

Which is exactly its limitation. Two things AUROC cannot tell you:

Where you will operate. A screening tool wants high sensitivity and accepts false alarms, because a missed cancer is worse than a recall visit. A confirmatory test wants the opposite. Reporting only AUROC leaves the reader to guess which regime the model was built for.

How it behaves when the classes are unbalanced. With 1% prevalence, the false-positive rate is computed against a very large denominator, so a model can look excellent while most of its positive calls are wrong. The precision–recall curve is the honest instrument there, and the next proposition is about why.

Corollary

Report the threshold, and report sensitivity and specificity at it, alongside AUROC. A deployed system does not sweep — it makes one decision with one threshold, and that is the number a clinician needs.

Sources

Depends on

Used by