AUC - ROC Curve

Beyond a Single Threshold

Accuracy fixes the threshold at 0.5. But a probabilistic classifier can use any threshold, trading false positives against false negatives. The ROC curve plots this trade-off across all thresholds, and AUC (Area Under the Curve) summarizes it in one number.

• True Positive Rate (recall) = TP / (TP + FN) — on the y-axis.
• False Positive Rate = FP / (FP + TN) — on the x-axis.

Reading AUC

AUC is the probability that the model ranks a random positive example higher than a random negative one. It ignores the threshold and, unlike accuracy, is not fooled by class imbalance.

Computing AUC and Plotting the ROC

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, RocCurveDisplay

X, y = make_classification(n_samples=1000, weights=[0.9, 0.1], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42)

model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
probs = model.predict_proba(X_test)[:, 1]   # probability of the positive class

print("AUC:", round(roc_auc_score(y_test, probs), 3))

RocCurveDisplay.from_predictions(y_test, probs)
plt.plot([0, 1], [0, 1], "k--", label="random")
plt.legend()
plt.show()

Why It Beats Accuracy on Imbalanced Data

With 99% negatives, a model that always predicts "negative" scores 99% accuracy while being useless. Its AUC would be 0.5, exposing the failure. That is why AUC is the standard metric for fraud, disease, and churn detection.

Precision-Recall Curve for Rare Positives

When the positive class is very rare and you care mostly about it, the Precision-Recall curve and its average precision (AP) are often more informative than ROC-AUC.

from sklearn.metrics import average_precision_score, PrecisionRecallDisplay

print("Average precision:", round(average_precision_score(y_test, probs), 3))
PrecisionRecallDisplay.from_predictions(y_test, probs)
plt.show()

Choosing an Operating Threshold

AUC evaluates every threshold, but in production you must pick one. Select it from the ROC/PR curve based on the costs of false positives vs false negatives.

from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_test, probs)
# Youden's J: threshold that maximizes TPR - FPR
best = thresholds[np.argmax(tpr - fpr)]
print("suggested threshold:", round(best, 3))

Best Practices

• Use probabilities/scores for AUC, never hard labels.
• Prefer AUC (or AP) over accuracy for imbalanced classification.
• Use the Precision-Recall curve when the positive class is rare and central.
• Choose the deployment threshold from business costs, not a default 0.5.