Python Tutorial
Cross Validation
Estimate model performance more reliably by testing on every part of the data in turn.
Why Not Just One Split?
A single train/test split gives one score that depends on which rows landed in the test set — unlucky splits mislead you, especially on small datasets. Cross-validation rotates the test set across the whole dataset and averages the results, giving a more stable estimate plus a sense of its variability.
K-Fold Cross Validation
K-fold splits the data into k equal parts. Each part serves as the test set once while the other k−1 parts train the model. You get k scores to average.
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X, y = load_iris(return_X_y=True)
model = LogisticRegression(max_iter=1000)
scores = cross_val_score(model, X, y, cv=5, scoring="accuracy")
print("fold scores:", scores.round(3))
print(f"mean: {scores.mean():.3f} std: {scores.std():.3f}")Report the mean ± standard deviation: a small std means the estimate is trustworthy.
Stratified K-Fold for Classification
For classification, StratifiedKFold keeps each class's proportion the same in every fold — the default for classifiers in cross_val_score. Explicit control:
from sklearn.model_selection import StratifiedKFold, cross_val_score
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf)
print(scores.mean().round(3))Choosing a CV Strategy
| Strategy | Use when |
|---|---|
| KFold | General regression, i.i.d. rows |
| StratifiedKFold | Classification, especially imbalanced classes |
| LeaveOneOut | Very small datasets (k = n) |
| GroupKFold | Rows share a group (same patient/user) that must not split across folds |
| TimeSeriesSplit | Temporal data — always train on the past, test on the future |
Never use plain KFold on time series: shuffling lets the model "see the future" and inflates scores. Use TimeSeriesSplit.
Preventing Leakage with Pipelines
Preprocessing must be re-fit inside every fold. Wrapping it in a pipeline makes cross-validation do this automatically.
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
print(cross_val_score(pipe, X, y, cv=5).mean().round(3))Nested Cross Validation
When you both tune hyperparameters and estimate performance, use nested CV: an inner loop selects hyperparameters, an outer loop measures generalization. This avoids the optimistic bias of tuning and testing on the same folds.
from sklearn.model_selection import GridSearchCV, cross_val_score
grid = GridSearchCV(LogisticRegression(max_iter=1000),
{"C": [0.1, 1, 10]}, cv=5)
nested = cross_val_score(grid, X, y, cv=5) # outer loop
print("unbiased estimate:", nested.mean().round(3))Best Practices
- Prefer 5- or 10-fold CV over a single split for small/medium data.
- Stratify for classification; respect groups and time order when present.
- Always cross-validate the whole pipeline, not just the estimator.
- Report mean and standard deviation, and use nested CV when tuning.
Try It Yourself
Exercise 1: Run 5-fold cross-validation for logistic regression on iris and print the mean score.
Show solution
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X, y = load_iris(return_X_y=True)
s = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=5)
print(s.mean().round(3))Exercise 2: Why is plain k-fold wrong for time-series data?
Show solution
Shuffling lets the model train on future data and test on the past, leaking information. Use TimeSeriesSplit instead.
📘 Real-World Deep Dive
Knowing <strong>ML Cross Validation (scikit-learn)</strong> well is what turns scikit-learn from a curiosity into a daily tool — you'll reach for it in nearly every real project.
Real-Life Scenario
An end-to-end usage of ML Cross Validation that you'd actually see in a data pipeline or analytics notebook.
Real-Life Example
import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
scores = cross_val_score(LogisticRegression(max_iter=200), X, y, cv=5)
print("mean:", round(scores.mean(), 3), "std:", round(scores.std(), 3))Expected Output
(see source)Common mistakes
fitexpects numeric arrays;OneHotEncoder/LabelEncoderare easy to forget for categorical features.- Calling
predicton a model trained on unscaled data and then scaling inputs at inference time silently degrades accuracy. train_test_split(X, y)requires both arrays;train_test_split(X)for unsupervised learning slips past static checkers.- Treating ML Cross Validation as a black box without reading the docs — the API has subtle defaults that bite when you scale.
🚀 Performance & Best Practices
- Wrap preprocessing + estimator in a
Pipelinesofit/predictstay reproducible. - Use
joblib/picklefor serialising models, not the entire Python state. - For > 100 k rows, switch to
HistGradientBoostingClassifierorcuML. - When working with scikit-learn, prefer vectorised / batched operations over Python loops.
🧪 Try It Yourself
- Reproduce the snippet on a representative slice of your own data.
- Profile the snippet with
cProfileortimeitand find the single biggest improvement. - Generalise the snippet into a small, reusable function you can drop into future projects.