Feature Scaling

Why Scaling Matters

Many algorithms compare features by magnitude. If one column is measured in kilograms (0–100) and another in cubic centimetres (1000–5000), the larger-valued feature dominates distance calculations and gradient steps. Scaling rewrites every feature onto a shared range so each contributes on its own merit.

Scaling is essential for k-nearest neighbours, k-means, SVMs, PCA, and any gradient-descent model (linear/logistic regression, neural networks). Tree-based models (decision trees, random forests, gradient boosting) split on thresholds and are not sensitive to scale.

Standardization (Z-score)

Standardization rescales a feature to have a mean of 0 and a standard deviation of 1 using the formula z = (x - mean) / std. It is the most common default.

import numpy as np
from sklearn.preprocessing import StandardScaler

X = np.array([[790, 5.6], [1160, 6.4], [820, 5.4], [1440, 6.7]])

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

print(X_scaled.round(2))
# each column now has mean ~0 and std ~1
print("means:", X_scaled.mean(axis=0).round(2))
print("stds: ", X_scaled.std(axis=0).round(2))

Min-Max Normalization

Min-max scaling squeezes values into a fixed range, usually 0 to 1, with x_scaled = (x - min) / (max - min). Use it when you need bounded inputs (e.g. image pixels, some neural nets).

from sklearn.preprocessing import MinMaxScaler

mm = MinMaxScaler()
X_mm = mm.fit_transform(X)
print(X_mm.round(2))   # every column ranges from 0 to 1

Fit on Train, Transform on Test

The single most important rule: learn the scaling parameters (mean, std, min, max) from the training set only, then apply them to the test set. Fitting on the full dataset leaks information from the test set into training and inflates your scores.

from sklearn.model_selection import train_test_split

X_train, X_test = train_test_split(X, test_size=0.25, random_state=42)

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)  # fit + transform
X_test_s  = scaler.transform(X_test)       # transform only

Predict on New, Scaled Data

Because the scaler stores the parameters it learned, you must scale new samples the same way before predicting.

from sklearn.linear_model import LinearRegression

y = np.array([120, 145, 110, 175])
model = LinearRegression().fit(scaler.fit_transform(X), y)

new_car = [[1000, 6.0]]
new_scaled = scaler.transform(new_car)
print(model.predict(new_scaled))

Choosing a Scaler

Advanced: Scaling Inside a Pipeline

Wrapping the scaler and model in a Pipeline guarantees that every cross-validation fold re-fits the scaler on just its training portion — the correct, leak-free way to tune.

from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier

pipe = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
scores = cross_val_score(pipe, X, y, cv=5)
print("CV accuracy:", scores.mean().round(3))

Best Practices

• Scale features for distance- and gradient-based models; skip it for tree ensembles.
• Always fit the scaler on training data only, then transform train and test.
• Use RobustScaler when outliers would distort the mean or range.
• Persist the fitted scaler (with joblib) alongside the model for production.