Standard Deviation
Measure how widely data points deviate from the mean to understand variability before training models.
Why Standard Deviation Matters
Standard deviation (SD) quantifies the spread of a dataset around its mean. High SD indicates dispersed values, while low SD signals values clustered near the mean. Feature scaling and outlier detection rely on this metric.
Formula
For a population with values xᵢ and mean μ:
σ = √( (1/N) · Σᵢ (xᵢ − μ)² )
For a sample, divide by N − 1 to obtain an unbiased estimator (Bessel's correction).
Computing Standard Deviation
import numpy as np
import pandas as pd
values = np.array([12, 15, 14, 10, 18, 12])
population_sd = values.std(ddof=0)
sample_sd = values.std(ddof=1)
series_sd = pd.Series(values).std() # Uses ddof=1 by default
print(population_sd, sample_sd, series_sd)NumPy's std defaults to population SD (ddof=0). Set ddof=1 for sample SD.
Standard Deviation and Scaling
Feature scaling methods such as standardization subtract the mean and divide by SD to create z-scores. Algorithms like logistic regression and k-means benefit from scaled features.
Best Practices
- Calculate SD after handling missing values and obvious data quality issues.
- Compare SD across groups to detect heteroscedasticity.
- Document whether you used population or sample SD (share conventions via
info.studygrid@gmail.com).
Next Steps
Move on to the percentile tutorial to analyze data ranks and thresholds.
Try It Yourself
Exercise 1: Compute the standard deviation of [10, 12, 23, 23, 16, 23, 21, 16] with NumPy.
Show solution
import numpy as np
print(round(np.std([10, 12, 23, 23, 16, 23, 21, 16]), 2)) # 4.9Exercise 2: Two datasets have the same mean but different SDs. What does a larger SD tell you?
Show solution
A larger standard deviation means the values are more spread out around the mean — greater variability and less consistency.
Key Takeaways
- Standard deviation measures spread around the mean.
- Variance is its square; both quantify variability.
- Low SD = consistent data; high SD = spread out.
📘 Real-World Deep Dive
Knowing <strong>ML Standard Deviation (scikit-learn)</strong> well is what turns scikit-learn from a curiosity into a daily tool — you'll reach for it in nearly every real project.
Real-Life Scenario
An end-to-end usage of ML Standard Deviation that you'd actually see in a data pipeline or analytics notebook.
Real-Life Example
import numpy as np
rng = np.random.default_rng(0)
data = rng.normal(0, 1, 1000)
print("std:", data.std().round(3))Expected Output
(see source)Common mistakes
fitexpects numeric arrays;OneHotEncoder/LabelEncoderare easy to forget for categorical features.- Calling
predicton a model trained on unscaled data and then scaling inputs at inference time silently degrades accuracy. train_test_split(X, y)requires both arrays;train_test_split(X)for unsupervised learning slips past static checkers.- Treating ML Standard Deviation as a black box without reading the docs — the API has subtle defaults that bite when you scale.
🚀 Performance & Best Practices
- Wrap preprocessing + estimator in a
Pipelinesofit/predictstay reproducible. - Use
joblib/picklefor serialising models, not the entire Python state. - For > 100 k rows, switch to
HistGradientBoostingClassifierorcuML. - When working with scikit-learn, prefer vectorised / batched operations over Python loops.
🧪 Try It Yourself
- Reproduce the snippet on a representative slice of your own data.
- Profile the snippet with
cProfileortimeitand find the single biggest improvement. - Generalise the snippet into a small, reusable function you can drop into future projects.