Data Distribution
Visualize and analyze how values are distributed to uncover structure, skew, and anomalies.
Why Distributions Matter
Model performance depends on understanding the underlying distribution of features and targets. Distribution analysis guides feature engineering, transformation choices, and model selection.
Histogram and Density Plots
import matplotlib.pyplot as plt
import seaborn as sns
fig, ax = plt.subplots(figsize=(8, 4))
sns.histplot(data, kde=True, bins=30, ax=ax)
ax.set_xlabel("Value")
ax.set_ylabel("Frequency")
plt.show()Overlay kernel density estimates (KDE) to smooth the histogram and highlight shape.
Cumulative Distribution Function
The CDF displays the probability of observing values less than or equal to a threshold. Use it to evaluate percentiles visually.
import numpy as np
sorted_values = np.sort(data)
cdf = np.arange(1, len(sorted_values) + 1) / len(sorted_values)
plt.plot(sorted_values, cdf)
plt.xlabel("Value")
plt.ylabel("Cumulative Probability")Box Plots and Violin Plots
Box plots summarize quartiles and potential outliers. Violin plots add density information, helping compare distributions across groups.
sns.boxplot(x="segment", y="score", data=df)
sns.violinplot(x="segment", y="score", data=df)Skewness and Kurtosis
Calculate skewness to understand asymmetry and kurtosis to evaluate tail heaviness:
from scipy.stats import skew, kurtosis
print(skew(data), kurtosis(data))Transform features (log, Box-Cox) if severe skew degrades model assumptions.
Best Practices
- Analyze distributions per cohort to detect hidden subpopulations.
- Profile distributions routinely; automate reporting and share updates via
info.studygrid@gmail.com. - Use stratified sampling when splitting data to preserve distribution characteristics.
Next Steps
Study the normal distribution to understand how Gaussian assumptions impact modeling choices.
Try It Yourself
Exercise 1: Generate 250 random floats between 0 and 5 and plot a histogram.
Show solution
import numpy as np
import matplotlib.pyplot as plt
data = np.random.uniform(0.0, 5.0, 250)
plt.hist(data, bins=20)
plt.show()Exercise 2: What shape would a histogram of fair dice rolls approach as you roll more times?
Show solution
A roughly flat (uniform) distribution — each face 1–6 becomes about equally likely.
Key Takeaways
- A data distribution describes how values are spread.
- Generate samples with NumPy and visualize with histograms.
- Understanding the distribution guides model choice and preprocessing.
📘 Real-World Deep Dive
Knowing <strong>ML Data Distribution (scikit-learn)</strong> well is what turns scikit-learn from a curiosity into a daily tool — you'll reach for it in nearly every real project.
Real-Life Scenario
An end-to-end usage of ML Data Distribution that you'd actually see in a data pipeline or analytics notebook.
Real-Life Example
import numpy as np
rng = np.random.default_rng(0)
data = rng.normal(50, 10, 1000)
print("mean:", data.mean().round(2))
print("std :", data.std().round(2))Expected Output
(see source)Common mistakes
fitexpects numeric arrays;OneHotEncoder/LabelEncoderare easy to forget for categorical features.- Calling
predicton a model trained on unscaled data and then scaling inputs at inference time silently degrades accuracy. train_test_split(X, y)requires both arrays;train_test_split(X)for unsupervised learning slips past static checkers.- Treating ML Data Distribution as a black box without reading the docs — the API has subtle defaults that bite when you scale.
🚀 Performance & Best Practices
- Wrap preprocessing + estimator in a
Pipelinesofit/predictstay reproducible. - Use
joblib/picklefor serialising models, not the entire Python state. - For > 100 k rows, switch to
HistGradientBoostingClassifierorcuML. - When working with scikit-learn, prefer vectorised / batched operations over Python loops.
🧪 Try It Yourself
- Reproduce the snippet on a representative slice of your own data.
- Profile the snippet with
cProfileortimeitand find the single biggest improvement. - Generalise the snippet into a small, reusable function you can drop into future projects.