Normal Distribution
Recognize Gaussian patterns, compute probabilities, and apply z-scores in machine learning workflows.
Characteristics
The normal distribution (also called Gaussian distribution) is symmetric around its mean μ with standard deviation σ. Many natural phenomena approximate this bell-shaped curve due to the Central Limit Theorem.
- Mean, median, and mode are all equal.
- Approximately 68% of data lies within μ ± σ (one standard deviation).
- Approximately 95% of data lies within μ ± 2σ (two standard deviations).
- Approximately 99.7% of data lies within μ ± 3σ (three standard deviations). This is known as the 68-95-99.7 rule.
The probability density function (PDF) is:
f(x) = (1 / (σ·√(2π))) · e^(−(x − μ)² / (2σ²))
- The distribution is completely determined by its mean μ and standard deviation σ, denoted as N(μ, σ²).
import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt
# Standard normal distribution (μ=0, σ=1)
mu, sigma = 0, 1
x = np.linspace(-4, 4, 1000)
pdf = norm.pdf(x, loc=mu, scale=sigma)
# Calculate the 68-95-99.7 rule probabilities
prob_1sigma = norm.cdf(mu + sigma, loc=mu, scale=sigma) - \
norm.cdf(mu - sigma, loc=mu, scale=sigma)
prob_2sigma = norm.cdf(mu + 2*sigma, loc=mu, scale=sigma) - \
norm.cdf(mu - 2*sigma, loc=mu, scale=sigma)
prob_3sigma = norm.cdf(mu + 3*sigma, loc=mu, scale=sigma) - \
norm.cdf(mu - 3*sigma, loc=mu, scale=sigma)
print(f"Probability within μ ± 1σ: {prob_1sigma:.4f} ({prob_1sigma*100:.1f}%)")
print(f"Probability within μ ± 2σ: {prob_2sigma:.4f} ({prob_2sigma*100:.1f}%)")
print(f"Probability within μ ± 3σ: {prob_3sigma:.4f} ({prob_3sigma*100:.1f}%)")
# Visualize with shaded regions
fig, ax = plt.subplots(figsize=(10, 6))
ax.plot(x, pdf, 'k-', linewidth=2, label='Normal PDF')
# Shade regions for 1σ, 2σ, 3σ
x1 = x[(x >= mu - sigma) & (x <= mu + sigma)]
ax.fill_between(x1, norm.pdf(x1, mu, sigma), alpha=0.3, color='blue', label='68% (1σ)')
x2 = x[(x >= mu - 2*sigma) & (x <= mu + 2*sigma)]
ax.fill_between(x2, norm.pdf(x2, mu, sigma), alpha=0.2, color='green', label='95% (2σ)')
x3 = x[(x >= mu - 3*sigma) & (x <= mu + 3*sigma)]
ax.fill_between(x3, norm.pdf(x3, mu, sigma), alpha=0.1, color='red', label='99.7% (3σ)')
ax.axvline(mu, color='black', linestyle='--', alpha=0.5, label=f'Mean (μ={mu})')
ax.set_xlabel('x')
ax.set_ylabel('Probability Density')
ax.set_title('Normal Distribution: 68-95-99.7 Rule')
ax.legend()
ax.grid(True, alpha=0.3)
plt.show()Central Limit Theorem
The Central Limit Theorem (CLT) states that the distribution of sample means approaches a normal distribution as the sample size increases, regardless of the original distribution's shape. This is why normal distributions are so prevalent in statistics and machine learning.
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import normaltest
# Generate non-normal data (uniform distribution)
np.random.seed(42)
original_distribution = np.random.uniform(0, 10, 1000)
print("Original distribution (uniform):")
print(f"Mean: {original_distribution.mean():.4f}")
print(f"Std: {original_distribution.std():.4f}")
# Demonstrate CLT: Take sample means from the uniform distribution
sample_means = []
sample_size = 30 # Sample size
num_samples = 1000 # Number of samples
for _ in range(num_samples):
sample = np.random.choice(original_distribution, size=sample_size, replace=True)
sample_means.append(sample.mean())
sample_means = np.array(sample_means)
print(f"\nDistribution of sample means (n={sample_size}):")
print(f"Mean: {sample_means.mean():.4f}")
print(f"Std: {sample_means.std():.4f}")
print(f"Expected std (σ/√n): {original_distribution.std() / np.sqrt(sample_size):.4f}")
# Test for normality
stat, p_value = normaltest(sample_means)
print(f"\nNormality test on sample means:")
print(f"p-value: {p_value:.6f}")
print(f"Approximately normal: {p_value > 0.05}")
# Visualize
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
axes[0].hist(original_distribution, bins=30, density=True, alpha=0.7, color='blue')
axes[0].set_title('Original Distribution (Uniform)')
axes[0].set_xlabel('Value')
axes[0].set_ylabel('Density')
axes[1].hist(sample_means, bins=30, density=True, alpha=0.7, color='green')
# Overlay normal distribution
mu_sample = sample_means.mean()
sigma_sample = sample_means.std()
x = np.linspace(sample_means.min(), sample_means.max(), 100)
axes[1].plot(x, norm.pdf(x, mu_sample, sigma_sample), 'r-', linewidth=2, label='Normal approximation')
axes[1].set_title(f'Distribution of Sample Means (n={sample_size})')
axes[1].set_xlabel('Sample Mean')
axes[1].set_ylabel('Density')
axes[1].legend()
plt.tight_layout()
plt.show()The CLT explains why many statistical methods assume normality: even when individual observations aren't normal, sample means often are. This is crucial for hypothesis testing and confidence intervals.
Generating and Visualizing
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
mu, sigma = 0, 1
values = np.random.normal(mu, sigma, size=10_000)
fig, ax = plt.subplots()
ax.hist(values, bins=50, density=True, alpha=0.6)
x = np.linspace(-4, 4, 200)
ax.plot(x, norm.pdf(x, mu, sigma), color="crimson")
ax.set_title("Standard Normal Distribution")Z-Scores and Standardization
Standardize values into z-scores to compare observations across features with different scales. The formula for a z-score is:
z = (x − μ) / σ
import numpy as np
from scipy.stats import zscore
# Example dataset
values = np.array([15, 18, 22, 25, 30, 32, 35, 40])
mu = values.mean()
sigma = values.std()
# Manual z-score calculation
z_scores_manual = (values - mu) / sigma
# Using scipy
z_scores_scipy = zscore(values)
print("Values:", values)
print("Z-scores (manual):", z_scores_manual)
print("Z-scores (scipy):", z_scores_scipy)
print("\nInterpretation:")
print(f"Value {values[0]} has z-score {z_scores_manual[0]:.2f}")
print(f"This means it's {abs(z_scores_manual[0]):.2f} standard deviations from the mean")Z-scores indicate how many standard deviations an observation lies from the mean. A z-score of 0 means the value equals the mean, while z-scores of ±1, ±2, and ±3 represent 1, 2, and 3 standard deviations away, respectively.
Working with CDF and Quantiles
The Cumulative Distribution Function (CDF) gives the probability that a random variable is less than or equal to a specific value. The Percent Point Function (PPF), also called the quantile function, does the inverse—it finds the value corresponding to a given probability.
from scipy.stats import norm
import numpy as np
# Standard normal distribution (mean=0, std=1)
mu, sigma = 0, 1
# CDF: Probability that value is <= x
prob_below_1 = norm.cdf(1, loc=mu, scale=sigma)
prob_below_negative_1 = norm.cdf(-1, loc=mu, scale=sigma)
prob_between = prob_below_1 - prob_below_negative_1
print(f"P(X ≤ 1): {prob_below_1:.4f}")
print(f"P(X ≤ -1): {prob_below_negative_1:.4f}")
print(f"P(-1 ≤ X ≤ 1): {prob_between:.4f} (approximately 68%)")
# PPF: Find value for a given probability
threshold_90 = norm.ppf(0.90, loc=mu, scale=sigma) # 90th percentile
threshold_95 = norm.ppf(0.95, loc=mu, scale=sigma) # 95th percentile
threshold_99 = norm.ppf(0.99, loc=mu, scale=sigma) # 99th percentile
print(f"\n90th percentile: {threshold_90:.4f}")
print(f"95th percentile: {threshold_95:.4f}")
print(f"99th percentile: {threshold_99:.4f}")
# PDF: Probability density at a specific point
pdf_at_0 = norm.pdf(0, loc=mu, scale=sigma)
pdf_at_1 = norm.pdf(1, loc=mu, scale=sigma)
print(f"\nPDF at x=0: {pdf_at_0:.4f}")
print(f"PDF at x=1: {pdf_at_1:.4f}")CDF (cdf) returns cumulative probabilities, PPF (ppf) yields quantile thresholds, and PDF (pdf) returns the probability density at a specific point. These functions are essential for statistical inference and hypothesis testing.
Practical Example: Quality Control
Normal distributions are commonly used in quality control. Suppose a manufacturing process produces items with a mean length of 100 mm and standard deviation of 2 mm. We can calculate probabilities and thresholds:
from scipy.stats import norm
# Manufacturing process parameters
mean_length = 100 # mm
std_length = 2 # mm
# Calculate probability that an item is within specifications (95-105 mm)
prob_acceptable = norm.cdf(105, loc=mean_length, scale=std_length) - \
norm.cdf(95, loc=mean_length, scale=std_length)
print(f"Probability of acceptable length (95-105 mm): {prob_acceptable:.4f}")
print(f"Expected defect rate: {(1 - prob_acceptable) * 100:.2f}%")
# Find rejection thresholds (items outside 3 standard deviations)
lower_threshold = mean_length - 3 * std_length
upper_threshold = mean_length + 3 * std_length
print(f"\nLower rejection threshold: {lower_threshold} mm")
print(f"Upper rejection threshold: {upper_threshold} mm")
# Probability of being within 3 standard deviations (99.7% rule)
prob_3sigma = norm.cdf(upper_threshold, loc=mean_length, scale=std_length) - \
norm.cdf(lower_threshold, loc=mean_length, scale=std_length)
print(f"Probability within 3σ: {prob_3sigma:.4f}")
# Find the length that 95% of items exceed
length_95th_percentile = norm.ppf(0.05, loc=mean_length, scale=std_length)
print(f"\n95% of items are longer than: {length_95th_percentile:.2f} mm")Testing for Normality
Before assuming data follows a normal distribution, it's important to test this assumption. Several statistical tests can help:
from scipy.stats import normaltest, shapiro, jarque_bera
import numpy as np
import matplotlib.pyplot as plt
# Generate sample data
np.random.seed(42)
normal_data = np.random.normal(100, 15, size=1000)
non_normal_data = np.random.exponential(2, size=1000)
# D'Agostino and Pearson's test
stat_norm, p_norm = normaltest(normal_data)
stat_non, p_non = normaltest(non_normal_data)
print("D'Agostino-Pearson test:")
print(f"Normal data: statistic={stat_norm:.4f}, p-value={p_norm:.4f}")
print(f" → Normal: {p_norm > 0.05}")
print(f"Non-normal data: statistic={stat_non:.4f}, p-value={p_non:.4f}")
print(f" → Normal: {p_non > 0.05}")
# Shapiro-Wilk test (for smaller samples, < 5000)
if len(normal_data) <= 5000:
stat_sw, p_sw = shapiro(normal_data[:1000])
print(f"\nShapiro-Wilk test (normal data):")
print(f"statistic={stat_sw:.4f}, p-value={p_sw:.4f}")
print(f" → Normal: {p_sw > 0.05}")
# Jarque-Bera test
jb_stat, jb_p = jarque_bera(normal_data)
print(f"\nJarque-Bera test (normal data):")
print(f"statistic={jb_stat:.4f}, p-value={jb_p:.4f}")
print(f" → Normal: {jb_p > 0.05}")
# Visual inspection with Q-Q plot
from scipy.stats import probplot
fig, ax = plt.subplots()
probplot(normal_data, dist="norm", plot=ax)
ax.set_title("Q-Q Plot: Normal Data")
plt.show()These tests check if your data follows a normal distribution. High p-values (typically > 0.05) suggest normality. Visual inspection using Q-Q plots and histograms should complement statistical tests.
Applications in Machine Learning
Normal distributions play a crucial role in machine learning:
- Feature Scaling: Standardizing features using z-scores improves convergence for gradient-based algorithms.
- Outlier Detection: Values beyond μ ± 3σ are often considered outliers.
- Model Assumptions: Many algorithms (linear regression, Naive Bayes) assume normally distributed errors or features.
- Sampling: Normal distributions are used in Bayesian inference and generative models.
from sklearn.preprocessing import StandardScaler
import numpy as np
import pandas as pd
# Example: Feature standardization
data = {
'feature1': np.random.normal(100, 15, 1000),
'feature2': np.random.normal(50, 5, 1000),
'feature3': np.random.normal(200, 30, 1000)
}
df = pd.DataFrame(data)
print("Original features:")
print(df.describe())
# Standardize features
scaler = StandardScaler()
df_scaled = pd.DataFrame(
scaler.fit_transform(df),
columns=df.columns
)
print("\nStandardized features:")
print(df_scaled.describe())
print(f"\nMean of standardized features: {df_scaled.mean().values}")
print(f"Std of standardized features: {df_scaled.std().values}")
# Outlier detection using z-scores
z_scores = np.abs((df['feature1'] - df['feature1'].mean()) / df['feature1'].std())
outliers = df[z_scores > 3]
print(f"\nNumber of outliers (|z| > 3): {len(outliers)}")
print(f"Percentage: {len(outliers) / len(df) * 100:.2f}%")Data Transformations
When data is not normally distributed, transformations can help achieve normality or improve model performance:
import numpy as np
from scipy import stats
from scipy.stats import normaltest
import matplotlib.pyplot as plt
# Generate skewed data (exponential distribution)
np.random.seed(42)
skewed_data = np.random.exponential(scale=2, size=1000)
print("Original skewed data:")
stat_before, p_before = normaltest(skewed_data)
print(f"Normality test p-value: {p_before:.6f}")
print(f"Mean: {skewed_data.mean():.2f}, Median: {np.median(skewed_data):.2f}")
# Log transformation
log_data = np.log1p(skewed_data) # log1p handles zeros
stat_log, p_log = normaltest(log_data)
print(f"\nAfter log transformation:")
print(f"Normality test p-value: {p_log:.6f}")
# Square root transformation
sqrt_data = np.sqrt(skewed_data)
stat_sqrt, p_sqrt = normaltest(sqrt_data)
print(f"\nAfter square root transformation:")
print(f"Normality test p-value: {p_sqrt:.6f}")
# Box-Cox transformation (requires positive values)
positive_data = skewed_data + 1 # Ensure all values are positive
transformed_data, lambda_param = stats.boxcox(positive_data)
stat_boxcox, p_boxcox = normaltest(transformed_data)
print(f"\nAfter Box-Cox transformation (λ={lambda_param:.4f}):")
print(f"Normality test p-value: {p_boxcox:.6f}")
# Choose best transformation
transformations = {
'Log': (log_data, p_log),
'Square Root': (sqrt_data, p_sqrt),
'Box-Cox': (transformed_data, p_boxcox)
}
best_transform = max(transformations.items(), key=lambda x: x[1][1])
print(f"\nBest transformation: {best_transform[0]} (p-value: {best_transform[1][1]:.6f})")Common transformations include logarithmic (for right-skewed data), square root (for moderate skew), and Box-Cox (optimal power transformation). Always validate normality after transformation using statistical tests and visual inspection.
When Assumptions Break
Not all data follows a normal distribution. Many real-world datasets exhibit skewness, heavy tails, or multimodality. When normality assumptions fail:
- Transformations: Apply log, Box-Cox, or other transformations to achieve approximate normality.
- Non-parametric Methods: Use algorithms that don't assume specific distributions (random forests, SVM with RBF kernel).
- Robust Methods: Choose algorithms less sensitive to distribution assumptions (median-based metrics, robust regression).
- Model Validation: Check residual distributions and model assumptions after fitting.
from scipy.stats import skew, kurtosis
import numpy as np
# Example: Detecting non-normality
np.random.seed(42)
normal_sample = np.random.normal(0, 1, 1000)
non_normal_sample = np.random.exponential(1, 1000)
def assess_normality(data):
"""Assess if data appears normally distributed."""
skewness = skew(data)
kurt = kurtosis(data) # Excess kurtosis (normal = 0)
print(f"Skewness: {skewness:.4f} (normal ≈ 0)")
print(f"Kurtosis: {kurt:.4f} (normal ≈ 0)")
if abs(skewness) > 1:
print(" → Highly skewed (non-normal)")
elif abs(kurt) > 1:
print(" → Heavy tails or peaked (non-normal)")
else:
print(" → Approximately normal")
print("Normal sample:")
assess_normality(normal_sample)
print("\nNon-normal sample (exponential):")
assess_normality(non_normal_sample)Skewness measures asymmetry (normal = 0), while kurtosis measures tail heaviness (normal excess kurtosis = 0). Document your distribution assumptions and transformation choices. Share guidelines and best practices via info.studygrid@gmail.com.
Next Steps
Transition to the scatter plot lesson to visualize bivariate relationships in your dataset.
Try It Yourself
Exercise 1: Draw 100000 samples from a normal distribution with mean 0 and SD 1, then histogram them.
Show solution
import numpy as np
import matplotlib.pyplot as plt
plt.hist(np.random.normal(0, 1, 100000), bins=100)
plt.show()Exercise 2: Roughly what fraction of values fall within one standard deviation of the mean?
Show solution
About 68% (the 68-95-99.7 rule).
Key Takeaways
- The normal (Gaussian) distribution is the symmetric bell curve.
- It is defined by its mean μ and standard deviation σ.
- The 68-95-99.7 rule describes how data clusters around the mean.
📘 Real-World Deep Dive
Knowing <strong>ML Normal Distribution (scikit-learn)</strong> well is what turns scikit-learn from a curiosity into a daily tool — you'll reach for it in nearly every real project.
Real-Life Scenario
An end-to-end usage of ML Normal Distribution that you'd actually see in a data pipeline or analytics notebook.
Real-Life Example
import numpy as np
rng = np.random.default_rng(0)
xs = rng.normal(0, 1, 5)
print(xs)
print("P(Z <= 1) =", 0.8413)Expected Output
(see source)Common mistakes
fitexpects numeric arrays;OneHotEncoder/LabelEncoderare easy to forget for categorical features.- Calling
predicton a model trained on unscaled data and then scaling inputs at inference time silently degrades accuracy. train_test_split(X, y)requires both arrays;train_test_split(X)for unsupervised learning slips past static checkers.- Treating ML Normal Distribution as a black box without reading the docs — the API has subtle defaults that bite when you scale.
🚀 Performance & Best Practices
- Wrap preprocessing + estimator in a
Pipelinesofit/predictstay reproducible. - Use
joblib/picklefor serialising models, not the entire Python state. - For > 100 k rows, switch to
HistGradientBoostingClassifierorcuML. - When working with scikit-learn, prefer vectorised / batched operations over Python loops.
🧪 Try It Yourself
- Reproduce the snippet on a representative slice of your own data.
- Profile the snippet with
cProfileortimeitand find the single biggest improvement. - Generalise the snippet into a small, reusable function you can drop into future projects.