Polynomial Regression
Extend linear regression to capture non-linear relationships by adding polynomial features and interaction terms.
Concept Overview
Polynomial regression extends linear regression to model non-linear relationships by creating polynomial features. While it uses a linear model under the hood, it can capture curves and complex patterns in the data.
For polynomial regression of degree n, the model becomes:
ŷ = β₀ + β₁x + β₂x² + β₃x³ + … + βₙxⁿ
This is still a linear model in terms of the coefficients βᵢ, but non-linear in terms of the features xⁱ.
When to Use:
- When scatter plots show non-linear patterns (curves, bends)
- When residual plots from linear regression show patterns
- When you need to capture accelerating or decelerating trends
- When domain knowledge suggests polynomial relationships
Challenges:
- Higher degrees can lead to overfitting
- Can create unstable predictions outside training range
- Requires careful degree selection
- May need regularization to prevent overfitting
Simple Polynomial Regression
Let's start with a simple polynomial regression using a single feature with polynomial terms.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
# Generate non-linear data
np.random.seed(42)
x = np.linspace(-3, 3, 100)
# Quadratic relationship with noise
y = 2 * x**2 - 3 * x + 1 + np.random.normal(0, 1, 100)
df = pd.DataFrame({'x': x, 'y': y})
# Split data
X = df[['x']]
y = df['y']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Compare different polynomial degrees
degrees = [1, 2, 3, 5, 10]
results = []
for degree in degrees:
# Create polynomial features
poly_features = PolynomialFeatures(degree=degree, include_bias=False)
X_train_poly = poly_features.fit_transform(X_train)
X_test_poly = poly_features.transform(X_test)
# Fit linear regression on polynomial features
model = LinearRegression()
model.fit(X_train_poly, y_train)
# Predictions
y_train_pred = model.predict(X_train_poly)
y_test_pred = model.predict(X_test_poly)
# Metrics
train_r2 = r2_score(y_train, y_train_pred)
test_r2 = r2_score(y_test, y_test_pred)
train_mse = mean_squared_error(y_train, y_train_pred)
test_mse = mean_squared_error(y_test, y_test_pred)
results.append({
'Degree': degree,
'Train R²': train_r2,
'Test R²': test_r2,
'Train MSE': train_mse,
'Test MSE': test_mse,
'Model': model,
'Poly Features': poly_features
})
print(f"Degree {degree}:")
print(f" Train R²: {train_r2:.4f}, Test R²: {test_r2:.4f}")
print(f" Train MSE: {train_mse:.4f}, Test MSE: {test_mse:.4f}")
print()
# Visualize different degrees
fig, axes = plt.subplots(2, 3, figsize=(18, 12))
axes = axes.flatten()
for idx, (degree, result) in enumerate(zip(degrees, results)):
ax = axes[idx]
# Plot data
ax.scatter(X_train, y_train, alpha=0.6, s=50, label='Training data')
ax.scatter(X_test, y_test, alpha=0.6, s=50, color='green', label='Test data')
# Plot polynomial curve
x_plot = np.linspace(X['x'].min(), X['x'].max(), 300).reshape(-1, 1)
x_plot_poly = result['Poly Features'].transform(x_plot)
y_plot = result['Model'].predict(x_plot_poly)
ax.plot(x_plot, y_plot, 'r-', linewidth=2, label=f'Degree {degree}')
ax.set_xlabel('X', fontsize=12)
ax.set_ylabel('Y', fontsize=12)
ax.set_title(f'Degree {degree} (Test R² = {result["Test R²"]:.4f})', fontsize=14)
ax.legend()
ax.grid(True, alpha=0.3)
# Remove last subplot
fig.delaxes(axes[5])
plt.tight_layout()
plt.show()
# Summary table
summary_df = pd.DataFrame([
{'Degree': r['Degree'],
'Train R²': r['Train R²'],
'Test R²': r['Test R²'],
'Difference': r['Train R²'] - r['Test R²']}
for r in results
])
print("\nSummary:")
print(summary_df)As the degree increases, the model becomes more flexible but may overfit. Notice how higher degrees fit the training data better but may perform worse on test data. Choose the degree that balances fit and generalization.
Using Pipeline for Polynomial Regression
Scikit-learn's Pipeline makes polynomial regression easier and prevents data leakage by combining feature transformation and model training.
import numpy as np
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split, cross_val_score
# Generate sample data
np.random.seed(42)
x = np.linspace(0, 10, 100)
y = 0.5 * x**2 - 2 * x + 3 + np.random.normal(0, 2, 100)
df = pd.DataFrame({'x': x, 'y': y})
X = df[['x']]
y = df['y']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Create pipeline: Polynomial features → Standard scaling → Linear regression
poly_degree = 2
poly_pipeline = Pipeline([
('poly', PolynomialFeatures(degree=poly_degree, include_bias=False)),
('scaler', StandardScaler()),
('regressor', LinearRegression())
])
# Train pipeline
poly_pipeline.fit(X_train, y_train)
# Make predictions
y_train_pred = poly_pipeline.predict(X_train)
y_test_pred = poly_pipeline.predict(X_test)
# Metrics
train_r2 = r2_score(y_train, y_train_pred)
test_r2 = r2_score(y_test, y_test_pred)
train_mse = mean_squared_error(y_train, y_train_pred)
test_mse = mean_squared_error(y_test, y_test_pred)
print(f"Polynomial Regression (Degree {poly_degree}):")
print(f"Train R²: {train_r2:.4f}")
print(f"Test R²: {test_r2:.4f}")
print(f"Train MSE: {train_mse:.4f}")
print(f"Test MSE: {test_mse:.4f}")
# Access model components
print(f"\nModel coefficients:")
print(poly_pipeline.named_steps['regressor'].coef_)
print(f"Intercept: {poly_pipeline.named_steps['regressor'].intercept_:.4f}")
# Cross-validation
cv_scores = cross_val_score(poly_pipeline, X_train, y_train, cv=5, scoring='r2')
print(f"\nCross-validation R² scores: {cv_scores}")
print(f"Mean CV R²: {cv_scores.mean():.4f} (+/- {cv_scores.std() * 2:.4f})")Pipelines ensure that polynomial features are created and scaled properly, preventing data leakage. The scaler is fit only on training data, and the same transformation is applied to test data.
Selecting Optimal Polynomial Degree
Choosing the right polynomial degree is crucial. Too low and you underfit; too high and you overfit. Use cross-validation to find the optimal degree.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split, cross_val_score
# Generate data with some noise
np.random.seed(42)
x = np.linspace(0, 10, 100)
true_y = 0.5 * x**2 - 2 * x + 3 # True quadratic relationship
y = true_y + np.random.normal(0, 2, 100) # Add noise
df = pd.DataFrame({'x': x, 'y': y})
X = df[['x']]
y = df['y']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Test different degrees
degrees = range(1, 11)
train_scores = []
test_scores = []
cv_scores_mean = []
cv_scores_std = []
for degree in degrees:
# Create pipeline
pipeline = Pipeline([
('poly', PolynomialFeatures(degree=degree, include_bias=False)),
('scaler', StandardScaler()),
('regressor', LinearRegression())
])
# Fit and predict
pipeline.fit(X_train, y_train)
y_train_pred = pipeline.predict(X_train)
y_test_pred = pipeline.predict(X_test)
# Calculate scores
train_r2 = r2_score(y_train, y_train_pred)
test_r2 = r2_score(y_test, y_test_pred)
train_scores.append(train_r2)
test_scores.append(test_r2)
# Cross-validation
cv_scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring='r2')
cv_scores_mean.append(cv_scores.mean())
cv_scores_std.append(cv_scores.std())
# Find best degree (highest CV score)
best_degree_idx = np.argmax(cv_scores_mean)
best_degree = degrees[best_degree_idx]
print(f"Best polynomial degree: {best_degree}")
print(f"Best CV R²: {cv_scores_mean[best_degree_idx]:.4f}")
# Visualize degree selection
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
# Plot R² scores
ax1.plot(degrees, train_scores, 'o-', label='Train R²', linewidth=2, markersize=8)
ax1.plot(degrees, test_scores, 's-', label='Test R²', linewidth=2, markersize=8)
ax1.plot(degrees, cv_scores_mean, '^-', label='CV R² (mean)', linewidth=2, markersize=8)
ax1.axvline(x=best_degree, color='red', linestyle='--', linewidth=2,
label=f'Best degree ({best_degree})')
ax1.fill_between(degrees,
np.array(cv_scores_mean) - np.array(cv_scores_std),
np.array(cv_scores_mean) + np.array(cv_scores_std),
alpha=0.3, label='CV R² ± 1 std')
ax1.set_xlabel('Polynomial Degree', fontsize=12)
ax1.set_ylabel('R² Score', fontsize=12)
ax1.set_title('R² vs Polynomial Degree', fontsize=14)
ax1.legend()
ax1.grid(True, alpha=0.3)
# Plot gap between train and test (overfitting indicator)
gap = np.array(train_scores) - np.array(test_scores)
ax2.plot(degrees, gap, 'o-', color='red', linewidth=2, markersize=8)
ax2.axvline(x=best_degree, color='blue', linestyle='--', linewidth=2,
label=f'Best degree ({best_degree})')
ax2.set_xlabel('Polynomial Degree', fontsize=12)
ax2.set_ylabel('Train R² - Test R²', fontsize=12)
ax2.set_title('Overfitting Indicator (Gap)', fontsize=14)
ax2.legend()
ax2.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
# Create summary table
summary = pd.DataFrame({
'Degree': degrees,
'Train R²': train_scores,
'Test R²': test_scores,
'CV R² (mean)': cv_scores_mean,
'CV R² (std)': cv_scores_std,
'Gap': gap
})
print("\nDetailed Results:")
print(summary.round(4))The optimal degree balances training performance and generalization. Cross-validation helps find the degree that generalizes best. Watch for the gap between train and test scores—a large gap indicates overfitting.
Regularized Polynomial Regression
Polynomial regression with high degrees can overfit. Adding regularization (Ridge or Lasso) helps control model complexity and improve generalization.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split, cross_val_score
# Generate data
np.random.seed(42)
x = np.linspace(0, 10, 100)
y = 0.5 * x**2 - 2 * x + 3 + np.random.normal(0, 2, 100)
df = pd.DataFrame({'x': x, 'y': y})
X = df[['x']]
y = df['y']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# High degree polynomial (without regularization - will overfit)
degree = 15
poly_unregularized = Pipeline([
('poly', PolynomialFeatures(degree=degree, include_bias=False)),
('scaler', StandardScaler()),
('regressor', LinearRegression())
])
poly_unregularized.fit(X_train, y_train)
pred_unreg = poly_unregularized.predict(X_test)
r2_unreg = r2_score(y_test, pred_unreg)
print(f"Unregularized Polynomial (degree {degree}):")
print(f"Test R²: {r2_unreg:.4f}")
# With Ridge regularization
poly_ridge = Pipeline([
('poly', PolynomialFeatures(degree=degree, include_bias=False)),
('scaler', StandardScaler()),
('regressor', Ridge(alpha=1.0))
])
poly_ridge.fit(X_train, y_train)
pred_ridge = poly_ridge.predict(X_test)
r2_ridge = r2_score(y_test, pred_ridge)
print(f"\nRidge Regularized Polynomial (α=1.0):")
print(f"Test R²: {r2_ridge:.4f}")
# With Lasso regularization
poly_lasso = Pipeline([
('poly', PolynomialFeatures(degree=degree, include_bias=False)),
('scaler', StandardScaler()),
('regressor', Lasso(alpha=0.1, max_iter=2000))
])
poly_lasso.fit(X_train, y_train)
pred_lasso = poly_lasso.predict(X_test)
r2_lasso = r2_score(y_test, pred_lasso)
print(f"\nLasso Regularized Polynomial (α=0.1):")
print(f"Test R²: {r2_lasso:.4f}")
# Visualize different regularizations
fig, axes = plt.subplots(1, 3, figsize=(18, 5))
models = [
(poly_unregularized, pred_unreg, r2_unreg, 'Unregularized', axes[0]),
(poly_ridge, pred_ridge, r2_ridge, 'Ridge (α=1.0)', axes[1]),
(poly_lasso, pred_lasso, r2_lasso, 'Lasso (α=0.1)', axes[2])
]
for model, pred, r2, title, ax in models:
ax.scatter(X_train, y_train, alpha=0.6, s=50, label='Training')
ax.scatter(X_test, y_test, alpha=0.6, s=50, color='green', label='Test')
x_plot = np.linspace(X['x'].min(), X['x'].max(), 300).reshape(-1, 1)
y_plot = model.predict(x_plot)
ax.plot(x_plot, y_plot, 'r-', linewidth=2, label='Prediction')
ax.set_xlabel('X', fontsize=12)
ax.set_ylabel('Y', fontsize=12)
ax.set_title(f'{title}\nTest R² = {r2:.4f}', fontsize=14)
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
# Compare coefficient magnitudes
coef_unreg = poly_unregularized.named_steps['regressor'].coef_
coef_ridge = poly_ridge.named_steps['regressor'].coef_
coef_lasso = poly_lasso.named_steps['regressor'].coef_
print(f"\nCoefficient Statistics:")
print(f"Unregularized - Max |coef|: {np.abs(coef_unreg).max():.2f}, Mean |coef|: {np.abs(coef_unreg).mean():.2f}")
print(f"Ridge - Max |coef|: {np.abs(coef_ridge).max():.2f}, Mean |coef|: {np.abs(coef_ridge).mean():.2f}")
print(f"Lasso - Max |coef|: {np.abs(coef_lasso).max():.2f}, Mean |coef|: {np.abs(coef_lasso).mean():.2f}")
print(f"Lasso - Non-zero coefficients: {(np.abs(coef_lasso) > 1e-5).sum()} / {len(coef_lasso)}")Regularization shrinks coefficients and helps prevent overfitting. Ridge shrinks all coefficients, while Lasso can eliminate polynomial terms entirely by setting coefficients to zero. Use cross-validation to tune the regularization parameter α.
Multiple Features with Polynomial Regression
Polynomial regression can include interaction terms and polynomial features for multiple variables, creating a more complex model.
import numpy as np
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
# Generate data with multiple features
np.random.seed(42)
n = 200
sqft = np.random.normal(1500, 500, n)
bedrooms = np.random.randint(1, 6, n)
age = np.random.normal(10, 5, n)
# Non-linear relationship with interactions
price = (100 * sqft + 20000 * bedrooms - 5000 * age +
0.01 * sqft * bedrooms - 50 * sqft * age / 1000 +
np.random.normal(0, 50000, n))
df = pd.DataFrame({
'sqft': sqft,
'bedrooms': bedrooms,
'age': age,
'price': price
})
X = df[['sqft', 'bedrooms', 'age']]
y = df['price']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Linear regression (baseline)
lr = LinearRegression()
lr.fit(X_train, y_train)
lr_pred = lr.predict(X_test)
lr_r2 = r2_score(y_test, lr_pred)
print("Linear Regression:")
print(f"Test R²: {lr_r2:.4f}")
# Polynomial regression (degree 2) - includes interactions and squared terms
poly_degree = 2
poly_pipeline = Pipeline([
('poly', PolynomialFeatures(degree=poly_degree, include_bias=False)),
('regressor', LinearRegression())
])
poly_pipeline.fit(X_train, y_train)
poly_pred = poly_pipeline.predict(X_test)
poly_r2 = r2_score(y_test, poly_pred)
print(f"\nPolynomial Regression (degree {poly_degree}):")
print(f"Test R²: {poly_r2:.4f}")
# Check what features were created
poly_features = poly_pipeline.named_steps['poly']
feature_names = poly_features.get_feature_names_out(X.columns)
print(f"\nNumber of polynomial features: {len(feature_names)}")
print(f"Original features: {X.columns.tolist()}")
print(f"\nPolynomial features (first 10):")
for i, name in enumerate(feature_names[:10]):
print(f" {i+1}. {name}")
print(f" ... (and {len(feature_names) - 10} more)")
# With Ridge regularization
poly_ridge_pipeline = Pipeline([
('poly', PolynomialFeatures(degree=poly_degree, include_bias=False)),
('scaler', StandardScaler()),
('regressor', Ridge(alpha=1.0))
])
poly_ridge_pipeline.fit(X_train, y_train)
poly_ridge_pred = poly_ridge_pipeline.predict(X_test)
poly_ridge_r2 = r2_score(y_test, poly_ridge_pred)
print(f"\nRegularized Polynomial Regression:")
print(f"Test R²: {poly_ridge_r2:.4f}")
# Compare feature importance
print("\nFeature importance comparison (coefficient magnitudes):")
coef_df = pd.DataFrame({
'Feature': feature_names,
'Unregularized': np.abs(poly_pipeline.named_steps['regressor'].coef_),
'Ridge': np.abs(poly_ridge_pipeline.named_steps['regressor'].coef_)
}).sort_values('Ridge', ascending=False)
print(coef_df.head(10))Polynomial features for multiple variables include interaction terms (e.g., x₁ × x₂) and squared terms (e.g., x₁²). This can dramatically increase the number of features, so regularization becomes even more important to prevent overfitting.
Best Practices
- Start simple: Begin with degree 2 or 3 and increase only if needed
- Use cross-validation: Find the optimal degree using cross-validation, not just training performance
- Regularize high degrees: Always use regularization (Ridge or Lasso) for degrees ≥ 5
- Standardize features: Scale features before polynomial transformation, especially for regularization
- Watch for overfitting: Monitor the gap between train and test scores
- Use pipelines: Prevent data leakage by using scikit-learn pipelines
- Check residuals: Verify that polynomial regression captures patterns in residual plots
- Validate assumptions: Ensure polynomial relationships make sense in your domain
- Consider alternatives: Sometimes transformations (log, square root) work better than polynomials
- Document choices: Record degree selection rationale and share insights through
info.studygrid@gmail.com
Next Steps
Learn about multiple regression to understand how to model relationships with multiple features simultaneously, building on the polynomial concepts learned here.
Try It Yourself
Exercise 1: Fit a degree-2 polynomial to the data and predict at x = 17.
Show solution
import numpy as np
x = [1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 19, 21, 22]
y = [100, 90, 80, 60, 60, 55, 60, 65, 70, 70, 75, 76, 78, 79, 90, 99, 99, 100]
model = np.poly1d(np.polyfit(x, y, 3))
print(round(model(17), 1))Exercise 2: Why can a very high-degree polynomial fit training data perfectly yet predict poorly?
Show solution
It overfits — it bends to pass through every training point, capturing noise instead of the true trend, so it generalizes badly to new data.
Key Takeaways
- Polynomial regression models curved relationships.
- Higher degree fits more closely but risks overfitting.
- Use R² and a test set to pick the right degree.
📘 Real-World Deep Dive
Knowing <strong>ML Polynomial Regression (scikit-learn)</strong> well is what turns scikit-learn from a curiosity into a daily tool — you'll reach for it in nearly every real project.
Real-Life Scenario
An end-to-end usage of ML Polynomial Regression that you'd actually see in a data pipeline or analytics notebook.
Real-Life Example
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
X = np.linspace(-3, 3, 30).reshape(-1, 1)
y = X[:,0]**2 + np.random.normal(0, 0.5, X.shape[0])
model = make_pipeline(PolynomialFeatures(2), LinearRegression()).fit(X, y)
print("r2:", model.score(X, y).round(3))Expected Output
(see source)Common mistakes
fitexpects numeric arrays;OneHotEncoder/LabelEncoderare easy to forget for categorical features.- Calling
predicton a model trained on unscaled data and then scaling inputs at inference time silently degrades accuracy. train_test_split(X, y)requires both arrays;train_test_split(X)for unsupervised learning slips past static checkers.- Treating ML Polynomial Regression as a black box without reading the docs — the API has subtle defaults that bite when you scale.
🚀 Performance & Best Practices
- Wrap preprocessing + estimator in a
Pipelinesofit/predictstay reproducible. - Use
joblib/picklefor serialising models, not the entire Python state. - For > 100 k rows, switch to
HistGradientBoostingClassifierorcuML. - When working with scikit-learn, prefer vectorised / batched operations over Python loops.
🧪 Try It Yourself
- Reproduce the snippet on a representative slice of your own data.
- Profile the snippet with
cProfileortimeitand find the single biggest improvement. - Generalise the snippet into a small, reusable function you can drop into future projects.