Lesson menuPython TutorialHideShow

Python Tutorial

Bootstrap Aggregation (Bagging)

Train many models on random resamples of the data and combine them to cut variance and boost accuracy.

← K-meansCross Validation →

The Core Idea

Bagging reduces variance. It draws many bootstrap samples (random samples of the training set, drawn with replacement), trains one model on each, and averages their predictions (regression) or takes a majority vote (classification). Averaging many noisy-but-unbiased models yields a smoother, more accurate estimator.

It works best with high-variance, low-bias base learners such as deep decision trees.

What a Bootstrap Sample Is

import numpy as np

data = np.array([11, 22, 33, 44, 55])
rng = np.random.default_rng(0)

# sample the same size, WITH replacement -> some repeat, some are left out
sample = rng.choice(data, size=len(data), replace=True)
print(sample)   # e.g. [55 22 22 44 11]

On average about 37% of rows are left out of each sample — these "out-of-bag" rows give a free validation estimate.

BaggingClassifier

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.metrics import accuracy_score

X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42)

bag = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=50,
    oob_score=True,       # evaluate on left-out rows
    random_state=42)
bag.fit(X_train, y_train)

print("OOB score:", round(bag.oob_score_, 3))
print("Test acc :", round(accuracy_score(y_test, bag.predict(X_test)), 3))

The out-of-bag (OOB) score is a built-in cross-validation-like estimate that needs no separate validation split.

Random Forest: Bagging + Feature Randomness

A Random Forest is bagging applied to decision trees, with an extra twist: each split considers only a random subset of features. That decorrelates the trees and usually beats plain bagging.

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(
    n_estimators=200, max_features="sqrt", random_state=42)
forest.fit(X_train, y_train)
print("Forest acc:", round(accuracy_score(y_test, forest.predict(X_test)), 3))

Bagging vs Boosting

BaggingBoosting
TrainingParallel, independent modelsSequential, each fixes prior errors
Mainly reducesVarianceBias
Base learnerDeep (low-bias) treesShallow (weak) trees
ExamplesRandom ForestAdaBoost, Gradient Boosting, XGBoost

Best Practices

  • Use deep base trees — bagging is meant to tame their variance.
  • More estimators help then plateau; 100–300 is a common sweet spot.
  • Use the OOB score for quick, honest validation.
  • Reach for Random Forest as a strong, low-tuning default on tabular data.

Try It Yourself

Exercise 1: Train a bagging classifier of 50 trees on wine and print the OOB score.

Show solution
from sklearn.datasets import load_wine
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = load_wine(return_X_y=True)
bag = BaggingClassifier(DecisionTreeClassifier(), n_estimators=50,
                        oob_score=True, random_state=0).fit(X, y)
print(round(bag.oob_score_, 3))

Exercise 2: Does bagging mainly reduce bias or variance?

Show solution

Variance — averaging many high-variance models (like deep trees) smooths out their fluctuations.

📘 Real-World Deep Dive

Knowing <strong>ML Bootstrap Aggregation (scikit-learn)</strong> well is what turns scikit-learn from a curiosity into a daily tool — you'll reach for it in nearly every real project.

Real-Life Scenario

An end-to-end usage of ML Bootstrap Aggregation that you'd actually see in a data pipeline or analytics notebook.

Real-Life Example

import numpy as np
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
clf = BaggingClassifier(estimator=DecisionTreeClassifier(), n_estimators=50,
                        random_state=0).fit(X, y)
print("score:", round(clf.score(X, y), 3))

Expected Output

(see source)

Common mistakes

  • fit expects numeric arrays; OneHotEncoder / LabelEncoder are easy to forget for categorical features.
  • Calling predict on a model trained on unscaled data and then scaling inputs at inference time silently degrades accuracy.
  • train_test_split(X, y) requires both arrays; train_test_split(X) for unsupervised learning slips past static checkers.
  • Treating ML Bootstrap Aggregation as a black box without reading the docs — the API has subtle defaults that bite when you scale.

🚀 Performance & Best Practices

  • Wrap preprocessing + estimator in a Pipeline so fit / predict stay reproducible.
  • Use joblib / pickle for serialising models, not the entire Python state.
  • For > 100 k rows, switch to HistGradientBoostingClassifier or cuML.
  • When working with scikit-learn, prefer vectorised / batched operations over Python loops.

🧪 Try It Yourself

  1. Reproduce the snippet on a representative slice of your own data.
  2. Profile the snippet with cProfile or timeit and find the single biggest improvement.
  3. Generalise the snippet into a small, reusable function you can drop into future projects.

FAQ: Bootstrap Aggregation (Bagging)

Common questions about this page.

What is Bootstrap Aggregation (Bagging)?

Bootstrap Aggregation (Bagging) is a Machine Learning lesson that explains bootstrap aggregation (bagging) in Python. Train many models on random resamples of the data and combine them to cut variance and boost accuracy. Copy the samples and run them in the Python editor. It is written for beginners who want a clear definition and working examples.

Should I run bootstrap aggregation (bagging) examples locally for better learning?

Yes. Use the browser editor on StudyGrid for a quick check, then Download the example and run it on your computer. Local runs show real errors and the real toolchain, which is one of the fastest ways to learn bootstrap aggregation (bagging) in this Machine Learning Python lesson (Bootstrap Aggregation (Bagging)).

How do I use bootstrap aggregation (bagging) in Python?

To use bootstrap aggregation (bagging) in Python, follow the examples on this StudyGrid page. Copy a snippet, run it in the browser, then Download and run it locally for better learning. Change the values and compare the output.

What is the syntax of bootstrap aggregation (bagging)?

This Bootstrap Aggregation (Bagging) tutorial shows bootstrap aggregation (bagging) syntax with short Python examples. Use the code blocks in this lesson for the exact statements, then try them in your editor.

Bootstrap Aggregation (Bagging) example for beginners

Yes. This page includes a beginner bootstrap aggregation (bagging) example you can copy and run. It is designed for searches such as "bootstrap aggregation (bagging) for beginners", "bootstrap aggregation (bagging) example", and "how to use bootstrap aggregation (bagging)".

What are common mistakes with bootstrap aggregation (bagging)?

Common bootstrap aggregation (bagging) mistakes include wrong syntax, mixing types, and skipping practice. Work through this Machine Learning chapter in order, run every example, and check the output before moving on.

Why should I learn bootstrap aggregation (bagging)?

Bootstrap Aggregation (Bagging) is used in real Python work. Learning bootstrap aggregation (bagging) helps you write clearer programs and continue the Machine Learning tutorial on StudyGrid.

Is Bootstrap Aggregation (Bagging) free to learn online?

Yes. You can learn bootstrap aggregation (bagging) free on StudyGrid (studygrid.in). This chapter is part of the Machine Learning path and includes examples, syntax, and next-step links.