Decision Tree

How a Decision Tree Works

A decision tree asks a sequence of yes/no questions about the features. Each internal node tests one feature against a threshold; each leaf holds a prediction. The algorithm greedily chooses the split that best separates the classes, measured by Gini impurity or entropy (information gain).

Trees handle numeric and categorical data, need no feature scaling, and are easy to interpret — you can read the exact rules that led to a prediction.

Building a Classifier

import pandas as pd
from sklearn.tree import DecisionTreeClassifier

data = {
    "Age":       [36, 42, 23, 52, 43, 44, 66, 35, 52, 35],
    "Experience":[10, 12,  4, 4,  21, 14, 3,  14, 13, 5],
    "Rank":      [9,  4,   6, 4,  8,  5,  7,  9,  7,  9],
    "Go":        [1,  0,   0, 0,  1,  0,  1,  1,  1,  1],
}
df = pd.DataFrame(data)

X = df[["Age", "Experience", "Rank"]]
y = df["Go"]

tree = DecisionTreeClassifier(criterion="gini", random_state=0)
tree.fit(X, y)

print(tree.predict([[40, 10, 7]]))   # 1 = go, 0 = don't go

Reading the Rules

Export the learned rules as text or a diagram to explain the model.

from sklearn.tree import export_text, plot_tree
import matplotlib.pyplot as plt

print(export_text(tree, feature_names=list(X.columns)))

plt.figure(figsize=(10, 6))
plot_tree(tree, feature_names=X.columns, class_names=["No", "Go"], filled=True)
plt.show()

Controlling Overfitting

An unrestricted tree grows until every leaf is pure, memorizing noise. Prune it with hyperparameters:

tree = DecisionTreeClassifier(max_depth=3, min_samples_leaf=2, random_state=0)
tree.fit(X, y)

Feature Importance

importances = pd.Series(tree.feature_importances_, index=X.columns)
print(importances.sort_values(ascending=False))

Higher values mean the feature contributed more to reducing impurity across the tree.

Advanced: From Trees to Forests

A single tree is high-variance — small data changes reshape it. Averaging many trees trained on bootstrapped samples (a Random Forest) dramatically improves accuracy and stability. Gradient boosting builds trees sequentially to correct prior errors.

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators=200, random_state=0)
forest.fit(X, y)
print(forest.predict([[40, 10, 7]]))

Best Practices

• Limit depth or leaf size to prevent overfitting on small datasets.
• Use DecisionTreeRegressor for continuous targets.
• Prefer ensembles (Random Forest, Gradient Boosting) when accuracy matters more than a single readable rule set.
• Evaluate on a held-out test set, not the training data.