Scikit Learn - Decision Trees

Scikit-Learn - Decision Trees

This chapter introduces Decision Trees, a fundamental non-parametric supervised learning method available in Scikit-Learn for both classification and regression tasks.

Decision Trees (DTs) are versatile machine learning algorithms that predict the value of a target variable by learning simple decision rules inferred from the data features. They create a model that resembles a tree structure. Each internal node in the tree represents a ‘test’ on an attribute (e.g., is feature X <= value Y?), each branch represents the outcome of the test, and each leaf node represents a class label (in classification) or a continuous value (in regression). The paths from root to leaf represent classification or regression rules.

Decision Tree Algorithms Overview

Several algorithms exist for building decision trees. Some historically significant ones include:

ID3 (Iterative Dichotomiser 3)

Developed by Ross Quinlan, ID3 typically uses Information Gain to select the best feature to split on at each node. It’s primarily designed for categorical features and targets.

C4.5

A successor to ID3, also by Quinlan. C4.5 can handle both continuous and categorical features, deals with missing values, and uses Gain Ratio as a splitting criterion to mitigate bias towards features with many values. It also prunes trees after construction.

C5.0

A later commercial version by Quinlan, often more accurate and memory-efficient than C4.5, supporting advanced features like boosting.

CART (Classification and Regression Trees)

Developed by Breiman, Friedman, Olshen, and Stone. CART constructs binary trees (each internal node has exactly two outgoing edges). For classification, it typically uses the Gini impurity as a splitting criterion, and for regression, it often uses variance reduction (e.g., Mean Squared Error). Scikit-Learn’s decision tree implementation is an optimized version of CART.

Decision Tree Classifier (DecisionTreeClassifier)

For classification tasks, Scikit-Learn provides sklearn.tree.DecisionTreeClassifier. It’s capable of multi-class classification.

Key Parameters

Important parameters for sklearn.tree.DecisionTreeClassifier:

Parameter	Description
criterion	{‘gini’, ‘entropy’, ‘log_loss’}, default=‘gini’. The function to measure the quality of a split. ‘gini’ for Gini impurity, ‘entropy’ or ‘log_loss’ for information gain.
splitter	{‘best’, ‘random’}, default=‘best’. The strategy used to choose the split at each node. ‘best’ chooses the best split among all features; ‘random’ chooses the best split among a random subset of features.
max_depth	int, default=None. The maximum depth of the tree. If None, nodes are expanded until all leaves are pure or contain less than min_samples_split samples.
min_samples_split	int or float, default=2. The minimum number of samples required to split an internal node.
min_samples_leaf	int or float, default=1. The minimum number of samples required to be at a leaf node.
min_weight_fraction_leaf	float, default=0.0. The minimum weighted fraction of the sum total of weights required to be at a leaf node.
max_features	int, float or {‘auto’, ‘sqrt’, ‘log2’}, default=None. The number of features to consider when looking for the best split. If None, max_features=n_features.
random_state	int, RandomState instance or None, default=None. Controls the randomness of the estimator (e.g., for splitter='random' or max_features < n_features).
max_leaf_nodes	int, default=None. Grow a tree with max_leaf_nodes in best-first fashion. If None, then an unlimited number of leaf nodes.
min_impurity_decrease	float, default=0.0. A node will be split if this split induces a decrease of the impurity greater than or equal to this value.
class_weight	dict, list of dicts, ‘balanced’ or None, default=None. Weights associated with classes. If ‘balanced’, weights are inversely proportional to class frequencies.
ccp_alpha	non-negative float, default=0.0. Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ccp_alpha will be chosen.

Key Attributes

After fitting DecisionTreeClassifier:

Attribute	Description
classes_	array of shape (n_classes,) or a list of such arrays. The class labels.
feature_importances_	array of shape (n_features,). The Gini importance (or impurity-based importance) of features.
max_features_	int. The inferred value of max_features.
n_classes_	int or list of int. The number of classes.
n_features_in_	int. Number of features seen during fit.
n_outputs_	int. The number of outputs when fit is performed.
tree_	Tree object. The underlying Tree object. Provides low-level access to tree structure.

Key Methods

Common methods:

Method	Description
fit(X, y[, sample_weight])	Build a decision tree classifier from the training set (X, y).
predict(X)	Predict class value for X.
predict_proba(X)	Predict class probabilities of the input samples X.
predict_log_proba(X)	Predict class log-probabilities of the input samples X.
score(X, y[, sample_weight])	Return the mean accuracy on the given test data and labels.
apply(X)	Return the index of the leaf that each sample is predicted as.
decision_path(X)	Return the decision path in the tree for each sample in X.
get_depth()	Return the depth of the decision tree.
get_n_leaves()	Return the number of leaves of the decision tree.
get_params([deep])	Get parameters for this estimator.
set_params(**params)	Set the parameters of this estimator.

Implementation Example: Iris Dataset

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score

# Load Iris dataset
iris = load_iris()
X, y = iris.data, iris.target
feature_names = iris.feature_names
target_names = iris.target_names

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize and fit Decision Tree Classifier
dt_clf = DecisionTreeClassifier(max_depth=3, random_state=42)
dt_clf.fit(X_train, y_train)

# Make predictions
y_pred = dt_clf.predict(X_test)
print(f"Accuracy on test set: {accuracy_score(y_test, y_pred):.4f}")

# Predict probability for a sample point
sample_point_proba = dt_clf.predict_proba(X_test[:1])
print(f"Predicted probabilities for first test sample: {sample_point_proba}")
print(f"Actual class for first test sample: {target_names[y_test[0]]}")
print(f"Predicted class for first test sample: {target_names[y_pred[0]]}")

Output Example:

Accuracy on test set: 0.9778
Predicted probabilities for first test sample: [[0.  0.90697674 0.09302326]]
Actual class for first test sample: versicolor
Predicted class for first test sample: versicolor

Visualizing the tree can be done using sklearn.tree.plot_tree (requires Matplotlib).

import matplotlib.pyplot as plt
from sklearn.tree import plot_tree

# plt.figure(figsize=(15,10))
# plot_tree(dt_clf, filled=True, feature_names=feature_names, class_names=list(target_names), rounded=True)
# plt.title("Decision Tree for Iris Dataset")
# plt.show()

The plot would display a tree structure showing decision nodes (conditions on features), branches, and leaf nodes (predicted class and sample distribution).

Decision Tree Regressor (DecisionTreeRegressor)

For regression tasks, where the target variable is continuous, Scikit-Learn provides sklearn.tree.DecisionTreeRegressor.

Key Parameters

Parameters are largely similar to DecisionTreeClassifier. Key differences:

• criterion: {‘squared_error’, ‘friedman_mse’, ‘absolute_error’, ‘poisson’}, default=‘squared_error’. Function to measure the quality of a split. ‘squared_error’ (MSE), ‘friedman_mse’ (MSE with Friedman’s improvement score), ‘absolute_error’ (MAE), ‘poisson’ (Poisson deviance).
• No class_weight parameter.

Key Attributes

Similar to DecisionTreeClassifier, but it lacks classes_ and n_classes_ attributes.

Key Methods

Similar to DecisionTreeClassifier, but lacks predict_proba() and predict_log_proba() methods.

Implementation Example: Simple Regression

from sklearn.tree import DecisionTreeRegressor
import numpy as np

# Sample data for regression
rng = np.random.RandomState(1)
X_reg = np.sort(5 * rng.rand(80, 1), axis=0)
y_reg = np.sin(X_reg).ravel() + rng.randn(80) * 0.1 # y = sin(x) + noise
y_reg[::5] += 3 * (0.5 - rng.rand(16)) # Add some noise/outliers

# Fit Decision Tree Regressor
dt_reg = DecisionTreeRegressor(max_depth=4, random_state=42)
dt_reg.fit(X_reg, y_reg)

# Predict on new data points
X_test_reg = np.arange(0.0, 5.0, 0.01)[:, np.newaxis]
y_pred_reg = dt_reg.predict(X_test_reg)

print(f"Shape of X_test_reg: {X_test_reg.shape}")
print(f"Shape of y_pred_reg: {y_pred_reg.shape}")
print(f"Prediction for X=2.5: {dt_reg.predict([[2.5]])}")

# Visualize the results (optional)
# plt.figure(figsize=(10,6))
# plt.scatter(X_reg, y_reg, s=20, edgecolor="black", c="darkorange", label="data")
# plt.plot(X_test_reg, y_pred_reg, color="cornflowerblue", label="max_depth=4", linewidth=2)
# plt.xlabel("data")
# plt.ylabel("target")
# plt.title("Decision Tree Regression")
# plt.legend()
# plt.show()

Output Example:

Shape of X_test_reg: (500, 1)
Shape of y_pred_reg: (500,)
Prediction for X=2.5: [0.70817608]

The plot for regression would show the original data points and a step-wise function representing the decision tree’s predictions.

Decision trees are intuitive and easy to visualize. However, single trees can be prone to overfitting. Ensemble methods like Random Forests and Gradient Boosting Trees, which build upon decision trees, often yield better performance. Minimal Cost-Complexity Pruning (ccp_alpha) is an important technique to control tree size and prevent overfitting. For more information, see the Scikit-Learn Decision Trees documentation.