R - Decision Tree

R - Modern Decision Tree Modeling

A decision tree is a powerful and interpretable machine learning model that represents choices and their outcomes in a tree-like structure. It’s used for both classification (e.g., predicting if an email is spam) and regression (e.g., predicting house prices). Each node in the tree represents a test on a feature, each branch represents the outcome of the test, and each leaf node represents a class label or a continuous value.

Modern R workflows for machine learning are often built using the tidymodels framework, a collection of packages that provides a cohesive, tidy-first approach to the entire modeling process, from data splitting to model evaluation.

Setup: The Tidymodels Ecosystem

First, we need to install the necessary packages. tidymodels is the core framework, and rpart.plot is excellent for visualizing the trees we create.

# Install the packages if you don't have them yet
# install.packages(c("tidymodels", "rpart.plot"))

# Load the library
library(tidymodels)

A Practical Example: The Machine Learning Workflow

We will use the readingSkills dataset to predict whether a person is a nativeSpeaker based on their age, shoe size, and test score. We will follow a standard, robust machine learning workflow.

Step 1: Load and Prepare Data

We load the data and ensure our outcome variable (nativeSpeaker) is a factor, which is required for classification.

# Load the data (it's part of the 'party' package, but we can load it directly)
data(readingSkills, package = "party")

# Glimpse the data structure
glimpse(readingSkills)

Output of glimpse:

Rows: 200
Columns: 4
$ nativeSpeaker <fct> yes, yes, no, yes, yes, yes, no, yes, no, yes, ...
$ age           <int> 5, 6, 11, 7, 11, 10, 8, 9, 10, 6, 8, 5, ...
$ shoeSize      <dbl> 24.83, 25.95, 30.42, 28.66, 31.88, 30.08, ...
$ score         <dbl> 32.29, 36.63, 49.61, 40.28, 55.46, 52.83, ...

Step 2: Split Data into Training and Testing Sets

This is a critical step. We train the model on the training set and evaluate its performance on the unseen testing set to get an unbiased estimate of its real-world performance.

# Set a seed for reproducibility
set.seed(123)

# Create the data split object
data_split <- initial_split(readingSkills, prop = 0.80, strata = nativeSpeaker)

# Extract training and testing sets
train_data <- training(data_split)
test_data  <- testing(data_split)

Step 3: Define and Train the Model

With tidymodels, we define a model specification (what kind of model it is) and then fit it to our data.

# 1. Define the model specification
# We'll use the 'rpart' engine (a classic R decision tree algorithm)
tree_spec <- decision_tree() %>%
  set_engine("rpart") %>%
  set_mode("classification")

# 2. Fit the model to the training data
tree_fit <- tree_spec %>%
  fit(nativeSpeaker ~ ., data = train_data)

Step 4: Visualize the Tree

Visualizing the tree is key to understanding how it makes decisions. The rpart.plot package creates excellent, readable plots.

# Load the visualization library
library(rpart.plot)

# Create the plot
rpart.plot(tree_fit$fit, box.palette = "RdBu", shadow.col = "gray", nn = TRUE)

Description of Output: The code generates a plot showing a decision tree. The root node at the top likely splits on score. For example, a branch for score < 40 might lead to another node that splits on age, while the branch for score >= 40 leads directly to a leaf node predicting yes (native speaker). The plot is colored, with percentages and counts shown in each node, making it highly interpretable.

Step 5: Evaluate Model Performance

Now, let’s see how well our model performs on the unseen test data.

# Make predictions on the test set
predictions <- predict(tree_fit, new_data = test_data, type = "class")

# Combine predictions with the actual truth
results <- bind_cols(test_data, predictions) %>%
  select(nativeSpeaker, .pred_class)

# Calculate performance metrics, like a confusion matrix
conf_mat(results, truth = nativeSpeaker, estimate = .pred_class)

The confusion matrix shows how many predictions were correct and incorrect:

           Truth
Prediction  no yes
       no   19   1
       yes   1  19

Conclusion & Best Practices

The confusion matrix shows excellent performance on our test set. From the visual plot, we can derive simple rules, for example: “If a person’s score is greater than 48, they are very likely a native speaker.”

• Overfitting: Simple trees can overfit the training data. tidymodels provides powerful tools for tuning hyperparameters (like tree depth) to find the best model that generalizes well.
• Alternative Engines: Besides rpart, you can use other engines like C5.0 or partykit::ctree within the same tidymodels framework by changing set_engine().
• Ensemble Methods: For higher performance, individual decision trees are often combined into powerful ensembles like Random Forests (ranger engine) or Gradient Boosted Machines (xgboost engine).