Keras - Regression Prediction using MPL

Keras - Regression Prediction using MLP

While previous examples focused on classification (predicting discrete categories), Keras is equally adept at regression tasks, where the goal is to predict continuous numerical values. A common example is predicting house prices based on features like size, location, and number of rooms.

In this chapter, we’ll build a Multi-Layer Perceptron (MLP) using tf.keras to predict house prices using the Boston Housing dataset, a classic regression benchmark dataset available in tf.keras.datasets.

Key differences for Regression Models:

• Output Layer: Typically has a single neuron (if predicting one value) and often uses a linear activation function (or no activation) because the output isn’t restricted to a specific range like probabilities.
• Loss Function: Common choices include Mean Squared Error ('mse') or Mean Absolute Error ('mae'). These measure the average difference between predicted and actual continuous values.
• Metrics: Evaluation often uses metrics like MAE (average absolute difference) or RMSE (Root Mean Squared Error) to understand the typical error magnitude in the original units of the target variable (e.g., dollars for house prices).

Conceptual MLP for Regression:

Input Features -> Dense Hidden Layer(s) (e.g., ReLU activation) -> Dense Output Layer (1 unit, linear activation) -> Predicted Value.

Model Specifications:

• Dataset: Boston Housing (13 input features, 1 target value: median house price).
• Preprocessing: Feature scaling (normalization/standardization) is crucial for regression with features on different scales.
• Architecture: MLP with two hidden Dense layers (e.g., 64 units, ReLU activation).
• Output Layer: Dense layer with 1 unit (no activation specified, defaults to linear).
• Compilation: mse loss, RMSprop or Adam optimizer, mae metric.
• Training: Use callbacks like EarlyStopping to prevent overfitting.

Step 1: Import Necessary Modules

import tensorflow as tf
from tensorflow.keras.datasets import boston_housing
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Normalization # Added Normalization layer
from tensorflow.keras.optimizers import Adam # Using Adam optimizer
from tensorflow.keras.callbacks import EarlyStopping
import numpy as np
import matplotlib.pyplot as plt # For plotting results

# For sklearn preprocessing (alternative to Normalization layer)
# from sklearn.preprocessing import StandardScaler

Step 2: Load Data

(x_train, y_train), (x_test, y_test) = boston_housing.load_data()

print(f"Training data shape: {x_train.shape}")
print(f"Test data shape: {x_test.shape}")
print(f"Training labels shape: {y_train.shape}")
# Example: Look at the first training sample features and label
# print("First training sample features:", x_train[0])
# print("First training sample label (price in $1000s):", y_train[0])

The dataset has 13 input features per sample.

Step 3: Preprocess the Data (Feature Scaling)

Features in the Boston Housing dataset have varying ranges. Scaling them (e.g., to have zero mean and unit variance) helps the model train more effectively. We can use either scikit-learn’s StandardScaler or Keras’s Normalization layer.

Method 1: Using Keras Normalization Layer (Recommended)

This layer can be included directly in the model. It adapts to the training data and applies the same transformation during inference.

# Initialize the Normalization layer
normalizer = Normalization(axis=-1) # Normalize features independently

# Adapt the layer to the training data (calculates mean and variance)
normalizer.adapt(x_train)

print('\nFeature means:', normalizer.mean.numpy())
print('Feature variances:', normalizer.variance.numpy())

# Note: We will add this 'normalizer' layer as the first layer in our model.

Method 2: Using Scikit-learn StandardScaler (Alternative)

This performs scaling outside the model.

# scaler = StandardScaler()
# x_train_scaled = scaler.fit_transform(x_train)
# x_test_scaled = scaler.transform(x_test) # Use transform, not fit_transform on test data!

# print("\nShape after scaling (sklearn):")
# print("Train:", x_train_scaled.shape)
# print("Test:", x_test_scaled.shape)

# If using this method, the model's input_shape would still be (13,)
# and you would pass x_train_scaled/x_test_scaled to fit/evaluate.

Step 4: Build the Regression Model

We’ll build a simple Sequential model, including the Normalization layer (if using Method 1).

input_shape = (x_train.shape[1],) # Should be (13,)

model = Sequential([
    tf.keras.Input(shape=input_shape), # Define input layer explicitly
    normalizer,                      # Add the adapted normalization layer (Method 1)
    Dense(64, activation='relu'),
    Dense(64, activation='relu'),
    Dense(1)                         # Output layer: 1 unit, linear activation (default)
], name="boston_housing_mlp")

model.summary()

If using scikit-learn scaling (Method 2), omit the normalizer layer from the model definition.

Step 5: Compile the Model

Compile with a suitable regression loss and metric.

model.compile(loss='mean_absolute_error', # Using MAE as loss
              optimizer=Adam(learning_rate=0.001),
              metrics=['mae', 'mse']) # Track both MAE and MSE

Step 6: Train the Model

Train the model, using EarlyStopping to prevent overfitting and find a good stopping point.

epochs = 500

# Define EarlyStopping callback
early_stopping = EarlyStopping(
    monitor='val_loss', # Monitor validation loss
    patience=20,        # Number of epochs with no improvement after which training will be stopped
    restore_best_weights=True # Restore model weights from the epoch with the best value of the monitored quantity
)

print("\nTraining Model...")
history = model.fit(
    x_train, y_train, # Pass original x_train if using Normalization layer
    # x_train_scaled, y_train, # Pass scaled data if using sklearn scaler
    epochs=epochs,
    validation_split=0.2, # Use 20% of training data for validation
    verbose=0, # Set verbose=0 for less output during training
    callbacks=[early_stopping]
)

print("\nTraining Complete.")
print(f"Stopped at epoch: {early_stopping.stopped_epoch}")

Step 7: Evaluate the Model

Evaluate performance on the unseen test set.

print("\nEvaluating on Test Data...")
results = model.evaluate(x_test, y_test, verbose=0)
# results = model.evaluate(x_test_scaled, y_test, verbose=0) # If using sklearn scaler

print(f"Test Loss (MAE): {results[0]:.2f}")
print(f"Test MAE: {results[1]:.2f}")
print(f"Test MSE: {results[2]:.2f}")

The ‘Test MAE’ tells you the average absolute difference between the predicted and actual house prices (in $1000s) on the test set.

Step 8: Make Predictions and Visualize

Predict on the test set and compare predictions to actual values.

test_predictions = model.predict(x_test).flatten()
# test_predictions = model.predict(x_test_scaled).flatten() # If using sklearn scaler

# Simple scatter plot of predictions vs. actual values
plt.figure(figsize=(8, 8))
plt.scatter(y_test, test_predictions)
plt.xlabel('True Values [Price in $1000s]')
plt.ylabel('Predictions [Price in $1000s]')
plt.axis('equal')
plt.axis('square')
plt.xlim([0, plt.xlim()[1]])
plt.ylim([0, plt.ylim()[1]])
_ = plt.plot([-100, 100], [-100, 100]) # Diagonal line
plt.title('Regression Predictions vs True Values')
plt.grid(True)
plt.show()

# Plot training & validation loss curves
plt.figure(figsize=(10, 6))
plt.plot(history.history['loss'], label='Training MAE Loss')
plt.plot(history.history['val_loss'], label='Validation MAE Loss')
plt.title('Training and Validation Loss (MAE)')
plt.xlabel('Epoch')
plt.ylabel('Loss (MAE)')
plt.legend()
plt.grid(True)
plt.show()

The scatter plot helps visualize the model’s performance. Points close to the diagonal line indicate accurate predictions. The loss curves show how the model learned over epochs and help diagnose overfitting or underfitting.