Skip to content
W3S101

Multi-Layer Perceptron Learning

TensorFlow - Multi-Layer Perceptron Learning with Keras

Section titled “TensorFlow - Multi-Layer Perceptron Learning with Keras”

A Multi-Layer Perceptron (MLP) is a fundamental type of artificial neural network. It consists of multiple layers of nodes (neurons), where each node in one layer connects with a certain weight to every node in the next layer. Typically, an MLP has at least three layers: an input layer, one or more hidden layers, and an output layer.

Conceptually, an MLP works as follows: The input layer receives the initial data. Each subsequent hidden layer transforms the data from the previous layer using an activation function. The output layer produces the final result, such as a classification or regression value. The network learns by adjusting the weights between neurons, typically using an algorithm like backpropagation.

MLPs are commonly used for supervised learning tasks, such as image classification, where the model learns from labeled data. The backpropagation algorithm is a cornerstone for training MLPs, allowing the network to adjust its weights based on the error in its predictions.

Let’s implement an MLP for an image classification problem using the MNIST dataset. We will use TensorFlow 2.x with the Keras API, which provides a high-level, user-friendly way to build and train neural networks.

# Import TensorFlow and other necessary libraries
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import matplotlib.pyplot as plt
import numpy as np


# Parameters
learning_rate = 0.001
training_epochs = 20
batch_size = 100
display_step = 1


# Network Parameters
n_hidden_1 = 256  # Number of features in the 1st hidden layer
n_hidden_2 = 256  # Number of features in the 2nd hidden layer
n_input = 784     # MNIST data input (image shape: 28*28 pixels)
n_classes = 10    # MNIST total classes (0-9 digits)


# Load MNIST data using Keras API
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()


# Preprocess the data
# Flatten the images from 28x28 to 784
x_train = x_train.reshape(-1, n_input).astype('float32') / 255.0
x_test = x_test.reshape(-1, n_input).astype('float32') / 255.0


# Convert labels to one-hot encoded vectors
y_train = keras.utils.to_categorical(y_train, num_classes=n_classes)
y_test = keras.utils.to_categorical(y_test, num_classes=n_classes)


# Build the MLP model using Keras Sequential API
model = keras.Sequential([
    layers.InputLayer(input_shape=(n_input,)),
    layers.Dense(n_hidden_1, activation='relu', name='hidden_layer_1'),
    layers.Dense(n_hidden_2, activation='relu', name='hidden_layer_2'),
    layers.Dense(n_classes, activation='softmax', name='output_layer')
])


# Compile the model
# We use Adam optimizer and CategoricalCrossentropy loss for multi-class classification.
# ReLU activation is common for hidden layers, and Softmax for the output layer in classification.
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
model.compile(optimizer=optimizer,
              loss='categorical_crossentropy',
              metrics=['accuracy'])


# Print model summary
model.summary()


# Training the model
print("Starting training...")
history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=training_epochs,
                    validation_data=(x_test, y_test),
                    verbose=1 # Set to 2 for one line per epoch, 0 for silent
                   )


print("Training phase finished.")


# Plotting training history (loss and accuracy)
epoch_set = range(1, training_epochs + 1)


plt.figure(figsize=(12, 4))


plt.subplot(1, 2, 1)
plt.plot(epoch_set, history.history['loss'], 'o-', label='Training Loss')
plt.plot(epoch_set, history.history['val_loss'], 'o-', label='Validation Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.title('Training and Validation Loss')
plt.legend()


plt.subplot(1, 2, 2)
plt.plot(epoch_set, history.history['accuracy'], 'o-', label='Training Accuracy')
plt.plot(epoch_set, history.history['val_accuracy'], 'o-', label='Validation Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.title('Training and Validation Accuracy')
plt.legend()


plt.tight_layout()
plt.show()


# Evaluate the model on the test set
print("\nEvaluating model on test data...")
loss, accuracy = model.evaluate(x_test, y_test, verbose=0)
print(f"Test Loss: {loss:.4f}")
print(f"Test Accuracy: {accuracy:.4f}")


# For more detailed exploration of training with Keras, see:
# https://www.tensorflow.org/guide/keras/train_and_evaluate

The code above defines an MLP, loads and preprocesses the MNIST dataset, then trains the model. The output will show the model summary, training progress per epoch (loss and accuracy), and finally the test accuracy. The plots will visualize how loss and accuracy change over epochs for both training and validation sets, which is crucial for diagnosing overfitting or underfitting.

Key changes from older TensorFlow versions:

  • We use tf.keras.datasets.mnist to load data.
  • The model is built using tf.keras.Sequential and tf.keras.layers.Dense.
  • Activation functions like ‘relu’ (Rectified Linear Unit) are commonly used for hidden layers due to their efficiency and ability to mitigate vanishing gradient problems. ‘softmax’ is used for the output layer for multi-class classification.
  • The model is compiled with an optimizer, loss function, and metrics.
  • Training is done using model.fit() and evaluation with model.evaluate().
  • TensorFlow 2.x operates in eager execution mode by default, making debugging more intuitive. Sessions and placeholders are no longer required for this style of programming.