Hidden Layers of Perceptron

TensorFlow - Hidden Layers of Perceptron for Function Approximation

In this chapter, we will explore how a Multi-Layer Perceptron (MLP) with hidden layers can learn to approximate a function. We’ll use a simple neural network with one hidden layer to learn a cosine function from a set of generated points (x, f(x)). This task is a form of regression.

The code below demonstrates building and training such an MLP using TensorFlow 2.x and the Keras API.

# Importing necessary modules
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
import math
import matplotlib.pyplot as plt

# Set a seed for reproducibility
np.random.seed(1000)
tf.random.set_seed(1000)

# Define the function to learn (cosine with some noise)
function_to_learn = lambda x: np.cos(x) + 0.1 * np.random.randn(*x.shape)

# Network and training parameters
num_hidden_neurons = 20 # Increased for better approximation capability
NUM_points = 1000
batch_size = 100
NUM_EPOCHS = 1000 # Reduced epochs, modern optimizers converge faster

# Generate data
all_x = np.float32(np.random.uniform(-2 * math.pi, 2 * math.pi, (NUM_points, 1)))
np.random.shuffle(all_x)

# Split data into training and validation sets
train_size = int(0.9 * NUM_points)
x_training = all_x[:train_size]
y_training = function_to_learn(x_training)

x_validation = all_x[train_size:]
y_validation = function_to_learn(x_validation)

# Plot the generated data
plt.figure(1)
plt.scatter(x_training, y_training, c='blue', label='Training Data', s=10)
plt.scatter(x_validation, y_validation, c='pink', label='Validation Data', s=10)
plt.legend()
plt.title('Generated Data for Function Approximation')
plt.xlabel('x')
plt.ylabel('f(x)')
plt.show()

# Build the model using Keras Sequential API
model = keras.Sequential([
    layers.InputLayer(input_shape=(1,)), # Input layer for single feature x
    layers.Dense(num_hidden_neurons, activation='tanh', name='hidden_layer'), # Using tanh for smoother approximation
    layers.Dense(1, name='output_layer') # Output layer with 1 neuron for f(x), linear activation by default
])

# Compile the model
# For regression, Mean Squared Error (MSE) is a common loss function.
optimizer = keras.optimizers.Adam(learning_rate=0.01) # Adam optimizer
model.compile(optimizer=optimizer, loss='mean_squared_error')

# Print model summary
model.summary()

# Train the model
print("Starting training...")
history = model.fit(x_training, y_training,
                    batch_size=batch_size,
                    epochs=NUM_EPOCHS,
                    validation_data=(x_validation, y_validation),
                    verbose=0 # Set to 1 or 2 to see epoch progress
                   )

print("Training finished.")

# Plot training and validation loss
plt.figure(2)
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.xlabel('Epochs')
plt.ylabel('Cost (MSE)')
plt.title('MLP Function Approximation Training Cost')
plt.legend()
plt.show()

# Predict on the validation set and plot against actual values
predictions = model.predict(x_validation)

plt.figure(3)
plt.scatter(x_validation, y_validation, c='pink', label='Actual Validation Data', s=10)
plt.scatter(x_validation, predictions, c='green', label='Model Predictions', s=10)
# For a clearer plot of the learned function, sort x_validation if it's not already
# and plot lines instead of scatter for predictions on a dense range of x.
x_dense = np.linspace(-2 * math.pi, 2 * math.pi, 200).reshape(-1, 1)
y_dense_pred = model.predict(x_dense)
plt.plot(x_dense, y_dense_pred, color='red', label='Learned Function Approximation')
plt.legend()
plt.title('Function Approximation: Actual vs. Predicted')
plt.xlabel('x')
plt.ylabel('f(x)')
plt.show()

# For more on regression with Keras:
# https://www.tensorflow.org/tutorials/keras/regression

Output and Interpretation

The script will generate three plots:

Generated Data: This plot shows the scattered training (blue) and validation (pink) data points derived from the cosine function with added noise. This visualizes the data our model will attempt to learn.
Training Cost: This plot displays the Mean Squared Error (MSE) for both the training and validation sets over epochs. Ideally, both losses should decrease and converge, indicating the model is learning effectively. A large gap between training and validation loss might suggest overfitting.
Function Approximation: This plot overlays the actual validation data points (pink), the model’s predictions on these points (green), and a smooth curve (red) representing the function learned by the model over a dense range of x values. A close match between the red curve and the general trend of the pink points indicates successful approximation.

The hidden layer with its non-linear activation function (tanh in this case, or ReLU, sigmoid) is crucial. It allows the MLP to learn complex, non-linear relationships in the data, which a simple linear model could not.