Implement Autoencoders Using Python

Implementing a Basic Autoencoder Using Python and Keras

Autoencoders are a type of artificial neural network (ANN) used in unsupervised learning to learn efficient data codings (representations). They achieve this by learning to compress input data into a lower-dimensional ‘latent space’ and then reconstruct the original input from this compressed representation. They have become an essential tool in machine learning for tasks like dimensionality reduction, feature learning, and anomaly detection. This chapter provides a step-by-step guide to implement a simple autoencoder in Python using TensorFlow/Keras, with the MNIST dataset of handwritten digits as our example.

We will cover the necessary setup, data preprocessing, model building, training, and visualization of the results. This example focuses on a basic dense autoencoder.

Step-by-Step Guide to Implement Autoencoders Using Python

Let’s explore the steps to implement a basic autoencoder using Python and the Keras API within TensorFlow.

Step 1: Setting Up the Environment

Before starting, ensure that the necessary libraries are installed. If not, you can use pip to install them. It’s highly recommended to use a virtual environment.

pip install numpy matplotlib tensorflow

Step 2: Importing Libraries

Once the installations are complete, we need to import the necessary modules from these libraries.

# Import necessary libraries
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense, Flatten, Reshape
from tensorflow.keras.optimizers import Adam

print(f"TensorFlow version: {tf.__version__}") # Optional: Check TensorFlow version

Step 3: Loading and Preprocessing the MNIST Dataset

In this step, we will load the MNIST handwritten digit dataset. The pixel values will be normalized to the range [0, 1]. Since this is a simple dense autoencoder, we will flatten the images.

# Load the MNIST dataset (we only need the images, not the labels for unsupervised learning)
(x_train, _), (x_test, _) = mnist.load_data()

# Normalize the data to the range [0, 1]
x_train = x_train.astype('float32') / 255.0
x_test = x_test.astype('float32') / 255.0

# Flatten the 28x28 images into 784-dimensional vectors
x_train_flat = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test_flat = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

print(f"x_train_flat shape: {x_train_flat.shape}")
print(f"x_test_flat shape: {x_test_flat.shape}")

Step 4: Building the Autoencoder Model

We will build the autoencoder model by defining the encoder and decoder parts. The encoder compresses the input, and the decoder attempts to reconstruct the original input from the compressed representation (latent code).

# Define the dimension of the latent space (compressed representation)
latent_dim = 64  # Example: compress 784 features to 64
input_dim = x_train_flat.shape[1] # Should be 784 for MNIST

# --- Encoder --- #
input_img = Input(shape=(input_dim,), name='encoder_input')
# Encoded representation
encoded = Dense(latent_dim, activation='relu', name='latent_vector')(input_img)

# --- Decoder --- #
# Decoded representation (reconstruction of the input)
decoded = Dense(input_dim, activation='sigmoid', name='decoder_output')(encoded) # Sigmoid for pixel values between 0 and 1

# --- Autoencoder Model --- #
# This model maps an input to its reconstruction
autoencoder = Model(input_img, decoded, name='autoencoder')

# Compile the autoencoder
# Using binary_crossentropy because pixel values are normalized (0-1) and sigmoid activation is used in the output layer.
# This treats each pixel as an independent Bernoulli distribution.
autoencoder.compile(optimizer=Adam(learning_rate=0.001), loss='binary_crossentropy')

# Print the summary of the autoencoder model
autoencoder.summary()

Step 5: Training the Autoencoder Model

Next, we train the autoencoder using the training data. The model learns to reconstruct x_train_flat from x_train_flat itself (i.e., the input is also the target output).

# Train the autoencoder
history = autoencoder.fit(x_train_flat, x_train_flat, # Input and target are the same
                        epochs=30,       # Number of epochs to train
                        batch_size=256,  # Batch size for training
                        shuffle=True,
                        validation_data=(x_test_flat, x_test_flat) # Validate on test data
                       )

Step 6: Visualizing Original and Reconstructed Data

In this final step, we will visualize some of the original test images and their reconstructions produced by the trained autoencoder to assess its performance.

# Predict the reconstructed images from the test set
reconstructed_imgs_flat = autoencoder.predict(x_test_flat)

# Reshape flattened images back to 28x28 to display
reconstructed_imgs = reconstructed_imgs_flat.reshape((len(x_test_flat), 28, 28))
original_imgs = x_test # Use the original x_test for comparison, not x_test_flat

# Number of digits to display
n = 10
plt.figure(figsize=(20, 4))

for i in range(n):
    # Display original image
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(original_imgs[i], cmap='gray')
    plt.title("Original")
    plt.axis('off')

    # Display reconstructed image
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(reconstructed_imgs[i], cmap='gray')
    plt.title("Reconstructed")
    plt.axis('off')

plt.suptitle('Original vs. Reconstructed MNIST Digits', fontsize=16)
plt.tight_layout(rect=[0, 0, 1, 0.96]) # Adjust layout to make space for suptitle
plt.show()

Complete Python Implementation Code

Below is the complete Python script combining all the above steps.

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf # Import tensorflow
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense # Flatten and Reshape not needed for this simple dense AE
from tensorflow.keras.optimizers import Adam

print(f"TensorFlow version: {tf.__version__}")

# Load the MNIST dataset
(x_train, _), (x_test, _) = mnist.load_data()

# Normalize the data
x_train = x_train.astype('float32') / 255.0
x_test = x_test.astype('float32') / 255.0

# Flatten the 28x28 images into 784-dimensional vectors
x_train_flat = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test_flat = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

print(f"x_train_flat shape: {x_train_flat.shape}")
print(f"x_test_flat shape: {x_test_flat.shape}")

# Define the dimension of the latent space
latent_dim = 64
input_dim = x_train_flat.shape[1]

# Encoder
input_img = Input(shape=(input_dim,), name='encoder_input')
encoded = Dense(latent_dim, activation='relu', name='latent_vector')(input_img)

# Decoder
decoded = Dense(input_dim, activation='sigmoid', name='decoder_output')(encoded)

# Autoencoder Model
autoencoder = Model(input_img, decoded, name='autoencoder')
autoencoder.compile(optimizer=Adam(learning_rate=0.001), loss='binary_crossentropy')
autoencoder.summary()

# Train the autoencoder
print("\nTraining the autoencoder...")
history = autoencoder.fit(x_train_flat, x_train_flat,
                        epochs=30, # Reduced epochs for quicker demo, 50 is also fine
                        batch_size=256,
                        shuffle=True,
                        validation_data=(x_test_flat, x_test_flat),
                        verbose=1 # Set to 1 or 2 to see training progress
                       )

print("\nTraining complete. Predicting on test data...")
# Predict the reconstructed images from the test set
reconstructed_imgs_flat = autoencoder.predict(x_test_flat)

# Reshape flattened images back to 28x28 to display
reconstructed_imgs = reconstructed_imgs_flat.reshape((len(x_test_flat), 28, 28))
original_imgs = x_test # Use the original x_test for comparison

# Plotting the results
print("Plotting original vs. reconstructed images...")
n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
    # Display original image
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(original_imgs[i], cmap='gray')
    plt.title("Original")
    plt.axis('off')

    # Display reconstructed image
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(reconstructed_imgs[i], cmap='gray')
    plt.title("Reconstructed")
    plt.axis('off')

plt.suptitle('Original vs. Reconstructed MNIST Digits by Autoencoder', fontsize=16)
plt.tight_layout(rect=[0, 0, 1, 0.96])
plt.show()

# Optional: Plot training & validation loss
plt.figure(figsize=(10, 5))
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('Model Loss During Training')
plt.xlabel('Epoch')
plt.ylabel('Loss (Binary Crossentropy)')
plt.legend()
plt.grid(True)
plt.show()

Output

After running the script, it will first print the TensorFlow version, data shapes, and the summary of the autoencoder model. Then, it will display the training progress for each epoch. Finally, it will show two plots:

1. A figure displaying a set of original MNIST digits from the test set in the top row, and their corresponding reconstructed versions by the autoencoder in the bottom row. Well-trained autoencoders will produce reconstructions that closely resemble the originals, demonstrating the model’s ability to learn and represent the data in a compressed form, though some loss of detail is expected due to the compression.

2. A plot of the training and validation loss over epochs, which helps to assess the learning process and check for overfitting.

Model: "autoencoder"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 encoder_input (InputLayer)  [(None, 784)]             0

 latent_vector (Dense)       (None, 64)                50240

 decoder_output (Dense)      (None, 784)               50960

=================================================================
Total params: 101,200
Trainable params: 101,200
Non-trainable params: 0
_________________________________________________________________

Training the autoencoder...
Epoch 1/30
235/235 [==============================] - 1s 2ms/step - loss: 0.2783 - val_loss: 0.1923
Epoch 2/30
235/235 [==============================] - 0s 2ms/step - loss: 0.1721 - val_loss: 0.1539
...
Epoch 30/30
235/235 [==============================] - 0s 2ms/step - loss: 0.0968 - val_loss: 0.0938

Training complete. Predicting on test data...
313/313 [==============================] - 0s 600us/step
Plotting original vs. reconstructed images...

The visual output would then show the comparison of original and reconstructed digits, followed by the loss curve plot. The reconstructed digits should be recognizable, albeit potentially blurrier than the originals.

Conclusion

Autoencoders are powerful tools for unsupervised learning, applicable to various tasks such as dimensionality reduction, feature extraction, image denoising, and anomaly detection. More complex architectures like Convolutional Autoencoders (for images) or Variational Autoencoders (for generative tasks) build upon these fundamental concepts.

In this chapter, we demonstrated how to implement a simple dense autoencoder using Python and Keras on the MNIST dataset. This involved setting up the environment, preprocessing the data, defining the model architecture (encoder and decoder), training the model, and visualizing the results to evaluate its reconstruction performance. Understanding these basic building blocks is essential for exploring more advanced autoencoder variants and their applications.