Theano - Trivial Training Example

Theano - Basic Training Example (Gradient Descent)

Context: This example demonstrates a fundamental machine learning concept: updating model parameters (weights) using gradient descent to minimize a cost function. While Theano was effective for this, modern frameworks offer more user-friendly APIs for training.

Theano’s strength lay in defining the computational graph, automatically calculating gradients (theano.grad), and specifying parameter updates (updates in theano.function), which it could then compile for efficient execution, especially over many training iterations.

Problem Statement

Let’s consider a very simple linear model. We have a set of input features (a vector x) and corresponding weights (a vector W). The model’s output y is the sum of the element-wise product of features and weights: y = sum(x * W).

Our goal is to adjust the weights W so that for a given input x (e.g., all ones), the output y gets closer to a specific target value (e.g., 100). We will use gradient descent to minimize the squared error cost function: cost = (target - y)^2.

Implementation Steps

1. Imports and Setup

import theano
import theano.tensor as tt
import numpy as np

print(f"Using Theano version: {theano.__version__}")

2. Define Symbolic Variables

We need symbolic variables for the input features, the target value, and the weights. The weights need special handling because their values must persist and be updated across multiple calls to the training function. For this, Theano uses shared variables.

# Symbolic input vector (e.g., 4 features, float32)
x = tt.fvector('x')

# Symbolic target value (scalar, float32)
target = tt.fscalar('target')

# Initial weights (as a NumPy array)
initial_W = np.array([0.1, 0.25, 0.15, 0.3], dtype='float32')

# Create a Theano SHARED variable for weights.
# Shared variables have persistent values that can be updated.
W = theano.shared(value=initial_W, name='W')

print(f"Initial weights (W): {W.get_value()}")

Shared variables (theano.shared) are central to training in Theano. They hold the model parameters that gradient descent will modify. You can access their current numerical value using .get_value() and update them via the updates mechanism in theano.function.

3. Define the Symbolic Computation Graph

Now, build the expressions for the model’s output, the cost function, and the gradients.

# Model output: y = sum(x * W)
y = (x * W).sum()

# Cost function: squared error
cost = tt.sqr(target - y)

# Calculate gradients: How cost changes with respect to W
# theano.grad(cost, wrt) computes the gradient of 'cost' with respect to 'wrt'
gradients = tt.grad(cost=cost, wrt=W)
# Note: tt.grad returns a symbolic variable representing the gradient

4. Define the Update Rule

Specify how the shared variable W should be updated using the computed gradients. This is the core of the gradient descent step: W_new = W_old - learning_rate * gradient.

# Define the learning rate
learning_rate = 0.1

# Define the update rule for the shared variable W
W_updated = W - (learning_rate * gradients)

# Create the 'updates' list for theano.function
# It's a list of tuples: (shared_variable_to_update, symbolic_expression_for_new_value)
updates = [(W, W_updated)]

5. Compile the Training Function

Compile a Theano function that takes input x and target, calculates the output y (optional, but useful to see progress), and applies the specified updates to the shared variable W.

# Compile the function
# Inputs: symbolic input x and target
# Outputs: symbolic output y (to monitor progress)
# Updates: the list defining how to update shared variables
train_step = theano.function(
    inputs=[x, target],
    outputs=y, # We want to see the output value during training
    updates=updates
)

6. Run the Training Loop

Call the compiled train_step function repeatedly with sample data. In each call, the function computes the output, calculates gradients internally, and updates the weights W according to the rule.

# Define the training data (using all 1s for features for simplicity)
training_x = np.array([1.0, 1.0, 1.0, 1.0], dtype='float32')
training_target = 100.0
num_iterations = 10

print(f"\nStarting training for {num_iterations} iterations...")
print(f"Target value: {training_target}")

for i in range(num_iterations):
    # Execute one training step
    current_output = train_step(training_x, training_target)

    # Print progress
    print(f"Iteration: {i}")
    print(f"  Current Output (y): {current_output:.4f}")
    print(f"  Updated Weights (W): {W.get_value()}")

Full Program Listing

import theano
import theano.tensor as tt
import numpy as np

print(f"Using Theano version: {theano.__version__}")

# 1. Define Symbolic Variables
x = tt.fvector('x')       # Input features (float32)
target = tt.fscalar('target') # Target value (float32)

# Initial weights
initial_W = np.array([0.1, 0.25, 0.15, 0.3], dtype='float32')
# Shared variable for weights (persistent and updatable)
W = theano.shared(value=initial_W, name='W')
print(f"Initial weights (W): {W.get_value()}")

# 2. Define the Symbolic Computation Graph
y = (x * W).sum()            # Model output
cost = tt.sqr(target - y)   # Cost function (squared error)
gradients = tt.grad(cost=cost, wrt=W) # Automatic differentiation

# 3. Define the Update Rule (Gradient Descent)
learning_rate = 0.1
W_updated = W - (learning_rate * gradients)
updates = [(W, W_updated)] # List of updates for theano.function

# 4. Compile the Training Function
train_step = theano.function(
    inputs=[x, target],
    outputs=y,          # Return the current output for monitoring
    updates=updates     # Apply the weight updates
)

# 5. Run the Training Loop
training_x = np.array([1.0, 1.0, 1.0, 1.0], dtype='float32')
training_target = 100.0
num_iterations = 10

print(f"\nStarting training for {num_iterations} iterations...")
print(f"Target value: {training_target}")

for i in range(num_iterations):
    current_output = train_step(training_x, training_target)
    print(f"Iteration: {i}")
    print(f"  Current Output (y): {current_output:.4f}")
    # Access the updated weights after the step
    print(f"  Updated Weights (W): {W.get_value()}")

print("\nTraining finished.")
print(f"Final weights: {W.get_value()}")
print(f"Final output for input {training_x}: {train_step(training_x, training_target):.4f}")

Expected Output Analysis

You should observe the following:

• The initial output y will be the sum of initial_W (0.1 + 0.25 + 0.15 + 0.3 = 0.8).
• In each iteration, the Current Output (y) will get progressively closer to the target value of 100.0.
• The Updated Weights (W) will change in each iteration. Since the input x is all ones and the target is 100, the weights should ideally converge towards values that sum close to 100 (e.g., roughly 25 each, although the exact values depend on the learning rate and number of iterations).

This example, though simple, encapsulates the core mechanics of how Theano was used for optimization in machine learning: defining the model and cost symbolically, letting Theano compute gradients, defining update rules for shared parameters, and compiling an efficient function to execute the training steps repeatedly.