Theano - Functions

Theano - Functions: Compiling and Executing Graphs

In Theano, defining symbolic variables and expressions only builds the computational graph. It describes what computation should be done, but doesn’t actually perform it. The bridge between the symbolic graph definition and its numerical execution is the theano.function.

theano.function takes your symbolic graph, compiles it into highly optimized executable code, and returns a callable Python object (a function) that you can use to run the computation with concrete data.

The Compilation Process

When you call theano.function, several things happen behind the scenes:

• Graph Analysis: Theano analyzes the input and output variables and the operations connecting them.
• Optimization: It applies numerous optimizations to the graph structure. This might involve algebraic simplification (e.g., removing redundant operations), rewriting expressions for numerical stability, fusing operations, and planning memory allocation.
• Code Generation: Parts of the optimized graph, especially those involving loops or tensor operations, may be compiled into C/C++ or CUDA code for speed.
• Function Creation: It packages the optimized, compiled code into a callable Python object.

This compilation step can take some time, especially for complex graphs, but the resulting function is typically much faster to execute repeatedly than interpreting the original Python/NumPy equivalent.

Basic Syntax

The basic syntax for creating a Theano function is:

import theano
import theano.tensor as tt

# 1. Define symbolic inputs
input_var1 = tt.dmatrix('in1')
input_var2 = tt.dscalar('in2')

# 2. Define symbolic expression(s) / output(s)
output_var = input_var1 * input_var2 # Example: scale matrix by scalar

# 3. Compile the function
compiled_function = theano.function(
    inputs=[input_var1, input_var2], # List of symbolic inputs
    outputs=output_var             # List (or single) symbolic output(s)
)

• inputs: A Python list containing the symbolic variables that serve as inputs to the computation. When you call the compiled function, you will provide concrete numerical values (e.g., NumPy arrays) corresponding to these symbolic inputs, in the same order.
• outputs: A Python list containing the symbolic variables whose computed values you want the function to return. If you only have one output, you can provide it directly without enclosing it in a list (though using a list is often clearer).

You can then call compiled_function with NumPy arrays or appropriate Python types:

import numpy as np

matrix_data = np.array([[1, 2], [3, 4]], dtype='float64')
scalar_data = 2.0

result = compiled_function(matrix_data, scalar_data)
print(result)
# Output:
# [[2. 4.]
#  [6. 8.]]

Advanced Features (Brief Overview)

theano.function supports several other important parameters, notably:

• updates: Used for implementing training loops. It takes a list of tuples, where each tuple contains a shared variable and a symbolic expression for its new value. After the function computes the outputs, it updates the shared variables according to these rules (e.g., gradient descent updates). See the Training Example chapter.
• givens: Allows substituting a specific expression within the graph with a different value or expression, often used for implementing techniques like dropout during testing or providing data during specific phases.
• mode: (Less relevant now) Allowed specifying optimization levels like FAST_COMPILE (faster compilation, less optimization) or FAST_RUN (slower compilation, more optimization). This could also be set globally via the THEANO_FLAGS environment variable. Modern frameworks handle optimization more automatically.

The updates mechanism is particularly crucial, as it allows theano.function to modify the state of shared variables (like model weights) as a side effect of its execution.

Role in Theano’s Workflow

theano.function is the linchpin of Theano’s workflow. It connects the abstract, symbolic world of graph definition with the concrete world of numerical computation. By compiling the graph, it unlocks the performance benefits that were Theano’s main selling point.

Understanding its role as a compiler and executor is key to grasping how Theano (and conceptually, parts of modern frameworks like TensorFlow’s tf.function or JAX’s jit) achieves efficiency.