IPython - Magic Commands

Magic commands are a standout feature of IPython, offering convenient shortcuts and control over the IPython environment and underlying system. They often simplify tasks that would be more verbose using standard Python syntax, making them particularly useful for interactive data analysis and workflow management.

These commands, prefixed with ’%’ or ’%%’, act like mini command-line tools within your IPython session (including Jupyter Notebooks and JupyterLab).

Types of Magic Commands

There are two primary types of magic commands:

Line Magics: Prefixed with a single ’%’, operate on a single line of input.
Cell Magics: Prefixed with ’%%’, operate on the entire code cell below the magic command.

Line Magics (%)

Line magics function like shell commands. They start with the ’%’ character, and the rest of the line contains their arguments, typically without parentheses or quotes. The output of a line magic can often be assigned to a Python variable.

Cell Magics (%%)

Cell magics, starting with ’%%’, apply to the entire block of code within the cell where they are invoked. They process the whole cell content as a single string, allowing for powerful transformations or execution in different contexts (e.g., running bash script, writing to a file).

You can list all available magic commands using %lsmagic. To get help on a specific magic command, type its name followed by a question mark, like %timeit? or %%writefile?.

Common Built-in Magic Commands

Here are some frequently used built-in magic commands:

%run

Executes an external Python script. Variables defined in the script become available in the current IPython namespace.

# Assuming hello.py contains: print('Hello from script!')
%run hello.py
# Output:
# Hello from script!

%load

Loads code from an external script into the current cell, ready for editing and execution.

# Assuming hello.py contains: print('Hello from script!')
%load hello.py

# Cell content becomes:
# # %load hello.py
# print('Hello from script!')

%timeit

Measures the execution time of a Python statement or expression by running it multiple times for accuracy. Can be used as both a line and cell magic.

# Line Magic:
%timeit [x**2 for x in range(1000)]
# Output: (e.g.) 85.6 µs ± 1.2 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

# Cell Magic:
%%timeit
result = []
for x in range(1000):
    result.append(x**2)
# Output: (e.g.) 93.1 µs ± 2.5 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%time

Times the execution of a single statement (useful for longer-running code where multiple loops aren’t necessary).

import time
%time time.sleep(1)
# Output: (e.g.) Wall time: 1 s

%history

Displays the command history of the current session. Use flags like -n to include line numbers.

a = 10
b = 20
%history -n 1-2
# Output:
#    1: a = 10
#    2: b = 20

%pwd, %cd

%pwd prints the current working directory. %cd <directory> changes the current working directory. Using %cd - goes back to the previous directory.

%pwd
# Output: (e.g.) /home/user/myproject

%cd ..
# Output: (e.g.) /home/user

%pwd
# Output: (e.g.) /home/user

%cd -
# Output: (e.g.) /home/user/myproject

%who, %whos

%who lists variables currently defined in the namespace. %whos provides more detailed information (type, value/info).

var_a = 'hello'
var_b = [1, 2, 3]

%who
# Output: var_a   var_b

%whos
# Output:
# Variable   Type    Data/Info
# ----------------------------
# var_a      str     hello
# var_b      list    n=3

%matplotlib

Configures Matplotlib integration. %matplotlib inline is commonly used in Jupyter Notebook/Lab to render plots directly within the output cell. Other backends (like qt, notebook) enable interactive plots.

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 10, 100)
plt.plot(x, np.sin(x))
# Output: [A static plot of a sine wave appears below the cell]

%load_ext

Loads IPython extensions, which can provide additional magic commands or functionality. A popular example is sql for running SQL queries or autoreload for automatically reloading modules before code execution.

%load_ext autoreload
%autoreload 2
# Enables automatic reloading of imported modules

%%writefile

A cell magic that saves the contents of the cell to a specified file.

%%writefile my_script.py

def greet(name):
    print(f'Hello, {name}!')

greet('World')

# Output: Writing my_script.py

%%bash / %%cmd / %%script

Cell magics to execute code using the system shell (bash on Linux/macOS, cmd on Windows) or any other interpreter specified with %%script <interpreter>.

# Example using bash (Linux/macOS)
%%bash
echo "Listing files:"
ls -l | head -n 3

# Output:
# Listing files:
# total 8
# -rw-r--r-- 1 user user 60 Dec 15 10:30 my_script.py
# ...

%env

Manages environment variables. %env lists all, %env VAR gets the value of VAR, %env VAR=value sets the value of VAR for the current process.

%env PATH
# Output: (Displays the current PATH environment variable)

%env MY_VAR=test
# Sets MY_VAR to 'test'

%env MY_VAR
# Output: test

%automagic

Toggles whether line magics can be called without the leading ’%’. It’s generally recommended to keep automagic OFF (the default is often ON) for clarity by explicitly using ’%’. Use %automagic 0 to disable or %automagic 1 to enable.

Creating Custom Magic Commands

You can define your own magic commands using decorators from the IPython.core.magic module. This allows you to encapsulate custom workflows into reusable commands.

from IPython.core.magic import register_line_magic

@register_line_magic
def hello_magic(line):
    """A simple line magic that greets someone."""
    name = line.strip() if line.strip() else 'World'
    print(f"Hello, {name} from custom magic!")

# Now you can use it:
%hello_magic
# Output: Hello, World from custom magic!

%hello_magic Alice
# Output: Hello, Alice from custom magic!

Similar decorators (@register_cell_magic, @register_line_cell_magic) exist for creating other types of custom magics.