Python Pandas - Iteration

Modern Pandas - Iteration Techniques

Iterating directly over Pandas objects behaves differently depending on the object type. Iterating over a Series yields its values, similar to a NumPy array. Iterating over a DataFrame, following dictionary-like conventions, yields its column labels (keys).

In summary, basic iteration (for item in object) produces:

• Series: values
• DataFrame: column labels

Iterating a DataFrame (Column Labels)

Directly iterating over a DataFrame provides the column names. This is useful for quickly accessing column labels but not the data itself.

import pandas as pd
import numpy as np

# Use a modern random number generator if possible, though np.random is still common
rng = np.random.default_rng(42) # Seed for reproducibility

N = 5 # Smaller N for clearer output
df = pd.DataFrame({
    'A': pd.date_range(start='2023-01-01', periods=N, freq='D'),
    'x': np.linspace(0, stop=N - 1, num=N),
    'y': rng.random(N),
    'C': rng.choice(['Low', 'Medium', 'High'], N).tolist(),
    'D': rng.normal(100, 10, size=N).tolist()
})

print('DataFrame:\n', df)
print('\nIterating over DataFrame (yields column labels):')
for col_label in df:
    print(col_label)

Example output:

DataFrame:
            A    x         y       C           D
0 2023-01-01  0.0  0.773956     Low  105.650547
1 2023-01-02  1.0  0.438878    High   94.485718
2 2023-01-03  2.0  0.858598     Low  104.112478
3 2023-01-04  3.0  0.697368  Medium  101.210774
4 2023-01-05  4.0  0.094177  Medium  104.057573

Iterating over DataFrame (yields column labels):
A
x
y
C
D

While direct iteration is simple, it’s often less efficient than vectorized operations. For row-wise operations when necessary, Pandas offers dedicated methods:

• .items(): Iterates over columns as (label, Series) pairs.
• .iterrows(): Iterates over rows as (index, Series) pairs. Caution: Often inefficient and may alter dtypes.
• .itertuples(): Iterates over rows as namedtuples. Generally more efficient than iterrows and preserves dtypes better.

Best Practice: Avoid iteration whenever possible. Use vectorized operations (applying functions directly to Series/DataFrames) for better performance. If iteration is unavoidable, prefer itertuples() over iterrows().

Iterating Columns: .items()

The .items() method iterates through the DataFrame’s columns, yielding each column label and the column data as a Series object. This is useful when you need to perform an operation on each column Series individually.

import pandas as pd
import numpy as np

rng = np.random.default_rng(42)
df = pd.DataFrame(rng.standard_normal((4, 3)), columns=['col1', 'col2', 'col3'])

print('DataFrame:\n', df)
print('\nIterating with .items():')
for label, content_series in df.items():
    print(f'\nColumn Label: {label}')
    print(content_series)

Example output:

DataFrame:
       col1      col2      col3
0  0.496714 -0.138264  0.647689
1  1.523030 -0.234153 -0.234137
2  1.579213  0.767435 -0.469474
3  0.542560 -0.463418 -0.465730

Iterating with .items():

Column Label: col1
0    0.496714
1    1.523030
2    1.579213
3    0.542560
Name: col1, dtype: float64

Column Label: col2
0   -0.138264
1   -0.234153
2    0.767435
3   -0.463418
Name: col2, dtype: float64

Column Label: col3
0    0.647689
1   -0.234137
2   -0.469474
3   -0.465730
Name: col3, dtype: float64

Observe how each column is yielded as a (label, Series) tuple.

Iterating Rows: .iterrows() (Use with Caution)

.iterrows() yields each row as a tuple containing the row’s index and a Series representing the row’s data. Warning: This method is notoriously slow for large DataFrames and can unexpectedly convert row data types (e.g., numbers to float) because the row is returned as a single Series. It should generally be avoided in favor of vectorized operations or itertuples().

import pandas as pd
import numpy as np

rng = np.random.default_rng(43)
df = pd.DataFrame(rng.standard_normal((3, 3)), columns=['col1', 'col2', 'col3'])

print('DataFrame:\n', df)
print('\nIterating with .iterrows():')
for index, row_series in df.iterrows():
    print(f'\nIndex: {index}')
    print(row_series)

Example output:

DataFrame:
       col1      col2      col3
0 -0.535952 -0.875490 -0.269001
1 -0.087176  0.125819  0.747178
2 -1.255743 -0.977989 -1.034384

Iterating with .iterrows():

Index: 0
col1   -0.535952
col2   -0.875490
col3   -0.269001
Name: 0, dtype: float64

Index: 1
col1   -0.087176
col2    0.125819
col3    0.747178
Name: 1, dtype: float64

Index: 2
col1   -1.255743
col2   -0.977989
col3   -1.034384
Name: 2, dtype: float64

Important Note: Because iterrows() returns each row as a Series, it does not preserve data types across the row if they are mixed. For example, an integer column might be cast to float within the row Series. This is a major reason to prefer itertuples() if row iteration is needed.

Iterating Rows Efficiently: .itertuples()

.itertuples() is the recommended method for row iteration when necessary. It yields each row as a lightweight namedtuple. The first element is the row’s index, and subsequent elements are the row values. This method is significantly faster than iterrows() and generally preserves data types.

import pandas as pd
import numpy as np

rng = np.random.default_rng(44)
df = pd.DataFrame(rng.standard_normal((3, 3)), columns=['col1', 'col2', 'col3'])

print('DataFrame:\n', df)
print('\nIterating with .itertuples():')
for row_tuple in df.itertuples(): # index=True is default
    print(row_tuple)

print('\nIterating with .itertuples(index=False, name="MyRow"):')
for row_tuple in df.itertuples(index=False, name='MyRow'): # Custom name, no index
    print(row_tuple)

Example output:

DataFrame:
       col1      col2      col3
0 -0.846179  0.058019 -0.616289
1  0.277575 -0.447319 -1.049108
2  0.223398 -0.204669 -0.358949

Iterating with .itertuples():
Pandas(Index=0, col1=-0.8461786296694368, col2=0.05801868623661116, col3=-0.6162891895287465)
Pandas(Index=1, col1=0.277575258121937, col2=-0.4473188014249641, col3=-1.0491081632783335)
Pandas(Index=2, col1=0.22339849625659426, col2=-0.20466887033086502, col3=-0.3589488659138892)

Iterating with .itertuples(index=False, name='MyRow'):
MyRow(col1=-0.8461786296694368, col2=0.05801868623661116, col3=-0.6162891895287465)
MyRow(col1=0.277575258121937, col2=-0.4473188014249641, col3=-1.0491081632783335)
MyRow(col1=0.22339849625659426, col2=-0.20466887033086502, col3=-0.3589488659138892)

You can access elements by attribute name (e.g., row_tuple.col1) or index (e.g., row_tuple[1], if index=True). The index=False option excludes the index from the tuple, and name allows customizing the namedtuple’s type name.

Mutation Warning

Crucial: Never modify the DataFrame you are iterating over within the loop (using .iterrows() or .itertuples()). These methods often return copies (or views, behavior can vary), not direct references to the original data. Modifications made to the row_series or row_tuple inside the loop will likely not change the original DataFrame.

import pandas as pd
import numpy as np

rng = np.random.default_rng(45)
df = pd.DataFrame(rng.standard_normal((3, 3)), columns=['col1', 'col2', 'col3'])

print('Original DataFrame:\n', df)

for index, row_series in df.iterrows():
    # Attempting to modify - this will likely affect only the copy
    row_series['col1'] = 1000

print('\nDataFrame after attempting modification during iterrows():')
print(df) # Observe: No changes reflected in the original df

# Correct way to modify requires assigning back or using other methods like .loc
# Example (not using iteration): df.loc[df['col1'] < 0, 'col1'] = 0

Example output:

Original DataFrame:
       col1      col2      col3
0 -0.815408  0.797837  1.232803
1 -0.228229  0.209149 -1.101992
2 -0.375991 -1.043386 -0.826451

DataFrame after attempting modification during iterrows():
       col1      col2      col3
0 -0.815408  0.797837  1.232803
1 -0.228229  0.209149 -1.101992
2 -0.375991 -1.043386 -0.826451

If you need to modify data based on conditions, use vectorized assignment with .loc or .iloc, or the .apply() method carefully.