Working with Text Data

Pandas - Working with Text Data

Text data (strings) is ubiquitous, and Pandas provides powerful and efficient tools for manipulating string data stored in Series or Index objects. These tools are accessed via the .str accessor.

A key advantage of using the .str accessor is that its methods automatically handle missing values (NaN), typically returning NaN for any operation involving a missing value, thus preventing errors.

Most .str methods mirror the standard Python string methods (like .lower(), .split(), .replace(), etc.) but are vectorized, meaning they operate on the entire Series efficiently without needing explicit Python loops.

Let’s explore some common string operations using a sample Series:

import pandas as pd
import numpy as np

s = pd.Series(['   Tom   ', 'William Rick', 'JOHN', ' Alber@t',
               np.nan, '12345', 'Steve Smith ',
               'Python & Pandas'])

print("Original Series:")
print(s)

Original Series Output:

0             Tom
1        William Rick
2                JOHN
3            Alber@t
4                 NaN
5               12345
6        Steve Smith
7    Python & Pandas
dtype: object

Common String Methods via .str Accessor

Method	Description
.str.lower()	Converts strings to lower case.
.str.upper()	Converts strings to upper case.
.str.len()	Computes the length of each string.
.str.strip()	Removes leading and trailing whitespace (including newlines).
.str.lstrip()	Removes leading whitespace.
.str.rstrip()	Removes trailing whitespace.
.str.split(pat=None, n=-1, expand=False)	Splits each string by the specified separator/pattern (pat). Returns a Series of lists by default, or a DataFrame if expand=True.
.str.cat(sep='')	Concatenates strings in the Series/Index with a given separator.
.str.contains(pat, case=True, regex=True)	Checks if a pattern (pat) is contained within each string. Returns a boolean Series.
.str.startswith(pat)	Checks if each string starts with the specified pattern (pat). Returns a boolean Series.
.str.endswith(pat)	Checks if each string ends with the specified pattern (pat). Returns a boolean Series.
.str.replace(pat, repl, n=-1, regex=True)	Replaces occurrences of pat with repl in each string.
.str.repeat(repeats)	Repeats each string repeats times.
.str.count(pat)	Counts occurrences of pat within each string.
.str.find(sub)	Returns the lowest index of the substring sub in each string; returns -1 if not found.
.str.findall(pat)	Finds all occurrences of the pattern pat (can be regex) in each string; returns a Series of lists.
.str.get(i)	Extracts the element at position i for each string (if it’s indexable).
.str.slice(start=None, stop=None, step=None)	Slices substrings from each element in the Series/Index.
.str.islower(), .str.isupper(), .str.isnumeric(), etc.	Checks character properties for each string (e.g., all lowercase, all uppercase, all numeric). Returns a boolean Series.
pd.get_dummies(s) or s.str.get_dummies(sep='|')	Converts categorical string data into dummy/indicator variables (One-Hot Encoding). Can split strings by sep first.

Examples

Let’s apply some of these methods to our sample Series s.

# Case Conversion
print("\n--- Case Conversion ---")
print("Lower Case:\n", s.str.lower())
print("\nUpper Case:\n", s.str.upper())

# Length
print("\n--- Length ---")
print("String Lengths:\n", s.str.len())

# Stripping Whitespace
print("\n--- Stripping Whitespace ---")
print("Stripped (both sides):\n", s.str.strip())
print("\nLeft Stripped:\n", s.str.lstrip())
print("\nRight Stripped:\n", s.str.rstrip())

# Splitting
print("\n--- Splitting ---")
# Split by space, return lists
print("Split by space (into lists):\n", s.str.split(' '))
# Split by space, expand into DataFrame columns
print("\nSplit by space (expand=True):\n", s.str.split(' ', expand=True))

# Checking Content
print("\n--- Checking Content ---")
print("Contains 'Rick'?\n", s.str.contains('Rick'))
print("\nStarts with 'P'?\n", s.str.startswith('P'))
print("\nEnds with ' '?\n", s.str.endswith(' '))

# Replacing
print("\n--- Replacing ---")
print("Replace '@' with '#':\n", s.str.replace('@', '#'))
print("\nReplace ' ' with '_':\n", s.str.replace(' ', '_')) # Note: affects internal spaces too

# Find / Count
print("\n--- Find / Count ---")
print("Find first 'e':\n", s.str.find('e'))
print("\nCount occurrences of 'n':\n", s.str.count('n'))

# Character Type Checks
print("\n--- Character Type Checks ---")
print("Is Lowercase?\n", s.str.islower())
print("\nIs Uppercase?\n", s.str.isupper())
print("\nIs Numeric?\n", s.str.isnumeric())

# Get Dummies (Example on a simpler series)
s_cat = pd.Series(['A|B', 'B', 'A|C', 'A'])
print("\n--- Get Dummies --- ")
print("Categorical Series:\n", s_cat)
print("\nDummy Variables (split by '|'):\n", s_cat.str.get_dummies(sep='|'))

Example Output Snippets:

--- Case Conversion ---
Lower Case:
 0             tom
1        william rick
2                john
3            alber@t
4                 NaN
5               12345
6        steve smith
7    python & pandas
dtype: object

Upper Case:
 0             TOM
1        WILLIAM RICK
2                JOHN
3            ALBER@T
4                 NaN
5               12345
6        STEVE SMITH
7    PYTHON & PANDAS
dtype: object

--- Length ---
String Lengths:
 0    9.0
1   12.0
2    4.0
3    8.0
4    NaN
5    5.0
6   12.0
7   15.0
dtype: float64

--- Stripping Whitespace ---
Stripped (both sides):
 0                Tom
1       William Rick
2               JOHN
3           Alber@t
4                NaN
5              12345
6        Steve Smith
7    Python & Pandas
dtype: object
...
--- Splitting ---
Split by space (into lists):
 0            [, , , Tom, , , ]
1             [William, Rick]
2                      [JOHN]
3              [, Alber@t]
4                        NaN
5                    [12345]
6           [Steve, Smith, ]
7    [Python, &, Pandas]
dtype: object

Split by space (expand=True):
          0       1        2
0                      Tom
1   William    Rick     None
2      JOHN    None     None
3              Alber@t     None
4      None    None     None
5     12345    None     None
6     Steve   Smith
7    Python       &   Pandas
...
--- Checking Content ---
Contains 'Rick'?
 0    False
1     True
2    False
3    False
4      NaN
5    False
6    False
7    False
dtype: object
...
--- Replacing ---
Replace '@' with '#':
 0           Tom
1      William Rick
2              JOHN
3          Alber#t
4               NaN
5             12345
6      Steve Smith
7    Python & Pandas
dtype: object
...
--- Find / Count ---
Find first 'e':
 0    -1.0
1     3.0
2    -1.0
3     4.0
4     NaN
5    -1.0
6     3.0
7    -1.0
dtype: float64

Count occurrences of 'n':
 0    0.0
1    1.0
2    1.0
3    0.0
4    NaN
5    0.0
6    0.0
7    2.0
dtype: float64
...
--- Get Dummies ---
Categorical Series:
 0    A|B
1      B
2    A|C
3      A
dtype: object

Dummy Variables (split by '|'):
    A  B  C
0  1  1  0
1  0  1  0
2  1  0  1
3  1  0  0

These vectorized string methods provided by the .str accessor are essential for cleaning, preparing, and analyzing text data efficiently within Pandas.