Python Pandas - Categorical Data

Pandas - Categorical Data

Real-world datasets often contain columns with repetitive text values representing distinct groups or categories, such as ‘Gender’ (Male/Female), ‘Country’ (USA/Canada/Mexico), or ‘Status’ (Pending/Completed/Failed). Pandas provides a dedicated Categorical data type for handling such data efficiently.

Categorical variables can only take on a limited, fixed set of possible values, known as ‘categories’. They might have a meaningful order (e.g., ‘Low’ < ‘Medium’ < ‘High’), but standard numerical operations (like addition) are not applicable. Using the Categorical dtype offers several advantages:

• Memory Efficiency: If a string column has many repetitions of a few unique values, converting it to Categorical can significantly reduce memory usage. Pandas stores the unique categories once and uses integer codes to represent the values.
• Logical Ordering: You can define a custom logical order for categories (e.g., [‘Small’, ‘Medium’, ‘Large’]). Sorting or finding the minimum/maximum will then use this logical order instead of the default alphabetical (lexical) order.
• Signal to Other Libraries: It clearly signals to other Python libraries (like visualization or statistical modeling libraries) that a column should be treated as categorical, enabling appropriate plot types or statistical methods.

Creating Categorical Data

Categorical data can be created in several ways:

Using dtype='category'

Specify dtype="category" when creating a Series or DataFrame:

import pandas as pd

# Create a Series with categorical data
s = pd.Series(["a", "b", "c", "a"], dtype="category")
print(s)

Output:

0    a
1    b
2    c
3    a
dtype: category
Categories (3, object): ['a', 'b', 'c']

Notice that although the Series has four elements, Pandas identifies only three unique categories: ‘a’, ‘b’, and ‘c’.

Using pd.Categorical()

You can explicitly create a Categorical object using the pd.Categorical constructor:

pd.Categorical(values, categories=None, ordered=False)

Example 1: Basic creation

import pandas as pd

cat_obj = pd.Categorical(['a', 'b', 'c', 'a', 'b', 'c'])
print(cat_obj)

Output:

['a', 'b', 'c', 'a', 'b', 'c']
Categories (3, object): ['a', 'b', 'c']

Example 2: Specifying categories

import pandas as pd

values = ['a', 'b', 'c', 'a', 'b', 'c', 'd'] # 'd' is not in categories
defined_categories = ['c', 'b', 'a']
cat_obj = pd.Categorical(values, categories=defined_categories)
print(cat_obj)

Output:

['a', 'b', 'c', 'a', 'b', 'c', NaN]
Categories (3, object): ['c', 'b', 'a']

Here, the categories argument explicitly defines the allowed categories. Any value in values not present in defined_categories (like ‘d’) becomes NaN (Not a Number).

Example 3: Specifying order

import pandas as pd

values = ['a', 'b', 'c', 'a', 'b', 'c', 'd']
defined_categories = ['c', 'b', 'a']
# Create an ordered categorical
cat_obj = pd.Categorical(values, categories=defined_categories, ordered=True)
print(cat_obj)

Output:

['a', 'b', 'c', 'a', 'b', 'c', NaN]
Categories (3, object): ['c' < 'b' < 'a']

Setting ordered=True establishes a logical order based on the sequence provided in categories. In this case, ‘c’ is considered less than ‘b’, and ‘b’ is less than ‘a’.

Using astype('category')

You can convert an existing Series or column to categorical type using the .astype() method. This is a very common approach.

import pandas as pd

s = pd.Series(["a", "b", "c", "a"])
print("Original Series:")
print(s)

s_cat = s.astype('category')
print("\nCategorical Series:")
print(s_cat)

Output:

Original Series:
0    a
1    b
2    c
3    a
dtype: object

Categorical Series:
0    a
1    b
2    c
3    a
dtype: category
Categories (3, object): ['a', 'b', 'c']

Working with Categorical Data

Describing Categorical Data

The .describe() method provides a summary for categorical data, including counts, unique categories, the most frequent category (top), and its frequency (freq).

import pandas as pd
import numpy as np

cat = pd.Categorical(["a", "c", "c", np.nan], categories=["b", "a", "c"])
df = pd.DataFrame({"cat_column": cat, "str_column": ["a", "c", "c", np.nan]})

print("Describe DataFrame (includes categorical summary):")
print(df.describe(include=['category']))

print("\nDescribe the categorical column directly:")
print(df["cat_column"].describe())

Output:

Describe DataFrame (includes categorical summary):
       cat_column
count           3
unique          2
top             c
freq            2

Describe the categorical column directly:
count     3
unique    2
top       c
freq      2
Name: cat_column, dtype: object

Accessing Categorical Properties

Use the .cat accessor to work with categorical properties:

import pandas as pd
import numpy as np

s = pd.Series(["a", "c", "c", np.nan], dtype=pd.CategoricalDtype(categories=["b", "a", "c"], ordered=True))

# Get categories
print("Categories:", s.cat.categories)

# Check if ordered
print("Is ordered:", s.cat.ordered)

Output:

Categories: Index(['b', 'a', 'c'], dtype='object')
Is ordered: True

Renaming Categories

Use the s.cat.rename_categories() method:

import pandas as pd

s = pd.Series(["a", "b", "c", "a"], dtype="category")
print("Original categories:", s.cat.categories)

# Rename categories using a dictionary or a function
s_renamed = s.cat.rename_categories({"a": "Alpha", "b": "Beta", "c": "Gamma"})
# Or using a function: s_renamed = s.cat.rename_categories(lambda c: f"Group {c.upper()}")

print("Renamed categories:", s_renamed.cat.categories)
print("\nSeries with renamed categories:")
print(s_renamed)

Output:

Original categories: Index(['a', 'b', 'c'], dtype='object')
Renamed categories: Index(['Alpha', 'Beta', 'Gamma'], dtype='object')

Series with renamed categories:
0    Alpha
1     Beta
2    Gamma
3    Alpha
dtype: category
Categories (3, object): ['Alpha', 'Beta', 'Gamma']

Adding New Categories

Use the s.cat.add_categories() method:

import pandas as pd

s = pd.Series(["a", "b", "c", "a"], dtype="category")
print("Original categories:", s.cat.categories)

s_new_cats = s.cat.add_categories(["d", "e"])
print("\nCategories after adding:", s_new_cats.cat.categories)

Output:

Original categories: Index(['a', 'b', 'c'], dtype='object')

Categories after adding: Index(['a', 'b', 'c', 'd', 'e'], dtype='object')

Removing Categories

Use the s.cat.remove_categories() method. Note that any existing data points belonging to the removed category will become NaN.

import pandas as pd

s = pd.Series(["a", "b", "c", "a"], dtype="category")
print("Original object:")
print(s)

# Remove category 'a'
s_removed = s.cat.remove_categories(["a"])
print("\nAfter removal:")
print(s_removed)
print("\nRemaining categories:", s_removed.cat.categories)

Output:

Original object:
0    a
1    b
2    c
3    a
dtype: category
Categories (3, object): ['a', 'b', 'c']

After removal:
0    NaN
1      b
2      c
3    NaN
dtype: category
Categories (2, object): ['b', 'c']

Remaining categories: Index(['b', 'c'], dtype='object')

Removing Unused Categories

If you filter data or remove categories, some categories might no longer be present in the data. Use s.cat.remove_unused_categories() to drop them from the .categories attribute.

import pandas as pd

s = pd.Series(pd.Categorical(["a", "b", "a"], categories=['a', 'b', 'c']))
print("Original Series:")
print(s)
print("Original Categories:", s.cat.categories)

# Remove unused categories ('c')
s_useful = s.cat.remove_unused_categories()
print("\nSeries after removing unused:")
print(s_useful)
print("Updated Categories:", s_useful.cat.categories)

Output:

Original Series:
0    a
1    b
2    a
dtype: category
Categories (3, object): ['a', 'b', 'c']
Original Categories: Index(['a', 'b', 'c'], dtype='object')

Series after removing unused:
0    a
1    b
2    a
dtype: category
Categories (2, object): ['a', 'b']
Updated Categories: Index(['a', 'b'], dtype='object')

Comparison of Categorical Data

Comparisons work under specific conditions:

• Equality (==, !=): Can compare a Categorical Series to a list-like object (list, Series, array) of the same length, or to a scalar.
• Ordered Comparisons (>, >=, <, <=): Possible only if ordered=True for the Categorical. You can compare:
• • An ordered Categorical Series to another ordered Categorical Series if they have the exact same categories and order.
• • An ordered Categorical Series to a scalar value that is one of its categories.

Example: Comparing ordered categoricals

import pandas as pd

# Define ordered categories
cat_dtype = pd.CategoricalDtype(categories=['low', 'medium', 'high'], ordered=True)

cat1 = pd.Series(['low', 'medium', 'high'], dtype=cat_dtype)
cat2 = pd.Series(['medium', 'medium', 'medium'], dtype=cat_dtype)

print("cat1 > cat2:")
print(cat1 > cat2)

print("\ncat1 == 'medium':")
print(cat1 == 'medium')

Output:

cat1 > cat2:
0    False
1    False
2     True
dtype: bool

cat1 == 'medium':
0    False
1     True
2    False
dtype: bool