Descriptive Statistics

Pandas - Descriptive Statistics

Pandas DataFrames and Series come equipped with a wide range of methods for computing descriptive statistics. These methods help summarize the central tendency, dispersion, and shape of a dataset’s distribution.

Most of these are aggregation methods (like .sum(), .mean(), .std()) that reduce the data to a single value or a Series of values. Some methods (like .cumsum()) produce an object of the same size as the input.

Many statistical methods accept an axis argument to specify the direction of the operation:

• axis=0 or axis='index' (default): Perform operation down the rows (aggregate across rows for each column).
• axis=1 or axis='columns': Perform operation across the columns (aggregate across columns for each row).

Let’s create a sample DataFrame to demonstrate these operations:

import pandas as pd
import numpy as np

# Create a dictionary of Series
data = {
    'Name': pd.Series(['Tom', 'James', 'Ricky', 'Vin', 'Steve', 'Smith', 'Jack',
                       'Lee', 'David', 'Gasper', 'Betina', 'Andres']),
    'Age': pd.Series([25, 26, 25, 23, 30, 29, 23, 34, 40, 30, 51, 46]),
    'Rating': pd.Series([4.23, 3.24, 3.98, 2.56, 3.20, 4.6, 3.8, 3.78, 2.98, 4.80, 4.10, 3.65])
}

# Create a DataFrame
df = pd.DataFrame(data)

print("Sample DataFrame:")
print(df)

Output:

Sample DataFrame:
      Name  Age  Rating
0      Tom   25    4.23
1    James   26    3.24
2    Ricky   25    3.98
3      Vin   23    2.56
4    Steve   30    3.20
5    Smith   29    4.60
6     Jack   23    3.80
7      Lee   34    3.78
8    David   40    2.98
9   Gasper   30    4.80
10  Betina   51    4.10
11  Andres   46    3.65

Common Aggregation Methods

.sum()

Calculates the sum of values. For numeric columns, it’s the arithmetic sum. For string/object columns, it performs concatenation.

# Sum along axis=0 (down the columns - default)
print("Sum of each column (axis=0):")
print(df.sum())

Output:

Sum of each column (axis=0):
Name      TomJamesRickyVinSteveSmithJackLeeDavidGasperBe...
Age                                                      382
Rating                                                 44.92
dtype: object

# Sum along axis=1 (across the columns)
# This often requires numeric data only, or can lead to errors/unexpected results
# We select only numeric columns first for a meaningful result
print("\nSum of each row for numeric columns (axis=1):")
print(df[['Age', 'Rating']].sum(axis=1))

Output:

Sum of each row for numeric columns (axis=1):
0     29.23
1     29.24
2     28.98
3     25.56
4     33.20
5     33.60
6     26.80
7     37.78
8     42.98
9     34.80
10    55.10
11    49.65
dtype: float64

.mean()

Calculates the arithmetic mean (average) of numeric columns.

# Calculate the mean of numeric columns (axis=0 default)
print("Mean of numeric columns:")
print(df.mean(numeric_only=True)) # numeric_only=True avoids warnings for non-numeric cols

Output:

Mean of numeric columns:
Age       31.833333
Rating     3.743333
dtype: float64

.std()

Calculates the sample standard deviation of numeric columns (measures data dispersion).

# Calculate the standard deviation of numeric columns
print("Standard deviation of numeric columns:")
print(df.std(numeric_only=True))

Output:

Standard deviation of numeric columns:
Age       9.232682
Rating    0.661628
dtype: float64

Other Useful Descriptive Functions

Pandas offers many other statistical functions:

Function	Description
.count()	Number of non-NA/null observations
.median()	Median (middle value) of numeric values
.mode()	Mode(s) (most frequent value(s)). Can return multiple values if ties exist.
.min()	Minimum value
.max()	Maximum value
.abs()	Absolute value (for numeric data)
.prod()	Product of values (for numeric data)
.cumsum()	Cumulative sum (returns object of same size)
.cumprod()	Cumulative product (returns object of same size)
.cummin()	Cumulative minimum (returns object of same size)
.cummax()	Cumulative maximum (returns object of same size)
.quantile(q=0.5)	Sample quantile (e.g., q=0.5 for median, q=0.25 for 1st quartile)
.var()	Sample variance
.skew()	Sample skewness (measure of asymmetry)
.kurt() / .kurtosis()	Sample kurtosis (measure of ‘tailedness’)

Note: Because DataFrames can contain heterogeneous data (different types in columns), some functions only work on numeric data (.abs(), .prod(), .var(), etc.) and will raise an error or require numeric_only=True if applied to a DataFrame containing non-numeric columns. Others like .count(), .sum(), and .mode() might work on non-numeric data but may yield results (like string concatenation for .sum()) that require careful interpretation.

Summarizing Data: .describe()

The .describe() method is extremely useful as it computes several common summary statistics for DataFrame columns in one go. By default, it summarizes numeric columns.

# Generate descriptive statistics for numeric columns
print("Describe numeric columns:")
print(df.describe())

Output:

Describe numeric columns:
             Age     Rating
count  12.000000  12.000000
mean   31.833333   3.743333
std     9.232682   0.661628
min    23.000000   2.560000
25%    25.000000   3.230000
50%    29.500000   3.790000
75%    35.500000   4.132500
max    51.000000   4.800000

This provides count, mean, standard deviation, minimum, maximum, and the 25th, 50th (median), and 75th percentiles (IQR).

You can use the include and exclude arguments to control which column types are summarized:

• include='object': Summarizes string or object columns (count, unique, top, freq).
• include='number': Summarizes numeric columns (default behavior).
• include='category': Summarizes categorical columns.
• include='datetime': Summarizes datetime columns (count, unique, first, last).
• include='all': Attempts to summarize all columns together (results may include NaNs where a statistic doesn’t apply to a column type).

Example: Summarizing object (string) columns

# Describe object columns (like 'Name')
print("\nDescribe object columns:")
print(df.describe(include=['object']))

Output:

Describe object columns:
         Name
count      12
unique     12
top     Ricky  # Top might vary if names weren't unique
freq        1

Example: Summarizing all columns

# Describe all columns
print("\nDescribe all columns:")
print(df.describe(include='all'))

Output:

Describe all columns:
         Name        Age     Rating
count      12  12.000000  12.000000
unique     12        NaN        NaN
top     Ricky        NaN        NaN
freq        1        NaN        NaN
mean      NaN  31.833333   3.743333
std       NaN   9.232682   0.661628
min       NaN  23.000000   2.560000
25%       NaN  25.000000   3.230000
50%       NaN  29.500000   3.790000
75%       NaN  35.500000   4.132500
max       NaN  51.000000   4.800000

Notice the NaN values where statistics don’t apply (e.g., mean for the ‘Name’ column, or unique for numeric columns in this combined view).