Understanding Data with Statistics

Understanding Data with Statistics in Python

Introduction

In machine learning, data is the foundation. The quality of your model heavily depends on the quality and understanding of your data. Before feeding data into any algorithm, it’s crucial to understand its characteristics.

While data visualization provides graphical insights (covered in another chapter), descriptive statistics offer a quantitative summary of the data’s properties. This initial analysis helps in identifying potential issues like outliers, missing values, or skewed distributions, guiding data preprocessing steps.

This chapter focuses on using Python, specifically the Pandas library, to perform fundamental statistical analysis on your dataset.

Peeking at Raw Data

The first step is always to look at the actual data values. This gives you a qualitative feel for the dataset and can reveal immediate issues like inconsistent formatting or unexpected values.

The head() and tail() methods in Pandas are perfect for this. Let’s load the Pima Indians Diabetes dataset (a common dataset for binary classification) and view the first few rows.

Example

Note: Ensure you have the ‘pandas’ and ‘scikit-learn’ libraries installed (pip install pandas scikit-learn). We’ll use scikit-learn’s utility to fetch the dataset easily, avoiding hardcoded file paths.

import pandas as pd
from sklearn.datasets import fetch_openml

# Fetch the Pima Indians Diabetes dataset from OpenML
# Note: fetch_openml might require an internet connection the first time
pima = fetch_openml(name='diabetes', version=1, as_frame=True, parser='pandas')
df = pima.frame

# Define more descriptive column names (based on dataset description)
df.columns = ['num_pregnancies', 'glucose_concentration', 'blood_pressure', 'skin_thickness', 'insulin', 'bmi', 'diabetes_pedigree', 'age', 'class']

# Display the first 10 rows
print("First 10 rows of the dataset:")
print(df.head(10))

# Display the last 5 rows
print("\nLast 5 rows of the dataset:")
print(df.tail(5))

Output (Illustrative)

First 10 rows of the dataset:
   num_pregnancies  glucose_concentration  blood_pressure  skin_thickness  insulin   bmi  diabetes_pedigree  age class
0              6.0                  148.0            72.0            35.0      0.0  33.6              0.627   50     1
1              1.0                   85.0            66.0            29.0      0.0  26.6              0.351   31     0
2              8.0                  183.0            64.0             0.0      0.0  23.3              0.672   32     1
3              1.0                   89.0            66.0            23.0     94.0  28.1              0.167   21     0
4              0.0                  137.0            40.0            35.0    168.0  43.1              2.288   33     1
5              5.0                  116.0            74.0             0.0      0.0  25.6              0.201   30     0
6              3.0                   78.0            50.0            32.0     88.0  31.0              0.248   26     1
7             10.0                  115.0             0.0             0.0      0.0  35.3              0.134   29     0
8              2.0                  197.0            70.0            45.0    543.0  30.5              0.158   53     1
9              8.0                  125.0            96.0             0.0      0.0   0.0              0.232   54     1

Last 5 rows of the dataset:
     num_pregnancies  glucose_concentration  blood_pressure  skin_thickness  insulin   bmi  diabetes_pedigree  age class
763             10.0                  101.0            76.0            48.0    180.0  32.9              0.171   63     0
764              2.0                  122.0            70.0            27.0      0.0  36.8              0.340   27     0
765              5.0                  121.0            72.0            23.0    112.0  26.2              0.245   30     0
766              1.0                  126.0            60.0             0.0      0.0  30.1              0.349   47     1
767              1.0                   93.0            70.0            31.0      0.0  30.4              0.315   23     0

Observing the first few rows reveals the data types (mostly numeric) and gives a sense of the value ranges. Notice that some columns like ‘blood_pressure’, ‘skin_thickness’, ‘insulin’, and ‘bmi’ have zero values, which might indicate missing data for these physiological measurements and require further investigation or imputation.

Checking Data Dimensions

Knowing the size of your dataset (number of rows/instances and columns/features) is essential for planning:

Too many features might lead to the ‘curse of dimensionality’ and require feature selection/engineering.
Too few instances might limit the model’s ability to generalize.
Large datasets require more computational resources and potentially different algorithmic approaches (e.g., incremental learning, distributed computing).

The shape attribute of a Pandas DataFrame provides this information.

Example

import pandas as pd
from sklearn.datasets import load_iris

# Load the Iris dataset
iris = load_iris(as_frame=True)
df_iris = iris.frame

# Add the target column for completeness (optional for shape)
df_iris['target'] = iris.target

print(f"Dataset dimensions (rows, columns): {df_iris.shape}")

Output

Dataset dimensions (rows, columns): (150, 5)

The output indicates the Iris dataset has 150 instances and 5 columns (4 features + 1 target).

Understanding Attribute Data Types

Machine learning algorithms often require specific data types. For instance, most algorithms expect numerical input. Understanding the data types (dtypes) helps identify columns that might need conversion (e.g., categorical strings to numerical representations like one-hot encoding or label encoding).

Pandas automatically infers data types upon loading, but it’s good practice to verify them using the dtypes attribute or the info() method (which also shows non-null counts).

Example

import pandas as pd
from sklearn.datasets import load_iris

# Load the Iris dataset
iris = load_iris(as_frame=True)
df_iris = iris.frame
df_iris['target'] = iris.target

print("Data types of each column:")
print(df_iris.dtypes)

print("\n--- More Detailed Info ---")
df_iris.info()

Output

Data types of each column:
sepal length (cm)    float64
sepal width (cm)     float64
petal length (cm)    float64
petal width (cm)     float64
target                 int64
dtype: object

--- More Detailed Info ---
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 150 entries, 0 to 149
Data columns (total 5 columns):
 #   Column             Non-Null Count  Dtype
---  ------             --------------  -----
 0   sepal length (cm)  150 non-null    float64
 1   sepal width (cm)   150 non-null    float64
 2   petal length (cm)  150 non-null    float64
 3   petal width (cm)   150 non-null    float64
 4   target             150 non-null    int64
dtypes: float64(4), int64(1)
memory usage: 6.0 KB

The output confirms all features are float64 and the target is int64, suitable for most numerical algorithms. The info() output also confirms there are no missing values in this particular dataset.

Descriptive Statistical Summary

Beyond dimensions and types, a summary of central tendency, dispersion, and shape of the distribution for each numerical attribute is vital. The describe() method in Pandas provides key statistics:

Count: Number of non-null observations.
Mean: Average value.
Std: Standard Deviation, measuring dispersion around the mean.
Min: Minimum value.
25%: First quartile (Q1).
50%: Median (Q2).
75%: Third quartile (Q3).
Max: Maximum value.

This summary helps identify the scale of different attributes (potentially requiring scaling/normalization), the presence of outliers (large differences between median and mean, or extreme min/max values), and the basic distribution.

Example

import pandas as pd
from sklearn.datasets import fetch_openml

# Fetch the Pima Indians Diabetes dataset from OpenML
pima = fetch_openml(name='diabetes', version=1, as_frame=True, parser='pandas')
df = pima.frame
df.columns = ['num_pregnancies', 'glucose_concentration', 'blood_pressure', 'skin_thickness', 'insulin', 'bmi', 'diabetes_pedigree', 'age', 'class']

# Set display options for better readability
pd.set_option('display.width', 100)
pd.set_option('display.precision', 3)

print("Dataset dimensions:")
print(df.shape)

print("\nStatistical Summary:")
print(df.describe())

Output

Dataset dimensions:
(768, 9)

Statistical Summary:
       num_pregnancies  glucose_concentration  blood_pressure  skin_thickness  insulin    bmi  diabetes_pedigree    age  class
count          768.000                768.000         768.000         768.000  768.000  768.000            768.000  768.000  768.000
mean             3.845                120.895          69.105          20.536   79.799   31.993              0.472   33.241    0.349
std              3.370                 31.973          19.356          15.952  115.244    7.884              0.331   11.760    0.477
min              0.000                  0.000           0.000           0.000    0.000    0.000              0.078   21.000    0.000
25%              1.000                 99.000          62.000           0.000    0.000   27.300              0.244   24.000    0.000
50%              3.000                117.000          72.000          23.000   30.500   32.000              0.372   29.000    0.000
75%              6.000                140.250          80.000          32.000  127.250   36.600              0.626   41.000    1.000
max             17.000                199.000         122.000          99.000  846.000   67.100              2.420   81.000    1.000

The summary confirms the 768 instances. Again, observe the minimum value of 0 for columns like ‘glucose_concentration’, ‘blood_pressure’, ‘skin_thickness’, ‘insulin’, and ‘bmi’. This reinforces the suspicion of missing values coded as zero, which needs careful handling (e.g., imputation).

Reviewing Class Distribution (for Classification)

In classification tasks, understanding the distribution of the target variable (class labels) is critical. Imbalanced classes (where one class significantly outnumbers others) can bias the model towards the majority class and may require specific techniques like resampling (oversampling/undersampling) or using different evaluation metrics (e.g., F1-score, AUC).

We can use the groupby() method followed by size() in Pandas to count occurrences of each class.

Example

import pandas as pd
from sklearn.datasets import fetch_openml

# Fetch the Pima Indians Diabetes dataset
pima = fetch_openml(name='diabetes', version=1, as_frame=True, parser='pandas')
df = pima.frame
df.columns = ['num_pregnancies', 'glucose_concentration', 'blood_pressure', 'skin_thickness', 'insulin', 'bmi', 'diabetes_pedigree', 'age', 'class']

# Convert class column to integer if it's not already (fetch_openml might load it as categorical)
df['class'] = pd.to_numeric(df['class'])

# Calculate class distribution
class_counts = df.groupby('class').size()

print("Class Distribution:")
print(class_counts)

Output

Class Distribution:
class
0    500
1    268
dtype: int64

The output shows 500 instances of class 0 (no diabetes) and 268 instances of class 1 (diabetes). While not extremely imbalanced, the difference is noticeable and might warrant consideration during modeling.

Reviewing Correlation Between Attributes

Correlation measures the linear relationship between pairs of numerical attributes. The Pearson correlation coefficient is commonly used:

Value near +1: Strong positive linear correlation.
Value near -1: Strong negative linear correlation.
Value near 0: Weak or no linear correlation.

Identifying highly correlated features is important because:

Redundancy: Highly correlated features might provide redundant information.
Model Stability: Some algorithms (like Linear Regression) can become unstable with highly correlated predictors (multicollinearity). Feature selection or dimensionality reduction techniques (like PCA) might be necessary.

The corr() method in Pandas calculates the pairwise correlation matrix.

Example

import pandas as pd
from sklearn.datasets import fetch_openml

# Fetch the Pima Indians Diabetes dataset
pima = fetch_openml(name='diabetes', version=1, as_frame=True, parser='pandas')
df = pima.frame
df.columns = ['num_pregnancies', 'glucose_concentration', 'blood_pressure', 'skin_thickness', 'insulin', 'bmi', 'diabetes_pedigree', 'age', 'class']
df['class'] = pd.to_numeric(df['class'])

# Set display options
pd.set_option('display.width', 100)
pd.set_option('display.precision', 3)

# Calculate correlation matrix
correlations = df.corr(method='pearson')

print("Correlation Matrix:")
print(correlations)

Output

Correlation Matrix:
                       num_pregnancies  glucose_concentration  blood_pressure  skin_thickness  insulin    bmi  diabetes_pedigree    age  class
num_pregnancies                1.000                  0.129           0.141          -0.082   -0.074  0.018             -0.034  0.544  0.222
glucose_concentration          0.129                  1.000           0.153           0.057    0.331  0.221              0.137  0.264  0.467
blood_pressure                 0.141                  0.153           1.000           0.207    0.089  0.282              0.041  0.240  0.065
skin_thickness                -0.082                  0.057           0.207           1.000    0.437  0.393              0.184 -0.114  0.075
insulin                       -0.074                  0.331           0.089           0.437    1.000  0.198              0.185 -0.042  0.131
bmi                            0.018                  0.221           0.282           0.393    0.198  1.000              0.141  0.036  0.293
diabetes_pedigree             -0.034                  0.137           0.041           0.184    0.185  0.141              1.000  0.034  0.174
age                            0.544                  0.264           0.240          -0.114   -0.042  0.036              0.034  1.000  0.238
class                          0.222                  0.467           0.065           0.075    0.131  0.293              0.174  0.238  1.000

The matrix shows the correlation between all pairs of attributes. For example, ‘age’ and ‘num_pregnancies’ have a moderate positive correlation (0.544). ‘glucose_concentration’ has the highest correlation with the target ‘class’ (0.467) among the features shown.

Reviewing Skew of Attribute Distributions

Skewness measures the asymmetry of a probability distribution. Many machine learning algorithms (especially linear models and those assuming normality) perform better when features have a symmetric (Gaussian-like) distribution.

Positive Skew (Right Skew): Tail is longer on the right side. Mean > Median.
Negative Skew (Left Skew): Tail is longer on the left side. Mean < Median.
Zero Skew: Symmetric distribution (like a perfect Gaussian).

Significant skew might necessitate data transformations (e.g., log, square root, Box-Cox) during preprocessing. The skew() method in Pandas calculates the skewness for each numerical column.

Example

import pandas as pd
from sklearn.datasets import fetch_openml

# Fetch the Pima Indians Diabetes dataset
pima = fetch_openml(name='diabetes', version=1, as_frame=True, parser='pandas')
df = pima.frame
df.columns = ['num_pregnancies', 'glucose_concentration', 'blood_pressure', 'skin_thickness', 'insulin', 'bmi', 'diabetes_pedigree', 'age', 'class']
df['class'] = pd.to_numeric(df['class'])

# Calculate skewness
skewness = df.skew()

print("Skewness of each attribute:")
print(skewness)

Output

Skewness of each attribute:
num_pregnancies          0.902
glucose_concentration    0.174
blood_pressure          -1.844
skin_thickness           0.109
insulin                  2.272
bmi                     -0.429
diabetes_pedigree       1.920
age                      1.130
class                    0.635
dtype: float64

Values close to 0 indicate low skew (e.g., ‘glucose_concentration’, ‘skin_thickness’). Values significantly different from 0 indicate skewness. For example, ‘insulin’ (2.272) and ‘diabetes_pedigree’ (1.920) show significant positive skew, while ‘blood_pressure’ (-1.844) shows significant negative skew (likely influenced by the zero values). These attributes might benefit from transformation.

Further Resources

Pandas Documentation: https://pandas.pydata.org/pandas-docs/stable/
Scikit-learn Datasets: https://scikit-learn.org/stable/datasets.html
OpenML Datasets: https://www.openml.org/