Analyzing Time Series Data

AI with Python – Analyzing Time Series Data

Predicting the next value or state in a given input sequence is a fundamental concept in machine learning. This chapter provides a detailed explanation of analyzing time series data, a common type of sequential data.

Introduction

Time series data consists of a series of data points indexed in time order, typically at uniform time intervals. When building sequence prediction models in machine learning, we often deal with sequential data, and time series data is a prominent example. The ordering of data is a crucial feature of sequential data, distinguishing it from other types of datasets.

Basic Concept of Sequence Analysis or Time Series Analysis

Sequence analysis, or specifically time series analysis in this context, aims to predict future values in a sequence based on previously observed values. The prediction can be for various types of sequential data: the next numerical value, a categorical symbol, tomorrow’s weather, the next word in a sentence, etc. Sequence analysis is highly valuable in applications such as stock market forecasting, weather prediction, demand forecasting, and anomaly detection.

Example

Consider a simple sequence of values: A, B, C, D. A sequence prediction model would analyze this pattern to predict the next value, E.

This could be visualized as: [A, B, C, D] -> Sequence Prediction Model -> [E]

Installing Useful Packages

For time series data analysis using Python, we will primarily use the following packages. Ensure you have them installed in your Python environment.

Pandas

Pandas is an open-source library providing high-performance, easy-to-use data structures and data analysis tools for Python. It’s essential for handling time series data. You can install Pandas using pip:

pip install pandas

If you are using Anaconda, you can install it using conda:

conda install pandas

NumPy

NumPy is the fundamental package for numerical computation in Python. Pandas is built on top of NumPy. It’s usually installed as a dependency of Pandas. If not, install it using pip:

pip install numpy

Matplotlib

Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python. It’s crucial for plotting time series data. Install it using pip:

pip install matplotlib

hmmlearn

For specific sequence modeling tasks using Hidden Markov Models (HMMs), the hmmlearn library is useful. It provides algorithms and models for HMMs. You can install it using pip:

pip install hmmlearn

If you are using Anaconda:

conda install -c conda-forge hmmlearn

yfinance (for stock data example)

To fetch financial data for one of our examples, the yfinance library is convenient. Install it using pip:

pip install yfinance

Note: Other libraries like statsmodels (for classical time series models like ARIMA, SARIMA) and deep learning frameworks (TensorFlow, PyTorch) are also widely used for advanced time series analysis, but are beyond the scope of this introductory section focusing on Pandas and HMMs.

Pandas: Handling, Slicing and Extracting Statistics from Time Series Data

Pandas is an indispensable tool for working with time series data in Python. Key capabilities include:

• Creating date ranges using pd.date_range().
• Indexing Pandas Series or DataFrames with datetime objects.
• Resampling time series data to different frequencies using resample().
• Performing time-based slicing and selection.

Example

The following example demonstrates handling and slicing time series data using Pandas. We’ll use the Monthly Arctic Oscillation (AO) data, which can typically be downloaded from sources like NOAA. For this example, assume you have downloaded the data into a text file named AO_monthly.txt. The file is expected to have data in columns, where one column (e.g., the third column) contains the AO index values.

You can find such data at sources like the National Oceanic and Atmospheric Administration (NOAA) Climate Prediction Center. For example, search for ‘monthly.ao.index.b50.current.ascii’. Ensure the data is saved as a plain text file where values are space or tab-delimited.

Handling time series data

Follow these steps to load and handle time series data:

1. Import necessary packages:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

2. Define a function to read data from the input file. This function assumes the relevant data is in the third column (index 2).

def read_data(input_file, column_index=2):
    # Load data, assuming space or tab delimiter
    input_data = np.loadtxt(input_file, delimiter=None)

    # Create a date range. Assuming data starts from January 1950 and has monthly frequency.
    # Adjust '1950-01' and freq as per your dataset's specifics.
    num_periods = input_data.shape[0]
    dates = pd.date_range('1950-01-01', periods=num_periods, freq='MS') # MS for Month Start

    # Create a Pandas Series with the date range as index
    # Ensure the selected column exists and contains the numeric data
    time_series_data = pd.Series(input_data[:, column_index], index=dates)
    return time_series_data

3. Specify the input file path and read the data:

# Replace with the actual path to your data file
input_file = "AO_monthly.txt"

# Assuming your data is in the file, and the AO index is in the third column (index 2)
# If your file structure is different, adjust column_index in read_data or the function itself
try:
    ao_timeseries = read_data(input_file, column_index=2)
except FileNotFoundError:
    print(f"Error: The file {input_file} was not found. Please check the path.")
    # As a fallback for demonstration, create some dummy data if file not found
    print("Using dummy data for demonstration.")
    dates_dummy = pd.date_range('1950-01-01', periods=100, freq='MS')
    data_dummy = np.random.randn(100)
    ao_timeseries = pd.Series(data_dummy, index=dates_dummy)
except IndexError:
    print(f"Error: Could not extract data from column. Check file format and column_index.")
    print("Using dummy data for demonstration.")
    dates_dummy = pd.date_range('1950-01-01', periods=100, freq='MS')
    data_dummy = np.random.randn(100)
    ao_timeseries = pd.Series(data_dummy, index=dates_dummy)

4. Plot and visualize the data:

plt.figure(figsize=(10, 6))
ao_timeseries.plot()
plt.title('Monthly Arctic Oscillation Index')
plt.xlabel('Date')
plt.ylabel('AO Index')
plt.grid(True)
plt.show()

This will display a plot of the Arctic Oscillation index over time. The x-axis represents dates, and the y-axis represents the AO index values, showing fluctuations and trends.

Slicing time series data

Slicing allows you to select a specific portion of the time series. For instance, to view data only from 1980 to 1990:

# Ensure ao_timeseries is loaded and is not empty
if not ao_timeseries.empty:
    try:
        ao_subset = ao_timeseries['1980':'1990']
        plt.figure(figsize=(10, 6))
        ao_subset.plot()
        plt.title('Monthly AO Index (1980-1990)')
        plt.xlabel('Date')
        plt.ylabel('AO Index')
        plt.grid(True)
        plt.show()
    except Exception as e:
        print(f"Error during slicing or plotting: {e}")
else:
    print("Time series data is empty, skipping slicing example.")

This code snippet will generate a plot focusing on the AO index values strictly between the start of 1980 and the end of 1990.

Extracting Statistics from Time Series Data

Pandas makes it easy to extract common descriptive statistics from time series data, which are crucial for understanding its characteristics.

Assuming ao_timeseries is your loaded Pandas Series:

Mean

Calculate the average value:

if not ao_timeseries.empty:
    mean_value = ao_timeseries.mean()
    print(f"Mean AO Index: {mean_value:.6f}")
else:
    print("Time series data is empty.")

Example output (will vary based on data):

Mean AO Index: -0.111431

Maximum

Find the highest value:

if not ao_timeseries.empty:
    max_value = ao_timeseries.max()
    print(f"Maximum AO Index: {max_value:.6f}")
else:
    print("Time series data is empty.")

Example output:

Maximum AO Index: 3.495300

Minimum

Find the lowest value:

if not ao_timeseries.empty:
    min_value = ao_timeseries.min()
    print(f"Minimum AO Index: {min_value:.6f}")
else:
    print("Time series data is empty.")

Example output:

Minimum AO Index: -4.265700

Getting all descriptive statistics at once

The describe() method provides a summary of statistics:

if not ao_timeseries.empty:
    desc_stats = ao_timeseries.describe()
    print("\nDescriptive Statistics:\n", desc_stats)
else:
    print("Time series data is empty.")

Example output:

Descriptive Statistics:
count    817.000000
mean      -0.111431
std        1.003151
min       -4.265700
25%       -0.649430
50%       -0.042744
75%        0.475720
max        3.495300
dtype: float64

Resampling

Resampling involves changing the frequency of your time series data. This is useful for aggregation (downsampling) or interpolation (upsampling). Common parameters are the target frequency and the aggregation method.

• Time period (e.g., ‘YE’ for year-end, ‘QE’ for quarter-end, ‘D’ for day).
• Aggregation method (e.g., mean(), median(), sum()).

Resampling to Annual Frequency with Mean

To resample the monthly data to annual data using the mean of values within each year:

if not ao_timeseries.empty:
    # 'YE' stands for Year End frequency. Pandas also accepts 'A'.
    ao_annual_mean = ao_timeseries.resample('YE').mean()
    plt.figure(figsize=(10, 6))
    ao_annual_mean.plot(style='g--')
    plt.title('Annual Mean AO Index')
    plt.xlabel('Year')
    plt.ylabel('Mean AO Index')
    plt.grid(True)
    plt.show()
else:
    print("Time series data is empty, skipping resampling example.")

This will plot the annually averaged AO index, showing a smoother trend compared to the monthly data.

Resampling to Annual Frequency with Median

Similarly, using the median for aggregation:

if not ao_timeseries.empty:
    ao_annual_median = ao_timeseries.resample('YE').median()
    plt.figure(figsize=(10, 6))
    ao_annual_median.plot()
    plt.title('Annual Median AO Index')
    plt.xlabel('Year')
    plt.ylabel('Median AO Index')
    plt.grid(True)
    plt.show()
else:
    print("Time series data is empty, skipping resampling example.")

This plot shows the median AO index for each year.

Rolling Mean (Moving Average)

A rolling mean calculates the average over a sliding window of a fixed size. This is often used to smooth out short-term fluctuations and highlight longer-term trends.

if not ao_timeseries.empty:
    # Calculate 12-month rolling mean
    ao_rolling_mean_12m = ao_timeseries.rolling(window=12, center=False).mean()
    plt.figure(figsize=(10, 6))
    ao_rolling_mean_12m.plot(style='-r')
    plt.title('12-Month Rolling Mean AO Index')
    plt.xlabel('Date')
    plt.ylabel('AO Index')
    plt.grid(True)
    plt.show()
else:
    print("Time series data is empty, skipping rolling mean example.")

The plot will display the 12-month moving average of the AO index, which can help identify trends more clearly than the raw monthly data.

Analyzing Sequential Data with Hidden Markov Models (HMM)

Hidden Markov Models (HMMs) are statistical models widely used for time series and sequential data where the system is assumed to be a Markov process with unobserved (hidden) states. Applications include speech recognition, bioinformatics, financial modeling, and more.

Understanding Hidden Markov Models (HMM)

An HMM is a stochastic model built upon the Markov chain concept. The core assumption is that the probability of transitioning to a future state depends only on the current state, not on the sequence of events that preceded it (the Markov property). The ‘hidden’ aspect refers to the fact that we don’t directly observe the states themselves, but rather some output or observation that is probabilistically dependent on the current hidden state.

For instance, in weather modeling, the hidden states might be ‘Sunny’, ‘Cloudy’, ‘Rainy’, and the observations could be ‘Wearing a coat’, ‘Carrying an umbrella’, ‘Wearing sunglasses’. We observe the clothing choices to infer the probable weather state.

Mathematically, an HMM is defined by:

States (S)

A set of N hidden or latent states: S = {s₁, s₂, …, sN}.

Output Symbols / Observations (O)

A set of M possible output symbols or observations: O = {o₁, o₂, …, oM}.

State Transition Probability Matrix (A)

An N x N matrix where Aᵢⱼ is the probability of transitioning from state sᵢ to state sⱼ.

Observation Emission Probability Matrix (B)

An N x M matrix where Bⱼ(k) is the probability of observing symbol oₖ when the system is in state sⱼ.

Initial State Probability Distribution (Π)

An N-dimensional vector where Πᵢ is the probability that the initial state is sᵢ.

Thus, an HMM is parameterized by λ = (A, B, Π), assuming S and O are known.

Example: Analyzing Stock Market Data with HMM

In this example, we’ll analyze historical stock market data to model potential underlying market regimes (e.g., high volatility, low volatility) using an HMM. We’ll use the hmmlearn library for the HMM implementation and yfinance to fetch stock data.

1. Import necessary packages:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import yfinance as yf
from hmmlearn.hmm import GaussianHMM
import warnings
import datetime

2. Load stock data. We’ll fetch data for Intel (INTC) as an example.

start_date_str = '2010-01-01'
end_date_str = '2020-12-31'
stock_ticker = 'INTC'

# Fetch data using yfinance
try:
    stock_data = yf.download(stock_ticker, start=start_date_str, end=end_date_str)
    if stock_data.empty:
        raise ValueError("No data fetched. Check ticker or date range.")
except Exception as e:
    print(f"Error fetching data for {stock_ticker}: {e}")
    # Fallback to dummy data if fetching fails
    print("Using dummy data for HMM example.")
    dates_dummy = pd.date_range(start_date_str, end=end_date_str, freq='B') # Business days
    dummy_close = np.random.rand(len(dates_dummy)) * 100 + 50
    dummy_volume = np.random.rand(len(dates_dummy)) * 1e6 + 1e5
    stock_data = pd.DataFrame({'Close': dummy_close, 'Volume': dummy_volume}, index=dates_dummy)

closing_prices = stock_data['Close']
volumes = stock_data['Volume']

3. Prepare features for the HMM. We’ll use daily percentage change in closing price and volume.

# Calculate daily percentage difference of closing prices
# Ensure closing_prices is a Pandas Series or NumPy array
diff_percentages = 100.0 * closing_prices.pct_change().fillna(0) # fillna(0) for the first value

# We need to align volumes with diff_percentages.
# pct_change() makes the first value NaN, which we filled with 0.
# Ensure volumes align or adjust if lengths differ after processing.
# For simplicity, ensure both features start from the same point after pct_change.
if len(diff_percentages) == len(volumes):
    aligned_volumes = volumes
elif len(diff_percentages) == len(volumes) -1 : # if pct_change created one less
    aligned_volumes = volumes[1:]
else: # Fallback or error if alignment is tricky
    aligned_volumes = volumes[-len(diff_percentages):] # A common way to align

# Create the training data: features are columns
# Using log of volume can sometimes help stabilize variance
training_data = np.column_stack([diff_percentages, np.log(aligned_volumes + 1e-6)]) # Add small constant to avoid log(0)

4. Create and train the Gaussian HMM. n_components is the number of hidden states (regimes).

# Let's assume 4 hidden market regimes (e.g., low-vol, med-vol, high-vol-up, high-vol-down)
num_hidden_states = 4
# 'diag' covariance means features are independent within each state
hmm_model = GaussianHMM(n_components=num_hidden_states, covariance_type='diag', n_iter=1000, random_state=42)

print("Training HMM...")
with warnings.catch_warnings():
    warnings.simplefilter('ignore') # HMMs can sometimes raise convergence warnings
    hmm_model.fit(training_data)
print("HMM training complete.")

5. Predict the hidden states for the training data and visualize.

hidden_states = hmm_model.predict(training_data)

plt.figure(figsize=(15, 8))
for i in range(num_hidden_states):
    # Plot data points belonging to each hidden state
    mask = hidden_states == i
    plt.plot(stock_data.index[mask], closing_prices[mask], '.', label=f'State {i}', alpha=0.6)

plt.title(f'{stock_ticker} Closing Prices by HMM Hidden State')
plt.xlabel('Date')
plt.ylabel('Closing Price')
plt.legend()
plt.grid(True)
plt.show()

# Visualize the hidden states over time as a sequence
plt.figure(figsize=(15, 4))
plt.plot(stock_data.index, hidden_states, drawstyle='steps-post')
plt.title(f'{stock_ticker} HMM Hidden States Sequence')
plt.xlabel('Date')
plt.ylabel('Hidden State')
plt.yticks(range(num_hidden_states))
plt.grid(True)
plt.show()

The first plot shows the closing prices color-coded by the HMM’s inferred hidden state. Different states might correspond to different market behaviors (e.g., periods of stability, volatility, upward or downward trends). The second plot shows the sequence of these hidden states over time.

Note: Interpreting the meaning of HMM states requires domain knowledge and further analysis of the characteristics (mean, variance of features) of each state.

Further Learning

Time series analysis is a vast field. For more advanced topics, consider exploring:

• Classical models: ARIMA, SARIMA, Exponential Smoothing (ETS) using the statsmodels library.
• State Space Models: Kalman filters, Unobserved Components Models.
• Machine Learning for Time Series: Using regression models with lagged features, tree-based models.
• Deep Learning for Time Series: Recurrent Neural Networks (RNNs, LSTMs, GRUs), Transformers using libraries like TensorFlow or PyTorch.
• Time Series Cross-Validation: Techniques like TimeSeriesSplit in scikit-learn.