R - Data Reshaping
Modern Data Reshaping with the Tidyverse
Section titled “Modern Data Reshaping with the Tidyverse”Data reshaping is the process of changing the structure of your data to make it suitable for analysis. The modern approach in R is guided by the principles of Tidy Data, where each variable has its own column, each observation has its own row, and each value has its own cell.
The Tidyverse is a collection of R packages designed for data science that share this underlying philosophy. We will use two key packages from it:
dplyr: For data manipulation tasks like combining datasets.tidyr: For reshaping data between long and wide formats (pivoting).
Combining Datasets: bind_rows() and bind_cols()
Section titled “Combining Datasets: bind_rows() and bind_cols()”The base R functions rbind() and cbind() can be unpredictable, sometimes coercing all your data into a less useful type (like characters). The dplyr functions bind_rows() and bind_cols() are safer, faster, and more flexible alternatives.
library(dplyr)
# Best practice: create data frames or tibbles directlyaddresses <- tibble( city = c("Tampa", "Seattle", "Hartford", "Denver"), state = c("FL", "WA", "CT", "CO"), zipcode = c(33602, 98104, 06161, 80294))
cat("### The First Tibble\n")print(addresses)
# Create another tibblenew_addresses <- tibble( city = c("Lowry", "Charlotte"), state = c("CO", "NC"), # Note: Changed FL to NC for a better example zipcode = c(80230, 28202))
cat("\n### The Second Tibble\n")print(new_addresses)
# Use bind_rows() to combine them. It's safer and more efficient than rbind().all_addresses <- bind_rows(addresses, new_addresses)
cat("\n### The Combined Tibble\n")print(all_addresses)When we execute the above code, it produces the following result:
### The First Tibble# A tibble: 4 × 3 city state zipcode <chr> <chr> <dbl>1 Tampa FL 336022 Seattle WA 981043 Hartford CT 61614 Denver CO 80294
### The Second Tibble# A tibble: 2 × 3 city state zipcode <chr> <chr> <dbl>1 Lowry CO 802302 Charlotte NC 28202
### The Combined Tibble# A tibble: 6 × 3 city state zipcode <chr> <chr> <dbl>1 Tampa FL 336022 Seattle WA 981043 Hartford CT 61614 Denver CO 802945 Lowry CO 802306 Charlotte NC 28202Joining (Merging) Data Frames
Section titled “Joining (Merging) Data Frames”Instead of the base merge() function, dplyr provides a family of join functions that are more expressive and faster. They allow you to combine data frames based on matching key columns.
inner_join(): Keeps only rows with matching keys in both tables.left_join(): Keeps all rows from the left table, and matching rows from the right.full_join(): Keeps all rows from both tables.anti_join(): Keeps rows from the left table that do not have a match in the right.
library(MASS)library(dplyr)
# The Pima.te and Pima.tr data frames are available in the MASS package.# Let's see what an inner join looks like based on bp and bmi.
merged_pima <- inner_join(Pima.te, Pima.tr, by = c("bp", "bmi"))
# The .x and .y suffixes are automatically added to distinguish non-join columns# with the same names from the original tables.print(head(merged_pima))cat("\nTotal rows in merged data:", nrow(merged_pima))When we execute the above code, it produces the following result:
npreg.x glu.x bp skin.x ped.x age.x type.x bmi npreg.y glu.y skin.y ped.y age.y type.y1 1 117 60 23 0.466 27 No 33.8 2 125 20 0.088 31 No2 2 75 64 24 0.370 33 No 29.7 2 100 23 0.368 21 No3 5 189 64 33 0.583 29 Yes 31.2 3 158 13 0.295 24 No4 4 117 64 27 0.230 24 No 33.2 1 96 27 0.289 21 No5 3 115 66 39 0.150 28 No 38.1 1 114 36 0.289 21 No6 2 100 68 25 0.324 26 No 38.5 7 129 49 0.439 43 Yes
Total rows in merged data: 17Pivoting: From Wide to Long and Back
Section titled “Pivoting: From Wide to Long and Back”The functions melt() and cast() from the older reshape2 package have been superseded by a more powerful and intuitive pair of functions in the tidyr package: pivot_longer() and pivot_wider().
From Wide to Long with pivot_longer()
Section titled “From Wide to Long with pivot_longer()”pivot_longer() makes a “wide” dataset longer by gathering multiple columns into two new columns: one for the names of the original columns (the variable) and one for their values.
library(tidyr)library(MASS)
# The original 'ships' dataset from MASS packageprint(head(ships))
# We want to 'lengthen' the data, gathering period, service, and incidents# into key-value pairs. The 'id' variables are type and year.ships_long <- ships |> pivot_longer( cols = -c(type, year), # Select all columns EXCEPT type and year names_to = "variable", # New column for the original column names values_to = "value" # New column for the values )
print(head(ships_long))The result is a tidy, long-format tibble, ideal for plotting or summarization.
# Original data:# type year period service incidents# 1 A 60 60 127 0# 2 A 60 75 63 0# 3 A 65 60 1095 3# ...
# Pivoted data:# A tibble: 6 × 4# type year variable value# <fct> <fct> <chr> <int># 1 A 60 period 60# 2 A 60 service 127# 3 A 60 incidents 0# 4 A 60 period 75# 5 A 60 service 63# 6 A 60 incidents 0From Long to Wide with pivot_wider()
Section titled “From Long to Wide with pivot_wider()”The old cast() function combined reshaping and aggregation. The modern tidyverse approach separates these concerns for clarity and flexibility. First, we aggregate the data, then we pivot it wider.
# Let's recreate the summary table from the original tutorial.# Step 1: Group by the identifiers and the variable we want to summarize.# Step 2: Summarize the values (in this case, sum them).# Step 3: Pivot the summarized data wider.
ships_summary_wide <- ships_long |> group_by(type, year, variable) |> summarise(total_value = sum(value), .groups = 'drop') |> pivot_wider( names_from = variable, # Column to get new column names from values_from = total_value # Column to get cell values from )
print(head(ships_summary_wide))This explicit, multi-step process is easier to read, debug, and adapt than the old single cast() function.
# A tibble: 6 × 5# Groups: type, year [6]# type year incidents period service# <fct> <fct> <int> <int> <int># 1 A 60 0 135 190# 2 A 65 7 135 2190# 3 A 70 24 135 4865# 4 A 75 11 135 2244# 5 B 60 68 135 62058# 6 B 65 111 135 48979