R - Web Data

R - Modern Web Data Acquisition

The web is a vast source of data, often available through structured files (CSV, JSON), APIs, or directly within HTML pages. Modern R provides a powerful ecosystem for programmatically acquiring this data. While older methods existed, today’s best practices revolve around the Tidyverse, a collection of packages designed for data science.

For web scraping and data retrieval, the key packages are:

• httr: For making sophisticated HTTP requests to servers and APIs.
• rvest: For easily harvesting (or ‘scraping’) data from HTML web pages.
• jsonlite: For working with JSON data, the standard for modern APIs.

Setup: Installing Modern Packages

We’ll use the tidyverse metapackage, which includes rvest and other essential tools like dplyr and stringr. httr is also commonly used for more direct web interactions.

# The tidyverse is a collection of essential packages for data science
install.packages("tidyverse")

# httr is the standard for handling HTTP requests
install.packages("httr")

# Load the packages into your R session
library(rvest)
library(dplyr)

Core Concept: Scraping HTML Tables

A common task is extracting tabular data directly from a webpage. The rvest package makes this incredibly straightforward. We will scrape a table of world population data from a Wikipedia page as our example.

Example: Scraping and Cleaning a Wikipedia Table

This code will read an HTML page, identify the main data table, and convert it directly into a clean R data frame (a tibble, to be precise).

# 1. Define the URL of the page to scrape
url <- "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population"

# 2. Read the HTML content from the URL
webpage <- read_html(url)

# 3. Target and extract the HTML table
# We use the `html_element` function with a CSS selector to find the first wikitable.
# Tip: Use the SelectorGadget browser extension to easily find CSS selectors.
population_table <- webpage %>%
  html_element(".wikitable") %>%
  html_table()

# 4. Inspect the raw extracted table
cat("--- Raw Table Head ---\n")
print(head(population_table))

# 5. Clean the data using dplyr for analysis
# The column names are messy, and data types need fixing.
world_population <- population_table %>%
  select( # Select and rename columns for clarity
    country_name = `Country / Dependency`,
    population = Population,
    percentage_of_world = `% of world`,
    date = Date
  ) %>%
  filter(country_name != "World") %>%
  mutate(
    # Remove comma separators and convert population to a number
    population = as.numeric(gsub(",", "", population)),
    # Remove the '%' sign and convert percentage to a numeric value
    percentage_of_world = as.numeric(gsub("%", "", percentage_of_world))
  )

# 6. Verify the cleaned data
cat("\n--- Cleaned Table Head ---\n")
print(head(world_population))

cat("\n--- Structure of Cleaned Table ---\n")
str(world_population)

Alternative Scenario: Downloading Files

If a page contains links to files (e.g., multiple CSVs), you can adapt the process to extract the URLs and download them. The original tutorial’s target URL is no longer active, but the modern workflow would look like this:

# Hypothetical example of downloading files
# library(purrr) # For iteration
# library(stringr) # For string matching
#
# # 1. Read the page
# page_with_links <- read_html("http://example.com/data_repository")
#
# # 2. Get all link URLs
# file_urls <- page_with_links %>%
#   html_elements("a") %>% # Find all anchor tags
#   html_attr("href") # Extract the 'href' attribute
#
# # 3. Filter for the CSV files you want
# csv_links <- file_urls[str_detect(file_urls, "\\.csv$")]
#
# # 4. Create a function to download a single file
# download_safely <- function(url) {
#   dest_file <- basename(url)
#   # Use tryCatch to handle potential download errors gracefully
#   tryCatch({
#     download.file(url, destfile = dest_file, mode = "wb")
#     cat("Successfully downloaded:", dest_file, "\n")
#   }, error = function(e) {
#     cat("Failed to download:", url, "\nError:", conditionMessage(e), "\n")
#   })
# }
#
# # 5. Apply the download function to each link
# # walk() is the modern equivalent of l_ply()
# walk(csv_links, download_safely)