repository·main·Indexed 23 days ago
https://github.com/tidyverse/rvestAn R package for scraping and harvesting data from web pages. It provides tools for reading HTML and extracting data using CSS selectors and XPath, including functions like read_html(), html_elements(), and html_table() to convert HTML tables into data frames.
rvest in conjunction with the polite package. The polite package helps ensure you are respecting robots.txt and avoids overwhelming websites with too many requests.You can install rvest either as part of the full tidyverse collection or as a standalone package using install.packages().
# The easiest way to get rvest is to install the whole tidyverse:
install.packages("tidyverse")
# Alternatively, install just rvest:
install.packages("rvest")The standard workflow for scraping data with rvest involves three steps:
read_html() to load a web page.html_elements() to find all nodes matching a CSS selector or XPath expression, or html_element() to find a single node.html_text2() for text content or html_attr() for attribute values.rvest is designed to work with magrittr pipes (|>) for easy expression of scraping tasks.
library(rvest)
# 1. Read an HTML page
starwars <- read_html("https://rvest.tidyverse.org/articles/starwars.html")
# 2. Find elements matching a CSS selector
films <- starwars |> html_elements("section")
# 3. Extract text from a specific element within those nodes
title <- films |>
html_element("h2") |>
html_text2()
# 4. Extract attribute values (returns a string)
episode <- films |>
html_element("h2") |>
html_attr("data-id") |>
readr::parse_integer()If a web page contains tabular data, you can use html_table() to convert an HTML element (representing a <table>) directly into a data frame (tibble).
html <- read_html("https://en.wikipedia.org/w/index.php?title=The_Lego_Movie&oldid=998422565")
html |>
html_element(".tracklist") |>
html_table()