polite R package

The politely() function is an 'adverb' (similar to purrr adverbs) that can wrap any web-accessing function to make it follow the four pillars of polite scraping: Introduce Yourself, Seek Permission, Take Slowly and Never Ask Twice.

To use it, wrap your existing function (like httr::GET or httr::POST) with politely() to create a new function. This new function will automatically handle:

• Providing a human-readable user-agent string.
• Consulting robots.txt for permissions.
• Respecting rate limits (crawl delays).
• Caching results (memoization).

Requirement: The wrapped function must accept a url argument (or have url as its first argument).

# Example: Wrapping httr::GET
polite_GET <- politely(httr::GET, verbose=TRUE)

# Use the new function instead of the original
res <- polite_GET("http://colormind.io/list")

If you are developing an R package that accesses the web, you can use polite::use_manners() to bootstrap a polite scraping workflow.

When called within your package or analysis directory, it creates R/polite-scrape.R containing template functions that act as drop-in replacements for common web-accessing tools. These functions use a polite_ prefix and are designed to look and feel like the original functions while adhering to polite scraping principles.

Common replacements include:

• read_html() $\rightarrow$ polite_read_html()
• download.file() $\rightarrow$ polite_download_file()

You can install the stable version of polite from CRAN or the development version from GitHub.

From CRAN:

install.packages("polite")

From GitHub (Development version):

install.packages("remotes")
remotes::install_github("dmi3kno/polite")

You can install the stable version of polite from CRAN:

install.packages("polite")

To install the development version from GitHub, use the remotes package:

install.packages("remotes")
remotes::install_github("dmi3kno/polite")

To perform a simple scrape, use bow() to create a session and scrape() to retrieve the page content. You can pass a query list to scrape() to handle URL parameters.

library(polite)
library(rvest)

session <- bow("https://www.cheese.com/by_type", force = TRUE)
result <- scrape(session, query=list(t="semi-soft", per_page=100)) %>%
  html_node("#main-body") %>%
  html_nodes("h3") %>%
  html_text()
head(result)

When navigating through a site (e.g., following "Older posts" links), use nod() to update the session to the new URL within a loop. This ensures every new URL is checked against robots.txt.

library(polite)
library(rvest)

hrbrmstr_posts <- data.frame()
url <- "https://rud.is/b/"
session <- bow(url)

while(!is.na(url)){
  message("Scraping ", url)
  
  # Update session to the new URL and scrape
  current_page <- nod(session, url) %>%
    scrape(verbose=TRUE)
  
  # Extract data
  hrbrmstr_posts <- current_page %>%
    html_nodes(".entry-title a") %>%
    polite::html_attrs_dfr() %>%
    rbind(hrbrmstr_posts)
  
  # Find the next URL
  url <- current_page %>%
    html_node(".nav-previous a") %>%
    html_attr("href")
}

tibble::as_tibble(hrbrmstr_posts)

By default, functions created via politely() will check robots.txt to ensure the requested URL is allowed. If you need to bypass this check (for example, if an API endpoint is disallowed in robots.txt but legal to access for small requests), you can set the robots argument to FALSE when creating the polite function.

library(polite)
library(httr)

# Create a GET function that ignores robots.txt
polite_GET_nrt <- politely(GET, verbose=TRUE, robots = FALSE)

# Use it to access a disallowed endpoint
beatles_lst <- polite_GET_nrt("https://musicbrainz.org/ws/2/artist/",
                   query=list(query="Beatles", limit=10),
                   httr::accept("application/json")) %>%
  httr::content(type = "application/json")