DOCSEARCHbyAlgolia
Submit Docs

CRAN Task View for WebTechnologies

repository·main·Indexed 41 days ago

https://github.com/cran-task-views/webtechnologies

A curated directory and categorization of R packages for interacting with, processing, or building web-related technologies. It covers five main functional areas: direct data download and ingestion, online services (AWS, Azure, Google Cloud), frameworks for web-based R applications, low-level operations (HTTP clients like httr2, httr, crul, and curl), and general resources. The view includes specialized tools for ethical web scraping with polite, XML/HTML parsing via xml2 and rvest, JSON validation with jsonlite, and programmatic access to research data and geocoding services.

Tokens
4.2K
Snippets
6
Records
25
Agent score
2%

What's inside CRAN Task View for WebTechnologies

  1. Overview of Web Technologies in R

    main

    The WebTechnologies CRAN Task View provides a curated selection of R packages and strategies for interacting with internet resources. The ecosystem is organized into five main functional areas:

    1. Direct data download and ingestion: Methods for pulling data from remote files or parsing structured web data.
    2. Online services: Tools for interacting with cloud computing and storage.
    3. Frameworks for building web-based R applications: Tools to create web interfaces for R code.
    4. Low-level operations: Direct manipulation of web protocols and connections.
    5. Resources: Additional guidance and documentation.

  2. Frameworks for building web-based R applications

    main

    The WebTechnologies task view identifies several frameworks for creating web interfaces and services using R:

    • Interactive Web Apps:

      • shiny: The standard for building interactive web applications.
      • dashR (via GitHub): A web framework for Python, R, and Julia with React.js components.
      • radiant: A Shiny-based GUI for R that runs in a browser.
      • rcloud (via GitHub): Provides an iPython notebook-style web interface for R.

    • Web APIs and REST Services:

      • plumber: Allows creating REST APIs by decorating existing R source code.
      • beakr: Provides functionality similar to Python's Flask or JavaScript's Express.js.
      • opencpu: An embedded statistical computation framework that exposes a web API for R, LaTeX, and Pandoc.
      • FastRWeb: Provides basic infrastructure for turning R code into a web API.

    • Low-level and Flexible Frameworks:

      • fiery: A flexible web framework (uses reqres for HTTP classes and routr for routing).
      • Rook: Provides specifications and software for building Rook applications.
      • httpuv: Low-level socket and protocol support for HTTP and WebSocket requests.
      • servr: A simple HTTP server to serve files from a directory using httpuv.

  3. Access the CRAN Task View: WebTechnologies

    main

    The CRAN Task View for WebTechnologies provides a curated collection of R packages related to web technologies. You can access the official view at the CRAN website to explore the categorized packages and documentation.

    https://CRAN.R-project.org/view=WebTechnologies

  5. Parse XML and HTML data

    main

    Web scraping often involves parsing XML or HTML. These tools can work with direct URLs or locally stored files.

    XML Parsing

    • xml2: A user-friendly package for XML/HTML parsing and XPath queries. It manages memory and namespaces well. Use xml2::read_html() as a starting point for HTML.
    • XML: A foundational package that supports general XML/HTML parsing and XPath. Unlike xml2, it supports de novo creation of XML nodes and documents.
    • XML2R: Coerces XML into data frames.
    • selectr: Translates CSS3 Selectors into XPath 1.0 expressions.
    • xslt: An extension for xml2 to transform XML documents using XSLT stylesheets.
    • XMLSchema (GitHub): Provides facilities for reading XML schema documents and creating R class definitions from them.

    HTML Scraping and Utilities

    • rvest: A high-level scraping package that uses pipes (|> or %>%) to express common web scraping tasks.
    • postlightmercury: A client for the 'Mercury' web service that converts web pages into structured, clean text.
    • boilerpipeR: Extracts main text content from HTML by removing ads, sidebars, and headers using the boilerpipe Java library.
    • htmltools: Used to create HTML elements.
    • RHTMLForms (GitHub): Reads HTML documents to describe forms, elements, and hidden fields.
    • htm2txt: Converts HTML to plain text by removing all tags using regex.
    • W3CMarkupValidator: Provides an R interface to W3C Markup Validation Services.
    • webshot: Uses PhantomJS to take screenshots of web pages (useful for testing Shiny apps).
    • rdom (GitHub): Uses PhantomJS to access a webpage's Document Object Model (DOM).

  6. Choose a core HTTP request client

    main

    R's web capabilities are primarily built on top of libcurl. Depending on your use case, you should choose one of the following core packages:

    • httr2 (Priority: core): The recommended user-facing client for modern HTTP requests. If you are developing a package that interacts with web services, refer to the httr2 vignettes.
    • httr (Priority: core): The predecessor to httr2, still widely used for user-facing HTTP requests.
    • crul (Priority: core): An R6-based client intended for use by other packages rather than end-users. It supports:
      • Asynchronous HTTP requests.
      • Pagination helpers.
      • HTTP mocking via webmockr.
      • Request caching for unit tests via vcr.
      • Note: crul does not currently support OAuth.
    • curl (Priority: core): A low-level interface to the libcurl C library. It is not intended for typical R users but is useful for specific operations like web-based XML or FTP.
    • utils / base: Standard R packages providing download.file() and url(), which also utilize libcurl.
    # Example of installing core packages
r pkg("httr2", priority = "core")
r pkg("crul", priority = "core")
r pkg("curl", priority = "core")

  7. Parse and validate JSON data

    main

    JSON is the standard for most web service APIs. Use the following packages for reading, writing, and validating JSON:

    • jsonlite (Recommended): The standard for converting JSON to R objects.
    • rjson & RJSONIO: Alternative JSON reading/writing packages.
    • jqr: Provides bindings to the fast 'jq' JSON processor.
    • ndjson: Supports the Newline Delimited JSON format.
    • jsonvalidate: Validates JSON against a schema using the is-my-json-valid JavaScript library.
    • ajv: Validates JSON against a schema using the ajv JavaScript library.

  8. Download a remote file for manual ingestion

    main

    If you need to process file types that aren't supported by direct URL ingestion (e.g., specialized formats like SPSS or SAS), use a two-step process:

    1. Download the file to a local path.
    2. Pass the local path to a specialized ingestion function (e.g., from haven or foreign).

    Available Download Functions

    • utils: download.file()
    • curl: curl_download(), curl_fetch_multi()
    • httr2: req_perform(path = <your_file_path>) or req_perform() |> resp_body_string()
    • httr: GET()
    • RCurl: getURL()

  9. Interface with social media platforms

    main

    Packages for accessing data from various social networks:

    • Twitter: streamR (accessing streams and parsing to data frames), graphTweets (network graphs), and twitterreport (report generation).
    • Other Platforms: bskyr/atrrr (Bluesky), Rfacebook (Facebook), instaR (Instagram), Rlinkedin (LinkedIn), vkR (VK), slackr (Slack), and rtoot (Mastodon).

  10. Work with Google Cloud and Google APIs in R

    main

    Several packages facilitate interaction with Google's ecosystem:

    • Cloud Infrastructure: googleComputeEngineR (Compute Engine API) and googleCloudStorageR (Cloud Storage).
    • Data & Analytics: bigrquery (BigQuery) and rrefine (Open Refine).
    • Drive & Sheets: googledrive (Google Drive) and googlesheets4 (Google Sheets - supports reading/writing by title, key, or URL). gsheet can also download sheets via sharing links.
    • General APIs: gargle provides a general interface to Google APIs.
    # Example: Interacting with Google Sheets
install.packages("googlesheets4")

  11. Mocking and testing HTTP requests in R

    main

    When developing packages that communicate with web services, use these tools to simulate or record network activity:

    • Mocking (Stubbing):

      • webmockr: Stubs HTTP requests and sets expectations. It integrates with crul and httr. It returns nothing when requests match expectations.
      • vcr: Caches HTTP requests in test suites. It relies on webmockr and integrates with crul and httr.
      • httptest: A framework for testing packages that use httr. It allows mocking APIs, recording real responses for use as mocks, and making assertions about HTTP requests without a live connection.

    • Usage Note: webmockr is primarily for stubbing, while httptest and vcr are used for more complex recording and assertion workflows.

  12. Handling OAuth authentication in R

    main

    Authentication for web resources can be complex. For OAuth:

    • Recommended Approach: Use httr2 for the most straightforward implementation. httr also provides excellent support with several built-in demos for OAuth 1.0 (LinkedIn, Twitter, Vimeo) and OAuth 2.0 (Facebook, GitHub, Google).
    • Specialized Providers:
      • googleAuthR: Specifically for Google web services (OAuth 2.0).
      • AzureAuth: Specifically for Azure Active Directory.
    • Alternative: ROAuth provides a separate R interface to OAuth.