DOCSEARCHbyAlgolia
Submit Docs

Reaver Clojure Library

repository·master·Indexed 41 days ago

https://github.com/mischov/reaver

A Clojure library for scraping and extracting structured data from HTML into EDN data structures. Reaver acts as a high-level wrapper around Jsoup, providing functions like `parse` and `extract-from` to pull data using CSS selectors.

Tokens
610
Snippets
2
Records
3
Agent score
50%

What's inside Reaver

  1. Overview of Reaver

    master
    Reaver is a Clojure library that wraps Jsoup to facilitate extracting data from HTML into EDN/Clojure maps. Unlike templating libraries (like Enlive or Laser) that focus on generating HTML, Reaver is purpose-built for data extraction by leveraging Jsoup's CSS selection mechanisms.

  3. Extract data from HTML using Reaver

    master

    Reaver provides an API to parse HTML and extract specific elements into Clojure data structures (EDN).

    Note: Reaver does not handle HTTP fetching. You must use a library like slurp, aleph, or clj-http to retrieve the HTML content first.

    To extract data, you typically follow this workflow:

    1. Parse the HTML string using parse.
    2. Use extract-from to define selectors and the mapping of data to extract.

    Commonly used functions:

    • parse: Converts an HTML string into a searchable document.
    • extract-from: The primary function for pulling structured data out of a parsed document using CSS selectors.
    • text: A helper to extract the text content of a selected element.
    • attr: A helper to extract the value of a specific attribute (e.g., :href).
    (require '[reaver :refer [parse extract-from text attr]])

; 1. Fetch the HTML (using slurp as an example)
(def html-content (slurp "https://example.com/"))

; 2. Parse and extract data
; The syntax for extract-from allows you to specify a base selector, 
; then a sequence of key-value pairs where each pair consists of 
; a selector and an extraction function (like `text` or `attr`).
(extract-from (parse html-content) 
              ".item-container" 
              [:title :link] 
              ".title" text 
              ".title" (attr :href))

; Result: ({:title "...", :link "..."}, ...)