scraper

By default, scraper uses Tendril types that are thread-local and !Send. If you need to use the parsed data across multiple threads, you must enable the atomic feature in your Cargo.toml to use the atomic counting version of Tendril which implements Send.

[dependencies]
scraper = { version = "0.27.0", features = ["atomic"] }

Parsing in scraper does not fail hard. If the input HTML is malformed, the parser will attempt to populate the tree as best as possible by setting the quirks_mode.

If the errors feature is enabled, you can access a Vec<Cow<'static, str>> via the errors field on the Html struct to inspect specific parsing errors encountered during the process.

The Node enum is the primary abstraction for any part of the HTML tree. It can represent a Document, Fragment, Doctype, Comment, Text, Element, or ProcessingInstruction.

You can check the type of a node using boolean methods like is_element(), is_text(), or is_comment(), and access the underlying specific data using as_element(), as_text(), etc., which return an Option of the specific type.

The Selectable trait allows you to write functions that are generic over different types of HTML collections, such as Html (the entire document/fragment) or ElementRef (a specific element). This is useful for creating helper functions that can query both a full document and a specific subtree using the same logic.

Both &Html and ElementRef implement Selectable, returning an iterator of ElementRef items that match the provided Selector.

use scraper::{selectable::Selectable, selector::Selector};

fn text_of_first_match<'a, S>(selectable: S, selector: &Selector) -> Option<String>
where
    S: Selectable<'a>,
{
    selectable.select(selector).next().map(|element| element.text().collect())
}

Once you have parsed an Html object and a Selector, use the .select(&selector) method to iterate over the elements that match the selector. The method returns an iterator over matching elements.

use scraper::{Html, Selector};

let html = "<ul><li class='foo'>Item</li></ul>";
let fragment = Html::parse_fragment(html);
let selector = Selector::parse("li").unwrap();

for element in fragment.select(&selector) {
    println!("Found element: {}", element.value().name());
}

You can convert an element back into a string representation:

• .html(): Returns the full HTML of the element, including its own tags.
• .inner_html(): Returns only the HTML content inside the element.

Use the .attr("attribute_name") method on an element's value to retrieve the value of a specific attribute. It returns an Option<&str>.

use scraper::{Html, Selector};

let fragment = Html::parse_fragment(r#"<input name="foo" value="bar">"#);
let selector = Selector::parse(r#"input[name="foo"]"#).unwrap();

let input = fragment.select(&selector).next().unwrap();
let value = input.value().attr("value"); // Returns Some("bar")

Use the .text() method on an element to get an iterator over the text nodes contained within that element and its descendants.

use scraper::{Html, Selector};

let fragment = Html::parse_fragment("<h1>Hello, <i>world!</i></h1>");
let selector = Selector::parse("h1").unwrap();
let h1 = fragment.select(&selector).next().unwrap();

// Collect text nodes into a vector
let text = h1.text().collect::<Vec<_>>();
assert_eq!(vec!["Hello, ", "world!"], text);

Use Selector::parse to create a CSS selector from a string. This method returns a Result, so you should handle potential parsing errors (e.g., using .unwrap() in simple scripts or proper error handling in production).

use scraper::Selector;

let selector = Selector::parse("h1.foo").unwrap();