DOCSEARCHbyAlgolia
Submit Docs

htmlquery

repository·master·Indexed 21 days ago

https://github.com/antchfx/htmlquery

A Go package for extracting data from HTML documents using XPath expressions. It provides functionality to load HTML from URLs, local files, or io.Reader sources, and query nodes using functions like Find, FindOne, QueryAll, and Query. The library includes utilities for extracting inner text and attributes, rendering nodes back to HTML strings, and an LRU cache for XPath query strings.

Tokens
2.1K
Snippets
10
Records
11
Agent score
24%

What's inside htmlquery

  1. Load HTML documents from different sources

    master

    You can load HTML documents into a doc object using several methods depending on your source:

    • From a URL: Use htmlquery.LoadURL(url).
    • From a local file: Use htmlquery.LoadDoc(filePath).
    • From a string: Use htmlquery.Parse(reader) where the reader is a strings.NewReader containing your HTML string.
    // Load from URL
doc, err := htmlquery.LoadURL("http://example.com/")

// Load from file
filePath := "/home/user/sample.html"
doc, err := htmlquery.LoadDoc(filePath)

// Load from string
s := "<html>...</html>"
doc, err := htmlquery.Parse(strings.NewReader(s))

  3. How NodeNavigator works with XPath

    master

    The NodeNavigator is an implementation of xpath.NodeNavigator that allows the xpath engine to traverse the html.Node tree. It maps HTML node types (Element, Text, Comment, etc.) to XPath node types.

    When using QuerySelector or QuerySelectorAll, the library internally creates a NodeNavigator using CreateXPathNavigator(top) to drive the selection process.

  4. Configure XPath query caching

    master

    htmlquery includes a built-in LRU cache for XPath query strings to avoid re-compiling expressions.

    To disable this caching mechanism, set the DisableSelectorCache package variable to true.

    htmlquery.DisableSelectorCache = true

  5. Evaluate XPath expressions manually

    master

    For advanced use cases like evaluating counts or other XPath functions, you can use the underlying xpath package with htmlquery.CreateXPathNavigator(doc).

    import "github.com/antchfx/xpath"

// Evaluate the number of all IMG elements
expr, _ := xpath.Compile("count(//img)")
v := expr.Evaluate(htmlquery.CreateXPathNavigator(doc)).(float64)
fmt.Printf("total count is %f", v)

  6. Query HTML elements using XPath

    master

    Use the following functions to extract nodes from a document:

    • htmlquery.Find(doc, expression): Returns a slice of matched nodes. Note: This will panic if the XPath expression is invalid.
    • htmlquery.FindOne(doc, expression): Returns the first matched node or nil.
    • htmlquery.QueryAll(doc, expression): Returns a slice of matched nodes and an error. Use this if you want to handle invalid XPath expressions gracefully instead of panicking.
    • htmlquery.QuerySelector(doc, expression) and htmlquery.QuerySelectorAll(doc, expression): Accept pre-compiled query expression objects to improve performance by avoiding re-compilation.
    // Returns matched elements or error
nodes, err := htmlquery.QueryAll(doc, "//a")
if err != nil {
	panic(`not a valid XPath expression.`)
}

// Returns all matched elements (panics on invalid XPath)
list := htmlquery.Find(doc, "//a")

// Returns the first matched element
a := htmlquery.FindOne(doc, "//a[3]")

  7. Extract text and attributes from nodes

    master

    Once you have a node, you can extract its content or attributes:

    • htmlquery.InnerText(n): Returns the text content of the node. If the node is an attribute (e.g., from a query like //a/@href), it returns the attribute value.
    • htmlquery.SelectAttr(n, attrName): Returns the value of the specified attribute for the given node.
    // Get href value from an attribute node
list := htmlquery.Find(doc, "//a/@href")
for _, n := range list {
    fmt.Println(htmlquery.InnerText(n))
}

// Get a specific attribute from an element node
img := htmlquery.FindOne(doc, "//img")
src := htmlquery.SelectAttr(img, "src")

  8. Query HTML nodes using XPath

    master

    Once you have an *html.Node tree, you can use XPath expressions to find specific elements.

    Error-handling variants

    • Query(top *html.Node, expr string): Returns the first matching *html.Node or nil. Returns an error if the XPath expression is invalid.
    • QueryAll(top *html.Node, expr string): Returns a slice of all matching *html.Node elements. Returns an error if the XPath expression is invalid.
    • FindOne(top *html.Node, expr string): Returns the first matching *html.Node. Panics if the XPath expression is invalid.
    • Find(top *html.Node, expr string): Returns a slice of all matching *html.Node elements. Panics if the XPath expression is invalid.
    // Find a single element
node, err := htmlquery.Query(doc, "//div[@id='content']")

// Find all matching elements
nodes, err := htmlquery.QueryAll(doc, "//a[@class='link']")

// Quick find (panics on invalid XPath)
node := htmlquery.FindOne(doc, "//h1")

  9. Load HTML documents from URLs or files

    master

    Use htmlquery to fetch and parse HTML documents from the web or local storage into an *html.Node tree.

    • LoadURL(url string): Fetches the HTML from the specified URL using the default HTTP client. It automatically handles gzip and deflate compression and detects character sets.
    • LoadURLWithClient(url string, client *http.Client): Fetches HTML using a custom *http.Client. It also enables gzip by default in the request headers.
    • LoadDoc(path string): Loads and parses an HTML document from a local file path.
    • Parse(r io.Reader): Parses HTML directly from any io.Reader.
    // Load from URL
doc, err := htmlquery.LoadURL("https://example.com")
if err != nil {
    log.Fatal(err)
}

// Load from local file
doc, err := htmlquery.LoadDoc("index.html")

// Parse from a reader
doc, err := htmlquery.Parse(strings.NewReader("<html><body></body></html>")

  10. Extract text and attributes from HTML nodes

    master

    After locating nodes, use these functions to extract their content:

    • InnerText(n *html.Node): Returns the concatenated text content of the node and all its descendants (excluding comments).
    • SelectAttr(n *html.Node, name string): Returns the value of the attribute named name. If the node is an attribute node itself or a root node matching the name, it returns the InnerText.
    • ExistsAttr(n *html.Node, name string): Returns true if the attribute name exists on the node.
    // Get text content
text := htmlquery.InnerText(node)

// Get attribute value
link := htmlquery.SelectAttr(node, "href")

// Check if attribute exists
if htmlquery.ExistsAttr(node, "class") {
    // ...
}

  11. Render HTML nodes to strings

    master

    Convert *html.Node objects back into HTML strings.

    • OutputHTML(n *html.Node, self bool):
      • If self is true, it renders the node n including its own tags.
      • If self is false, it renders only the children of node n.
    // Render the node and its children
htmlStr := htmlquery.OutputHTML(node, true)

// Render only the children of the node
innerHtml := htmlquery.OutputHTML(node, false)