goquery

goquery provides a syntax and feature set similar to jQuery for the Go language. It is built on top of Go's net/html package and the cascadia CSS selector library.

Key Mental Models

• DOM vs. Nodes: Unlike jQuery, which operates on a full-featured DOM tree, goquery is based on net/html nodes. Consequently, stateful manipulation functions that rely on a full DOM (such as height(), css(), or detach()) are not available.
• Encoding: goquery requires UTF-8 encoded HTML. It is the responsibility of the caller to ensure the source document is UTF-8 encoded.
• API Style: The library uses a chainable interface and attempts to maintain the same function names as jQuery where possible to provide a familiar developer experience.

goquery is built around three primary types that facilitate HTML manipulation:

1. Document: Acts as the entry point. Unlike jQuery, which implicitly acts on the current DOM, goquery requires a Document to define the root node being manipulated.
2. Selection: Represents a set of nodes (similar to a jQuery object) that you can traverse, filter, and manipulate.
3. Matcher: An interface used for CSS selector matching.

Because goquery is statically typed, it uses specific naming conventions to provide different versions of jQuery-like methods based on the argument type provided.

Because goquery parses static HTML, it cannot execute JavaScript. If the content you need is rendered dynamically via JavaScript, goquery alone will not see it.

To handle these pages, consider these alternatives:

• Use a headless browser (e.g., webloop).
• Use a Go JavaScript parser package (e.g., otto).

To install goquery, use the following command:

$ go get github.com/PuerkitoBio/goquery

Go Version Requirements

Depending on the version of goquery you are using, you must meet the following minimum Go version requirements:

The underlying go.net/html package used by goquery requires HTML documents to be UTF-8 encoded. If you are scraping a page with a different charset, you must convert the response body to UTF-8 before passing it to goquery.NewDocumentFromReader.

You can use the github.com/djimenez/iconv-go package to create a UTF-8 reader from the original response body.

// Load the URL
res, err := http.Get(url)
if err != nil {
    // handle error
}
defer res.Body.Close()

// Convert the designated charset HTML to utf-8 encoded HTML.
// `charset` being one of the charsets known by the iconv package.
utfBody, err := iconv.NewReader(res.Body, charset, "utf-8")
if err != nil {
    // handler error
}

// use utfBody using goquery
doc, err := goquery.NewDocumentFromReader(utfBody)
if err != nil {
    // handler error
}
// use doc...

If you have cloned the repository locally, you can run tests and benchmarks using the standard Go toolchain.

To run unit tests:

$ cd $GOPATH/src/github.com/PuerkitoBio/goquery
$ go test

To run benchmarks (warning: this may take several minutes):

$ cd $GOPATH/src/github.com/PuerkitoBio/goquery
$ go test -bench=".*"

The Matcher interface defines how HTML nodes are matched against selectors. While standard selectors find all matches, you can optimize performance on large documents by using goquery.Single(selector) or goquery.SingleMatcher(matcher).

These return a Matcher that stops searching after the first match is found. This is semantically equivalent to calling .First() on a standard selection but is more efficient.

Note: The 'single-match' behavior only applies to selection methods. If used in a Filter method, it will still return all nodes in the selection that match the criteria.

// Standard way (finds all, then takes first)
sel1 := doc.Find("a").First()

// Optimized way (stops after first match)
sel2 := doc.FindMatcher(goquery.Single("a"))

goquery uses a stack-based approach for filtering. Methods like Filter, Not, FilterSelection, etc., create a new Selection that represents a subset of the current nodes. These operations can be chained together to drill down into specific parts of the DOM.

To 'escape' a sub-selection and return to the parent selection in a chain, use the End() method. End() pops the most recent filtering state off the internal stack, returning you to the Selection you had before the last filtering operation was applied.

While goquery provides Each and Map for iteration, you can use a standard Go for loop to iterate over the nodes in a selection. This is useful when you want to avoid the functional style of Each. You can access individual nodes using the .Eq(i) method on the selection.

sel := Doc().Find(".selector")
for i := range sel.Nodes {
	single := sel.Eq(i)
    // use `single` as a selection of 1 node
}

Use FilterFunction or NotFunction to filter elements based on a predicate function. The function receives the current index and the Selection object.

• FilterFunction(f func(int, *Selection) bool): Keeps elements where the function returns true.
• NotFunction(f func(int, *Selection) bool): Keeps elements where the function returns false.

// Example usage of FilterFunction
// selection.FilterFunction(func(i int, s *Selection) bool {
//     return i%2 == 0
// }) // Keeps only even-indexed elements