soup

The Root struct is the primary object returned by parsing and DOM navigation methods. It contains:

• Pointer: The pointer to the current HTML node.
• NodeValue: The current node's value (tag name for ElementNode, text for TextNode).
• Error: An error object if an operation failed, otherwise nil.

If Error is not nil, you can check the Type field (of type ErrorType) to identify the specific issue. Supported error types include:

• ErrUnableToParse
• ErrElementNotFound
• ErrNoNextSibling / ErrNoPreviousSibling / ErrNoNextElementSibling / ErrNoPreviousElementSibling
• ErrCreatingGetRequest / ErrInGetRequest / ErrReadingResponse

Install the soup web scraper package using the standard Go get command:

go get github.com/anaskhan96/soup

This example demonstrates the full workflow: fetching a page, parsing it, finding a specific container, and iterating through its children to extract text and attributes.

package main

import (
	"fmt"
	"github.com/anaskhan96/soup"
	"os"
)

func main() {
	resp, err := soup.Get("https://xkcd.com")
	if err != nil {
		os.Exit(1)
	}
	doc := soup.HTMLParse(resp)
	links := doc.Find("div", "id", "comicLinks").FindAll("a")
	for _, link := range links {
		fmt.Println(link.Text(), "| Link :", link.Attrs()["href"])
	}
}

Use the following functions to perform HTTP requests and retrieve HTML strings:

• Get(url string) (string, error): Fetches the HTML content of the provided URL.
• GetWithClient(url string, client *http.Client): Fetches HTML using a custom *http.Client.
• Post(url string, bodyType string, payload interface{}) (string, error): Sends a POST request with a specified bodyType and payload.
• PostForm(url string, body url.Values): Sends a POST request with application/x-www-form-urlencoded content type.

To configure request headers or cookies globally before making a request, use the Headers and Cookies maps, or use the individual setter functions:

Once you have a Root object, you can navigate the DOM using several methods:

• Find(args ...string): Finds the first occurrence of a tag. You can optionally provide attribute names and values (e.g., Find("div", "class", "container")).
• FindAll(args ...string): Returns a slice of all matching Root elements.
• FindStrict(args ...string): Finds the first occurrence where all provided attributes are an exact match.
• FindAllStrict(args ...string): Returns a slice of all matching elements with exact attribute matches.

Arguments follow the pattern: tagName, attributeName, attributeValue.

Navigate relative to a specific Root element using these methods:

• Children(): Returns a slice of all direct children.
• FindNextSibling(): Returns the next sibling node.
• FindPrevSibling(): Returns the previous sibling node.
• FindNextElementSibling(): Returns the next sibling that is an ElementNode (skipping text/comment nodes).
• FindPrevElementSibling(): Returns the previous sibling that is an ElementNode (skipping text/comment nodes).

Use Post(url string, bodyType string, body interface{}) to perform a POST request using the default client. The body can be a map[string]string (serialized to JSON), netURL.Values (form encoded), []byte (JSON), or string (JSON).

For convenience with form data, use PostForm(url string, data netURL.Values).

For advanced usage, PostWithClient(url string, bodyType string, body interface{}, client *http.Client) allows providing a custom client.