DOCSEARCHbyAlgolia
Submit Docs

Colly Web Scraping Framework for Go

repository·master·Indexed 12 days ago

https://github.com/gocolly/colly

A fast and elegant web scraping framework for Go (Gophers) used to build crawlers, spiders, and scrapers. Colly v2 features high performance (>1k requests/sec), concurrency and flow control, session management, and built-in robots.txt support. It provides a clean interface for extracting structured data via HTML and XML callbacks, supports sync, async, and parallel scraping modes, and includes a functional options pattern for configuring collectors.

Tokens
8.4K
Snippets
37
Records
53
Agent score
93%

What's inside Colly

  1. Key features of Colly

    master

    Colly is a fast scraping framework for Go that includes the following capabilities:

    • Performance: Capable of >1k requests/sec on a single core.
    • Concurrency & Flow Control: Manages request delays and maximum concurrency per domain.
    • Session Management: Automatic cookie and session handling.
    • Scraping Modes: Supports sync, async, and parallel scraping.
    • Compliance: Built-in robots.txt support.
    • Advanced Capabilities: Caching, distributed scraping, automatic encoding of non-unicode responses, and extensibility via extensions.
    • Configuration: Can be configured via environment variables.

  3. Run Colly examples

    master

    You can execute the provided code snippets in the _examples directory to understand how to use Colly. To run a specific example, use the go run command followed by the path to the example file.

    Example command:

    go run [example/example.go]
    go run rate_limit/rate_limit.go

  4. Use Context to pass data between callbacks

    master

    The Context type provides a thread-safe mechanism for storing and retrieving data during a scraping session. It is primarily used to pass state or custom data between different Colly callbacks (e.g., from an OnRequest callback to an OnHTML callback).

    Key methods:

    • Put(key string, value interface{}): Stores a value associated with a key.
    • Get(key string) string: Retrieves a value specifically as a string. Returns an empty string if the key is not found.
    • GetAny(key string) interface{}: Retrieves a value of any type. Returns nil if the key is not found.
    • Clone() *Context: Creates a new Context instance containing a copy of all current key-value pairs. This is useful when you want to branch off a state without modifying the original context.
    // Example of passing data between callbacks using Context
colly.OnRequest(func(c *colly.Context) {
    c.Put("request_id", "12345")
})

colly.OnHTML("a", func(c *colly.Context) {
    id := c.Get("request_id")
    fmt.Println("Processing request:", id)
})

  5. Use XMLElement for XML and HTML data extraction

    master

    The XMLElement type is a wrapper used to navigate and extract data from XML or HTML documents within Colly. It provides a unified interface for querying elements using XPath, regardless of whether the underlying document is HTML or XML.

    Key properties include:

    • Name: The name of the tag.
    • Text: The inner text of the element.
    • Request & Response: The associated Colly request and response objects.
    • DOM: The underlying node (either *html.Node or *xmlquery.Node).
    • Index: The position of the element within the set of elements matched by an OnXML callback.

  6. Configure connection restrictions with LimitRule

    master

    The LimitRule struct allows you to define rate limiting and parallelism constraints for specific domains. You can target domains using either a regular expression (DomainRegexp) or a glob pattern (DomainGlob).

    Two types of limitations can be applied:

    • Parallelism: Limits the number of concurrent requests to matching domains using the Parallelism field.
    • Delay: Forces a wait period between requests using the Delay field. If Delay is set, Parallelism effectively becomes 1.

    To add randomized jitter to your delays, use the RandomDelay field.

    Important: After creating or cloning a LimitRule, you must call Init() before using it to compile patterns and initialize internal synchronization channels. However, if you are adding rules via the Collector's backend (e.g., using Limit() or Limits()), the library calls Init() for you automatically.

    rule := &colly.LimitRule{
	DomainGlob: "*.example.com",
	Delay:      2 * time.Second,
	Parallelism: 1,
}

  7. How Collector callbacks work

    master

    The Collector manages a lifecycle of events for every URL visited. When a request is triggered (e.g., via Visit):

    1. OnRequest and OnRequestHeaders are triggered.
    2. The HTTP request is performed.
    3. OnResponseHeaders is triggered. If the user calls Request.Abort() here, the body download is cancelled.
    4. OnResponse is triggered once the response is received.
    5. If the content type is HTML, OnHTML callbacks are executed for matching selectors.
    6. If the content type is XML, OnXML callbacks are executed for matching XPath queries.
    7. Finally, OnScraped is triggered.

    This sequence allows you to intercept requests, inspect headers to avoid heavy downloads, and extract data from the resulting body.

    // Example of aborting a download based on headers
c.OnResponseHeaders(func(r *colly.Response) {
    if strings.Contains(r.Headers.Get("Content-Type"), "application/pdf") {
        r.Abort()
    }
})

  8. Basic scraping example with Colly

    master

    This example demonstrates how to initialize a new collector, visit a URL, handle requests, and extract links from HTML elements using OnHTML and OnRequest callbacks.

    import (
	"fmt"

	"github.com/gocolly/colly/v2"
)

func main() {
	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("http://go-colly.org/")
}

  9. Sanitize file names for safe storage

    master

    When scraping data that you intend to save as files, use SanitizeFileName to ensure the resulting string is safe for use as a filename on most filesystems. It replaces dangerous characters and ensures a valid extension is present.

    safeName := SanitizeFileName("../../etc/passwd") 
// Result will be a sanitized version suitable for a filename

  10. Initialize a new Context with NewContext

    master

    To create a fresh, empty Context instance, use the NewContext function. This initializes the internal map and the mutex required for thread-safe operations.

    ctx := colly.NewContext()

  11. Abort an ongoing request in OnRequest

    master

    If you are inside an OnRequest callback and decide that a specific request should not proceed (e.g., due to a specific URL pattern or condition), you can call r.Abort(). This sets the internal abort flag, preventing the HTTP request from being executed.

    c.OnRequest(func(r *colly.Request) {
    if r.URL.Host == "forbidden.com" {
        r.Abort()
    }
})

  12. Initialize a LimitRule with Init()

    master

    The Init() method prepares a LimitRule for use by compiling the DomainRegexp or DomainGlob and initializing the internal waitChan.

    Init() is idempotent: if it has already been successfully initialized, subsequent calls do nothing. This makes it safe to share a single *LimitRule across multiple Collector instances via Collector.Limit.

    err := rule.Init()
if err != nil {
	// handle error (e.g., invalid regex or glob)
}