DOCSEARCHbyAlgolia
Submit Docs

grab

repository·master·Indexed 23 days ago

https://github.com/cavaliergopher/grab

A Go package for high-concurrency file downloading featuring progress monitoring, automatic download resumption, rate limiting, and checksum validation. It provides a simple API via grab.Get() for one-off downloads, as well as a more granular grab.Client for managing multiple requests, batch downloads via DoBatch(), and channel-based processing with DoChannel().

Tokens
4.4K
Snippets
8
Records
29
Agent score
81%

What's inside grab

  1. How grab handles state and resuming

    master

    Grab is designed to be stateless. It does not create auxiliary state files (like .crdownload) on your filesystem. Instead, it relies on the local and remote files being immutable.

    Auto-resume: If a download is interrupted, running the same request again will automatically attempt to resume the incomplete download.

    Warning: If the local file or the remote file is modified by another process outside of grab, resuming may result in file corruption. To avoid this, ensure remote files are immutable or disable the resume feature.

  2. Monitor download progress and use grab.Client

    master

    For granular control, such as monitoring progress or managing multiple requests, use grab.NewClient() and grab.NewRequest(destination, url).

    Key features for monitoring:

    • client.Do(req): Starts the download and returns a response object.
    • resp.BytesComplete(): Returns the number of bytes transferred so far.
    • resp.Size: The total size of the file in bytes.
    • resp.Progress(): Returns the progress as a float (0.0 to 1.0).
    • resp.Done: A channel that is closed when the download is complete.
    • resp.Err(): Returns any error encountered during the download process.
    • resp.Filename: The path to the saved file.
    package main

import (
	"fmt"
	"os"
	"time"

	"github.com/cavaliergopher/grab/v3"
)

func main() {
	// create client
	client := grab.NewClient()
	req, _ := grab.NewRequest(".", "http://www.golang-book.com/public/pdf/gobook.pdf")

	// start download
	fmt.Printf("Downloading %v...\n", req.URL())
	resp := client.Do(req)
	fmt.Printf("  %v\n", resp.HTTPResponse.Status)

	// start UI loop
	t := time.NewTicker(500 * time.Millisecond)
	defer t.Stop()

Loop:
	for {
		select {
		case <-t.C:
			fmt.Printf("  transferred %v / %v bytes (%.2f%%)\n",
				resp.BytesComplete(),
				resp.Size,
				100*resp.Progress())

		case <-resp.Done:
			// download is complete
			break Loop
		}
	}

	// check for errors
	if err := resp.Err(); err != nil {
		fmt.Fprintf(os.Stderr, "Download failed: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("Download saved to ./%v \n", resp.Filename)
}

  4. Use Hooks to extend request lifecycle

    master

    A Hook is a callback function of type func(*Response) error that allows you to execute custom logic at specific stages of a request's lifecycle. If a hook returns a non-nil error, the request is canceled and that error is returned on the Response object.

    Important: Hooks are called synchronously. Do not perform blocking operations inside a hook that wait for the download to complete (e.g., calling Response.Wait), as this will cause a deadlock. To cancel a download from within a hook, simply return an error.

    Available hook points on the Request struct:

    • BeforeCopy: Called immediately before the download starts.
    • AfterCopy: Called immediately after a successful download, but before checksum validation and closure.

  5. Use the Response object to monitor and manage downloads

    master

    The Response object represents the state of a download request. It provides methods to check progress, calculate ETA, and manage the lifecycle of the transfer (waiting, canceling, or retrieving the data).

    Key Lifecycle Methods:

    • Wait(): Blocks until the download is finished.
    • Err(): Blocks until the download is finished and returns any error encountered. Use this to check for success or failure.
    • Cancel(): Cancels the ongoing transfer. It blocks until the transfer is closed and typically returns context.Canceled.
    • IsComplete(): Returns true if the transfer has finished (successfully or with an error).

    Progress Tracking:

    • Progress(): Returns a float64 ratio (0.0 to 1.0) of bytes downloaded. Multiply by 100 for percentage.
    • BytesPerSecond(): Returns the current transfer speed using a moving average.
    • ETA(): Returns the estimated time of completion based on current speed.
    • Size(): Returns the total expected size in bytes. Returns -1 if the size is unknown and the transfer is incomplete.

  6. Perform a simple download with grab.Get()

    master

    For quick, one-off downloads, use the grab.Get(destination, url) function. This function returns a response object containing the filename and an error if the request fails. It automatically handles filename guessing from the URL or content headers.

    resp, err := grab.Get(".", "http://www.golang-book.com/public/pdf/gobook.pdf")
if err != nil {
	log.Fatal(err)
}

fmt.Println("Download saved to", resp.Filename)

  7. Configure Client settings

    master

    You can customize the Client behavior using the following fields:

    FieldTypeDescription
    HTTPClientHTTPClientAn interface for performing HTTP requests. Allows providing a custom http.Client for custom proxy, timeout, or cookie settings.
    UserAgentstringThe User-Agent string set in headers for all requests. Can be overridden per request.
    BufferSizeintThe size in bytes used for transferring files. Larger buffers increase throughput but use more memory. Default: 32KB. Can be overridden per Request.

  8. Manage Request context with WithContext

    master

    A Request carries a context.Context which controls cancellation and timeouts.

    • Use r.Context() to retrieve the current context (defaults to context.Background() if not set).
    • Use r.WithContext(ctx) to create a shallow copy of the request with a new context. The provided ctx must be non-nil.

  9. Perform a single file download with Do()

    master

    The Do(req *Request) method sends a file transfer request and returns a *Response.

    Behavior:

    • It blocks while the transfer is initiated (performing initial state machine steps like statFileInfo or headRequest).
    • It returns as soon as the transfer has started in a background goroutine or if it failed early.
    • The actual file copying happens in a separate goroutine.

    Error Handling:

    • Errors caused by client policy (like CheckRedirect) or HTTP/IO errors are returned via Response.Err.
    • Response.Err is a blocking call that will wait until the transfer is completed (successfully or otherwise) before returning the error.

  10. Execute batch downloads with DoBatch()

    master

    Use DoBatch(workers int, requests ...*Request) to execute multiple requests concurrently using a specified number of workers.

    • Workers: If workers is less than 1, a separate worker is created for every request (all requests run concurrently).
    • Concurrency Control: Control is returned to the caller immediately after workers are initiated.
    • Output: It returns a read-only channel <-chan *Response. This channel is closed only after all requests have completed, successfully or otherwise.
    • Error Handling: Errors for individual transfers are accessed via the Err() method on the associated Response objects received from the channel.

  11. Initialize a Client with NewClient()

    master

    Use NewClient() to create a new file download Client with default configuration. The default configuration includes a UserAgent of "grab" and an http.Client configured to use proxies from the environment.

    Clients are safe for concurrent use by multiple goroutines.

  12. Handle server HTTP status code errors

    master
    When a server response returns a status code outside the 200-299 range (after following redirects), grab returns a StatusCodeError. This error type is an integer representing the HTTP status code. You can identify these errors using the IsStatusCodeError helper function to distinguish them from other error types like checksum or file existence errors.