DOCSEARCHbyAlgolia
Submit Docs

spatie/crawler

repository·main·Indexed 25 days ago

https://github.com/spatie/crawler

A PHP package for crawling websites featuring concurrent requests via Guzzle and JavaScript rendering support through Browsershot, Puppeteer, or Cloudflare. It includes tools for link extraction, sitemap parsing, custom request handlers, and the ability to resume long-running crawls using serialized queues.

Tokens
22.9K
Snippets
71
Records
119
Agent score
82%

What's inside spatie/crawler

  1. Quickstart: Crawl a website and handle responses

    main

    Use Crawler::create() to initialize a crawl starting from a specific URL. You can use the onCrawled() method to register a callback that executes whenever a page is successfully crawled. This callback receives the URL as a string and a CrawlResponse object, which you can use to inspect the response (e.g., checking the HTTP status code). Call start() to begin the process.

    use Spatie\
Crawler\\Crawler;
use Spatie\\Crawler\\CrawlResponse;

Crawler::create('https://example.com')
    ->onCrawled(function (string $url, CrawlResponse $response) {
        echo "{$url}: {$response->status()}\n";
    })
    ->start();

  2. Use new v9 features: Closures, Throttling, and Extraction

    main

    v9 introduces several new ways to interact with the crawler:

    Closure Callbacks: Use onCrawled, onFailed, and onFinished for quick implementations without creating observer classes.

    Throttling: Control request frequency using throttle() with FixedDelayThrottle or AdaptiveThrottle.

    Resource Extraction: Use alsoExtract() or extractAll() to discover images, scripts, and stylesheets.

    Retry Logic: Use retry(times: int, delayInMs: int) to automatically retry failed requests (5xx or connection errors).

    // Example of new features
Crawler::create('https://example.com')
    ->onCrawled(function (string $url, CrawlResponse $response, CrawlProgress $progress) {
        echo $url . ': ' . $response->status();
    })
    ->throttle(new AdaptiveThrottle(minDelayMs: 50, maxDelayMs: 5000))
    ->alsoExtract(ResourceType::Image, ResourceType::Script)
    ->retry(times: 3, delayInMs: 500)
    ->start();

  4. Implement graceful shutdown for long-lived crawler processes

    main

    When running the crawler as a long-lived CLI process, you can stop it cleanly using Ctrl+C (SIGINT) or SIGTERM instead of killing it mid-request.

    If the pcntl PHP extension is available, the crawler automatically registers signal handlers. Upon receiving a signal, the crawler:

    1. Stops yielding new requests.
    2. Allows in-flight requests to complete normally.
    3. Calls the finishedCrawling() method on your observers with FinishReason::Interrupted.
    4. Causes the start() method to return FinishReason::Interrupted.

    This behavior ensures the crawl queue remains in a consistent state, which is particularly useful when using the 'crawling across requests' feature to allow for resuming later.

    use Spatie\
Crawler\\Crawler;
use Spatie\\Crawler\\Enums\\FinishReason;

$reason = Crawler::create('https://example.com')
    ->start();

if ($reason === FinishReason::Interrupted) {
    echo "Crawl was interrupted by a signal\n";
}

  5. Configure Browsershot for JavaScript rendering

    main

    To customize how JavaScript is rendered using Browsershot, create a configured Browsershot instance and pass it to a new BrowsershotRenderer. This allows you to use Browsershot features like noSandbox() or waitUntilNetworkIdle() during the crawling process.

    use Spatie\Crawler\Crawler;
use Spatie\Crawler\JavaScriptRenderers\BrowsershotRenderer;
use Spatie\Browsershot\Browsershot;

$browsershot = (new Browsershot())
    ->noSandbox()
    ->waitUntilNetworkIdle();

Crawler::create('https://example.com')
    ->executeJavaScript(new BrowsershotRenderer($browsershot))
    ->start();

  6. Migrate from v3 to v4

    main
    When upgrading from version 3 to version 4, CrawlObserver and CrawlProfile have changed from interfaces to abstract classes. You must convert your existing observers and profiles to extend these new abstract classes. Additionally, the crawled method now receives every successfully crawled URI, and crawlFailed receives every failed URI.

  7. Migrate from v2 to v3

    main

    When upgrading from version 2 to version 3:

    • PHP Requirement: PHP 7.1 is now the minimum required version.
    • URI Implementation: The custom \Spatie\Crawler\Url object has been replaced by Psr\Http\Message\UriInterface. The concrete implementation used is \GuzzleHttp\Psr7\Uri.
    • Observer/Profile Updates: Custom Profiles and Observers must be updated to use the correct arguments and return types compatible with Psr\Http\Message\UriInterface.

  8. Upgrade from v8 to v9: Breaking Changes

    main

    v9 is a major rewrite. Key breaking changes include:

    • Redirects: Now followed by default with tracking enabled. To disable, pass RequestOptions::ALLOW_REDIRECTS => false in client options.
    • Client Options: Custom options are now merged with defaults instead of replacing them. Defaults include 10s timeouts, enabled cookies, and enabled redirects.
    • Non-parseable Responses: Responses with MIME types not in allowedMimeTypes now trigger crawled() on observers with an empty body. Ensure your observer logic handles empty bodies.
    • Entry Point: Use Crawler::create($url)->start() instead of startCrawling($url).
    • Default Scheme: Defaults to https instead of http. Use ->defaultScheme('http') to restore old behavior.
    • JavaScript Rendering: setBrowsershot() is removed. Use executeJavaScript(new BrowsershotRenderer($browsershot)) instead.
    • Removed Classes: Spatie\ Crawler\Url and Spatie\Crawler\ResponseWithCachedBody have been removed.

  9. Extract all resource types

    main

    To instruct the crawler to extract every supported resource type (links, images, scripts, stylesheets, and Open Graph images) simultaneously, use the extractAll() method.

    Crawler::create('https://example.com')
    ->extractAll()
    ->onCrawled(function (string $url, CrawlResponse $response) {
        // $response->resourceType() tells you what kind of resource this is
    })
    ->start();

  11. Implement a custom failed request handler

    main

    To customize how failed requests and errors are processed, create a class that extends Spatie\Crawler\Handlers\CrawlRequestFailed. The class must implement the __invoke method with the signature public function __invoke(Exception $exception, mixed $index): void.

    After defining your class, register it with the crawler using the failedHandler() method.

    Note: The class you pass must extend the base handler class, otherwise an InvalidCrawlRequestHandler exception will be thrown.

    use Exception;
use Spatie
	Crawler
	Handlers
	CrawlRequestFailed;

class MyFailedHandler extends CrawlRequestFailed
{
    public function __invoke(Exception $exception, mixed $index): void
    {
        // your custom logic here

        parent::__invoke($exception, $index);
    }
}

use Spatie
	Crawler
	Crawler;

Crawler::create('https://example.com')
    ->failedHandler(MyFailedHandler::class)
    ->start();

  12. Create custom crawl profiles

    main

    For reusable filtering logic, implement the Spatie\Crawler\CrawlProfiles\CrawlProfile interface. Your class must implement the shouldCrawl(string $url): bool method. Pass the instance to the crawler using crawlProfile().

    use Spatie\Crawler\CrawlProfiles\CrawlProfile;

class MyCustomProfile implements CrawlProfile
{
    public function shouldCrawl(string $url): bool
    {
        return parse_url($url, PHP_URL_HOST) === 'example.com'
            && !str_contains($url, '/private');
    }
}

// Usage
use Spatie\Crawler\Crawler;

Crawler::create('https://example.com')
    ->crawlProfile(new MyCustomProfile())
    ->start();