Roach PHP Core

repository·main·Indexed 23 days ago

https://github.com/roach-php/core

A comprehensive web scraping toolkit for PHP inspired by Python's Scrapy. It provides a structured framework for building crawlers and scrapers, featuring the RoachPHP\Http\Client for making concurrent HTTP requests via Guzzle and shell commands like fetch and roach:run for inspecting pages and executing spiders.

Tokens
679
Snippets
1
Records
8
Agent score
78%

What's inside roach-php/core

  1. Execute concurrent requests with pool()

    main

    The pool() method allows you to execute a list of RoachPHP\Http\Request objects concurrently.

    Parameters:

    • array $requests: A list of Request objects to be processed.
    • ?callable $onFulfilled: A callback executed when a request succeeds. It receives a RoachPHP\Http\Response object.
    • ?callable $onRejected: A callback executed when a request fails. It receives a RoachPHP\Http\RequestException (or a Response object if the failure was a BadResponseException).

    Behavior:

    • If a request results in a BadResponseException (e.g., a 4xx or 5xx error with a response body), the client wraps the response in a RoachPHP\Http\Response object so it can be processed by middleware.
    • For all other errors, it throws a RoachPHP\Http\RequestException wrapping the original Guzzle exception.
  2. Use the RoachPHP Client to make requests

    main

    The RoachPHP\Http\Client class is the primary entrypoint for making HTTP requests during scraping. It implements ClientInterface and wraps a Guzzle client. You can instantiate it with a custom GuzzleHttp\Client instance if you need specific configuration, or let it default to a new Guzzle client.

    To handle multiple requests efficiently, use the pool() method, which executes requests concurrently using a Guzzle Pool.

  3. Use the fetch command to retrieve a URL

    main

    The fetch command allows you to perform a GET request to a specified URL within the Roach shell environment. When executed, it populates the shell's scope with two new variables: $response (a RoachPHP\Http\Response object) and $html (the raw HTML string of the response). This is useful for quickly inspecting a page's structure before writing scraping logic.

    Usage: fetch <url>

  4. Run a spider using the roach:run command

    main
    Use the roach:run command to start a spider execution for a specific spider class. You must provide the spider class name as a required argument. The command uses the configured NamespaceResolverInterface to resolve the provided spider name into a fully qualified class name. If the spider class cannot be resolved, it will throw an InvalidSpiderException and display an error message.