Roach PHP Core
repository·main·Indexed 23 days ago
https://github.com/roach-php/coreA comprehensive web scraping toolkit for PHP inspired by Python's Scrapy. It provides a structured framework for building crawlers and scrapers, featuring the RoachPHP\Http\Client for making concurrent HTTP requests via Guzzle and shell commands like fetch and roach:run for inspecting pages and executing spiders.
What's inside roach-php/core
- Roach is a complete web scraping toolkit for PHP. It is heavily inspired by the Scrapy package for Python, providing a structured framework for building web crawlers and scrapers.
Install Roach PHP core via Composer
mainTo use Roach in your PHP project, install the core package using Composer:
composer require roach-php/coreExecute concurrent requests with pool()
mainThe
pool()method allows you to execute a list ofRoachPHP\Http\Requestobjects concurrently.Parameters:
array $requests: A list ofRequestobjects to be processed.?callable $onFulfilled: A callback executed when a request succeeds. It receives aRoachPHP\Http\Responseobject.?callable $onRejected: A callback executed when a request fails. It receives aRoachPHP\Http\RequestException(or aResponseobject if the failure was aBadResponseException).
Behavior:
- If a request results in a
BadResponseException(e.g., a 4xx or 5xx error with a response body), the client wraps the response in aRoachPHP\Http\Responseobject so it can be processed by middleware. - For all other errors, it throws a
RoachPHP\Http\RequestExceptionwrapping the original Guzzle exception.
Use the RoachPHP Client to make requests
mainThe
RoachPHP\Http\Clientclass is the primary entrypoint for making HTTP requests during scraping. It implementsClientInterfaceand wraps a Guzzle client. You can instantiate it with a customGuzzleHttp\Clientinstance if you need specific configuration, or let it default to a new Guzzle client.To handle multiple requests efficiently, use the
pool()method, which executes requests concurrently using a Guzzle Pool.Reference: fetch command arguments and variables
mainThefetchcommand uses the following input argument and provides the following scope variables upon successful execution.Arguments for the roach:run command
mainThe
roach:runcommand accepts the following argument:Argument Requirement Description spiderRequired The spider class to execute (resolved via the namespace resolver) Use the fetch command to retrieve a URL
mainThe
fetchcommand allows you to perform a GET request to a specified URL within the Roach shell environment. When executed, it populates the shell's scope with two new variables:$response(aRoachPHP\Http\Responseobject) and$html(the raw HTML string of the response). This is useful for quickly inspecting a page's structure before writing scraping logic.Usage:
fetch <url>Run a spider using the roach:run command
mainUse theroach:runcommand to start a spider execution for a specific spider class. You must provide the spider class name as a required argument. The command uses the configuredNamespaceResolverInterfaceto resolve the provided spider name into a fully qualified class name. If the spider class cannot be resolved, it will throw anInvalidSpiderExceptionand display an error message.