DOCSEARCHbyAlgolia
Submit Docs

Roach PHP

website·Indexed 33 days ago

https://roach-php.dev

A framework-agnostic PHP web scraping and crawling library. It features a flexible architecture with support for spiders, item pipelines, and middleware for both downloaders and spiders. Roach PHP provides official adapters for Laravel and Symfony, an interactive shell (REPL), and a robust system for processing responses and managing items.

Tokens
11.8K
Snippets
69
Records
94
Agent score
67%

What's inside Roach PHP

  1. Overview of Roach PHP web scraping toolkit

    Roach PHP is a complete web scraping toolkit for PHP, heavily inspired by the Python package Scrapy. It provides a full ecosystem for defining spiders to crawl and scrape web documents, as well as a processing pipeline to clean, persist, and process the extracted data.

  2. Overview of Downloader Middleware

    Downloader middleware sits between the Roach engine and the Downloader (the component responsible for sending HTTP requests and receiving responses). Every outgoing request and incoming response passes through this middleware. Middleware classes should implement DownloaderMiddlewareInterface, which combines RequestMiddlewareInterface and ResponseMiddlewareInterface. Roach uses an adapter to ensure that middleware implementing only one of these interfaces still satisfies the full DownloaderMiddlewareInterface.

  3. Overview of Roach PHP web scraping toolkit

    Roach PHP is a complete web scraping toolkit for PHP, heavily inspired by the Python Scrapy package. It allows developers to define spiders to crawl and scrape web documents and provides a full pipeline to clean, persist, and process the extracted data.

  4. Overview of RoachPHP Spider Middleware

    Spider middleware sits between the Roach engine and the spiders. It allows developers to intercept and process three distinct types of data: incoming responses from the downloader, outgoing requests emitted by the spider, and items emitted by the spider. While the SpiderMiddlewareInterface combines all three concerns, Roach allows you to implement only the specific interfaces relevant to your middleware's purpose; the framework automatically wraps these in an adapter to provide missing methods.

  5. Integrate Roach with a web framework

    Roach provides first-party packages to reduce configuration and boilerplate when used within a web framework. Integration is available for Laravel, with Symfony support currently in progress.

  6. Manage extracted data using Items in RoachPHP

    Items in RoachPHP are abstractions around PHP arrays used to represent data extracted from web documents. While developers typically create items using a spider's item() method, they are primarily interacted with via the ItemInterface when writing ItemProcessors. An Item allows for structured data retrieval, assignment, and existence checks.

  7. Understand the difference between Scraping and Crawling in Roach

    In the context of the Roach framework, 'scraping' and 'crawling' refer to two distinct processes:

    1. Scraping: The process of extracting specific data (e.g., author, date, title) from a web document. Scrapers are typically bespoke to the specific document structure they process. A scraper is only responsible for extraction, not for retrieving the document itself.
    2. Crawling: The process of iteratively finding and following web links starting from one or more URLs. Crawlers focus on the structure of the web to discover new pages, though they perform a minimal amount of scraping to extract the URLs needed for navigation.

  8. Create Request Middleware to modify outgoing requests

    To process outgoing requests before they are sent, implement the RequestMiddlewareInterface. The handleRequest method receives a Request object and must return a Request object. Downloader request middleware runs after spider request middleware, except for initial requests which bypass spider middleware entirely.
    interface RequestMiddlewareInterface extends ConfigurableInterface
{
    public function handleRequest(Request $request): Request;
}

  9. Implement a custom Item Processor in RoachPHP

    An item processor is a PHP class that implements the ItemProcessorInterface. Processors are executed sequentially after a spider yields an item. They should be focused on a single task (e.g., saving to a database, validating data, or adding metadata) to remain simple and testable.
    use RoachPHP\ItemPipeline\ItemInterface;
use RoachPHP\ItemPipeline\Processors\ItemProcessorInterface;

class MyProcessor implements ItemProcessorInterface
{
    public function processItem(ItemInterface $item): ItemInterface
    {
        // Process the item here
        return $item;
    }
}

  10. Dispatch new requests and extract items using ParseResult

    The parse callback must return a Generator that yields ParseResult objects. A ParseResult can represent either a new request to be crawled or an item to be sent to the processing pipeline. Use $this->request() to schedule new URLs and $this->item() to extract data.
    use RoachPHP\Http\Response;
use RoachPHP\Spider\BasicSpider;

class MySpider extends BasicSpider
{
    public function parse(Response $response): \Generator
    {
        // 1. Extract data and send to item pipeline
        $headline = $response->filter('h1')->text();
        yield $this->item(['headline' => $headline]);

        // 2. Dispatch a request to a detail page
        $detailPageLink = $response->filter('a#details')->link();
        yield $this->request(
            $detailPageLink->getUri(),
            'parseDetailPage'
        );
    }

    public function parseDetailPage(Response $response): \Generator
    {
        // Process detail page...
    }
}

  11. Install Roach as a standalone package

    To use Roach as a standalone package without a specific framework adapter, install the core package using Composer.
    composer require roach-php/core

  12. Implement Response Middleware to process incoming responses

    Response middleware handles responses returned by the downloader before they are passed to the spider's parse callback. To create one, implement the ResponseMiddlewareInterface. You can drop a response by calling the drop() method on the Response object, which prevents further middleware from running and stops the response from reaching the spider. Dropping a response triggers a ResponseDropped event.
    <?php

use RoachPHP\Http\Response;
use RoachPHP\Spider\Middleware\ResponseMiddlewareInterface;
use RoachPHP\Support\Configurable;

class MyResponseMiddleware implements ResponseMiddlewareInterface
{
    use Configurable;
    
    public function handleResponse(Response $response): Response
    {
        // To drop the response and prevent it from reaching the spider:
        return $response->drop('Computer says no.');
    }
}