DOCSEARCHbyAlgolia
Submit Docs

Reworkd Documentation

website·Indexed 37 days ago

https://docs.reworkd.ai/

Documentation for Reworkd, a platform to extract web data at scale. Includes guides on scraping SDKs (Harambe), schema definitions, job scheduling, deduplication, and API references for managing cookies, local storage, and data exports.

Tokens
4.3K
Snippets
19
Records
42
Agent score
63%

What's inside Reworkd

  1. Overview of the Harambe Scraping SDK

    Harambe is a custom web scraping SDK used by Reworkd for code generation. It provides built-in utilities for saving and validating data against schemas, enqueuing URLs with automatic formatting, de-duplicating data, and handling common scraping challenges such as pagination and file downloads.

  2. Overview of Reworkd web data extraction

    Reworkd is a platform that leverages Large Language Models (LLMs) to parse, understand, and interact with web pages. It is designed for extracting web data at scale, enabling users to collect millions of rows of data for building data-constrained products, fine-tuning domain-specific language models, and enriching data pipelines.

  3. Understand how Reworkd handles data deduplication

    Reworkd automatically deduplicates data when scrapers re-run by using a unique key (or composite key) based on record fields. The system determines the action based on the key's existence in the database:

    • New data: If the key is not found, Reworkd inserts the data and marks it as a CREATE change.
    • Duplicate data: If the exact same data is saved again, the insertion is skipped to prevent duplicates.
    • Updated data: If the key exists but the data has changed, Reworkd updates the existing record and marks it as an UPDATE change.

  4. Use Templates to reuse scrapers across identical websites

    Templates allow you to save and reuse pre-built scraper code for multiple websites that share the same underlying structure or web provider. This prevents the need to write new scrapers for every site, reducing time and LLM token consumption. A key advantage is centralized maintenance: if the website structure changes, updating the template once automatically fixes the scraper for all associated sites.

    Note: Templates are not available for customers on the hobby plan.

  5. Define data extraction schemas in Reworkd

    Schemas in Reworkd define the exact data format expected from websites within a specific group. Every row of processed data undergoes a strict schema validation process to ensure consistency with the defined schema. If a field is missing from a page, it will be returned as null in the output, or as an empty array if the field is defined as an array.

  6. Organize scraping jobs using Groups

    In Reworkd, a Group is the primary organizational unit used to collect source URLs and scraping jobs that share a common schema and scraping frequency. For example, if you need to scrape data from multiple different online bookstores, you would create a single 'Bookstore' group and add all relevant bookstore URLs to it.

  7. Understand the role of a Run in scraping jobs

    A Run represents a single execution of a scraping job. Runs are used to track the status and results of individual scraping attempts, allowing for consistent data collection and processing. If a run fails, it can be retried to improve data accuracy. Each run typically generates a list of outputs containing the extracted data or links for further processing.

  8. Understand the Job and Stage hierarchy in scraping

    In the scraping workflow, a Job represents a distinct source URL within a scraping group. Jobs are organized hierarchically: the initial URL is the source job, and any subsequent URLs enqueued by that job are child jobs.

    Jobs are processed through Stages, which define the role of the page being scraped. A typical e-commerce scraping flow follows these stages:

    1. Category Stage: Scrapes a high-level category page and enqueues the associated listing pages.
    2. Listing Stage: Scrapes a list of products (e.g., a specific category) and enqueues the individual product detail pages.
    3. Detail Stage: The final stage where the specific product information is extracted and saved.

  9. Export data using the Reworkd Outputs API

    The primary method for ingesting data from Reworkd is via the API endpoints. To export data from your groups, use the Outputs API. An API key is required to authenticate these requests.

  10. Optimize schemas for higher extraction accuracy

    To improve scraping reliability and reduce LLM failures, follow these schema design guidelines:

    1. Keep it simple: Minimize the number of fields to reduce errors.
    2. Avoid speculative fields: Only include fields currently needed; do not add 'nice-to-have' fields that rarely appear on target sites.
    3. Capture raw data: Extract fields exactly as they appear on the page. Use a post-processing layer in your own application to convert arbitrary strings into domain-specific enums.
    4. Avoid derived fields: Do not create fields that are generated from other fields within the schema; handle this in your application code.
    5. Remove ambiguity: Use clear field names, provide detailed descriptions, and include example values. Explain industry jargon and potential aliases for the field.
    6. Prefer advanced fields: Use specialized field types (like URL or Phone Number) to abstract transformation complexity away from the LLM.

  11. File storage retention policy

    Files downloaded by Reworkd are stored in S3 buckets with a guaranteed retention period of 90 days. For requirements exceeding 90 days, contact Reworkd support.

  12. Configure automatic file downloads in Reworkd

    To enable Reworkd to automatically download files on your behalf, you must configure a specific field in your schema and associate it with a job. Note that downloads happen asynchronously, so files may not appear in exports immediately after a job runs.
    1. Create a field in your schema with the URL type.
    2. In the field settings, set Download file from URL to True.
    3. Create a job that saves the download URL to this field.