DOCSEARCHbyAlgolia
Submit Docs

Scraperr Documentation

repository·master·Indexed 26 days ago

https://github.com/jaypyles/scraperr

A self-hosted web scraping solution for extracting data using XPath without writing code. Features include job queue management, domain spidering, custom JSON headers, media downloading, and data export to Markdown and CSV. The system includes a FastAPI backend with AI-related endpoints and a React-based web application.

Tokens
3.6K
Snippets
9
Records
32
Agent score
89%

What's inside Scraperr

  1. Overview of Scraperr features

    master

    Scraperr is a self-hosted web scraping solution designed to scrape websites without writing code. Key capabilities include:

    • XPath-Based Extraction: Target specific page elements using XPath.
    • Queue Management: Submit and manage multiple scraping jobs.
    • Domain Spidering: Scrape all pages within a specific domain.
    • Custom Headers: Include JSON headers in scraping requests.
    • Media Downloads: Automatically download images, videos, and other media.
    • Results Visualization: View scraped data in structured tables.
    • Data Export: Export results to Markdown and CSV formats.
    • Notification Channels: Receive completion notifications via various channels.

  5. Run the application using start.sh

    master

    The start.sh script is the entrypoint for launching the application. It performs database migrations using Alembic and starts the backend job worker.

    By default, the script enables recordings. You can control this behavior using the RECORDINGS_ENABLED environment variable.

    • If RECORDINGS_ENABLED is true (default): The script starts an Xvfb virtual framebuffer and an x11vnc server on port 5900 before launching the worker. This is used for visual recording/scraping tasks.
    • If RECORDINGS_ENABLED is false: The script launches the worker directly without the virtual display and VNC server.

  6. Deploy Scraperr using Docker Compose

    master

    You can deploy the Scraperr application and its API using Docker Compose. The setup consists of two main services: scraperr (the web application) and scraperr_api (the backend API).

    Ensure you have an OPENAI_KEY set in your environment before running, as it is required by the API service.

    services:
  scraperr:
    image: jpyles0524/scraperr:latest
    container_name: scraperr
    command: ["npm", "run", "start"]
    environment:
      - NEXT_PUBLIC_API_URL=http://scraperr_api:8000
      - SERVER_URL=http://scraperr_api:8000
    ports:
      - 80:3000
    networks:
      - web
  scraperr_api:
    init: True
    image: jpyles0524/scraperr_api:latest
    environment:
      - LOG_LEVEL=INFO
      - OPENAI_KEY=${OPENAI_KEY}
    container_name: scraperr_api
    ports:
      - 8000:8000
    volumes:
      - "$PWD/data:/project/app/data"
      - "$PWD/media:/project/app/media"
    networks:
      - web

networks:
  web:

  7. Configure Scraperr Web Application environment variables

    master

    The scraperr service (web application) uses the following environment variables for connectivity:

    • NEXT_PUBLIC_API_URL: The URL of the API used by the client-side application (e.g., http://scraperr_api:8000).
    • SERVER_URL: The URL of the API used by the server-side application (e.g., http://scraperr_api:8000).

    By default, the web application maps host port 80 to container port 3000.

  8. Configure Scraperr API environment variables and volumes

    master

    The scraperr_api service requires the following configuration:

    Environment Variables:

    • LOG_LEVEL: Sets the logging verbosity (e.g., INFO).
    • OPENAI_KEY: Your OpenAI API key, passed via the host environment ${OPENAI_KEY}.

    Volumes: Data and media are persisted in the following local directories relative to the project root:

    • $PWD/data maps to /project/app/data
    • $PWD/media maps to /project/app/media

    Ports:

    • The API is exposed on host port 8000.

  9. Configure application logging via LOG_LEVEL

    master
    The backend application uses the LOG_LEVEL environment variable to determine the logging verbosity. The value is processed by get_log_level before being applied to the standard Python logging configuration. The log format is: %(levelname)s: %(asctime)s - [%(name)s] - %(message)s.

  10. Configure recordings via RECORDINGS_ENABLED

    master

    The RECORDINGS_ENABLED environment variable determines whether the application initializes a virtual display environment (Xvfb and x11vnc) for the job worker.

    VariableDefaultDescription
    RECORDINGS_ENABLEDtrueSet to false to disable Xvfb and VNC server startup.

    When enabled, the VNC server is accessible on port 5900.

  11. Configure CsvTableProps

    master

    The CsvTable component accepts the following props:

    • csv: An object containing the data to be rendered.
      • rows: An array of CsvRow objects. Each object is a dictionary where keys are column values and values are strings.
      • headers: An array of strings representing the column names.
    • className (optional): A string used to apply custom CSS classes to the component container.