DOCSEARCHbyAlgolia
Submit Docs

ScrapeGraphAI

repository·main·Indexed 12 days ago

https://github.com/scrapegraphai/scrapegraph-ai

A Python web scraping library based on LangChain that uses LLMs and graph logic to create automated scraping pipelines. Version 2.1.6 supports extracting structured data from websites and documents (HTML, JSON, XML, PDF, DOCX) using natural language prompts. It includes specialized graphs such as SmartScraperGraph, DepthSearchGraph, OmniScraperGraph, and DocumentScraperGraph, as well as utilities for converting websites to Markdown.

Tokens
16.2K
Snippets
62
Records
100
Agent score
95%

What's inside ScrapeGraphAI

  1. Available Scrapegraph-ai Example Use Cases

    main

    The repository provides several specialized graph implementations for different scraping and automation tasks:

    • smart_scraper/: Advanced web scraping with intelligent content extraction.
    • search_graph/: Web search and data retrieval.
    • script_generator_graph/: Automated script generation.
    • depth_search_graph/: Deep web crawling and content exploration.
    • csv_scraper_graph/: Scraping and processing data into CSV format.
    • xml_scraper_graph/: XML data extraction and processing.
    • speech_graph/: Speech processing and analysis.
    • omni_scraper_graph/: Universal web scraping for multiple data types.
    • omni_search_graph/: Comprehensive search across multiple sources.
    • document_scraper_graph/: Document parsing and data extraction.
    • custom_graph/: Custom graph implementation examples.
    • code_generator_graph/: Code generation utilities.
    • json_scraper_graph/: JSON data extraction and processing.

  2. Overview of available ScrapeGraphAI pipelines

    main

    ScrapeGraphAI provides several specialized graph pipelines for different scraping needs:

    Pipeline NameDescription
    SmartScraperGraphSingle-Page-Scraper requiring only a user prompt and an input source.
    SearchGraphMulti-Page-Scraper that extracts information from the top search engine results.
    SpeechGraphSingle-Page-Scraper that extracts info and generates an audio file.
    ScriptCreatorGraphSingle-Page-Scraper that extracts info and generates a Python script.
    SmartScraperMultiGraphMulti-Page-Scraper that extracts info from multiple pages using a prompt and a list of sources.
    ScriptCreatorMultiGraphMulti-Page-Scraper that generates a Python script to extract info from multiple pages/sources.

    Note: For every graph, there is a Multi-Version available which enables parallel LLM calls.

  3. Overview of ScrapeGraphAI scraping processes

    main

    ScrapeGraphAI provides several specialized graph processes depending on your scraping needs:

    Process NameDescription
    SmartScraperGraphSingle-page scraper requiring a user prompt and a source input.
    SearchGraphMulti-page scraper that extracts information from the top n search engine results.
    SpeechGraphSingle-page scraper that extracts information and generates an audio file.
    ScriptCreatorGraphSingle-page scraper that extracts information and generates a Python script.
    SmartScraperMultiGraphMulti-page scraper that extracts information from multiple pages given a single prompt and a list of sources.
    ScriptCreatorMultiGraphMulti-page scraper that extracts information from multiple pages/sources and generates a Python script.

    Note: Most processes have 'multi' versions that allow for parallel LLM calls.

  4. Overview of ScrapeGraphAI pipelines

    main

    ScrapeGraphAI provides several specialized pipeline classes (graphs) for different scraping tasks:

    Pipeline NameDescription
    SmartScraperGraphSingle-page scraper requiring only a user prompt and input source.
    SearchGraphMulti-page scraper that extracts information from the top $n$ search engine results.
    SpeechGraphSingle-page scraper that extracts information and generates an audio file.
    ScriptCreatorGraphSingle-page scraper that generates a Python script from a website.
    SmartScraperMultiGraphMulti-page scraper that extracts information from a list of sources using a single prompt.
    ScriptCreatorMultiGraphMulti-page scraper that generates Python scripts from multiple pages and sources.

    Note: Most graphs have multi-version variants that allow for parallel LLM calls.

  5. Overview of available ScrapeGraph pipelines

    main

    ScrapeGraphAI provides several specialized graph classes for different scraping tasks:

    NameDescription
    SmartScraperGraphSingle-page scraper requiring a prompt and an input source.
    SearchGraphMulti-page scraper that extracts info from the first $n$ search engine results.
    SpeechGraphSingle-page scraper that extracts info and generates an audio file.
    ScriptCreatorGraphSingle-page scraper that extracts info and generates a Python script.
    SmartScraperMultiGraphMulti-page scraper that extracts info from multiple sources using a single prompt.
    ScriptCreatorMultiGraphMulti-page scraper that generates a Python script for multiple sources.

    Note: Most graphs have a "multi" version that allows parallel LLM calls for improved performance.

  6. Use Mock HTTP Server for consistent testing

    main

    The project includes a mock HTTP server located in tests/fixtures/mock_server/ to allow testing without external network dependencies. It supports simulating various web conditions including slow responses, errors, and rate limiting.

    Available Endpoints:

    • / - Home page
    • /products - Product listings (prices/stock)
    • /projects - Project listings
    • /api/data.json, /api/data.xml, /api/data.csv - Data endpoints
    • /slow - Simulates a 2-second delay
    • /error/404, /error/500 - Error page simulation
    • /rate-limited - Simulates rate limiting (max 5 requests)
    • /dynamic - Dynamically generated content
    • /pagination?page=N - Paginated content

  7. Understand Scrapegraph AI scraping modes

    main

    Scrapegraph AI provides two primary scraping patterns demonstrated in the examples:

    1. Smart Scraper

    Used for extracting structured data from a single specific website. You provide a natural language prompt, and the AI transforms the webpage content into a structured JSON format.

    2. Search Scraper

    Used for complex queries that require searching the internet. It performs the following workflow:

    • Searches the internet for relevant information.
    • Extracts structured data from multiple different sources.
    • Merges and analyzes information from those sources to provide a comprehensive answer.

  8. Understand the Semantic Commit Format

    main

    This repository follows the Conventional Commits specification to automate semantic versioning via @semantic-release/commit-analyzer. Commit messages must follow a specific structure to trigger appropriate version bumps in the release cycle.

    Commit Structure

    <type>(<scope>): <description>

    Valid Commit Types

    • feat: New features (triggers a Minor version bump).
    • fix: Bug fixes (triggers a Patch version bump).
    • docs: Documentation changes (no version bump, but appears in the changelog).
    • chore: Maintenance tasks.
    • refactor: Code refactoring.
    • perf: Performance improvements.
    • test: Test changes.

    Note on Documentation: If documentation introduces or exposes a new user-facing feature (e.g., documenting a previously undocumented configuration option), it should be categorized as a feat rather than docs to ensure the feature is properly versioned.

  9. Compare Open Source vs Managed API

    main

    ScrapeGraphAI offers two distinct ways to use the technology:

    Open Source (scrapegraphai)

    • Best for: Full control, self-hosting, using local LLMs (Ollama), and fine-grained cost tuning.
    • Responsibility: You manage infrastructure, browsers (Playwright), proxies, anti-bot measures, and scaling.
    • LLM: Bring your own (OpenAI, Groq, Gemini, Azure, or local via Ollama).

    Managed API (scrapegraph-py / scrapegraph-js SDKs)

    • Best for: Zero infrastructure, production-ready speed, and managed scaling.
    • Responsibility: Fully managed by ScrapeGraphAI cloud. Includes stealth browser rendering, proxies, and anti-bot protection.
    • Capabilities: Includes advanced features like Crawl and scheduled Monitor jobs.
    • Auth: Uses SGAI_API_KEY.

  11. Best practices for managing timeouts

    main

    To ensure robust scraping pipelines, follow these best practices:

    1. Tailor timeouts to content type:
      • Fast APIs: 5-10 seconds
      • Regular websites: 15-30 seconds
      • Large PDFs or slow sites: 60+ seconds
    2. Handle TimeoutError: Always wrap node.execute() or graph.run() in a try-except block to catch TimeoutError and handle it gracefully (e.g., logging or retrying).
    3. Use specialized nodes: Instead of one global timeout, use separate FetchNode instances with different node_config settings for different tasks (e.g., one for fast web pages and one for slow PDFs).
    4. Monitor failures: Log timeout occurrences to identify problematic target URLs or sources.