Crawl4AI

Crawl4AI provides a comprehensive suite of features for LLM-friendly web scraping:

Markdown Generation

• Clean Markdown: Structured formatting.
• Fit Markdown: Heuristic filtering to remove noise.
• Citations: Converts links into numbered reference lists.
• BM25 Algorithm: Filtering for core information extraction.

Structured Data Extraction

• LLM-Driven: Supports all major LLMs.
• Chunking: Topic-based, regex, or sentence-level strategies.
• Semantic Extraction: Uses Cosine Similarity to find relevant content.
• CSS/XPath: Fast schema-based extraction.

Browser & Crawling

• Managed Browsers: Full control over Chromium, Firefox, and WebKit.
• Stealth Mode: Mimics real users to avoid detection.
• Dynamic Content: Executes JS, handles lazy loading, and simulates scrolling for infinite pages.
• Media & IFrame: Extracts images, audio, video, and content from embedded iframes.

In Crawl4AI, storage_state is a mechanism used to preserve and reuse session data, such as cookies and localStorage, across multiple crawler runs. This allows you to start a crawl in an authenticated or pre-configured state (e.g., already logged in) without repeating the login flow every time.

storage_state can be provided in two formats:

1. A dictionary: Containing cookies and origins (which holds localStorage data).
2. A file path: A string pointing to a JSON file that holds the session information.

{
  "cookies": [
    {
      "name": "session",
      "value": "abcd1234",
      "domain": "example.com",
      "path": "/",
      "expires": 1675363572.037711,
      "httpOnly": false,
      "secure": false,
      "sameSite": "None"
    }
  ],
  "origins": [
    {
      "origin": "https://example.com",
      "localStorage": [
        { "name": "token", "value": "my_auth_token" },
        { "name": "refreshToken", "value": "my_refresh_token" }
      ]
    }
  ]
}

Crawl4AI supports state persistence to resume crawls if a process is interrupted. This is achieved using two parameters in your deep crawl strategy:

1. on_state_change: An async callback function that is triggered after every URL is processed. Use this to save the current state to a database or file.
2. resume_state: A dictionary containing a previously saved state. When passed to the strategy, the crawler will skip already-visited URLs and continue from the pending queue.

State Structure

The state dictionary is JSON-serializable and includes:

• strategy_type: "bfs", "dfs", or "best_first"
• visited: List of already crawled URLs
• pending: List of {"url": "...", "parent_url": "..."} objects
• depths: Mapping of url: depth
• pages_crawled: Integer counter

Manual State Export

If you have on_state_change enabled, you can manually retrieve the current state using strategy.export_state().

# Callback to save state
async def save_state_to_redis(state: dict):
    await redis.set("crawl_state", json.dumps(state))

# Strategy with recovery
strategy = BFSDeepCrawlStrategy(
    max_depth=3,
    on_state_change=save_state_to_redis,
)

# Resuming later
saved_state = json.loads(await redis.get("crawl_state"))
strategy = BFSDeepCrawlStrategy(
    max_depth=3,
    resume_state=saved_state,
    on_state_change=save_state_to_redis,
)

Crawl4AI provides a hook system via crawler.crawler_strategy.set_hook() to intercept and modify different stages of the crawling lifecycle. This is ideal for handling authentication, blocking resources (like images/ads), injecting custom headers, or performing final page manipulations before HTML retrieval.

Hook Lifecycle and Recommended Use Cases

Key Implementation Details

• Authentication: The recommended place for login flows is on_page_context_created. This ensures the context is configured before arun() navigates to the target URL.
• Concurrency: When using arun_many(), hooks are triggered in parallel for each URL. Ensure hook logic is async-safe.
• Session Management: To reuse a session across multiple arun() calls, pass session_id= in your CrawlerRunConfig.

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from playwright.async_api import Page, BrowserContext

async def main():
    browser_config = BrowserConfig(headless=True)
    crawler_run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
    crawler = AsyncWebCrawler(config=browser_config)

    # Define a hook
    async def on_page_context_created(page: Page, context: BrowserContext, **kwargs):
        # Example: Block images
        async def route_filter(route):
            if route.request.resource_type == "image":
                await route.abort()
            else:
                await route.continue_()
        await context.route("**", route_filter)
        return page

    # Attach the hook
    crawler.crawler_strategy.set_hook("on_page_context_created", on_page_context_created)

    await crawler.start()
    result = await crawler.arun("https://example.com", config=crawler_run_config)
    await crawler.close()

if __name__ == "__main__":
    asyncio.run(main())