DOCSEARCHbyAlgolia
Submit Docs

Reader Documentation

repository·main·Indexed 19 days ago

https://github.com/vakra-dev/reader

An open-source, production-grade web scraping engine for LLMs and AI agents. Reader provides high-fidelity scraping, crawling, and stealthed browser sessions with clean markdown output. It features anti-bot bypass, proxy management, and browser scaling, with built-in integrations for RAG frameworks like LangChain and LlamaIndex, and vector stores such as Pinecone and Qdrant.

Tokens
53.5K
Snippets
168
Records
228
Agent score
68%

What's inside @vakra-dev/reader

  1. Overview of Reader Example Categories

    main

    The examples/ directory is organized into several functional categories to demonstrate different capabilities of the Reader library:

    • basic/: Core scraping patterns including single URL scraping, concurrent batch scraping, large-scale scraping (1000+ URLs), browser pool configuration, proxy rotation, Cloudflare bypass, and website crawling.
    • ai-tools/: Integrations with AI frameworks and vector stores, such as OpenAI, Anthropic, Vercel AI SDK, LangChain, LlamaIndex, Pinecone, and Qdrant.
    • production/: Real-world server implementations, such as a REST API server using Express.
    • deployment/: Guides for deploying to various environments including Docker, AWS Lambda (container), and Vercel functions.

  2. Scraping and Crawling capabilities

    main

    The basic examples demonstrate several core scraping and crawling patterns:

    Scraping

    • basic-scrape.ts: Scrape a single URL and display markdown output.
    • batch-scrape.ts: Scrape multiple URLs concurrently with progress tracking.
    • all-formats.ts: Output content in all supported formats (markdown, html).

    Crawling

    • crawl-website.ts: Crawl a website to discover and optionally scrape pages.

  3. Choose between standard and premium proxy types

    main

    Reader supports different proxy tiers based on your needs for speed, cost, and detection avoidance:

    • Standard (type: "standard"): These are typically Datacenter proxies. They are fast and cheap but easily detected. Best for sites without bot protection.
    • Premium (type: "premium"): These are typically Residential proxies. They use real IPs, are hard to detect, and are trusted by services like Cloudflare. They are slower and more expensive but best for sensitive scraping.
    • Mobile Proxies: Highest trust level, used for the most aggressive anti-bot systems, but the most expensive.

  4. When to use BrowserPool vs ReaderClient

    main

    Choosing between ReaderClient and BrowserPool depends on your level of required control over the browser lifecycle:

    • Use ReaderClient (Recommended) for most use cases, including simple scraping/crawling, scripts, and CLI tools. It manages the Playwright browser pool lifecycle automatically.
    • Use BrowserPool when you need custom browser control, low-level page interaction, or are building high-performance Express/production servers where you need direct access to browser pages.

  5. How Reader bypasses anti-bot protections

    main

    Reader uses Playwright with stealth enhancements to avoid detection. The anti-bot bypass mechanism includes:

    • Fingerprint Generation: Uses fingerprint-generator to provide randomized browser fingerprints.
    • Navigator Spoofing: Sets webdriver=false and spoofs device memory and hardware concurrency.
    • Stealth Scripts: Injects scripts to patch common automation indicators.
    • WebRTC IP Masking: Prevents IP leaks through WebRTC connections.
    • Proxy Chaining: Routes traffic through a proxy-chain for IP management.

  7. Understand Browser Session Stealth Features

    main

    Every browser session automatically includes anti-bot stealth features injected at the browser level via Page.addScriptToEvaluateOnNewDocument. These apply to all pages created within the session (including those via Playwright/Puppeteer).

    Active Stealth Features:

    • navigator.webdriver = false: Hides the automation flag.
    • Navigator spoofing: Provides realistic deviceMemory, hardwareConcurrency, and platform values.
    • WebGL/Canvas fingerprinting: Uses randomized rendering signatures.
    • WebRTC IP masking: Prevents real IP leaks through WebRTC.
    • Chrome plugin array: Simulates the presence of real Chrome extensions.
    • Permission API behavior: Matches real Chrome permission responses.

  8. How the shared browser pool improves performance

    main

    The Express server is designed for high efficiency by initializing a single ReaderClient at startup and reusing it across all incoming requests. This differs from a per-request client approach in several ways:

    • Eliminates cold starts: No browser startup delay per request.
    • Reduces memory usage: A single Core instance is shared across all requests.
    • Improves throughput: Requests do not wait for Core initialization.
    ApproachStartup TimeMemoryBest For
    Per-request client~2-5sHigh (each request)Scripts, CLI
    Shared poolOnce at startupShared across requestsServers

    This pattern is recommended for server-side deployments to optimize resource utilization.

  9. How the Scraper class works

    main

    The Scraper class is responsible for URL scraping. When calling scrape(), it follows these steps:

    1. Initializes a browser pool based on the provided concurrency settings.
    2. Processes URLs using p-limit to maintain concurrency control.
    3. For each URL, it performs the following sequence:
      • Acquires a browser instance from the pool.
      • Navigates to the URL.
      • Detects and waits for Cloudflare challenges if present.
      • Extracts HTML and metadata.
      • Cleans the content (removing navigation, ads, etc.).
      • Formats the output into the requested formats (e.g., Markdown, HTML).
      • Releases the browser back to the pool.
    4. Aggregates results and metadata into a ScrapeResult object.

    Each URL is processed in its own browser instance from the shared pool to ensure isolation.

    // Conceptual usage of the Scraper class
const scraper = new Scraper({ urls: ["https://example.com"], formats: ["markdown"] });
const result = await scraper.scrape();

  10. How Reader's three core primitives work

    main

    Reader provides three primary ways to interact with the web, abstracting away browser management, anti-bot bypass, and content cleaning:

    1. scrape: Converts specific URLs into clean markdown or HTML. This is the fastest way to get structured data from a page.
    2. crawl: Discovers and scrapes pages by following links on a site with configurable depth and limits.
    3. browser: Provides a stealthed Chrome session via a CDP WebSocket endpoint. This allows you to use your existing Playwright or Puppeteer scripts while benefiting from Reader's anti-detection features (TLS fingerprinting, navigator spoofing, etc.).
    import { ReaderClient } from "@vakra-dev/reader";
import { chromium } from "playwright-core";

const reader = new ReaderClient();

// 1. Scrape URLs → clean markdown
const result = await reader.scrape({ urls: ["https://example.com"] });

// 2. Crawl a site → discover + scrape pages
const pages = await reader.crawl({
  url: "https://example.com",
  depth: 2,
  scrape: true,
});

// 3. Browser session → full Playwright control with stealth
const session = await reader.browser();
const browser = await chromium.connectOverCDP(session.wsEndpoint);
// ... use browser ...
await session.close();

  11. Use Browser Sessions for advanced automation

    main

    Use the browser() method when you need to perform complex browser automation that goes beyond simple scraping or crawling. This is the correct primitive for:

    • Clicking buttons, filling forms, and navigating multi-page flows.
    • Scraping pages that require authentication/login.
    • Taking screenshots or generating PDFs.
    • Running existing Playwright or Puppeteer scripts with built-in anti-bot stealth.

    Unlike scrape() (for markdown extraction) or crawl() (for page discovery), browser() returns a session object containing a CDP (Chrome DevTools Protocol) WebSocket URL that you connect to using your preferred automation library.

    const session = await reader.browser();

  12. How job queues work for scraping

    main

    For high-volume scraping, a job queue architecture allows you to:

    • Process requests asynchronously
    • Handle retries automatically
    • Scale workers independently
    • Monitor job progress
    • Avoid overwhelming target sites

    The architecture typically consists of an API Server that enqueues jobs into a Redis Queue, which are then consumed by one or more Workers. Results are stored in a Results Store.