DOCSEARCHbyAlgolia
Submit Docs

HeadlessX Documentation

repository·main·Indexed 24 days ago

https://github.com/saifyxpro/headlessx

A self-hosted platform for automated web extraction, search, and agentic workflows. It features the headlessx-pro package (v2.1.2), a CLI for managing scraping, crawling, and search operators (Google AI Search, YouTube, Tavily, and Exa), and headfox-js for high-fidelity browser automation using Camoufox and Playwright.

Tokens
48.4K
Snippets
109
Records
310
Agent score
83%

What's inside HeadlessX

  1. Overview of HeadlessX capabilities

    main

    HeadlessX is a self-hosted scraping and automation platform. It provides a web dashboard, a protected API, and queue-backed workflows for various operators.

    Core Operator Surfaces:

    • Website operator: Supports scraping, crawling, mapping, content extraction, and screenshots.
    • Search & Discovery: Google AI Search, Tavily, and Exa.
    • Media: YouTube.
    • Infrastructure: Queue jobs, logs, API key management, proxy management, and a remote MCP endpoint via /mcp.

    Operator-specific requirements:

    • Google AI Search: Requires a one-time Build Cookies run in the dashboard to establish a shared persistent browser profile.
    • YouTube: The workspace is only active if YT_ENGINE_URL points to a healthy yt-engine service.

  2. Design specification for the /logs dashboard page

    main

    The /logs dashboard is designed to provide a first-class interface for monitoring request logs and summary statistics. It leverages existing backend endpoints to display real-time operational data without requiring backend schema changes.

    Key Features

    • Summary Metrics: High-level cards showing total requests, success rate, failures, and average latency.
    • Log Management: A paginated table with search and status filtering.
    • Detailed Inspection: An inline row detail panel/drawer for inspecting specific log entries.

    Data Integration

    • Summary Stats: Fetched from /api/logs/stats.
    • Paginated Logs: Fetched from /api/logs?page=&limit=.

    Implementation Constraints & Limitations

    • Client-side Filtering: Currently, search and status filters are applied client-side to the current page of results only. Full-text search and advanced filtering are not yet supported by the backend API.
    • Visual Language: The UI must follow the existing flat bordered dashboard design system and use the PageHeader component.

  3. Understand the Logs UX Redesign approach

    main

    The /logs page has been redesigned to move away from a split-screen layout (request table + persistent right-side details panel) to a full-width, two-stage layout. This design prioritizes scan density and dashboard width rhythm.

    Key Layout Changes:

    • Primary Surface: The request table now spans the full available width.
    • Inspection Model: Instead of a permanent side panel, log inspection is handled via a centered overlay dialog (modal). This allows users to inspect metadata without losing context of the table.
    • Layout Structure:
      1. Top stats row.
      2. Full-width logs workspace card containing an integrated filter toolbar and the request table.
    • Visual Style: The design uses a flat, border-led system with no shadows. It utilizes white primary cards on a soft slate background and pale slate nested surfaces for metadata.

    Interaction Model:

    • Clicking a table row opens a modal overlay with that row's metadata.
    • Closing the overlay returns the user to the exact same table state.
    • Search is performed client-side on the currently loaded page.
    • Filter and page-size controls are integrated inline with the search surface.

  4. How Google cookie bootstrap works

    main

    The Google cookie bootstrap is a mechanism to transition from a 'cold' browser profile to a 'trusted' human-primed profile.

    Core Concepts

    • Shared Persistent Profile: Instead of creating a Google-specific profile, HeadlessX uses a single, long-lived browser profile (managed via BrowserService.ts) for all operations. This profile is persisted in the browser_profile volume in Docker or the local apps/api/data/browser-profile/default directory.
    • Interactive Mode: During bootstrap, the browser is launched in an interactive mode that allows a human to interact with the UI.
    • Virtual Display: In headless environments (like Docker or VPS), the system automatically manages a virtual display to facilitate the interactive session. This display is torn down once the bootstrap session is stopped.
    • State Persistence: Any cookies, storage, or reCAPTCHA solutions achieved during the bootstrap session are saved directly into the shared profile, making them available for subsequent automated searches.

  5. WebGL data support and fingerprint fidelity

    main
    For high-fidelity fingerprinting, headfox-js utilizes a bundled WebGL sampling database. This requires an optional SQLite dependency. If the SQLite dependency is missing (e.g., in a stripped Docker image), headfox-js will gracefully disable WebGL for that launch instead of failing.

  7. Website Workflow Tools: Scrape, Crawl, and Map

    main

    The HeadlessX website playground is designed around three primary operational modes for website interaction:

    1. Scrape: Performs one-page extraction. It supports multiple output formats including HTML, Rendered HTML, Markdown, and Screenshot.
    2. Crawl: Executes queued multi-page markdown jobs. This is a background process backed by BullMQ and Redis.
    3. Map: Performs fast link discovery. It returns a list of links discovered from the current page and optional sitemap data, presented in a card/list view with link counts and source badges.

  8. Handle Server-Sent Events (SSE) streams

    main

    Several endpoints use Server-Sent Events (SSE) to provide real-time updates via the text/event-stream content type.

    Common SSE event names:

    • start
    • progress
    • result
    • error
    • done (Note: The Google AI Search endpoints use end instead of done to signal the end of a stream).

  9. Configure Google Search targeting

    main

    The Google operator supports advanced targeting controls to refine search results. These include:

    • Region: Select a specific geographic region via a dropdown.
    • Language: Select a specific language via a dropdown.
    • Time Filter: Filter results by time (located in advanced settings).

    These settings are applied using the gl (region), hl (language), and tbs (time) parameters. If not explicitly set, the operator defaults to google.com with AI mode as the primary flow.

  10. Legal Disclaimer and User Responsibility

    main

    HeadlessX is provided "as is", without warranty of any kind. By using this software, you acknowledge and agree to the following:

    • Sole Responsibility: You are entirely responsible for your actions. The developers of HeadlessX accept no liability for legal consequences, damages, or bans resulting from your use of the tool.
    • Compliance Duty: It is your responsibility to ensure all scraping activities comply with local, national, and international laws, as well as the target website's Terms of Service.
    • No Guarantee of Access: While the project strives for a 0% detection rate, anti-bot technologies evolve. Access to any specific target website is not guaranteed.

  11. Configure CLI output format

    main

    The HeadlessX CLI supports two primary output modes:

    • Default (Markdown/Text): Optimized for LLM-facing use and human readability in the terminal.
    • JSON: Use the --json flag when you require structured, machine-readable output for programmatic processing.

  12. Responsible Scraping Guidelines for HeadlessX

    main

    To ensure ethical and legal use of HeadlessX, follow these four core principles when performing website extraction or automation:

    1. Respect robots.txt: Always check the target website's robots.txt file to understand their crawling policies and respect the site owner's wishes.
    2. Implement Request Throttling: Avoid overwhelming target servers. Use delays between actions to minimize load on the target infrastructure and prevent degrading service for legitimate users.
    3. Protect Personal Data & Privacy: Avoid scraping sensitive Personally Identifiable Information (PII) without explicit consent or a legitimate legal basis (e.g., GDPR compliance). Handle or delete accidentally collected personal data immediately.
    4. Respect Copyright & Intellectual Property: Be aware that data scraped from public websites may still be protected by copyright laws. Do not republish content in a way that infringes on the original owner's rights.