DOCSEARCHbyAlgolia
Submit Docs

OpenSEO Documentation

repository·main·Indexed 26 days ago

https://github.com/every-app/open-seo

OpenSEO is an open-source, pay-as-you-go SEO tool designed as a lightweight alternative to suites like Semrush and Ahrefs. It features an MCP server for AI agent integration (supporting Claude Code, OpenClaw, and Hermes) and supports self-hosting via Docker or Cloudflare. Users provide their own DataForSEO API key to manage data costs.

Tokens
31K
Snippets
80
Records
210
Agent score
95%

What's inside OpenSEO

  1. Overview of OpenSEO features and workflows

    main

    OpenSEO is an AI-native SEO platform designed for keyword research, domain research, backlinks, rank tracking, site audits, and Google Search Console integration. It is built to support both manual workflows and AI-agent SEO workflows via MCP.

    Core Workflows:

    • Keyword Research: Expand seed topics, compare volume, difficulty, CPC, intent, and SERP context.
    • Domain Overview: Analyze organic footprints and ranking keywords for any domain (including competitors).
    • Backlink Research: Inspect backlinks, referring domains, target URLs, and link quality signals.
    • Rank Tracking: Monitor keyword positions over time with configurable schedules and device types.
    • Site Audit: Crawl pages to inspect technical signals like status codes, meta tags, indexability, and Lighthouse findings.
    • Google Search Console: Access search performance data (clicks, impressions, CTR, position) and URL inspection data (index status, crawl info, etc.).

  2. Overview of OpenSEO

    main
    OpenSEO is an open-source, pay-as-you-go alternative to SEO suites like Semrush and Ahrefs. It is designed to be lightweight, focused on specific workflows, and compatible with AI agents via an MCP (Model Context Protocol) server. Instead of a fixed subscription, self-hosted users bring their own DataForSEO API key and pay only for the data they consume.

  3. Overview of OpenSEO features and deployment

    main

    OpenSEO is an all-in-one open source SEO platform providing keyword research, rank tracking, backlinks, site audits, AI brand visibility, and an AI search prompt explorer.

    Deployment Options:

    • Self-host via Docker
    • Self-host via Cloudflare Workers
    • Use the hosted version at openseo.so

    Data Requirements: OpenSEO relies on DataForSEO for SEO data. Users must provide their own API key, and costs are based on pay-as-you-go usage.

  4. Understand the OpenSEO Site Audit Architecture

    main

    OpenSEO uses an in-house Workers crawler (Option A) for site audits. This architecture prioritizes low marginal costs and full control over crawl behavior, UA identity, and politeness, rather than relying on third-party vendors like DataForSEO.

    Key Architectural Characteristics:

    • Crawler Engine: Runs on Cloudflare Workers.
    • Storage: Uses Cloudflare D1 for storing audit data (tables: audit_pages, audit_links).
    • Identity: Uses a dedicated User-Agent: OpenSEOBot/1.0.
    • Rendering: Currently lacks JS rendering (SPA/CSR sites may receive degraded audits). JS rendering is a future roadmap item via Cloudflare Browser Rendering or a local npx agent.
    • Constraints: The crawl is subject to a step-budget (currently ~1,024 steps). The binding constraint is often the Lighthouse strategy all, which can consume the budget quickly.

  5. Understand the OpenSEO Site Audit architecture and limitations

    main

    The OpenSEO site audit currently consists of a crawler and Lighthouse integration, but lacks a full rule engine for identifying SEO issues.

    Current Capabilities:

    • Extraction of per-page fields.
    • Filterable tables for extracted data.
    • Lighthouse/Core Web Vitals (CWV) analysis (considered a strength).

    Current Limitations & Technical Constraints:

    • Link Graph Persistence: The link graph is currently discarded after extraction. Only link counts are persisted to D1, making features like broken link detection, orphan page identification, and redirect chain analysis impossible without a schema change.
    • Memory & Step Constraints:
      • runCrawlBatch returns full page arrays (links, images) as durable step state, which is subject to a ~1MiB/step cap.
      • allPages accumulates data in 128MB Worker memory.
      • batchWriteResults writes all rows within a single finalize step. Large link graphs may exceed the DB_BATCH_SIZE=100 limit.
    • Crawl Size Caps: The existing workflow allows ~1,024 steps per instance. A typical 3-step per 25-page batch configuration limits crawls to approximately 8.5k pages. Using the Lighthouse strategy all (2 steps per 10-URL batch) reduces this limit to approximately 5k pages.
    • Security (SSRF): The url-policy.ts validates the starting URL, but discovered links are currently fetched without validation.

  6. Understand the Onboarding Agent implementation

    main

    The onboarding process consists of two main components:

    1. Seed Function: An asynchronous process that runs once when onboarding begins. It performs domain discovery (sitemap/robots.txt), scrapes 3–5 pages to markdown using the BROWSER binding, executes DataForSEO calls for overview and keyword research, and synthesizes a strategy via OpenRouter. The result is saved as the first version of the Project Context.
    2. Streaming Chat: A user-facing chat interface powered by the Vercel AI SDK (streamText) over OpenRouter. Users can refine their strategy, and the update_project_context tool allows the chat to write new versions of the project context.

    Note: The implementation uses a versioned, append-only log for Project Context. Markdown blobs are stored in R2, and metadata is stored in D1. Reverting a change involves inserting a new row that reuses a previous R2 blob key.

  7. Quickstart OpenSEO with Docker Compose

    main

    Run OpenSEO locally using Docker Compose and the published GHCR image. Note that in Docker mode, authentication is disabled (AUTH_MODE=local_noauth) and the local admin user is admin@localhost. Only expose this instance behind an auth-protected reverse proxy, tunnel, or private network.

    Prerequisites:

    • Docker Desktop (or Docker Engine + Docker Compose)
    • A DataForSEO API key

    Steps:

    1. Clone the repository.
    2. Copy the example environment file to .env.
    3. Start the containers.
    4. Configure your DATAFORSEO_API_KEY in the .env file.
    5. Access the application at http://localhost:3001 (or your configured PORT).
    git clone https://github.com/every-app/open-seo.git
cd open-seo
cp .env.example .env
docker compose up -d

  8. Quickstart OpenSEO with Docker

    main

    To run OpenSEO locally using Docker, follow these steps:

    1. Copy the example environment file: cp .env.example .env.
    2. Set your DATAFORSEO_API_KEY in the .env file.
    3. Start the services: docker compose up -d.
    4. Access the application at http://localhost:<PORT> (the default port is 3001).

    Security Warning: In Docker mode, OpenSEO uses AUTH_MODE=local_noauth, which disables application authentication (local admin is admin@localhost). Only expose this instance behind an auth-protected reverse proxy, tunnel, or private network.

    cp .env.example .env
docker compose up -d

  9. Set up self-hosted Google Search Console integration

    main
    Connecting Google Search Console (GSC) allows OpenSEO to pull real clicks, impressions, positions, and URL inspection data. This is an optional step. To enable it, you must create a Google Cloud project, enable the Search Console API, configure an OAuth consent screen, and create an OAuth client ID with the correct redirect URI.

  10. Understand the Onboarding Agent workflow

    main

    The Onboarding Agent is a guided pipeline designed to convert new signups into active users by analyzing a website live and proposing a tailored SEO strategy. The process follows these stages:

    1. Profile (Stage 0): Collects domain, experience level, primary goal, country, and language.
    2. Discover (Stage 1): Uses robots.txt and sitemaps (or a shallow crawl) to map the site structure.
    3. Read (Stage 2): Scrapes 3–5 key pages (home, top product/nav pages) to markdown using Cloudflare Browser Rendering.
    4. Signal (Stage 3): Fetches SEO data via DataForSEO (domain rank overview and keyword ideas).
    5. Synthesize (Stage 4): An LLM call processes the profile, scraped markdown, and keyword data to generate a positioning statement, themes, a starter keyword table, and a prioritized 'do this next' list.
    6. Persist (Stage 5): Saves the strategy as a 'Project Context' markdown artifact.

  11. Rank tracking with SerpBear

    main

    SerpBear is a specialized tool for rank tracking that allows for unlimited keywords and domains.

    Key Features:

    • SERP API Integration: Supports eight different providers; you pay for usage based on your chosen provider.
    • Google Search Console Integration: Free integration to view real GSC data.
    • Keyword Research: Free feature, but requires integration with the Google Ads API (which requires an application and approval process).

    Deployment: Self-host via Docker.

  12. Perform a low-downtime D1 to Postgres delta sync

    main

    To avoid downtime caused by large tables (like keyword_metrics or audit_pages), use a delta catch-up approach instead of a full write-freeze.

    1. Bulk Copy: Run the full migration (as described in the standard migration guide) while the app is live.
    2. Delta Sync: Run the script with the --update flag to upsert changes that occurred during the bulk copy. Use --since-hours to cover the gap between the bulk copy and the current time:
      pnpm exec tsx scripts/migrate-d1-to-postgres.ts --update --since-hours 12
    3. Cut over: Deploy using pnpm deploy:postgres.

    Note on Deletes: The --update flag does not sync deletes. If a row is deleted in D1 during the window, it will remain in Postgres. If absolute parity including deletes is required, you must perform a full copy under a complete write-freeze.

    pnpm exec tsx scripts/migrate-d1-to-postgres.ts --update --since-hours 12