Deepcrawl Documentation

Deepcrawl is an agent-oriented website data context extraction platform designed for high-frequency agent workloads. Unlike tools focused on bypassing anti-bot measures, Deepcrawl is optimized to scrape public pages and transform them into LLM-friendly formats.

Key outputs include:

• Cleaned Markdown: Page content converted to markdown to minimize token costs.
• Hierarchical Links Tree: An agent-favored structure of links for easier navigation.
• Metadata: Structured data designed to reduce context switching and hallucinations in LLMs.

WARNING: The project is under rapid development and is subject to change. It is not currently recommended for production use.

The create-deepcrawl CLI is a scaffolding tool designed to provision Cloudflare resources, write configuration, and deploy Deepcrawl components in a single guided flow.

Currently, the CLI focuses on the V0 API Worker only (running on Cloudflare Workers) with authentication set to either none or jwt.

Upon successful deployment, the CLI provides:

1. The Worker URL.
2. A link to documentation.
3. The locations of two secret files (apps/workers/v0/.dev.vars and apps/workers/v0/.dev.vars.production) which contain the JWT_SECRET.
4. A summary of provisioned resources.
5. An option to run an immediate test, which auto-mints a 15-minute token (if JWT is required) and prints a copyable curl command for testing the API.

The @deepcrawl/types package provides TypeScript types and Zod schemas used throughout the Deepcrawl API ecosystem. It ensures type-safe contracts between clients and servers and provides runtime validation for API requests and responses.

Key Features:

• Runtime validation: Zod schemas for validating API requests and responses.
• Type-safe contracts: Shared types used across both client and server implementations.
• oRPC integration: Type definitions specifically for oRPC endpoints.
• Service schemas: Validation schemas for all core Deepcrawl services.
• Configuration types: Definitions for environment and configuration settings.

Deepcrawl is an open-source, agents-oriented web page context extraction platform designed to make website data AI-ready. It converts messy web pages into cleaned markdown, agent-navigable link trees, and page metadata to reduce token costs and hallucinations in LLM prompts.

Warning: Deepcrawl is under active development. All APIs and SDKs are subject to change. Use at your own risk.

Key Capabilities

• Markdown Extraction: Compresses web content into structured markdown optimized for LLM consumption.
• Link Mapping: Maps domain link topologies so agent workflows can understand site structure before browsing.
• Type-Safe Integration: Provides production-ready APIs and SDKs designed to feel native within agent frameworks like ai-sdk.

The @deepcrawl/types package acts as the central source of truth for the entire Deepcrawl ecosystem. It provides unified type definitions to maintain consistency across four key layers:

1. oRPC contracts: Ensures type-safe API definitions.
2. Service implementations: Provides types for business logic validation.
3. Client libraries: Guarantees type safety within SDKs.
4. Database schemas: Maintains consistency for data models.

Deepcrawl is a pnpm + Turbo monorepo designed to keep the dashboard, Workers, and SDKs in sync. The architecture is divided into three main areas:

• Apps and Workers: Contains the user-facing Next.js dashboard (apps/app) and the core Cloudflare Workers (apps/workers/auth for authentication and apps/workers/v0 for crawl jobs/queues).
• Shared Packages: Contains the logic used by both apps and workers, including packages/auth (session/token helpers), packages/contracts (oRPC definitions for type-safe API calls), packages/db (Prisma clients and migrations), and packages/sdks/js-ts (the public TypeScript SDK).
• Tooling: Uses Turborepo for optimized builds and Biome for linting and formatting.

By using shared oRPC contracts in packages/contracts, every API call between the dashboard, Workers, and the SDK remains type-safe.

The GetOneLogResponse object contains metadata about the request and the resulting payload. The response field's shape depends on the path value:

• read-getMarkdown: response is the markdown string or GetMarkdownResponse (string).
• read-readUrl: response narrows to ReadUrlResponse.
• links-getLinks / links-extractLinks: response narrows to GetLinksResponse or ExtractLinksResponse respectively.

Example JSON Response

{
  "id": "log_123",
  "path": "read-readUrl",
  "success": true,
  "requestOptions": {
    "url": "https://example.com",
    "markdown": true
  },
  "response": {
    "markdown": "# Example Page",
    "links": ["https://example.com/about"]
  },
  "requestTimestamp": "2025-01-10T14:20:00.000Z"
}

Deepcrawl uses scoped Conventional Commits following the pattern: <scope>[:subscope]:<type> message.

Examples:

• app:feat add job status badge to dashboard
• pkg:ui:fix correct Button loading state
• workers:v0:chore add queue retries
• scripts:ref log slow routes

Guidelines:

• Keep the subject line $\le$ 72 characters and use the imperative mood.
• Reference issues (e.g., #123) in the body when relevant.