DOCSEARCHbyAlgolia
Submit Docs

DevDocs Documentation

repository·feature-main·Indexed 24 days ago

https://github.com/cyberagiinc/devdocs

An intelligent documentation crawling and processing platform that converts technical documentation into LLM-ready data. DevDocs features intelligent crawling to discover website structures, exports content in MD or JSON formats, and integrates via MCP servers for AI querying. It includes a backend API for page discovery and crawling, a frontend UI, and integration with Crawl4AI.

Tokens
35.7K
Snippets
59
Records
163
Agent score
84%

What's inside DevDocs

  1. Features of Anthropic Claude in AG2

    feature-main

    When using Anthropic models with AG2, the following capabilities are supported:

    • Function/tool calling: Agents can interact with external tools.
    • Structured Outputs: Support for generating data in specific formats (see Structured Outputs notebook example).
    • Accurate Cost Tracking: Token usage and cost calculations are aligned with Anthropic's API costs (as of December 2024).

  2. Understand the DevDocs directory structure

    feature-main

    The DevDocs codebase is organized by file purpose to reduce root directory clutter and improve maintainability. Use the following mapping to locate files:

    • scripts/: Executable files, subdivided into general/, test/, docker/, and mcp/.
    • docs/: Documentation, subdivided into general/, docker/, and mcp/.
    • docker/: Docker-related files, including dockerfiles/ (definitions) and compose/ (Compose files).
    • config/: Project configuration files (e.g., tsconfig.json, next.config.mjs).
    • app/, backend/, components/: Application source code (unchanged).

  3. Understand the DevDocs Docker Architecture

    feature-main

    The DevDocs platform uses a hybrid containerized and host-based architecture.

    • Containerized Components: The Next.js Frontend, FastAPI Backend, and Crawl4AI Service all run within Docker containers and communicate via a dedicated Docker bridge network (devdocs-network).
    • Host-based Component: The Fast-Markdown-MCP Server runs directly on the host system (not in a container) to maintain direct integration with the host file system.

    Communication Flow:

    1. The Frontend communicates with the Backend via http://backend:24125.
    2. The Backend communicates with the Crawl4AI service via http://crawl4ai:11235.
    3. The Backend communicates with the MCP server on the host using the special DNS host.docker.internal.

  4. How real-time crawl status monitoring works

    feature-main

    The system provides visibility into URL discovery and content crawling using a Backend State + Frontend Polling model.

    1. Job Initiation: When a user starts a discovery via /api/discover, the backend generates a unique job_id and initializes a CrawlJobStatus object in an in-memory store.
    2. Background Processing: The discovery and crawling tasks run as background tasks. As the crawler processes URLs, it updates the status of each URL and the overall job status in the backend state.
    3. Frontend Polling: The frontend component (e.g., CrawlStatusMonitor) receives the job_id and initiates a polling loop (typically every 3 seconds) using setInterval. It calls GET /api/crawl-status/{job_id} to fetch the latest state and update the UI.
    4. Completion: Once the overall_status reaches a terminal state (like completed or error), the frontend stops polling.

  5. How Discovery Error Tooltips work in the Crawl Queue UI

    feature-main

    The Discovery Error Tooltip feature provides specific error feedback for URLs that encounter a discovery_error status during the crawling process.

    When a URL fails discovery, the backend captures a concise error message and stores it alongside the URL status. The API then exposes this message via the errorMessage field in the URL details. On the frontend, the CrawlUrls component detects the discovery_error status and wraps the status Badge in a Tooltip to display the captured error message to the user.

  6. Understand the In-Memory File Processing Architecture

    feature-main

    The proposed architecture for DevDocs optimizes storage by keeping individual crawled files in memory while only writing consolidated files to the physical disk. This reduces disk clutter in the storage/markdown directory.

    Workflow Comparison:

    • Current: Crawl $\rightarrow$ Process $\rightarrow$ Write individual files to disk $\rightarrow$ Consolidate $\rightarrow$ Write consolidated file to disk.
    • Proposed: Crawl $\rightarrow$ Process $\rightarrow$ Store individual files in memory $\rightarrow$ Consolidate $\rightarrow$ Write consolidated file to disk. Virtual file metadata is generated for the in-memory files to ensure they remain visible to the system.

  7. Workflow for Selective URL Consolidation

    feature-main

    The Selective URL Consolidation feature allows users to discover all reachable internal URLs within a site and then manually choose which specific pages to crawl and consolidate into a single markdown file.

    The workflow follows these steps:

    1. Discovery: Enter a root URL and depth, then trigger discovery. The backend finds reachable internal URLs and marks them as pending_crawl. No content is fetched during this phase.
    2. Selection: Once the status reaches discovery_complete, the UI displays a list of discovered URLs with checkboxes. Users can select individual URLs or use a "Select All" option.
    3. Selective Crawl & Consolidation: Triggering the crawl for selected URLs sends the jobId and the list of selected URLs to the /api/crawl endpoint. The backend fetches content for only the selected URLs using the crawl4ai service and appends the resulting markdown to a consolidated file (e.g., storage/markdown/<root_url_filename>.md).
    4. Results: Upon completion, a ConsolidatedFiles component displays metadata (filename, page count, total size, last updated) and provides access to the raw markdown and JSON metadata.

  8. Data Structure for URL Status and HTTP Codes

    feature-main

    To support displaying HTTP status codes in the Crawl Queue UI, the data model for URL tracking must transition from a simple string-based status to a structured object.

    Backend (Python/Pydantic): Use a UrlDetails model to ensure type safety when storing status information.

    class UrlDetails(BaseModel):
    status: str
    statusCode: Optional[int] = None

    The CrawlJobStatus model should then be updated to use this structure: urls: dict[str, UrlDetails]

    Frontend (TypeScript): Define a corresponding UrlDetails interface to match the backend structure:

    interface UrlDetails {
  status: UrlStatus;
  statusCode: number | null;
}

    Update the CrawlJobStatus and CrawlUrlsProps interfaces to use this new record type: urls: Record<string, UrlDetails>

    class UrlDetails(BaseModel):
    status: str
    statusCode: Optional[int] = None

interface UrlDetails {
  status: UrlStatus;
  statusCode: number | null;
}

  9. Understand the simplified file storage architecture

    feature-main

    The DevDocs architecture has been simplified to eliminate ephemeral in-memory storage. The system now follows a direct disk-based flow:

    1. Crawl Process: The Crawl4ai service writes files (often UUID-named) to ./crawl_results via volume mounts.
    2. Consolidation: The backend processes these and writes consolidated, URL-named markdown files into the storage/markdown directory.
    3. Retrieval: The Frontend requests file content by providing a path relative to storage/markdown. This request flows through a Frontend API proxy to the Backend API, which reads the file directly from the filesystem.

    This approach removes the need for complex monkey-patching of the open function and reduces memory overhead by relying on persistent disk storage.

  10. How the Backend Crawl Kill Switch works

    feature-main

    The kill switch mechanism relies on three components working together:

    1. API Endpoint: Receives the cancellation request and calls status_manager.request_cancellation(job_id).
    2. StatusManager: Tracks cancellation requests using an internal dictionary (_cancellation_requests). It updates the job status to cancelling and sets a flag for the specific job_id.
    3. Crawler Logic: The core crawling loop (e.g., discover_and_process_url) must periodically check status_manager.is_cancellation_requested(job_id). If true, the crawler logs the event, breaks the current loop, performs resource cleanup, and updates the job status to cancelled.

    Note: This mechanism does not terminate the FastAPI process or Docker container; it only stops the specific background task. Additionally, tasks already submitted to external tools like Crawl4AI might not be immediately stoppable via this mechanism.

  11. Redirect file operations to memory in crawler.py

    feature-main

    The file redirection system in backend/app/crawler.py must be modified to intercept file access. The logic follows these rules:

    1. Writes to individual files: Intercept and return a MemoryFileObject.
    2. Writes to consolidated files: Allow standard disk writes.
    3. Reads: Check the in_memory_files storage first; if the file is not found in memory, fall back to reading from the disk.
    # Pseudo-code for the modified file redirection
def redirecting_open(file, mode='r', *args, **kwargs):
    if 'w' in mode and is_individual_file(file):
        # Return a memory file object instead of a real file
        return MemoryFileObject(file)
    elif 'w' in mode and is_consolidated_file(file):
        # Allow writes to consolidated files
        return original_open(file, mode, *args, **kwargs)
    else:
        # For read operations, check if file exists in memory first
        if is_individual_file(file) and file in in_memory_files:
            return MemoryFileObject(file)
        return original_open(file, mode, *args, **kwargs)

  12. Display HTTP Status Codes in the Crawl Queue UI

    feature-main

    To show HTTP status codes in the CrawlUrls component table, follow these steps:

    1. Data Access: Update component logic to access the descriptive status via urls[url].status and the numerical code via urls[url].statusCode.
    2. Helper Functions: Ensure existing helpers like getStatusBadgeStyle and getStatusTooltip continue to receive the UrlStatus string from urls[url].status.
    3. Table Updates:
      • Add a new <TableHead> labeled "Code" (or similar).
      • Add a new <TableCell> in the table body to display urls[url].statusCode.
    4. Null Handling: If statusCode is null or undefined, render a placeholder such as '-' or 'N/A' to maintain table consistency.