deepwiki-rs (Litho)

Litho provides several core and advanced capabilities for automated documentation:

Core Capabilities

• AI-Driven Architecture Generation: Creates documentation based on codebase analysis.
• C4 Model Diagrams: Automatically generates Context, Container, Component, and Code diagrams.
• Intelligent Extraction: Pulls code comments, structures, and dependency relationships.
• Multi-language Support: Extensible language processors.
• Customizable Templates: Supports custom output styles via a template system.

Advanced Features

• External Knowledge Mounting: Supports attaching PDF, Markdown, and SQL files as additional knowledge sources.
• Database Documentation: Analyzes SQL projects to generate ER diagrams and database architecture docs.
• Git History Analysis: Tracks architectural evolution over time.
• Cross-referencing: Links code elements directly to documentation.
• CI/CD Integration: Can be integrated into pipelines to automate documentation updates on every commit.

When integrating or deploying deepwiki-rs, keep the following architectural decisions in mind:

• Multi-Agent AI Pipeline: Uses 8 specialized research agents operating at different C4 abstraction levels to reduce hallucinations and improve coherence.
• Read-Only Analysis: The system strictly enforces read-only file system access to target projects to ensure safety and prevent source code corruption.
• Dual LLM Provider Support: Supports both Cloud APIs (via HTTPS/JSON) and local Ollama inference (via HTTP) to accommodate enterprise security and air-gapped environments.
• Language-Agnostic Preprocessing: Uses a trait-based architecture supporting 12+ languages through pluggable modules.
• File-Based Caching: Implements persistent MD5-based caching for LLM responses to reduce costs and enable offline replay.
• Privacy & Security: Local Ollama support allows for zero-egress environments. The system requires no network ingress and operates without privilege escalation.

The Core Generation Domain (located in src/generator/) is the orchestration layer of the deepwiki-rs engine. It manages the end-to-end AI-powered documentation pipeline, bridging infrastructure (LLM providers, caching, configuration) with domain-specific processing stages: Preprocessing, Research, Composition, and Output.

Key Architectural Characteristics

• Thread-Safe Resource Management: Uses Arc<RwLock<T>> for concurrent access to shared resources across async boundaries.
• Dual Execution Models: Supports Imperative patterns (direct AgentExecutor usage for fine-grained control) and Declarative patterns (using the StepForwardAgent trait for research and synthesis).
• Type-Safe Agent Framework: Provides compile-time guarantees for agent outputs via associated types and JSON Schema validation.
• Intelligent Caching: Implements a cache-aside pattern with token usage tracking.
• Multilingual Support: Supports integrated internationalization with 8-language prompt generation.

The system processes data through three primary layers:

• Input Layer: Consumes CLI arguments, configuration files, the target project, and existing knowledge bases.
• Processing Layer: Executes the core logic through the sequence of Preprocessing $\rightarrow$ Research $\rightarrow$ Composition.
• Output Layer: Handles Document Persistence and the generation of Summary Reports.

flowchart LR
    subgraph Input["Input Layer"]
        A[CLI Arguments] --> B[Configuration]
        C[Target Project] --> D[Knowledge Base]
    end
    
    subgraph Processing["Processing Layer"]
        E[Preprocessing] --> F[Research]
        F --> G[Composition]
    end
    
    subgraph Output["Output Layer"]
        G --> H[Document Persistence]
        H --> I[Summary Reports]
    end
    
    Input --> Processing
    Processing --> Output
    
    style Processing fill:#e3f2fd,stroke:#1976d2
    style Input fill:#e8f5e9,stroke:#388e3c
    style Output fill:#fff3e0,stroke:#f57c00

The Preprocessing Module is the first stage of the Litho four-stage pipeline. Its purpose is to transform raw source code files into structured information (project structure, code insights, directory dossiers, and dependency relationships) that subsequent AI Agents can use for analysis. It does not produce final documentation itself, but provides the essential "raw materials" for the research and orchestration stages.

Core Functions

• Project Structure Scanning: Uses StructureExtractor to traverse the directory tree and calculate importance scores for files and directories.
• Multi-language Analysis: Uses LanguageProcessor implementations to parse language-specific structures (classes, functions, traits, etc.) for 14 supported languages.
• Directory Importance Scoring: Uses DirectoryScorer (via LLM) to distinguish core directories (e.g., src/) from auxiliary ones (e.g., assets/).
• Directory Summarization: Uses DirectorySummarizer to perform deep analysis of core files, generating DirectoryDossier and FileInsight objects.
• Dependency Analysis: Uses RelationshipsAnalyzer to identify import/use statements and map module dependencies.

The system uses a hybrid storage strategy combining in-memory state and file-system-based persistence. It does not require an external database.

Memory Scopes (Runtime)

Data is managed in three distinct lifecycles aligned with pipeline stages. Scopes are cleared after consumption to manage memory footprint:

• PREPROCESSING Scope: Contains CodeInsight and ProjectStructure.
• STUDIES_RESEARCH Scope: Contains SystemContextReport, DomainModulesReport, and ArchitectureReport.
• DOCUMENTATION Scope: Contains DocTree and DocumentMetadata.

File System Cache

• LLM Response Cache: Uses MD5 keys (hash of prompt content + model configuration) to map to JSON responses with TTL-based expiration. This follows a Cache-Aside pattern.
• Knowledge Base Cache: Stores chunked documents and sync metadata.

Output Artifacts

Final results are persisted as:

• Markdown Files: With localized naming.
• Mermaid Diagrams: With fixed syntax.
• Summary Reports: Including performance metrics.