DOCSEARCHbyAlgolia
Submit Docs

ArkhamFrame Core Infrastructure

repository·main·Indexed 19 days ago

https://github.com/mantisfury/arkhammirror

The immutable core infrastructure for SHATTERED, a modular, local-first intelligence analysis platform. ArkhamFrame provides a service-oriented architecture featuring an EventBus for inter-shard communication, a Vector Service using pgvector, an OpenAI-compatible LLM Service, and a PostgreSQL-backed Worker Pool system. It manages critical layers including infrastructure, AI & search, data, communication, and output, allowing developers to extend the platform by implementing custom ArkhamShards.

Tokens
241.9K
Snippets
699
Records
1.1K
Agent score
64%

What's inside arkham-frame

  1. Overview of ArkhamFrame core infrastructure

    main

    ArkhamFrame is the immutable core infrastructure for the SHATTERED modular architecture. It follows the Voltron philosophy, providing plug-and-play components that combine into a unified application.

    Key architectural principles include:

    • Immutable Core: Shards (feature modules) depend on the Frame, but the Frame never depends on shards.
    • Service-Oriented: All functionality is exposed through well-defined service interfaces.
    • Auto-Discovery: Shards are automatically discovered and loaded using Python entry points.
    • Event-Driven: Shards are loosely coupled via an EventBus.

  2. Overview of Documents Shard capabilities

    main

    The Documents shard is a component of the SHATTERED/ArkhamFrame ecosystem designed for managing and interacting with document data. Its primary functions include:

    • Document Browsing: Searching and filtering documents.
    • Document Viewing: Navigating through document pages.
    • Metadata Management: Editing document metadata.
    • Chunk & Entity Analysis: Browsing document chunks and associated entities.
    • Status Tracking: Monitoring document processing status.

  5. Overview of the Documents Shard

    main

    The arkham-shard-documents shard provides a document browser, viewer, and metadata editor. It is the primary interface for interacting with documents within the ArkhamFrame system.

    Key Capabilities:

    • Document Browsing: Filtering and searching through documents.
    • Document Viewer: Navigating through document pages.
    • Metadata Management: Editing metadata and managing custom fields.
    • Content Exploration: Browsing document chunks and extracted entities.
    • Status Tracking: Monitoring document processing status.

  6. Overview of the Entities Shard

    main

    The entities shard is a Data category shard (order 14) responsible for managing entities, their relationships, and mentions within the Arkham ecosystem. It provides a structured way to handle entity types, relationship types, and merging logic to resolve duplicates.

    Key Capabilities:

    • entity_management
    • entity_merging
    • relationship_editing
    • canonical_resolution
    • mention_tracking

    Dependencies:

    • Required: database, events
    • Optional: vectors, entities

  7. Overview of arkham-shard-parse package structure

    main

    The arkham-shard-parse package is responsible for parsing and extracting structured data from unstructured inputs. It is composed of several core modules:

    • ParseShard (shard.py): The main orchestration class.
    • API Endpoints (api.py): FastAPI-based endpoints for service interaction.
    • Data Models (models.py): Pydantic/Data models defining the schema for parsed data.
    • Extractors: Specialized modules for Named Entity Recognition (NER), dates, locations, and relations.
    • Linkers: Modules for entity linking and coreference resolution.
    • TextChunker (chunker.py): Handles text segmentation for processing.

  8. Overview of arkham-shard-credibility

    main

    The arkham-shard-credibility package provides source reliability assessment and deception detection. It uses weighted factors for credibility scoring and implements intelligence tradecraft checklists (MOM, POP, MOSES, EVE) to detect deception.

    Key Capabilities:

    • Credibility Scoring: Assess reliability (0-100) using weighted factors.
    • Source Assessment: Track reliability across documents, entities, claims, and external sources.
    • Deception Detection: Use MOM, POP, MOSES, and EVE checklists.
    • LLM Analysis: AI-powered checklist completion and analysis.
    • History Tracking: Monitor credibility changes over time.

  9. Overview of the Graph Shard (arkham-shard-graph)

    main

    The Graph Shard (arkham-shard-graph) provides entity relationship visualization and graph analysis capabilities for ArkhamFrame. It constructs entity graphs based on document co-occurrences and implements various graph algorithms (pathfinding, centrality, community detection) designed for investigative journalism workflows.

    Key Capabilities:

    • Graph Construction: Builds graphs from entity co-occurrences with configurable filtering (entity type, degree, weight).
    • Path Analysis: Uses BFS to find shortest paths between entities with depth limits.
    • Centrality Metrics: Calculates Degree, Betweenness, and PageRank to identify influential or connecting entities.
    • Community Detection: Uses Louvain-style modularity optimization to find clusters.
    • Exporting: Supports JSON, GraphML (for Gephi/Cytoscape), and GEXF formats.

  10. Overview of arkham-shard-shell

    main

    The arkham-shard-shell is the unified React/TypeScript frontend application for the SHATTERED architecture. It acts as a thin rendering layer (following the Voltron philosophy) that integrates various shard UIs into a single cohesive application without containing business logic.

    Key capabilities include:

    • Dynamic Navigation: Sidebar navigation is rendered automatically from shard manifests fetched via the /api/shards/ API.
    • Layout Management: Provides a standard structure with a collapsible sidebar, top bar, and content area.
    • Generic Components: Offers data-driven list and form components for shards that do not require custom UIs.
    • Custom Shard Pages: Supports full React page implementations for complex shard interfaces.
    • Shared Context: Provides system-wide services like toast notifications, confirmations, badges, theming, and project scope management.

  11. Overview of arkham-shard-graph

    main

    The arkham-shard-graph package (version 0.1.0) is a component of Project Shattered designed to build and analyze entity relationship graphs from parsed documents. It implements the ArkhamShard interface and provides tools for graph construction, complex algorithm execution (like PageRank and community detection), and data export.

    Key Capabilities:

    • Graph Building: Constructs graphs based on entity co-occurrences with support for filtering by entity type, document ID, and minimum co-occurrence thresholds.
    • Algorithms: Includes Shortest Path (BFS), Degree/Betweenness Centrality, PageRank, Louvain-style Community Detection, and Connected Components.
    • Exporting: Supports JSON, GraphML (for Gephi/Cytoscape), and GEXF formats.
    • Integration: Designed to work within the arkham-frame ecosystem, using event-driven invalidation for caches when entities or documents change.

  12. Overview of arkham-shard-entities

    main

    The arkham-shard-entities package provides entity browser and management functionality for ArkhamFrame. It enables comprehensive entity management, including:

    • Browsing: Viewing and searching for entities.
    • Merging: Resolving duplicate entities into canonical records.
    • Relationships: Managing connections between different entities.
    • Mention Tracking: Tracking where entities are mentioned across documents.

    Note: As of version 0.1.0, this shard is a Blueprint/Stub Implementation. While the interfaces, structure, and API endpoints are defined, the core business logic (database operations, merge algorithms, etc.) is currently stubbed.