CortexDB

repository·main·Indexed 42 days ago

https://github.com/liliang-cn/cortexdb

A pure-Go, single-file AI memory and knowledge graph system providing a unified storage layer for vectors, hybrid RAG search, episodic memory, and an RDF/SPARQL knowledge graph. It is designed to be embedded in Go applications or used as a gRPC sidecar for AI agents, featuring specialized sub-clients for knowledge, memory, graph, graphrag, tools, and administration.

Tokens
166.8K
Snippets
438
Records
616
Agent score
57%

What's inside CortexDB

  1. Overview of CortexDB Core Features

    main

    CortexDB is a pure Go, single-file AI memory and knowledge graph system. It uses a SQLite file to store:

    • Vectors & Hybrid RAG: Supports vector, lexical, and hybrid retrieval (RRF, weighted RRF, DBSF) with metadata filtering.
    • KnowledgeMemory: High-level API for Recall, Remember, Reflect, and Consolidate operations.
    • Knowledge Graph: RDF triples/quadruples with a SPARQL subset, RDFS-lite reasoning, and SHACL-lite validation.
    • Ontology: Palantir-style object/link/interface types with governed action types and schema diffing.
    • Storage: Defaults to SQLite, but can be switched to PostgreSQL + pgvector by changing the DSN.
    • Pipelines: Built-in flows for memory (memoryflow), graph (graphflow), and data import (importflow).
  2. Capabilities demonstrated by Scale + Analytics

    main

    The Scale + Analytics example showcases several core CortexDB capabilities:

    • Throughput Monitoring: Reports ingest metrics including rows/s, chunks/s, and triples/s.
    • Large-Scale Graph Analytics: Executes complex queries including GROUP BY, HAVING subqueries, RDFS-lite inference materialization, and SHACL validation.
    • PII Redaction: Demonstrates free-text redaction (e.g., phone numbers and emails) via a privacy connector before data is indexed.
    • CDC (Change Data Capture): Uses Postgres logical replication to stream live UPDATE events into the brain while maintaining concurrent query performance, measuring how quickly the brain catches up to the source.
  3. Overview of the gRPC Sidecar and Rust Client Architecture

    main

    CortexDB implements a gRPC sidecar pattern to expose the pkg/cortexdb facade over gRPC. This architecture consists of:

    1. gRPC Sidecar Server (cmd/cortexdb-grpc): A Go-based server that uses a proto contract located in proto/cortexdb/v1/ to mirror the existing pkg/cortexdb Request/Response structs 1:1. It uses a conversion layer in pkg/rpcserver to delegate calls to the core *cortexdb.DB instance. Authentication is handled via bearer tokens.
    2. Rust Client (clients/rust/cortexdb-client): A typed Rust crate that provides access to the sidecar. It includes an optional managed-server feature that can automatically download and spawn the sidecar process.

    Note on Rust Implementation: To ensure compatibility with docs.rs and offline builds, the Rust crate commits pre-generated tonic code rather than relying on build.rs and protoc at compile time.

  4. Overview of CortexDB Architecture and Layers

    main

    CortexDB is a pure Go-implemented single-file AI memory and knowledge graph library. It uses SQLite (via modernc.org/sqlite) as its storage engine, allowing a single database file to contain vectors, FTS5 full-text indices, RAG knowledge, scoped agent memory, RDF knowledge graphs, and MCP tool definitions.

    Key Architectural Layers:

    • Engine Layer (pkg/core & pkg/graph): Provides SQLiteStore, vector embedding support, FTS5, and various vector indices (HNSW, IVF, Flat). The graph engine provides property graphs, RDF triples, SPARQL queries, RDFS reasoning, and SHACL validation.
    • Public Facade (pkg/cortexdb): The primary entry point that encapsulates core and graph. Application code should depend on this layer.
    • Workflow Layer:
      • pkg/memoryflow: Handles dialogue transcription ingestion, memory awakening, and journaling.
      • pkg/graphflow: Manages extraction, construction, analysis, reporting, and exporting of knowledge graphs from corpora.
      • pkg/importflow: Handles structured imports from CSV, MySQL, and PostgreSQL.
    • Specialized Modules:
      • pkg/agentmem: Provides SQL-based agent memory without requiring embedding models (uses FTS5 trigram and importance-based time decay).
      • pkg/hindsight: A recall strategy plugin for memoryflow.
      • pkg/semantic-router: Provides intent routing before retrieval.
  5. Overview of Palantir-style Ontology (v2) Implementation

    main

    CortexDB is transitioning from a 'lite' ontology (simple entity and relation types) to a robust, Palantir Foundry-style ontology. This new system (v2) introduces:

    • Typed Object Types: Objects with specific data types and mandatory primary keys.
    • Link Types with Cardinality: Links that define independent cardinality (ONE or MANY) for each side of the relationship.
    • Interface Polymorphism: Support for interfaces and type inheritance.
    • ObjectSet Algebra: A recursive union type for composable queries that pushes down to existing engines (Vector, FTS5, and Graph traversal).
    • Action-based Writes: A governed write path with auditing, controlled by a strict_actions flag in the schema.
    • Typed MCP Tools: Automatically generated Model Context Protocol (MCP) tools based on the schema.

    Key Architectural Constraints:

    • All logic resides in the pkg/cortexdb facade layer.
    • Uses a new dedicated table ontology_schemas_v2; the old graph_ontology_schemas table is deprecated and will not be read or written.
    • API names are stored exactly as provided but are case-insensitive during lookups to prevent collisions like Airport vs airport.
  6. Overview of the importflow workflow layer

    main

    The importflow package is an opt-in workflow layer designed to import external structured data (such as CSV files or MySQL/PostgreSQL SQL dumps) into CortexDB. It performs a single-pass import that builds both RAG (Retrieval-Augmented Generation) foundations using vector/FTS5 and Knowledge Graph (KG) foundations using RDF triples.

    Key architectural components include:

    • Source: A plugin interface (e.g., CSVSource, SQLDumpSource) that yields a uniform stream of Record objects.
    • MappingPlan: A declaration of how columns from a source table route to RAG content and KG entities/relations.
    • Mapper: Applies the MappingPlan to a Record to produce RAG chunks and RDF triples.
    • Sinks: RAGSink and KGSink which write the processed data to CortexDB via the pkg/cortexdb facade.
    • Importer: The orchestrator for Plan, Run, and AutoImport operations.

    AI assistance for mapping inference, triple extraction, and text refinement is provided via optional interfaces (MappingInferer, TextRefiner, and LLMExtractor) and does not require an LLM SDK to be present within the pkg/ directory.

  7. Understand the CortexDB Layered Architecture

    main

    CortexDB is organized into specialized packages. When building an agent, choose the layer that matches your requirements:

    • pkg/cortexdb: The main facade. Use this for vector, text/RAG, knowledge, memory, KG, ontology, and MCP tools.
    • pkg/memoryflow: High-level agent memory workflows (transcript ingestion, recall, promotion).
    • pkg/graphflow: Data pipeline for extracting and building knowledge graphs from corpora.
    • pkg/importflow: Tools to import CSV, SQL dumps, or online Postgres/MySQL into RAG + KG.
    • pkg/connector: Privacy gate for importflow (PII masking, manual signing, CDC sync).
    • pkg/graph: Low-level RDF/SPARQL/RDFS/SHACL and property graph support.
    • pkg/core: The storage engine (SQLite or PostgreSQL), embeddings, FTS5, and vector indexing (HNSW/IVF/Flat).
  8. Implement the Knowledge Contract for auditable records

    main

    To ensure multi-producer reliability, CortexDB uses a Knowledge Contract. Every record includes metadata keys prefixed with _ to define its provenance and reliability. Producers should call ValidateContract(meta) before writing to ensure compliance.

    Metadata Keys

    KeyMeaning
    _sourceSource (URL, file, job, engagement). Never contains credentials.
    _chunkChunk index in the source (-1 if no chunking).
    _producerMethod of production (e.g., llm-extract, ddl, tabular, graph-import, human, measured, compiled).
    _gradeThe reliability level (see below).
    _state, _whyProducer's claim on record state and reasons for rejection.
    _by, _atWho asserted the record and when (mandatory if _producer is human).
    _contradictsID of a record that cannot be true simultaneously with this one.

    Reliability Grades (from strongest to weakest)

    1. verified: Reviewed and confirmed by a named human.
    2. self_consistent: Deterministically derived from existing data (e.g., CREATE TABLE statements).
    3. asserted: Stated by a model or human without verification (the default for unverified data).
    4. held: Pending processing.
    5. refused: Rejected by the contract (reason specified in _why).

    Auditing and Inspection

    • db.ContractTally(ctx) (contract_tally): Counts nodes and edges by contract grade, including untagged records.
    • db.NeedsAttention(ctx, limit) (contract_needs_attention): Retrieves records requiring review.
    • db.FactProvenanceFor(ctx, edgeID, true) (fact_provenance): Retrieves the provenance and original citations for a specific fact.
  9. Corpus-to-graph workflow with 05_graphflow

    main

    The 05_graphflow example uses pkg/graphflow to transform documents or model extraction outputs into inspectable graphs.

    Key Features:

    • Canonical extraction schema.
    • Deterministic build, analyze, and report pipeline.
    • Exports: graph.json, GRAPH_REPORT.md, and HTML visualizations.
  10. Ontology Schema Validation Rules

    main

    When defining an OntologySchema in CortexDB, the schema must pass several validation constraints to ensure data integrity and API consistency.

    Object Type Constraints

    • Primary Key: Every OntologyObjectType must declare a primary_key. This key must be a declared property of the object, it must be marked as Required, and its data type must be one of: OntologyDataString, OntologyDataInteger, or OntologyDataLong.
    • Title Property: If a title_property is declared, it must correspond to an existing property within the object.
    • Property Uniqueness: All properties within an object must have unique API names.

    Interface Type Constraints

    • Uniqueness: Interface type API names must be unique and cannot collide with existing ObjectTypes.
    • Implementation: If an object type implements an interface, the interface must be defined in the schema.
    • Side Uniqueness: Each side of a link (OntologyLinkSide) must have a distinct API name.
    • Cardinality & Foreign Keys:
      • A side must declare a cardinality of either OntologyCardinalityOne or OntologyCardinalityMany.
      • If a side declares a ForeignKeyProperty, its cardinality must be OntologyCardinalityOne. (A MANY side cannot host a foreign key).
      • The declared ForeignKeyProperty must exist as a property on the associated ObjectType.

    Data Type Constraints

    • Supported Kinds: OntologyDataString, OntologyDataInteger, OntologyDataLong, OntologyDataDouble, OntologyDataDecimal, OntologyDataBoolean, OntologyDataDate, OntologyDataTimestamp, OntologyDataGeoPoint, OntologyDataGeoShape, OntologyDataMarking.
    • Vectors: Must declare a positive dimension.
    • Arrays: Must declare an item_type.
    • Structs: Must declare one or more fields.
  11. Understand Listing Cursors and Keyset Pagination

    main

    CortexDB uses keyset pagination (rather than offset pagination) for bulk operations like memory_list_all and graph_list_all. This prevents records from being skipped or duplicated when the underlying data changes (e.g., new memories are written) while a client is reading the brain.

    Key Concepts

    • Opaque Cursor: The cursor is a string that encodes the sort key of the last row scanned. It is intended to be opaque to the caller; clients should treat it as a single token and not attempt to parse or perform arithmetic on it.
    • Keyset vs. Offset: Unlike offsets, which can silently skip records if rows are inserted behind the current position, a keyset cursor uses the actual values (like timestamps and IDs) to resume the scan exactly where it left off.
    • Truncated Responses: When a result set is too large for a single gRPC message, the response is marked as Truncated and includes a NextCursor. The caller must use this NextCursor in their subsequent request to continue reading.
  12. Understand the Ontology Action Types model

    main

    CortexDB implements a 'governed write' model inspired by Palantir Foundry. Instead of allowing agents to perform arbitrary upserts, the system exposes a whitelist of OntologyActionTypes. This ensures all graph edits are auditable and follow strict business logic.

    Core Constraints

    1. Link Creation: create_link is strictly for many-to-many relationships. For one-to-many or one-to-one relationships, you must use modify_object to update the foreign key property.
    2. Atomicity/Ordering: modify_object or delete_object cannot reference an object that was just created within the same action.
    3. Validation Scope: Validation only checks parameters and submission criteria; it does not query existing data (e.g., it does not check for primary key uniqueness during the validation phase).

    Key Components

    • Action Types: A set of governed edits defined by parameters, rules, and submission criteria.
    • Rules: Define the actual edits (e.g., create_object, modify_object, delete_link).
    • Submission Criteria: Predicates that gate whether an action can be submitted based on parameter values (e.g., regex matching).
    • Value Sources: Determines where a rule gets its data (e.g., from a parameter, a static value, or current_time).
    // Example of an Action Type definition
    OntologyActionType{
        APIName:     "registerAirport",
        DisplayName: "Register Airport",
        Parameters: []OntologyActionParameter{
            {APIName: "iataCode", DataType: OntologyDataType{Kind: OntologyDataString}, Required: true},
            {APIName: "airportName", DataType: OntologyDataType{Kind: OntologyDataString}, Required: true},
        },
        Rules: []OntologyActionRule{
            {
                Kind:       ActionRuleCreateObject,
                ObjectType: "Airport",
                PropertyValues: map[string]OntologyValueSource{
                    "iataCode":    {Kind: ValueSourceParameter, Parameter: "iataCode"},
                    "airportName": {Kind: ValueSourceParameter, Parameter: "airportName"},
                },
            },
        },
        SubmissionCriteria: []OntologySubmissionCriterion{
            {
                Parameter:      "iataCode",
                Op:             PredicateContains,
                Regex:          "^[A-Z]{3}$",
                FailureMessage: "IATA code must be three uppercase letters.",
            },
        },
    }