OntoCast

repository·main·Indexed 41 days ago

https://github.com/growgraph/ontocast

An agentic system for extracting RDF knowledge graphs from unstructured documents. OntoCast uses a parallel pipeline to co-evolve domain ontologies and fact graphs, ensuring schema and instance data alignment. It supports multiple triple store backends including Apache Fuseki and in-memory pyoxigraph, and optional vector stores like Qdrant or LanceDB. Version 0.6.4.

Tokens
71.7K
Snippets
170
Records
282
Agent score
50%

What's inside OntoCast

  1. Determine what makes a rendered unit acceptable

    main

    A render is considered acceptable if it has no material_defects(). The following signals affect acceptance:

    SignalBlocks Acceptance?
    Deterministic finding with mandatory=TrueAlways
    Critic TripleFix at or above FACTS_ACCEPT_BLOCKING_SEVERITY (default critical)Yes
    Critic TripleFix with action=REMOVENever (the repair contract forbids resolving findings by deleting statements)
    Advisory findings (numeric_coverage)No
    The critic's score / successNo (used for telemetry only)

    Configuration:

    • To let deterministic findings be the sole gatekeeper, set FACTS_ACCEPT_BLOCKING_SEVERITY=never.
  2. How OntoCast manages ontologies

    main

    OntoCast uses an OntologyManager to handle ontologies with automatic versioning and lineage tracking. Key behaviors include:

    • Identity: Catalog entries are keyed by the ontology IRI. Aliases like ontology_id and author Turtle prefixes (e.g., observation / obs) are supported.
    • Versioning: Uses Semantic Versioning (MAJOR/MINOR/PATCH) driven by change analysis. Changes produce new versions rather than in-place edits.
    • Lineage: Parent hashes are used to track ontology evolution.
    • Storage: Versions are stored as separate named graphs (using Fuseki or in-memory pyoxigraph).
    • Timestamps: created_at is synced as dcterms:created.
    • Contexts: When using multi-ontology prompt snapshots, OntoCast uses OntologySnapshot (containing the graph + source_iris) to avoid inventing a single catalog identity from a union graph.
    • Prefixes: During graph merges, conflicting prefixes are renamed rather than silently overridden.
  3. Configure the chunk preparation pipeline

    main

    The Chunk node follows a multi-step pipeline that can be configured via environment variables:

    1. Segment: Partitions text into sections.
      • CHUNK_SEGMENTER=semantic (Default): Uses the outline to partition markdown and splits text at section boundaries.
      • CHUNK_SEGMENTER=docling: Uses Docling HybridChunker structural segments.
    2. Coalesce: Merges undersized segments into neighbors without crossing section boundaries.
    3. Tag: Applies the classification cascade. Segments at or above CHUNK_SECTION_TAG_MIN_CHARS reach the LLM tier if enabled.
    4. Filter: Uses target_sections (allowlist) and exclude_sections (denylist) to prune chunks.
      • If exclude_sections is unset, the active schema's default_exclude applies.
      • To keep all sections, pass an explicit empty value to exclude_sections.
      • If filtering results in an empty graph, CHUNK_SECTION_FILTER_ON_EMPTY determines if the process warns (default) or errors (fails).
    5. Size: Splits oversized segments or merges consecutive same-label chunks to min_size / max_size.
  4. Understand the three layers of facts validation

    main

    OntoCast uses a three-layer validation strategy to ensure that LLM-rendered facts are correct. The system prioritizes deterministic machine repairs over expensive LLM calls.

    LayerNameLocationLLM Cost
    1Machine repairagent/render_facts.py::_normalize_and_repair_graphNone (performed at parse time on every graph)
    2Critic passstategraph/atomic.py::run_unit_loopOne per FACTS_CRITIC_PASSES (the critique itself costs a call)
    3Post-merge gateVALIDATE_FACTS nodeNone (performed once per document)

    Key distinction between retry settings:

    • MAX_VISITS: Retries a render that failed entirely. Use this only if renders are failing outright.
    • FACTS_CRITIC_PASSES: Performs review-and-patch cycles. Each pass runs deterministic checks for free, sends findings to the critic, and applies the fix. Set FACTS_CRITIC_PASSES=0 to limit extraction to exactly one LLM call per unit.
  5. How SHACL shapes are stored and managed

    main

    Shapes are stored as a deployment artifact in a dedicated tenancy partition ({tenant}--{project}--shapes) within the triple store. This keeps them structurally separate from ontologies to prevent shapes from being indexed as first-class schema atoms.

    Sources of shapes:

    1. The shapes partition: Seeded at startup from the FACTS_SHAPES_DIR. This partition is mutable via the [/shapes] API.
    2. The ontology context: Shapes can be embedded inline within the ontology itself using sh:NodeShape declarations (the zero-config path).

    Addressing shapes:

    • Shapes are addressed by the ontology IRI they declare. Re-uploading a shape replaces the existing one.
    • Shapes without a header are named using their seed path (urn:shapes:<relative/path.ttl>) or their filename, ensuring stable replacement rather than accumulation of stale copies.

    Flushing shapes: By default, POST /flush retains the shapes partition. To drop shapes along with facts and ontologies, use the query parameter ?include_shapes=true.

  6. View the OntoCast configuration hierarchy

    main

    The configuration is structured as a nested hierarchy. The root Config object contains a tool_config section (of type ToolConfig), a server section (of type ServerConfig), a logging_level string, and a clean boolean. The tool_config is further subdivided into specialized configurations for various components such as llm_config, chunk_config, embedding, vector_store, and more.

    Config
    ├── tool_config: ToolConfig
    │   ├── llm_config: LLMConfig
    │   ├── chunk_config: ChunkConfig
    │   ├── converter_config: ConverterConfig
    │   ├── path_config: PathConfig
    │   ├── fuseki: FusekiConfig
    │   ├── domain: DomainConfig
    │   ├── web_search: WebSearchConfig
    │   ├── aggregation: AggregationConfig
    │   ├── embedding: EmbeddingConfig
    │   ├── patch_retrieval: PatchRetrievalConfig
    │   ├── vector_store: VectorStoreConfig
    │   ├── qdrant: QdrantConfig
    │   ├── lancedb: LanceDBConfig
    │   └── facts_validation: FactsValidationConfig
    ├── server: ServerConfig
    ├── logging_level: str | None
    └── clean: bool
  7. Understand document schemas and the partition model

    main

    Labels are relative to a schema. Choosing the correct schema is critical for accurate extraction. OntoCast uses a partition model where every document belongs to exactly one exclusive cell defined by a document_profile.

    | Cell | Description | |---|---|academic | Scholarly reporting of original experiments or reviews. | | financial | Corporate disclosure of results to shareholders/regulators. | | clinical | Protocols or reports governing human participant treatment. | | legal | Binding agreements between parties. | | patent | Invention disclosures with numbered claims. | | standard | Normative requirements for implementers. | | manual | Instructions for a user operating a system. | | fiction | Narrative prose with characters and a story. | | news | Announcements of recent events with datelines/quotes. | | general | The residual cell for everything else (cannot be inferred). |

    Schema Precedence (from highest to lowest):

    1. Explicit section_schema_id provided by the caller.
    2. document_type_hint (e.g., 10-Q resolves to financial).
    3. Automatic detection via CHUNK_SECTION_SCHEMA_DETECT (Lexical $\rightarrow$ Semantic $\rightarrow$ Content).
    4. Manifest default (academic).
  8. Manage Ontology Context Size with Triple Limits

    main

    The ONTOLOGY_CONTEXT_MAX_TRIPLES environment variable is the primary knob for controlling context blow-up. It limits how many triples are serialized into each prompt.

    Behavior by Mode:

    • selected_single_ontology / fixed_single_ontology: Bounded by ONTOLOGY_CONTEXT_MAX_TRIPLES.
    • selected_vector_search_ontology: Bounded first by VECTOR_STORE_INDUCED_SUBGRAPH_MAX_TOTAL_TRIPLES (default 1200), then by ONTOLOGY_CONTEXT_MAX_TRIPLES as a backstop.
    • Facts prompts: Bounded by ONTOLOGY_CONTEXT_MAX_TRIPLES.

    How Triples are Dropped (Best-Effort): If the budget is exceeded, triples are dropped in this order:

    1. Header and RDF-list noise (owl:versionInfo, owl:imports, etc.).
    2. Redundant structure (generic rdf:type, stub/orphaned blank nodes).
    3. Glosses (rdfs:comment, skos:definition, etc.).

    Note: rdfs:label, rdf:type, and hierarchy triples are never dropped. If the graph still exceeds the budget after these steps, a WARNING is issued. If you see this warning, consider splitting your catalog or switching to selected_vector_search_ontology rather than simply increasing the limit.

    ONTOLOGY_CONTEXT_MAX_TRIPLES=500
  9. Understand the API response format

    main

    When processing a document via the API, the server returns a JSON object containing the extracted facts and ontology artifacts in Turtle (TTL) format.

    {
        "data": {
            "facts": "# facts in turtle format",
            "ontology_artifacts": [
                {"iri": "https://...", "title": "...", "ttl": "# ontology update in turtle"}
            ]
        }
    }
  10. Understand Ontology Identity and Lifecycle

    main

    Ontologies in OntoCast are identified by several keys and follow a specific lifecycle for updates:

    Identity Keys

    • Canonical catalog key: The ontology IRI (owl:Ontology subject).
    • ontology_id and prefix: Aliases registered during ingestion (e.g., lifecycle vs life).
    • OntologySnapshot: A view of catalog graphs or stitched ensembles. These have source_iris / writable_iris provenance but no catalog ID. They are used for prompting and are never registered in the catalog.

    The Update Lifecycle

    The process follows an Assemble → Propose → Apply pattern:

    1. Assemble: Catalog ontologies $O^*$ $\rightarrow$ OntologySnapshot $S$.
    2. Propose: $(S, \text{text}) \rightarrow$ GraphUpdate $U$ (Note: $U$ must contain only complements; do not restate $S$).
    3. Apply: $(U, O^) \rightarrow$ versioned catalog ontologies $O^{'}$.

    Writeback Behavior

    When performing writebacks ($U \rightarrow O^*$), complement inserts ($U \setminus S$) are partitioned by namespace ownership onto writable catalog IRIs and merged onto each ontology's freshest terminal.

  11. Understand how LLM cache keys are generated

    main

    Cache keys are generated by hashing the normalized prompt text along with all parameters that influence the provider's response. A change in any of these will result in a cache miss:

    • Provider, model name, temperature, and base URL.
    • Ollama generation knobs: think, num_predict, num_ctx.
    • Cloud reasoning knobs: reasoning_effort and thinking_budget (only when set; unset values use provider defaults).
    • The output schema name (for structured extract calls).
    • The cache_format_version constant (used to invalidate old cache entries when the entry shape changes).

    Binary inputs (like PDFs) are hashed as raw bytes.

  12. Use the GraphUpdate system for incremental modifications

    main

    To save tokens during LLM interactions, OntoCast uses a GraphUpdate system instead of regenerating full graphs.

    • Mechanism: The LLM outputs a GraphUpdate containing ordered TripleOp insert/delete patches.
    • Formats: Supports compact JSON-LD by default, or Turtle strings via the LLM_GRAPH_FORMAT environment variable.
    • Execution: At runtime, these patches are compiled into rdflib UPDATE queries to apply changes to the graph.