DOCSEARCHbyAlgolia
Submit Docs

kaelio/ktx

repository·main·Indexed 23 days ago

https://github.com/kaelio/ktx

A compute package and CLI for the ktx ecosystem providing semantic layer operations, database introspection, LookML parsing, and embedding computations. It features connection-centric ingestion for data sources including MongoDB (context-only) and Postgres (supporting query-history ingest via pg_stat_statements), as well as Orbit-style relationship discovery.

Tokens
147K
Snippets
227
Records
741
Agent score
79%

What's inside ktx

  1. What is ktx?

    main

    ktx is an open-source, self-improving context layer designed for data agents. It transforms warehouse metadata, BI tool definitions, query history, documentation, and approved metric definitions into reviewable, version-controlled files (YAML and Markdown) that agents can search and execute.

    Key capabilities include:

    • Context as code: Semantic-layer definitions and wiki pages are managed as git-based files for review and merging.
    • Self-improving ingest: Automatically reconciles new evidence from data stacks with accepted context.
    • Executable semantics: Allows agents to use approved measures, joins, filters, dimensions, and segments instead of writing raw SQL.
    • Agent-native access: Provides CLI and MCP (Model Context Protocol) tools for searching context, compiling semantic queries, and running read-only SQL.

  2. What is ktx and how does it work?

    main

    ktx is a self-improving context layer designed to teach data agents how to query your warehouse accurately. It automates the creation of warehouse context by ingesting approved metric definitions, joinable columns, and business knowledge.

    Core Capabilities

    • Learns from company knowledge: Ingests and organizes wiki content, removing duplicates and flagging contradictions.
    • Maps the data stack: Samples tables, captures metadata, and detects joinable columns to help agents write better queries.
    • Builds a semantic layer: Combines raw tables and high-level metrics through a join graph that resolves chasm and fan traps, allowing agents to fetch metrics declaratively.
    • Serves agents at execution: Exposes CLI and MCP (Model Context Protocol) tools for combined full-text and semantic search.

    Key Characteristics

    • Read-only by design: ktx connections are read-only; it never writes to your database.
    • Local execution: ktx runs locally. The only data leaving your machine is what you send to your configured LLM provider.
    • Unified surface: It ingests existing layers (like dbt or MetricFlow) and combines them with raw-table introspection and wiki content into a single searchable surface.

  3. Use ktx-daemon in one-shot mode

    main

    The ktx-daemon is a portable Python compute package for ktx. It can be used in 'one-shot' mode, where you pipe a JSON payload into a specific command via stdin. This mode is used by default by the @kaelio/ktx CLI.

    Available one-shot commands include:

    • semantic-query: Perform semantic queries.
    • semantic-generate-sources: Generate semantic-layer sources from schema scan data.
    • database-introspect: Introspect a Postgres database schema.
    • lookml-parse: Parse LookML projects into resolved, KSL-ready structures.
    • embedding-compute: Compute a single text embedding.
    • embedding-compute-bulk: Compute multiple text embeddings in bulk.
    • code-execute: Execute Python code within the current in-process boundary.

  4. Use `ktx admin` for low-level project management

    main

    ktx admin is a CLI tool for low-level project initialization, managing the bundled Python runtime, and managing local search indexes.

    Note: Most users should use ktx setup for standard workflows. Use ktx admin specifically for:

    • Preparing local fixtures
    • Checking the bundled Python runtime
    • Rebuilding local indexes
    • Debugging runtime state
    ktx admin <subcommand> [options]

  5. Understand Orbit's product workflow and target audience

    main

    Orbit is procurement workflow and spend-control software designed to route purchase requests, collect approvals, onboard suppliers, and issue purchase orders.

    Primary Buyers: Finance, Procurement, and Business Operations. Daily Users: Department admins, office managers, IT leads, and legal ops partners.

    Core Workflow Steps:

    1. Request: Requester submits a purchase request.
    2. Approval: Routing collects the necessary decisions.
    3. Onboarding: Supplier invite and onboarding occur before work begins.
    4. PO Creation: A purchase order is generated from the approved request.
    5. Renewal: Handoff processes manage the ongoing relationship.

  6. Compare Agent Client Integrations

    main

    ktx supports several AI agent clients, each with different methods for tool access (MCP), analytics skills, and rule/instruction configuration. Use the following comparison to determine the best integration for your workflow:

    ClientMCP toolsAnalytics skillAdmin CLI skillsGlobal installRule/Instruction fileSkill file
    Claude CodeYes.claude/skills/ktx-analytics/SKILL.mdOptionalYes.claude/rules/ktx.md.claude/skills/ktx/SKILL.md
    Claude DesktopLocal stdio via claude_desktop_config.jsonUpload .ktx/agents/claude/ktx-analytics.zipOptional .ktx/agents/claude/ktx.zipClaude Desktop configNot separatektx/SKILL.md inside ktx.zip
    CursorYes.cursor/rules/ktx-analytics.mdcOptional (.mdc)No.cursor/rules/ktx.mdcNot separate
    CodexSnippet.agents/skills/ktx-analytics/SKILL.mdOptionalYes.codex/instructions/ktx.md.agents/skills/ktx/SKILL.md
    OpenCodeSnippet.opencode/commands/ktx-analytics.mdOptionalNo.opencode/commands/ktx.mdNot separate

  8. What is the ktx context layer?

    main

    The ktx context layer is a trusted knowledge surface that sits between your data stack and AI agents. It provides the business logic and semantic meaning that a raw database connection lacks, such as canonical metrics, safe join paths, and business definitions (e.g., what constitutes an "active customer").

    Unlike schema-only access, the context layer acts as a contract that prevents agents from generating plausible but incorrect SQL by defining:

    • Which tables are the source of truth for specific metrics.
    • Which joins are safe versus those that cause measure fanout.
    • How to handle specific data nuances (e.g., excluding test accounts or handling refunds).
    • The board-approved definitions for key business terms like ARR.

  9. What is Wiki Capture and when to use it

    main

    Wiki Capture is ktx's knowledge base for storing durable, reusable business knowledge. It is used to capture rules, definitions, and conventions that answer "how should this concept be handled in this organization?" so they can be reused across different chat sessions.

    What to capture

    • Metric definitions: e.g., "revenue means booked revenue after refunds".
    • Filters or conventions: e.g., "exclude test accounts when reporting ARR".
    • Mappings or aliases: e.g., "mood_stress_sleep = Oxytocin protocol".
    • Domain rules: e.g., "status = 'T' means terminated".
    • External system conventions: e.g., primary key mappings for external EMRs.

    What NOT to capture

    • One-off or temporary instructions: Requests scoped only to the current chat.
    • Information already in the semantic layer: Column names, join paths, or measure formulas belong in the Semantic Layer (SL), not the wiki.
    • Query results or snapshots: Do not capture specific numbers or time-bounded benchmarks (e.g., "Oct 2025: 25%") as they become stale. Instead, capture the rule and use sl_refs to point to the live data source.
    • Interpretive narratives: Avoid commentary tied to a specific data snapshot.

  10. Understand the semantic layer directory structure

    main

    The ktx planner relies on structured YAML files organized by connection to build the semantic context. The following directories are used to manage this data:

    • raw-sources/<connection>/: Contains scan evidence from your warehouse, including schemas, columns, keys, samples, and observed usage patterns.
    • wiki/: Contains business language, definitions, and caveats. While the planner doesn't read this at compile time, AI agents use it to anchor measure names and dimensions to team-specific terminology.
    • semantic-layer/<connection>/: Holds the structured sources, joins, grain, measures, and segments that the planner actually compiles against.

  11. Handling derived metrics spanning multiple semantic_models

    main

    The ktx Semantic Layer does not natively support cross-source references for derived metrics. If a MetricFlow derived metric spans two different semantic_models (e.g., a margin metric using revenue from sales and cost from costs), you must use one of two strategies:

    1. Domain Ownership: Pick the source that 'owns' the metric domain and write the derived measure there, assuming the operands are resolvable in that context.
    2. SQL Standalone Fallback: Create a new standalone ktx object with source_type: sql. This object should perform the necessary JOIN between the two tables to bring the required columns into a single context, then define the measures based on that joined result.

    When using the SQL fallback, it is recommended to document the cross-source origin in a wiki page.

    # <connId>/margin.yaml - standalone because it spans two tables
# <!-- from: .../models/sales.yml#L1-8 -->
# <!-- from: .../models/costs.yml#L1-8 -->
# <!-- from: .../metrics/margin.yml#L1-8 -->
name: margin
source_type: sql
sql: |
  SELECT s.period_id, s.revenue_cents, COALESCE(c.cost_cents, 0) AS cost_cents
  FROM analytics.sales s
  LEFT JOIN analytics.costs c ON c.period_id = s.period_id
grain: [period_id]
columns:
  - {name: period_id, type: string}
  - {name: revenue_cents, type: number}
  - {name: cost_cents, type: number}
measures:
  - {name: revenue, expr: "sum(revenue_cents)"}
  - {name: cost, expr: "sum(cost_cents)"}
  - {name: margin, expr: "sum(revenue_cents) - sum(cost_cents)"}

  12. Capture durable business knowledge from Notion

    main

    When using notion_synthesize, focus on capturing reusable company knowledge while skipping transient noise.

    What to Capture

    • Definitions: Metric formulas, KPI definitions, and named business concepts.
    • Processes: Workflows, policies, ownership rules, and approval conventions.
    • Data Mapping: Source-of-truth mappings, cross-system aliases (e.g., connecting Notion terms to dbt or Looker names), and table/column descriptions.
    • Context: Caveats, conflicts, and product assumptions.

    What to Skip

    • Meeting notes without reusable rules.
    • Task lists and project status updates.
    • Transient announcements or time-bounded snapshots.
    • Duplicate documentation that adds no new facts.