DOCSEARCHbyAlgolia
Submit Docs

Homer Telecom Observability Platform

repository·homer11·Indexed 24 days ago

https://github.com/sipcapture/homer

An all-in-one telecom observability platform for HEP (Homer Event Protocol) capture and API services. It utilizes a columnar data lake architecture with DuckDB, Apache Arrow, and Parquet for high-performance SIP and VQRTCP monitoring. The system consists of four main modules: Ingest, Storage (DuckLake), Node (Airport), and Coordinator. It supports tiered storage via RustFS, OTLP data ingestion, and customizable Lua scripts for call-id correlation and Loki label-shaping.

Tokens
106.7K
Snippets
170
Records
451
Agent score
84%

What's inside Homer

  1. Overview of the Homer MCP Module

    homer11

    The MCP (Model Context Protocol) module exposes Homer Core as an MCP server, allowing LLM clients (such as Cursor, Claude Desktop, or custom agents) to search HEP (Homer Event Processing) data using natural language.

    Key characteristics:

    • Transport: Uses stdio (standard input/output) for JSON-RPC communication.
    • Backend: Connects to the Homer Core Coordinator HTTP API (/api/v4/transactions/search, /api/v4/query).
    • Parsing: Uses a deterministic regex parser by default, which can be augmented by an OpenAI-compatible LLM for better natural language understanding.
    • Security: It is a read-only interface. A SQL validator ensures only SELECT or WITH statements are executed against the homer_lake.main.hep_proto_1_call table, blocking all destructive commands (e.g., DROP, DELETE, UPDATE).
    • Availability: The module is disabled by default and must be enabled via mcp.enable=true in the configuration.

  2. Overview of DuckLake Storage

    homer11

    DuckLake is a lakehouse-style storage backend for Homer HEP (Homer Event Packet) data. Unlike standard Arrow/Parquet storage, DuckLake provides advanced features including:

    • Time travel and Snapshots: Ability to query data at specific points in time.
    • ACID transactions: Ensures data integrity.
    • Multi-table architecture: Automatically routes packets to separate tables based on proto_type and SIP method.
    • Schema evolution: Handles changes to data structures automatically.
    • S3 support: Built-in support for cloud object storage.
    • Multi-writer support: Available when using PostgreSQL or MySQL as the catalog.

    It combines Parquet files for data storage with a SQL catalog database (DuckDB, PostgreSQL, MySQL, or SQLite) for metadata management.

  3. Overview of Chess and NetChess Dashboard Widgets

    homer11

    Homer includes two new dashboard widgets under the Games category:

    1. chess: A single-player widget where users play against a built-in minimax bot or an LLM opponent (if the MCP LLM backend is enabled).
    2. netchess: A two-player network chess widget. Unlike the netris relay model, NetChess uses the server as the authoritative game state, meaning the server validates all moves, manages clocks, and detects game outcomes (mate/draw) to prevent cheating and synchronization issues.

    Both widgets utilize chess.js on the UI and notnil/chess on the server, sharing a common ChessBoard.tsx presentational component.

  4. Homer Core MCP module overview

    homer11

    The Homer Core MCP (Model Context Protocol) module is a Go-based integration for homer-core. It allows LLMs to interact with Homer via natural language (NL) by translating queries into API calls or SQL. It is built on mark3labs/mcp-go and integrates with the writer, node, and coordinator stack.

    Available Tools

    • homer_search_transactions: Translates NL into a POST /api/v4/transactions/search request.
    • homer_query_sql: Translates NL into SQL, then executes via POST /api/v4/query.
    • homer_query: A hybrid mode that defaults to structured search but uses SQL if the query explicitly requests it.

  5. Quick links for Homer 11 documentation

    homer11

    The following topics are covered in the Homer 11 documentation suite:

  6. Review UI Parity Matrix for Homer v11

    homer11
    The UI Parity Matrix tracks the migration status of features from the legacy Homer UI to the new homer-ui (located in homer-core/src/ui). It categorizes features into Settings pages, Dashboard widgets, Transaction details, and Access control to identify which legacy components have been implemented, are partially implemented, or are still missing in the new UI.

  7. Operational notes for the OTLP Receiver

    homer11

    When working with the OpenTelemetry Protocol (OTLP) receiver in Homer, keep the following operational constraints and behaviors in mind:

    • No HEP transcoding: Homer does not convert OTLP spans into HEP type 100 LOGs. If you require OTLP data to appear in existing Homer SIP dashboards, you must perform the conversion in your client SDK or an OpenTelemetry Collector before sending it to Homer.
    • Data Source of Truth: The raw column is the source of truth. While schema columns provide convenient shortcuts for indexing and filtering, any rare attributes not promoted to top-level columns can always be recovered using json_extract(raw, '$. ...').
    • Manual Schema Migrations: Unlike the Line Protocol receiver, the OTLP receiver does not automatically extend schemas. If you need to add a new top-level column to the otlp_traces, otlp_metrics, or otlp_logs tables, you must perform a manual ALTER TABLE command.
    • Capacity Planning and Retention: Signal tables are partitioned by date. To manage data retention, use the standard DuckLake compaction TTL on the homer_lake.otlp_* tables.
    • Testing: End-to-end tests for the receiver are located in src/otlpreceiver/{http,grpc}_test.go and test both transport protocols against an in-process server.

  8. Use the otlp_synthetic tool to export OTLP data

    homer11

    The otlp_synthetic tool is a Go-based utility designed to generate and POST rich OTLP/HTTP exports (including traces, metrics, and logs) to an OTLP receiver, such as homer-core running on port 4318. It is useful for testing OTLP ingestion pipelines.

    Prerequisites

    • Go 1.22 or higher
    • A running OTLP HTTP endpoint (default: http://127.0.0.1:4318) supporting the following paths:
      • /v1/traces
      • /v1/metrics
      • /v1/logs
    cd examples/otlp_synthetic
make build          # binary: bin/otlp_synthetic
make dry-run        # marshal only, print payload sizes (no network)
make run            # POST JSON to http://127.0.0.1:4318

  9. Overview of the OTLP Receiver

    homer11

    The otlpreceiver.Module in homer-core provides a first-class ingest path for OpenTelemetry (OTLP) signals. It accepts traces, metrics, and logs via three canonical transports:

    • OTLP/gRPC on port :4317
    • OTLP/HTTP + protobuf on port :4318
    • OTLP/HTTP + JSON on port :4318

    Signals are stored in dedicated DuckLake tables (otlp_traces, otlp_metrics, and otlp_logs) and preserve full OTel fidelity (resource attributes, scope info, severity, etc.). Note that OTLP signals are not transcoded into HEPv3; they land directly in their own storage layout.

  10. Overview of Homer 11 Architecture

    homer11

    Homer 11 is an all-in-one HEP (Homer Event Protocol) capture and API server designed for telecom observability. The system functions as a data lake composed of four primary stages:

    1. Ingest: Receives and processes HEP packets.
    2. Storage (DuckLake): Manages data persistence using DuckDB.
    3. Node (FlightSQL): Provides a gRPC/HTTP interface for data access via Node.
    4. Coordinator: Provides a REST API for management and querying.

    Developers can use Homer 11 to ingest telecom signaling data, store it in a high-performance data lake, and query it via REST, gRPC, or SQL interfaces.

  11. Understand the Homer Server DuckLake storage layout

    homer11

    Homer Server uses DuckLake, a lakehouse format that combines Parquet files for data storage, a SQLite catalog for metadata (snapshots, schemas, file statistics), and Hive-style partitioning for efficient time-range queries.

    Directory Structure

    Data is typically stored under /data/homer/:

    • homer_catalog.sqlite: The DuckLake catalog containing metadata.
    • parquet/: The data directory containing partitioned Parquet files organized by schema (e.g., main/) and protocol type (e.g., hep_proto_1_call/).
    /data/homer/
├── homer_catalog.sqlite          # DuckLake catalog (metadata)
└── parquet/                    # Data directory
    └── main/                   # Schema "main"
        ├── hep_proto_1_call/           # SIP calls (INVITE, BYE, etc.)
        │   ├── date=2025-01-26/
        │   │   ├── data_00001.parquet
        │   │   └── data_00002.parquet
        │   └── date=2025-01-27/
        │       ├── data_00003.parquet
        │       └── data_00004.parquet
        ...

  12. How lazy payload hydration works in DuckLake

    homer11

    To prevent OOM caused by decompressing large payload and data_extra columns during wide-row searches, Homer uses Lazy Payload Hydration (enabled by default via storage.ducklake.search.lazy_payload: true).

    Instead of decompressing the entire scanned range, the search happens in two phases:

    1. Search/sort over a narrow projection: The engine filters, orders, and applies LIMIT using all columns except the heavy payload and data_extra blobs. This keeps memory usage flat.
    2. By-uuid hydration: Once the final LIMIT rows are identified, a bounded point-lookup (SELECT uuid, payload, data_extra ... WHERE uuid IN (...)) re-attaches the wide columns only for the specific rows being returned.

    This ensures that the heavy payload column is decompressed for at most LIMIT rows, rather than for every row in the scanned range. This applies to default full-row searches; custom select or group_by queries bypass this mechanism.