DOCSEARCHbyAlgolia
Submit Docs

IotEdgeDB Documentation

website·Indexed 31 days ago

http://docs.lmgateway.com

A time-series database designed for IIoT, featuring a DuckDB SQL engine and Parquet-based storage. It supports high-performance data ingestion via MQTT, Telegraf, and REST APIs, and offers advanced data lifecycle management including continuous queries, tiered storage, and automatic compaction. IotEdgeDB provides enterprise features such as RBAC, audit logging, and high-availability clustering with Raft consensus and automatic failover.

Tokens
44K
Snippets
220
Records
298
Agent score
52%

What's inside IotEdgeDB

  2. Overview of IEDB Tiered Storage

    IEDB implements a two-tier storage model to reduce storage costs by 60-80%. Recent data is kept in a high-cost, low-latency 'Hot Storage' layer (local), while historical data is automatically migrated to a low-cost 'Cold Storage' layer (S3/MinIO) once it exceeds a configurable age threshold (default is 30 days). Key features include automatic lifecycle-based migration, per-database policy overrides, a Cron-based scheduler, manual migration triggers via API, zero-compression overhead during migration, and transparent querying across both layers.

  3. Overview of Query Governance

    Query Governance allows administrators to control resource usage by enforcing limits at the API Token level. This prevents runaway queries, ensures fair resource distribution among teams, and enables multi-tenant isolation or Service Level Agreements (SLAs) for service providers. Key enforcement areas include:

    • Rate Limits: Maximum queries per minute and hour.
    • Query Quotas: Maximum queries per hour and day.
    • Row Limits: Maximum number of rows returned per single query.
    • Custom Policies: Token-specific overrides for global defaults.
    • Usage Monitoring: Tracking current consumption and remaining quotas.

  4. Overview of IEDB Data Deletion

    IEDB supports precise data deletion using a rewrite-based method. This approach ensures zero overhead for subsequent write and query operations by physically removing data from Parquet files. The process involves identifying affected files via DuckDB, filtering out the rows matching the WHERE clause, writing the remaining data to temporary files, and performing an atomic replacement of the original files using os.replace(). Files that become empty after filtering are completely deleted.

  6. Overview of IEDB Role-Based Access Control (RBAC)

    IEDB uses a hierarchical RBAC model built on top of token-based authentication to manage access to deployments. The hierarchy is structured as: Organization $\rightarrow$ Team $\rightarrow$ Role $\rightarrow$ Database/Measurement Table.

    Key capabilities include:

    • Organizations: Highest level grouping (e.g., a company).
    • Teams: Functional groupings of users (e.g., Engineering, Analytics).
    • Roles: Define permissions for specific databases and optional measurement limits.
    • Table-level Permissions: Use wildcard patterns to restrict access to specific tables within a database.

  7. Overview of iedb-agent

    iedb-agent is a lightweight Rust service acting as an edge collection proxy for IotEdgeDB. It collects, buffers, and forwards time-series data from edge devices to an IotEdgeDB server. It is specifically designed for resource-constrained ARM devices (ARM32/ARM64) and unstable network environments, ensuring reliable data upload through local Write-Ahead Logging (WAL) and batching data into Parquet files before transmission.

  8. Overview of IEDB Automatic Compaction

    IEDB's automatic compaction system optimizes query performance (improving speed by 10-50x) by merging small Parquet files into larger, optimized files. This process reduces the number of S3/MinIO API calls, improves storage efficiency (up to 80.4% savings using ZSTD), and enables more effective partition pruning for the analysis engine. The system is enabled by default in production environments and runs non-blockingly, meaning queries can still function during compaction.

  9. Overview of IEDB Caching Layers

    IEDB utilizes three complementary internal cache layers to optimize repeated query performance, particularly for dashboards and monitoring scenarios.

    Cache LayerTTLPurposePerformance Gain
    SQL Transformation Cache60sCaches SQL → read_parquet() conversion49-104x acceleration
    Partition Path Cache60sCaches partition pruning results50-100ms per query
    Glob Cache30sCaches filesystem glob results5-10ms per query

  10. Overview of Telegraf integration with IotEdgeDB

    Telegraf is a plugin-driven metrics collection agent that can gather data from over 300 sources (systems, databases, cloud services, containers, IoT devices) and write them directly into IotEdgeDB's high-performance time-series storage. Key technical advantages include the use of MessagePack for efficient transmission, gzip compression to reduce bandwidth, and Parquet storage for open-format data analysis via SQL.

  11. Overview of IotEdgeDB for IIoT Analysis

    IotEdgeDB is a high-performance columnar database specifically designed for IIoT environments to handle massive volumes of multi-dimensional time-series data from sensors and devices. It utilizes columnar storage, high compression rates, and vectorized queries to provide real-time insights for industrial operations, product interaction analysis, system observability, and AI/ML feature storage.

  12. Understand IEDB's architectural performance drivers

    IEDB achieves high performance through three primary architectural choices:

    1. Query Engine: Uses DuckDB's columnar execution engine featuring vectorized execution, parallel query execution, SIMD instructions, and predicate/filter pushdown.
    2. Columnar Storage: Utilizes Parquet format with Snappy/ZSTD compression (reducing data size by ~80%) and predicate pushdown to skip irrelevant row groups.
    3. Go Runtime: Leverages Goroutines for high concurrency, a single binary deployment for zero dependencies, and efficient garbage collection for sub-millisecond pauses.