DOCSEARCHbyAlgolia
Submit Docs

InfluxDB 3 Core

repository·main·Indexed 12 days ago

https://github.com/influxdata/influxdb

A high-performance, open-source time series database built on Apache Arrow, DataFusion, and Parquet. It supports SQL and InfluxQL for real-time monitoring, analytics, and event processing. The ecosystem includes the influxdb_iox_client and influxdb2_client Rust libraries, a pure Rust parser for InfluxDB Line Protocol, and Catalog v3 for cluster metadata management.

Tokens
70.8K
Snippets
210
Records
308
Agent score
99%

What's inside InfluxDB

  1. Overview of Catalog v3

    main

    Catalog v3 is the single source of truth for InfluxDB cluster metadata, including databases, tables, columns, field families, caches, triggers, tokens, nodes, retention, and storage configuration.

    Key improvements over v2 include:

    • Forward compatibility: Uses version-gated records and a cluster-wide committed feature level to allow mixed-version clusters to upgrade without a coordinated all-node restart. An UPGRADE_SAFE flag allows older nodes to safely ignore newer records.
    • Persistence boundary: Separates frozen CatalogRecord types (on-disk format) from runtime types (in-memory API), allowing them to evolve independently.
    • Compact binary format: Uses bitcode-encoded record bodies in a fixed framing format, which is approximately 5–6× smaller than v2 JSON and avoids full-JSON deserialization during loading.

  2. Overview of oss/core shared infrastructure

    main
    The oss/core directory contains low-level, product-agnostic shared crates used across InfluxDB 3 Core, InfluxDB 3 Enterprise, and InfluxDB 3 IOx. These crates provide the foundational building blocks for the InfluxDB ecosystem, including query execution, schema definitions, data types, communication protocols (HTTP/gRPC), and observability tools. They also provide critical support for Arrow and Parquet infrastructure.

  3. Overview of InfluxDB 3 Core

    main

    InfluxDB 3 Core is an open-source time series database designed for real-time events, analytics, and monitoring. It is optimized for high-speed ingestion and low-latency queries (e.g., <10ms for last-value queries).

    Key technical specifications:

    • Storage Format: Apache Parquet on object storage (S3, Azure, GCP) or local disk.
    • Query Languages: SQL, InfluxQL, and Flight SQL.
    • Write Format: Line protocol.
    • API Interface: HTTP on port 8181.
    • Core Technologies: Built with Rust, Apache Arrow, and DataFusion.
    • Compatibility: Supports InfluxDB 1.x and 2.x write APIs, and the InfluxDB 1.x query API (InfluxQL).

  5. Supported Operating Systems and Architectures for InfluxDB 3

    main

    Official InfluxDB builds are portable across several platforms and use a consistent CPython distribution from python-build-standalone. Supported environments include:

    • Linux: amd64 and arm64 (available as tar.gz, deb, and rpm). Targets glibc 2.23+.
    • Darwin (macOS): arm64 (tar.gz).
    • Windows: amd64 (zip). Requires Windows 8 / Windows Server 2012 or newer.
    • Docker: Linux amd64 and arm64.

  6. Use the InfluxDB IOx Rust client

    main

    The influxdb_iox_client is a Rust library designed for connecting to InfluxDB IOx. It supports both HTTP-based APIs (such as the write API) and gRPC-based APIs. Note that the client is under active development and may not support 100% of all available APIs at all times.

    To write data to InfluxDB IOx, you must create a connection and a write client, then send data formatted in InfluxDB Line Protocol.

  8. How Catalog v3 handles writes and transactions

    main

    There are two primary paths for mutating the catalog:

    1. Op Path (Catalog::update::<Op>())

    Used for explicit DDL mutations such as creating/deleting databases, registering/stopping nodes, managing caches, triggers, tokens, retention, or configuration. This path acquires a write_permit, prepares the change, serializes it, and persists it to the object store. If a write fails due to an AlreadyExists error, the system retries by loading remote logs and re-preparing.

    2. Transaction Path

    Used for schema-on-write during line-protocol ingestion and for DDL table creation.

    • A DatabaseCatalogTransaction is opened via Catalog::begin_database_transaction().
    • Changes (like CreateTable or AddColumns) are accumulated through a TableTransaction.
    • TableTransaction enforces column, tag, and field-family limits.
    • On commit(), the accumulated RecordBatch is gated, serialized, persisted, applied, and broadcast. If the catalog sequence has advanced since the transaction began, the commit returns a retry prompt.

  9. How the InfluxDB 3 Processing Engine works

    main
    The InfluxDB 3 Processing engine is an embedded Python Virtual Machine (VM) used for running code within InfluxDB. It utilizes python-build-standalone to provide a portable, maintained, and permissively licensed Python runtime. This approach ensures a consistent experience across different operating systems and architectures without requiring users to manage a system-level Python installation that meets specific requirements.

  10. Understand the InfluxDB 3 Processing Engine

    main
    The InfluxDB 3 Processing engine is an embedded Python VM that allows you to run code directly inside the database to process and transform data. It is implemented using the pyo3 crate, which embeds Python into the InfluxDB binary. By default, the processing engine is enabled during the build process.

  11. Handle Catalog Restoration with RestoreCatalog records

    main

    The RestoreCatalog record (Enterprise ID 0x8006) is a special record used to restore the entire catalog state from a backup.

    Key characteristics:

    • Non-mutating: Applying this record does not mutate the InnerCatalog directly; instead, it loads a backup snapshot and log files from the paths provided in the record body.
    • Record Body: Contains { time_ns, restore_id, checkpoint_path, log_paths }.
    • Async Loading: Because loading involves async I/O against the object store, the apply driver uses a two-step process:
      1. Call preload_restore_for_records (async, off-lock) to pre-load the backup state.
      2. The synchronous apply_records then consumes the pre-loaded InnerCatalog when it encounters the record.
    • Lifecycle: The RestoreCatalog record is not retained in ordered_records. Catalog::restore forces a checkpoint immediately after application, so the next snapshot captures the restored state directly, and the backup paths in the persisted log are no longer needed for cold-starting peers.

  12. Understand the Feature Level and Forward Compatibility Mechanism

    main

    To prevent nodes from writing record types that other nodes cannot understand, InfluxDB v3 uses a Feature Level mechanism.

    Key Concepts

    • Feature Level: A FeatureLevel { core: u16, enterprise: u16 } representing the highest sequential record ID known to a binary. This is computed at startup.
    • Committed Feature Level: A cluster-wide value stored in InnerCatalog. It represents the highest feature level that all running nodes are guaranteed to understand.
    • Write-path Gate: Before persisting, the system checks if a record's ID exceeds the committed_feature_level. If it does (and it is not marked UPGRADE_SAFE), the write is rejected with RecordExceedsCommittedFeatureLevel.
    • UPGRADE_SAFE Flag: Records can be marked with RecordFlags::UPGRADE_SAFE if they add state that old code doesn't read and don't modify state that old code depends on. These records bypass the write-path gate during upgrade windows.

    Node Behavior

    • Startup: If a node's derived level is lower than the committed level, it fails immediately with NodeBelowCommittedFeatureLevel.
    • Advancement: When a node registers, the cluster evaluates the minimum feature level across all running nodes. If this minimum is higher than the committed level, an AdvanceFeatureLevel record is written to raise the cluster-wide level.