DOCSEARCHbyAlgolia
Submit Docs

KurrentDB Documentation

repository·master·Indexed 25 days ago

https://github.com/kurrent-io/kurrentdb

An event-native data platform and database designed for event-driven architectures. KurrentDB features an integrated streaming engine, gRPC and HTTP client protocols, and official SDKs for Python, Node.js, Java, .NET, Go, and Rust. Documentation covers server diagnostics, statistics monitoring, operational maintenance (scavenging, backup, and restore), and plugin configuration for Kontext RAG capabilities.

Tokens
119.2K
Snippets
275
Records
699
Agent score
86%

What's inside KurrentDB

  1. Overview of KurrentDB

    master

    KurrentDB is an event-native database designed for modern software applications and event-driven architectures. It features an integrated streaming engine to handle distributed messaging and ensure data consistency.

    Licensing: KurrentDB is licensed under Kurrent License v1 (KLv1). While the source code is available, KLv1 is not an OSI-approved Open Source License. Enterprise features can be enabled using a valid license key.

  3. Overview of KurrentDB AtomPub HTTP API

    master

    KurrentDB provides a native AtomPub interface over HTTP, compatible with the Atom Protocol (RFC 4287). It uses content type negotiation via the Accept header or URL parameters (e.g., ?format=xml) to serve serialized events.

    Warning: The AtomPub protocol is disabled by default and is slated for deprecation. For new applications, it is recommended to use the gRPC protocol (available since v20) for more reliable real-time event streaming.

    Supported Content Types

    For POST requests:

    • application/xml
    • application/vnd.eventstore.events+xml
    • application/json
    • application/vnd.kurrent.events+json
    • application/vnd.eventstore.events+json
    • text/xml

    For GET requests:

    • application/xml
    • application/atom+xml
    • application/json
    • application/vnd.kurrent.atom+json
    • application/vnd.eventstore.atom+json
    • text/xml
    • text/html
    • application/vnd.kurrent.streamdesc+json
    • application/vnd.eventstore.streamdesc+json

  4. Overview of KurrentDB.Projections.Management

    master
    KurrentDB.Projections.Management is the top-level orchestration layer for the KurrentDB projections subsystem. It is responsible for managing the lifecycle of projections, providing API endpoints, coordinating subsystem startup/shutdown, and managing worker distribution. Projections are designed to run only on the leader node, and the management layer handles leader election awareness.

  5. Overview of KurrentDB Connectors

    master

    KurrentDB Connectors are server-side components designed to integrate data from KurrentDB into external systems. They automate the process of creating data pipelines by eliminating the need for users to host and maintain custom subscription services.

    How Connectors Work:

    1. Subscription: A connector uses a catch-up subscription to receive events from KurrentDB streams.
    2. Filter/Transform: The connector can filter or transform the received events.
    3. Sink: The connector pushes the processed data to an external system via a sink.

    Key Benefits:

    • Reduced Maintenance: Removes the need to develop, host, and observe custom subscription infrastructure.
    • Automatic Checkpointing: Eliminates the manual management of checkpoints required by self-hosted solutions.
    • High Availability: By running natively within KurrentDB, connectors avoid the single-point-of-failure risks associated with single-instance self-hosted subscription services.

  6. Understand Projection Modes

    master

    KurrentDB supports three distinct modes for projections, which determine their lifecycle and persistence:

    • Transient: Ephemeral, ad-hoc queries with no persistence. These can be created and controlled by standard users.
    • OneTime: Batch projections that run once and stop at the end of the stream (EOF). They support optional checkpoints.
    • Continuous: Long-running, persistent projections that use checkpoints to track progress. These require Admin or Operations roles to manage.

  7. Authentication methods in KurrentDB

    master

    KurrentDB supports several authentication mechanisms to secure connections to the database:

    • Basic authentication: Enabled by default; uses usernames and passwords managed within KurrentDB.
    • User X.509 certificates: Authenticates users using X.509 certificates (used in addition to basic authentication).
    • LDAP authentication: Authenticates users against an LDAP server (e.g., OpenLDAP or Active Directory).
    • OAuth authentication: Authenticates users against an identity provider (e.g., Auth0 or Identity Server).

    Note: Authentication only verifies identity; users must still be authorized to perform specific actions.

  8. Understand KurrentDB Projections

    master

    Projections is a KurrentDB subsystem used to append new events or link existing events to streams in a reactive manner. They are specifically designed for 'temporal correlation queries'—complex queries that look for patterns or relationships between events over time (e.g., finding specific sequences of medical diagnoses or sensor readings).

    Key Requirements & Characteristics:

    • Data Format: The event body must be in JSON.
    • Continuous Querying: Projections can be configured to run once (returning current results) or to run continuously, finding new results as they happen and updating the result set. The output of a projection is a stream that can be listened to like any other stream.
    • Execution: Projections run exclusively on the leader node of a cluster to ensure consistency, which increases CPU and IO load on that node.

  9. Understand Connector Checkpointing

    master

    Connectors periodically store the position of the last successfully processed event in a dedicated system stream: $connectors/{connector-id}/checkpoints. This allows connectors to resume processing from where they left off after a restart.

    • Default Behavior: If no checkpoint exists, the connector starts from the latest position in the stream.
    • Custom Start Position: Use the initialPosition property in settings, or use the Start API with the position parameter to begin from a specific point.

  10. Understand the OpenTelemetry Collector Proto package structure

    master

    The OpenTelemetry Collector Proto package defines the protocols used for OpenTelemetry services. It is organized into four primary packages:

    • common: Contains shared messages used across different services.
    • trace: Contains the Trace Service protocol definitions.
    • metrics: Contains the Metrics Service protocol definitions.
    • logs: Contains the Logs Service protocol definitions.

  11. Understand the KurrentDB Event Processing Pipeline

    master

    KurrentDB processes projection events through a multi-stage pipeline designed for consistency and scalability. The flow follows these high-level steps:

    1. Event Reading: The EventReader reads from the Event Store (TF/Streams) and distributes events via ReaderSubscriptionMessage.CommittedEventDistributed.
    2. Routing: EventReaderCoreService routes to an IReaderSubscription, which performs filtering, tagging, and checkpoint suggestions.
    3. Dispatching: The ReaderSubscriptionDispatcher routes events to the correct subscriptionId.
    4. Work Item Creation: The EventSubscriptionBasedProjectionProcessingPhase creates a CommittedEventWorkItem and enqueues it to the CoreProjectionQueue.
    5. Staged Processing: Work items move through a StagedProcessingQueue in five stages:
      • Stage 0: RecordEventOrder: Ensures event ordering consistency.
      • Stage 1: GetStatePartition: Determines the partition (e.g., via StatePartitionSelector).
      • Stage 2: Load: Loads partition state from cache or the event store.
      • Stage 3: ProcessEvent: Executes the logic via IProjectionStateHandler (JS or native).
      • Stage 4: WriteOutput: Finalizes processing and writes results/emitted events via ResultWriter.

  12. Use User Defined Indexes in KurrentDB 26.0

    master
    KurrentDB 26.0 introduces user-defined secondary indexes. These allow you to define custom secondary indexes based on record content to enable fast, field-based reads, subscriptions, and UI queries (e.g., "orders-by-country"). Indexes are stored separately on each node and follow the log, providing targeted access without increasing the log size.