Nakadi Documentation

Nakadi is a distributed event bus broker that provides a RESTful API abstraction over Kafka-like queues. It is designed to decouple microservices by allowing them to send, receive, and analyze streaming data in real time using reliable and highly available data streams.

Key features include:

• REST Abstraction: Decouples services from specific broker technologies (like Kafka).
• Schema Registry: Ensures data quality by validating events against defined schemas.
• High-Level Subscription API: Supports automatic partition redistribution and server-side cursor management via commits.
• Security: Supports OAuth2 authentication and per-event type authorization.
• Low Latency: Supports push-based consumption via streaming HTTP connections.

Both are API-based brokers built on Kafka, but they differ in delivery and schema support:

• Delivery Mechanism: Hermes uses webhooks (callback URLs) to deliver messages. Nakadi maintains a streaming connection to push events.
• Ordering: Nakadi guarantees delivery in arrival order for each partition (similar to Kafka). Hermes' ordering guarantees are not explicitly defined in its API.
• Schema Support: Hermes supports both JSON Schema and Avro. Nakadi currently only supports JSON Schema.
• Event Types: Nakadi provides inbuilt event types for business processes; Hermes does not.
• Client Support: Hermes provides a Java client driver; Nakadi does not ship with a built-in client.

When performing repartitioning in Nakadi, be aware of the following side effects and limitations:

• Partition Mapping: If using a hash partition strategy, publishing events to the event type will change the partitions in which they previously appeared.
• Ordering Guarantees: Nakadi guarantees ordering per partition per batch. Increasing the partition count from 1 to more will break the total order of events.
• Directional Limit: Repartitioning only allows you to increase the number of partitions.
• Subscription Impact: All consuming subscriptions are disconnected once the repartitioning process is finished.

Both Nakadi and Google Cloud Pub/Sub provide HTTP APIs suitable for microservices backplanes, but they differ in delivery and state management:

• Acknowledgment Model: Pub/Sub allows acknowledging every message individually. Nakadi uses a logical log model where consumers checkpoint their position (typically per batch).
• Subscription Management: Pub/Sub requires a subscription to be set up before consumption to manage delivery state. In Nakadi, consumers are currently expected to manage their own offsets.
• Delivery Model: Pub/Sub uses a polling model (consumers request pages of messages). Nakadi maintains a streaming connection and pushes events to consumers as they arrive.

Nakadi publishes events in the order they appear in the posted array. It does not re-order events based on data properties.

Partial Ordering (Per Entity)

To ensure events for a specific entity (e.g., a specific user or order) are ordered, configure the event type with the hash partitioning strategy and specify the fields used to construct the key. This provides partial ordering for that key.

Total Ordering

Total ordering is generally not achievable unless the partition size is set to 1. However, setting partition size to 1 is discouraged as it limits scalability and can cause cluster hot spotting.

Nakadi groups events into batch responses. Each batch is emitted on a single line in the stream.

A batch object contains:

• cursor: An object describing the partition and the offset for this specific batch. This allows clients to checkpoint their position.
• events: An array of event objects. These are ordered by arrival time. Note that Nakadi may regroup events from producers into different batches than those originally sent by the producer.

Important: Individual events do not have cursors; cursors are properties of the batch.

{
  "cursor": {
    "partition": "0",
    "offset": "4"
  },
  "events": [...]
}

Nakadi constrains all events on the platform to one of three specific categories via its HTTP API. When defining an Event Type, you must categorize it into one of the following:

• Data Change Event: Represents a change to a record or other entity.
• Business Event: An event that is part of, or drives, a business process.
• Undefined Event: Used for other types of events where schema validation is still required but doesn't fit the other two categories.

Every event sent to a stream is validated against the effective schema for that event type's category.

Validation behavior depends on the category:

• Category-specific requirements: For example, events in the business category must include a metadata object containing eid and occurred_at fields, in addition to the fields defined in the event type's original schema.
• Rejection: If an event does not conform to the effective schema, Nakadi will reject it. If it is valid, it is placed into a partition for consumers.