Apache Pulsar

The Pulsar Java Client V5 API is a new, non-breaking replacement for the existing pulsar-client-api. It is designed to support Scalable Topics (where topic segments are managed dynamically by the broker) and provides a cleaner, more type-safe interface by separating different consumption models into distinct consumer types.

Key improvements include:

• Opaque Topic Model: Removes partitioning details (like partition indexes) from the client API, treating topics as single endpoints.
• Purpose-built Consumers: Replaces the single Consumer interface with specialized types for streaming, queuing, and checkpointing.
• Modern Java Idioms: Uses Duration, Instant, Optional, and ServiceLoader instead of legacy patterns.
• Side-by-side Coexistence: The V5 API is shipped in new modules (pulsar-client-api-v5 and pulsar-client-v5) and can coexist with the existing v4 client in the same JVM.

<!-- Dependency structure -->
<dependency>
    <groupId>org.apache.pulsar</groupId>
    <artifactId>pulsar-client-api-v5</artifactId>
</dependency>
<dependency>
    <groupId>org.apache.pulsar</groupId>
    <artifactId>pulsar-client-v5</artifactId>
</dependency>

PIP-478 introduces three new Service Provider Interfaces (SPIs) to modernize the Pulsar Java client and resolve architectural issues in the v4 surface. This proposal is a breaking change that replaces older, synchronous, or poorly segregated interfaces with more robust, asynchronous, and purpose-driven ones.

Key changes include:

• v5 Authentication SPI: An asynchronous, capability-segregated replacement for the synchronous Authentication stub introduced in PIP-466. It addresses issues where the Netty event loop would block during token refreshes.
• PulsarTlsFactory SPI: A purpose-driven SPI for TLS material loading, replacing the older PulsarSslFactory (PIP-337).
• PulsarHttpClient SPI: A framework-managed SPI for HTTP operations.
• Security Defaults: Pulsar 5.0 TLS is now secure by default, enabling hostname verification and requiring Subject Alternative Names (SAN).

Note on Breaking Changes: This PIP removes the PIP-337 PulsarSslFactory entirely. Developers using custom SSL loading logic via PulsarSslFactory must migrate to the new PulsarTlsFactory SPI.

PIP-264 is a parent Proposal Improvement Plan (PIP) aimed at modernizing the Pulsar metrics system by consolidating it under OpenTelemetry (OTel).

Core Objectives

• Scalability: Enable monitoring of brokers with very large topic counts (10k, 100k, and potentially 1M+) by implementing internal aggregation and fine-grained filtering.
• Standardization: Replace fragmented metric libraries with a single, industry-standard implementation using OpenTelemetry.
• Improved Observability: Solve issues related to high cardinality, non-conformant histograms, and inconsistent naming conventions.
• Plugin & Function Support: Provide a consistent way for plugin authors and Pulsar Function developers to integrate and report custom metrics.

Key Design Shifts

• Aggregation & Filtering: To handle high cardinality, metrics will be aggregated into topic groups and filtered to prevent overwhelming monitoring systems.
• Metric Types: Moving from Summary types to Histogram types at the namespace and broker levels to improve visualization and compatibility with tools like Prometheus.
• Eliminating Delta Resets: Removing the practice of delta-resetting metrics to prevent data loss during scrapes.

The Pulsar Edge Component is designed to provide the following functional capabilities:

• Function Management: A REST interface to submit function packages (e.g., .jar files, .py files) and a REST interface to start, stop, create, and configure Pulsar functions.
• CLI Parity: Support for the same interface as the Pulsar CLI, including secure SSL/TLS communication for REST endpoints.
• Runtime & Execution: A dedicated runtime environment for executing Pulsar functions locally.
• Data Forwarding: The ability to forward messages to multiple topics on a downstream Pulsar cluster over TCP.
• Throughput Optimization: Support for message batching before forwarding to the Pulsar cluster to improve throughput in high-volume environments.
• Bidirectional Communication (Proposed): A communication back-channel allowing Pulsar clients to send instructions back to the Edge component (e.g., sending a command to a device after an alert is detected).

The migration to OpenTelemetry (OTel) aims to modernize Pulsar's observability with the following objectives:

• Cardinality Management: Introduce 'Topic Metric Groups' to allow monitoring 10k to 1M topics without high cardinality costs.
• Granular Filtering: Allow filtering metrics at the namespace, topic metric group, topic, consumer, producer, or subscription level.
• Standardization:
  • Replace Summary with explicit bucket Histograms.
  • Consolidate all metric usage into the OpenTelemetry library.
  • Adhere to OTel Semantic Conventions for naming.
  • Support the OTLP protocol for efficient exporting.
• Robustness:
  • Stop using delta resets (ensure counters only increase).
  • Eliminate static metric registries to prevent flaky tests.
  • Provide a rich, typed interface for Plugin and Function authors.
  • Ensure histogram reporting is Prometheus-conformant.

PIP-45 introduces a unified, pluggable interface designed to abstract all Pulsar metadata interactions. Currently, Pulsar relies heavily on ZooKeeper for metadata and coordination, with ZooKeeper client API calls scattered throughout the codebase. This interface consolidates those accesses into a single generic MetadataStore interface.

Key Characteristics:

• Backward Compatibility: The default implementation remains based on ZooKeeper, maintaining 100% compatibility with existing metadata locations and formats.
• Pluggable Backends: The architecture allows for multiple backend implementations, including:
  • ZooKeeper (Default)
  • Etcd
  • In-memory (for unit testing)
  • Local disk (for Pulsar standalone usage)
• API Status: The interface is considered "Beta" and may undergo breaking changes until multiple concrete implementations are stabilized. Consequently, it is initially treated as an internal Pulsar API and is not yet open for external user plugins.

The Scalable Topic Controller is a broker-side component introduced as part of the Scalable Topics initiative (PIP-460). It manages the lifecycle and topology of scalable topics, which are built on a Directed Acyclic Graph (DAG) of range segments that can be split or merged to scale independently.

The controller is responsible for:

• Segment lifecycle management: Creating, terminating, and deleting the underlying segment topics.
• Segment layout coordination: Executing split and merge operations with strict ordering guarantees to prevent message loss and ensure subscription cursors are present before producer redirection.
• Consumer assignment: Coordinating which consumers receive messages from specific segments for a given subscription.
• Leader election: Ensuring exactly one broker acts as the controller for each scalable topic with automatic failover.
• DAG watch sessions: Pushing topology updates to connected producers and consumers whenever the segment layout changes.