DOCSEARCHbyAlgolia
Submit Docs

Apache Pinot Documentation

repository·master·Indexed 27 days ago

https://github.com/apache/pinot

Apache Pinot is a real-time distributed OLAP datastore designed for low-latency, high-concurrency analytics on massive datasets, supporting both streaming and batch ingestion. This documentation covers building and deploying Pinot via Docker, configuring Pinot-Superset, exposing metrics to Prometheus via JMX Exporter, and performing compatibility regression testing.

Tokens
67.7K
Snippets
107
Records
323
Agent score
86%

What's inside Apache Pinot

  1. Overview of the Pinot connectors module

    master

    The pinot-connectors module is the centralized location for implementing Apache Pinot connectors for various streaming sources. Each specific stream implementation (e.g., Kafka 3.0) resides in its own sub-module within this directory.

    When developing or extending connectors:

    • Shared dependencies across all connector sub-modules should be defined in pinot-connectors/pom.xml.
    • Dependencies specific to a single connector should be defined in that connector's individual pom.xml.

  2. Overview of Apache Pinot

    master
    Apache Pinot is a real-time distributed OLAP (Online Analytical Processing) datastore designed for low-latency, scalable real-time analytics. It supports ingestion from both batch data sources (Hadoop HDFS, Amazon S3, Azure ADLS, Google Cloud Storage) and streaming sources (Apache Kafka, Apache Pulsar, AWS Kinesis).

  3. Key Features of Apache Pinot

    master

    Apache Pinot provides several core capabilities for high-performance analytics:

    • Fast Queries: P90 latencies in the tens of milliseconds for petabyte-scale datasets.
    • High Concurrency: Capable of serving hundreds of thousands of concurrent queries per second.
    • SQL Query Interface: Standard SQL accessible via a built-in query editor and REST API.
    • Versatile Joins: Supports fact/dimension and fact/fact joins.
    • Column-oriented Storage: Uses various compression schemes like Run Length and Fixed Bit Length.
    • Pluggable Indexing: Supports timestamp, inverted, StarTree, Bloom filter, range, text, JSON, and geospatial indexes.
    • Real-time & Batch Ingestion: Combines streaming (Kafka, Pulsar, Kinesis) and batch (Hadoop, Spark, S3) sources into single tables.
    • Upsert Support: Enables at-scale data updates during real-time ingestion.
    • Multitenancy: Isolated logical namespaces for secure resource management.
    • Cloud-native: Supports horizontal scaling and fault tolerance, with Helm charts for Kubernetes deployment.

  4. Understand the Pinot Materialized View (MV) design and runtime contract

    master

    Apache Pinot's Materialized View subsystem currently supports time-windowed MVs. The system relies on two primary metadata types and a specific partitioning model:

    Metadata Types

    • MaterializedViewDefinitionMetadata: Persisted under /CONFIGS/MATERIALIZED_VIEW/DEFINITION/<viewTableNameWithType>. Contains user-supplied SQL, source-table references, source partition expressions, rewriteEnabled, and stalenessThresholdMs. This is immutable after creation, except for the rewrite flag and SLO knobs.
    • MaterializedViewRuntimeMetadata: Persisted under /CONFIGS/MATERIALIZED_VIEW/RUNTIME/<viewTableNameWithType>. Contains watermarkMs and a Map<Long, PartitionInfo> keyed by bucketStartMs. This is updated by the minion executor and the MaterializedViewConsistencyManager.

    Partitioning and Constraints

    • Time Buckets: MVs use uniform-width time buckets defined by bucketTimePeriod in the task configuration.
    • Time Column Requirement: The MV's designated time column must be either an identity passthrough of the base time column or a DATETRUNC where the unit matches the bucket width. This is enforced by TimeExprValidator during creation.
    • Unsupported Base Tables: To ensure correctness, upsert, dedup, dim, and REFRESH base tables are rejected by the analyzer.

  5. Join the Apache Pinot community

    master

    You can engage with the Apache Pinot community through Slack, mailing lists, or local meetup groups.

    Slack

    Mailing Lists

    • dev-subscribe@pinot.apache.org: Subscribe to the pinot-dev mailing list.
    • dev@pinot.apache.org: Post to the pinot-dev mailing list.
    • users-subscribe@pinot.apache.org: Subscribe to the pinot-user mailing list.
    • users@pinot.apache.org: Post to the pinot-user mailing list.

    Meetups

  6. Supported Materialized View Partition Shapes

    master

    Apache Pinot Materialized Views (MVs) support three distinct partition shapes for data aggregation and rollups:

    • TIME: Time-windowed buckets keyed by bucketStartMs. Use this for time-series rollups, such as per-day-per-carrier flight data.
    • CATEGORICAL: N fixed buckets keyed by a declared partition column value. Use this for dimension-based rollups, such as per-tenant rollups where each bucket corresponds to a tenant_id.
    • SINGLETON: Exactly one bucket representing a full-table aggregate. This is a degenerate case of CATEGORICAL with a single fixed key. Use this for global aggregates, such as Top-K customers across an entire warehouse.

  7. QuickStart: Run Pinot stack with Docker Compose

    master

    The provided docker-compose.yml sets up a full Pinot stack including ZooKeeper, Kafka (in KRaft mode), Controller, Broker, and Server.

    To start the stack:

    docker-compose -f docker-compose.yml up

    To create a table and load data from Kafka (Example: airlineStats):

    1. Create the table:
    docker run --network=pinot_default apachepinot/pinot:latest AddTable -schemaFile examples/stream/airlineStats/airlineStats_schema.json -tableConfigFile examples/stream/airlineStats/docker/airlineStats_realtime_table_config.json -controllerHost pinot-controller -controllerPort 9000 -exec
    1. Ingest data into Kafka:
    docker run --network=pinot_default apachepinot/pinot:latest StreamAvroIntoKafka -avroFile examples/stream/airlineStats/rawdata/airlineStats_data.avro -kafkaTopic flights-realtime -kafkaBrokerList kafka:9092

    Accessing Pinot: Open http://localhost:9000 in your browser to query.

  8. Quickstart Pinot on Kubernetes using Helm

    master

    To run Apache Pinot on Kubernetes using Helm charts, follow the official Kubernetes quickstart guide. This guide provides the necessary steps to deploy Pinot components into a Kubernetes cluster.

    https://docs.pinot.apache.org/basics/getting-started/kubernetes-quickstart

  9. Concurrency State Management Principles

    master

    When working with Apache Pinot's state management and concurrency, adhere to these core principles to prevent data races and atomicity violations:

    • Atomic Transitions: Never wipe old metadata before the new state is durably installed. Use the pattern: prepare-new $\rightarrow$ swap-reference $\rightarrow$ cleanup-old. Avoid eager deletes.
    • Version-Checked Writes: All shared state in ZooKeeper (such as IdealState, IdealStateConfig, TableConfig, and Schema) must be written using optimistic locking via the ZK node version. Reject blind writes.
    • Atomic Operations over Check-then-Act: Avoid racy patterns like if (!map.containsKey(k)) map.put(k, v). Instead, use atomic methods like putIfAbsent or computeIfAbsent on concurrent maps.
    • Thread-Safety Conservatism: Default to explicit synchronization. When using AtomicReference or ConcurrentHashMap.compute, ensure the visibility story is verified across all callers.
    • Idempotent Observers: Handlers for shared observers (e.g., MetricsRegistry, segment lifecycle listeners) must be idempotent and their mutable state must be published safely.
    • Executor Lifecycle: Every ExecutorService must have a clear shutdown path (e.g., calling awaitTermination followed by shutdownNow) within its close() or stop hook.

  10. Quick Start with Spark-Pinot Connector

    master

    Use the Spark-Pinot connector to read data from Pinot realtime, offline, or hybrid tables. The connector supports distributed parallel scans, SQL support, and column/filter pushdown.

    To read a table, use the pinot format and specify the table and tableType (e.g., offline, realtime, or hybrid).

    import org.apache.spark.sql.SparkSession

val spark: SparkSession = SparkSession
      .builder()
      .appName("spark-pinot-connector-test")
      .master("local")
      .getOrCreate()

import spark.implicits._

val data = spark.read
  .format("pinot")
  .option("table", "airlineStats")
  .option("tableType", "offline")
  .load()
  .filter($"DestStateName" === "Florida")

data.show(100)

  11. Analyze interleavings when changing lock granularity

    master

    When transitioning from coarse-grained locking (e.g., a global lock) to fine-grained locking (e.g., per-table locks), you must exhaustively analyze all possible thread interleavings. A common error is moving a dependent operation (like an IdealState update) outside of the new, narrower lock, which can allow other threads to observe inconsistent or stale states.

    Best Practice: Keep all logically coupled operations under the same lock to ensure atomicity.