DOCSEARCHbyAlgolia
Submit Docs

Apache Druid

repository·master·Indexed 11 days ago

https://github.com/apache/druid

A high-performance, real-time analytics database designed for low-latency queries and high-concurrency ingestion. It serves as an open-source alternative to traditional data warehouses and includes a web console (version 39.0.0) and various extensions for Consul discovery, gRPC queries, moving average calculations, and OpenTelemetry emitting.

Tokens
339.3K
Snippets
835
Records
1.3K
Agent score
96%

What's inside Druid

  1. What is Quidem UT

    master

    Quidem UT is a module designed to enable easy writing of SQL-level tests. It allows developers to write tests against existing test backends via a ComponentSupplier.

    Key capabilities include:

    • Test Capture Mode: Captures real-world Druid usage from external tools. These captured tests can be validated later to ensure results remain consistent, acting as an early warning system for regressions.
    • Backend Integration: Tests can be moved closer to the exercised code by utilizing existing ComponentSupplier implementations.

  2. Overview of the Druid Tasks API

    master

    The Tasks API provides endpoints for managing the lifecycle of Druid tasks. Tasks are individual jobs executed by Druid to perform operations such as data ingestion, querying, and compaction. Through this API, users can retrieve information about tasks, submit new tasks, and delete existing tasks.

    When making API calls, use the Router service address and port. For a standard quickstart configuration, the base URL is http://localhost:8888.

  3. Overview of Spectator Histogram module

    master

    The Spectator Histogram module provides approximate histogram aggregators and percentile post-aggregators based on Spectator fixed-bucket histograms. It is designed as a more storage-efficient alternative to the DataSketches extension for computing percentile approximations.

    Key Benefits

    • Reduced Storage Footprint: Uses significantly fewer bytes per row compared to quantilesDoublesSketch (e.g., ~6 bytes vs ~48 bytes in some datasets).
    • Performance: Smaller segment sizes lead to faster loading from deep storage and lower memory usage.
    • Accuracy: Maintains high absolute accuracy for smaller values and maintains relative accuracy across the range.

    When to use Spectator Histogram vs DataSketches

    • Use Spectator Histogram when you need a compact representation for positive long integer values and want to minimize storage costs.
    • Use DataSketches if you need to support negative values, decimal values, or if the specific limitations of Spectator buckets do not meet your accuracy requirements.

  5. Introduction to Apache Druid

    master
    Apache Druid is a real-time analytics database optimized for fast slice-and-dice (OLAP) queries on large, event-oriented datasets. It is designed for high-concurrency environments where real-time ingestion, fast query performance, and high uptime are critical. It is commonly used as a backend for analytical GUIs or highly-concurrent APIs requiring fast aggregations.

  6. Ways to extend Druid using JavaScript

    master

    JavaScript can be injected dynamically at runtime to extend Druid without deploying new extensions. Common use cases include:

    • Aggregators: Creating custom JavaScript aggregators.
    • Extraction functions: Defining custom dimension extraction logic.
    • Filters: Implementing custom filtering logic.
    • Post-aggregators: Implementing custom post-aggregation logic.
    • Router strategy: Defining custom routing logic.
    • Worker select strategy: Defining custom worker selection logic.

  8. Write an ingestion spec overview

    master

    An ingestion spec is a JSON configuration that defines how Druid should ingest data. This tutorial focuses on the native batch indexing task, which is used for loading static datasets (like files).

    When defining a spec, you must consider several key components:

    • Datasource name: The name of the target datasource in Druid.
    • Schema definition: Including the time column and column types.
    • Rollup: Deciding whether to aggregate data during ingestion.
    • Task type: Specifying the type of ingestion (e.g., native batch).
    • Input source: Where the data is coming from.
    • Data format: The structure of the input data (e.g., JSON, CSV).

  12. Join the Apache Druid community

    master

    The Druid community provides several channels for support and discussion:

    For Users:

    • Mailing List: Join the druid-user mailing list on Google Groups for general help.
    • Slack: Use the #troubleshooting channel for technical discussions.

    For Developers:

    • Mailing List: Join the druid-dev mailing list (dev@druid.apache.org). You can subscribe by emailing dev-subscribe@druid.apache.org.
    • Slack: Join the #dev channel for live technical conversations.

    For a full list of community resources, visit the official community page.