DOCSEARCHbyAlgolia
Submit Docs

Apache Doris

repository·master·Indexed 12 days ago

https://github.com/apache/doris

An open-source MPP (Massively Parallel Processing) database designed for real-time analytics, lakehouse querying, and hybrid search, including vector and text data.

Tokens
142.8K
Snippets
414
Records
603
Agent score
97%

What's inside Apache Doris

  2. Overview of Apache Doris

    master
    Apache Doris is an open-source, real-time analytics and search database built on Massively Parallel Processing (MPP) architecture. It is designed to provide fast SQL analytics, lakehouse query acceleration, and hybrid search capabilities across structured, text, and vector data. It can be used for customer-facing analytics, data warehousing, observability, and AI workloads.

  3. Overview of Doris FE SQL Parser

    master

    The fe-sql-parser is a standalone ANTLR4-based syntax parser for Apache Doris SQL. It generates an ANTLR parse tree (CST) for any Doris-dialect SQL string.

    Important Note: This parser performs no semantic analysis. It does not resolve identifiers, validate tables or columns, or check data types. It is strictly a syntax parser.

  4. Overview of the fe-foundation module

    master

    The fe-foundation module is a zero-dependency, lightweight shared library designed for the Apache Doris Frontend (FE) ecosystem. It sits at the bottom of the dependency hierarchy to provide essential, general-purpose utilities that can be safely used by both core Doris modules and external SPI (Service Provider Interface) plugins without pulling in heavy transitive dependencies like Guava, Hadoop, or Gson.

    Key Characteristics:

    • Zero third-party dependencies: It uses only pure JDK utilities.
    • No business logic coupling: It contains no references to Doris-specific components like catalogs, optimizers, or planners.
    • Stability: It provides a stable API intended for long-term use across the ecosystem.

  5. Overview of Iceberg and Paimon Schema Evolution and Time Travel Test Suites

    master

    This documentation outlines the regression testing coverage for Apache Doris when interacting with external Iceberg and Paimon tables. It specifically focuses on the intersection of Schema Evolution (changes to table structure), Partition Evolution (changes to how data is partitioned), and Time Travel (accessing historical snapshots, tags, or branches).

    Key areas of coverage include:

    • Schema Operations: Renaming, promotion, dropping, and re-adding fields, as well as metadata atomicity (comments, defaults, nullability).
    • Partition Evolution: Adding, dropping, or replacing partition specs and how they interact with runtime filters and scanners.
    • Historical References: Using snapshots, tags, and branches to access data at specific points in time.
    • Delete Mechanisms: Handling Equality Deletes, Position Deletes, and Deletion Vectors (DV) across schema and partition changes.
    • Catalog Types: Support for REST, Filesystem, and JDBC catalogs.

  6. Iceberg Write P0 Coverage Matrix Overview

    master

    This document defines the P0 (Priority 0) testing coverage for writing data from Apache Doris to Apache Iceberg tables. The coverage ensures correctness, compatibility, and failure atomicity across various scenarios including schema evolution, partition evolution, and row-level DML operations.

    Coverage Scope

    The matrix validates the interaction between:

    • Schema Changes: Adding, renaming, dropping, or promoting types.
    • Partition Evolution: Changes to partition specs and transforms.
    • Data Models: Support for Duplicate, Unique (MOW/MOR), and Aggregate models.
    • DML Operations: INSERT, OVERWRITE, DELETE, UPDATE, and MERGE.
    • Iceberg Features: Snapshots, tags, and branches.
    • Data Types: Primitive types, complex types (ARRAY, MAP, STRUCT), and NULL semantics.

    Validation Principles

    • Positive Suites: Results are compared row-by-row against Spark for the same table logic after the final write. For suites involving transform metadata, physical partition values are also compared.
    • Historical References: Validation includes checking the isolation of snapshots, tags, and branches.
    • Failure Atomicity: Ensures that if an operation fails (e.g., due to type mismatch or constraint violation), the snapshot, files, and data remain unchanged.

  8. Understand the purpose of default certificates in conf/mysql_ssl_default_certificate

    master

    The directory conf/mysql_ssl_default_certificate contains default certificates generated for testing or initial setup.

    Important Security Warning: These certificates are generated by default and cannot be used in a production environment.

    Specifically, the certificates located in ./client_certificate are used to verify the identity of the client. For production-grade security, you must replace these with certificates issued by a trusted Certificate Authority (CA).

  9. Understand the Spring Boot + MyBatis + Doris Integration Demo

    master

    This demo project demonstrates how to integrate Apache Doris with a Spring Boot application using MyBatis and JDBC to access Doris data and expose it via RESTful APIs.

    Key components of the architecture include:

    • Data Source Management: Uses datasource for dynamic switching between multiple data sources.
    • Data Access: Uses mapper (MyBatis interfaces) and domain (Entity Java beans) to interact with Doris tables.
    • API Layer: Uses controller to provide RESTful endpoints.
    • Configuration: Managed via application.yml (Spring Boot), application-druid.yml (Database connections), and MyBatis XML files for data access mapping.

  10. Understand the Doris UI project structure

    master

    The UI project follows a standard web application directory structure:

    • public/: Contains static resources.
    • src/: The main development directory.
      • assets/: Static resources and images (managed via webpack).
      • components/: Common reusable components.
      • pages/: Subpages, which may contain their own subcomponents.
      • utils/: Public utility methods.
    • webpack.config.js: The Webpack configuration file.

    The technology stack uses react, react-router-dom, ant-design, and rxjs.

  11. Understand the fe-connector plugin framework

    master

    The fe-connector framework allows Apache Doris FE to serve metadata, scans, and writes for external data sources without the core engine needing internal knowledge of those sources.

    Core Design Principles:

    1. Isolation: Connectors must never import fe-core internals. All interaction must occur through the SPI (org.apache.doris.connector.*) or shared neutral types.
    2. Property Ownership: fe-core does not parse connector properties. Metadata-connection properties are parsed within the connector (typically via a metastore layer), while storage properties are handled by fe-filesystem.
    3. Agnosticism: Shared planning code remains source-agnostic. Per-source behavior is accessed via the SPI, and optional features are enabled via ConnectorCapability (opt-in mechanism).

  12. How statistics are analyzed and loaded

    master

    Analysis Flow

    When an analysis job is created via DdlExecutor, the AnalysisManager validates partitions and creates tasks for each column (and Materialized View indexes).

    • Synchronous tasks are executed immediately.
    • Asynchronous tasks are persisted to the StatisticsRepository, scheduled via AnalysisTaskScheduler, and executed by AnalysisTaskExecutor in a thread pool. The tasks collect data from the BE (Backend) nodes, refresh the StatisticsCache, and update the job status upon completion.

    Loading Flow

    When the StatsCalculator requests statistics:

    1. It checks the StatisticsCache.
    2. If cached, the stats are returned immediately.
    3. If not cached, the cache returns UNKNOWN and submits an asynchronous load task.
    4. The load task executes a statistic query via StatisticsUtil. If successful, the results are deserialized and cached as normal statistics. If an exception occurs, UNKNOWN is cached for that column to prevent repeated failed attempts.