DOCSEARCHbyAlgolia
Submit Docs

Databend Documentation

website·Indexed 32 days ago

https://docs.databend.cn/

Documentation for Databend, featuring a comprehensive SQL reference (DDL, DML, functions, and stored procedures), user guides for cloud configuration, data loading, and query optimization, and developer resources including APIs, SDKs, and contribution guides for the open-source project.

Tokens
410.5K
Snippets
2.3K
Records
3.1K
Agent score
51%

What's inside Databend

  1. Overview of Databend Stage operations

    In Databend, a Stage is used as a temporary storage location for data files that need to be loaded into a table or unloaded from a table. Stage operations are categorized into management, file manipulation, and information retrieval.

  2. Overview of Databend Security and Reliability Features

    Databend provides a multi-layered security mechanism to ensure data integrity, compliance, and business continuity. Key security capabilities include:

    • Access Control: Fine-grained management of user permissions via roles and object ownership.
    • Audit Trail: Comprehensive logging of database activities for security monitoring, compliance, and performance analysis.
    • Network Policy: Restriction of network access to specific IP ranges to prevent unauthorized connections.
    • Password Policy: Enforcement of password complexity, regular rotation, and account lockout strategies.
    • Data Protection: Implementation of row-level filtering and column-level masking/desensitization.
    • Fail-safe: Prevention of data loss by recovering accidentally deleted data from S3-compatible storage.
    • Recovery from Operational Errors: Ability to restore deleted databases/tables or undo incorrect data modifications.

  3. Overview of Databend Query Operator Types

    Databend supports several categories of query operators for data manipulation and filtering:

    • Arithmetic Operators: Used for mathematical calculations (e.g., +, -, *, /, %, DIV).
    • Comparison Operators: Used to compare values (e.g., =, !=, <, >, <=, >=, BETWEEN, IN).
    • Logical Operators: Used for boolean logic (e.g., AND, OR, NOT, XOR).
    • JSON Operators: Used for operating on JSON data (e.g., ::, ->, ->>, @>, <@).
    • Set Operators: Used to combine results from multiple queries (e.g., UNION, INTERSECT, EXCEPT).
    • Subquery Operators: Used for nested queries (e.g., EXISTS, IN, ANY, ALL, SOME).

  4. Overview of Databend Fuse Engine

    The Fuse Engine is the default storage engine for Databend. It provides a Git-like data management system with the following characteristics:

    • Snapshot-based Architecture: Allows querying and recovering data from any point in time with a full history of changes.
    • High Performance: Optimized for analytical workloads (OLAP) using automatic indexing and Bloom Filters.
    • Efficient Storage: Uses the Parquet format for high compression and storage efficiency.
    • Cloud Optimized: Specifically designed for object storage.
    • Data Maintenance: Includes automatic data retention, snapshot management, and change tracking.

  5. Overview of Databend Cloud Task Flow

    Task Flow is a built-in workflow orchestration feature in Databend Cloud. It allows users to organize multiple SQL tasks into a Directed Acyclic Graph (DAG) to unify scheduling, monitoring, and management. Each node in the DAG is a Task containing a SQL statement and scheduling configuration. Databend Cloud automatically determines the execution order based on defined dependencies. Key capabilities include a visual graph editor, version history with rollback, Stream event triggering, and batch operations.

  6. Introduction to Databend Cloud

    Databend Cloud is a fully managed version of Databend that eliminates the need for manual deployment, upgrades, or scaling. It is currently available in specific AWS regions and includes all features provided in the Databend Enterprise edition.

  7. Overview of Databend Community Edition

    Databend Community is the open-source version of Databend, released under the Apache 2.0 license. It is free for both commercial and non-commercial use. It shares the same core analytical engine as Databend Enterprise and Databend Cloud but does not include enterprise-specific features.

  8. Overview of Databend External Functions

    External Functions in Databend allow you to extend SQL with custom logic that executes on a remote server. Communication between Databend and the remote handler is performed using the Apache Arrow Flight protocol. While the remote handler can be implemented in any language, Python is the most common choice, and the databend-udf package is available for rapid development.

  9. Overview of Kafka Consumer tasks

    Kafka Consumer tasks allow for the continuous consumption of JSON messages from Kafka topics. Unlike S3, MySQL, or PostgreSQL integration tasks, Kafka Consumer tasks do not write directly to target tables. Instead, they save message batches as objects in the internal object storage (tenant Stage) at the path @kafka_consumer/<task_name>/. Users can then query these objects using SQL and perform INSERT INTO ... SELECT or COPY INTO operations to move data into business tables.

  10. Overview of Databend Cloud Agent

    Cloud Agent is an AI assistant embedded in the Databend Cloud console. It allows users to perform tasks using natural language, such as executing SQL, rendering charts, managing resources, and building data pipelines. It operates within the user's existing role permissions and provides a safety mechanism where read-only operations execute immediately, while any operation that modifies account or data requires explicit confirmation via a pop-up box.

  11. Overview of the Databend Formal Type System redesign

    The legacy DataType system relied on an enum, which required frequent pattern matching and prevented its use as a generic parameter. The new system replaces this with a trait-based approach where DataType is a trait, allowing for better extensibility, support for nullable properties, and a more logical structure for nested types like Structs.

  12. Overview of Join Semantics and Algorithms

    Join operations in SQL are categorized by semantics and implemented via specific algorithms.

    Join Semantics:

    • INNER JOIN: Returns tuples satisfying the join condition.
    • LEFT OUTER JOIN: Returns all tuples from the left table and matching tuples from the right table.
    • RIGHT OUTER JOIN: Returns all tuples from the right table and matching tuples from the left table.
    • FULL OUTER JOIN: Returns all tuples from both tables where the join condition is met, or where no match exists in the opposite table.
    • CROSS JOIN: Returns the Cartesian product of the tables.
    • Semi-join / Anti-join: Used to implement IN, EXISTS, NOT IN, and NOT EXISTS subqueries.

    Join Algorithms:

    • Nested Loop Join: The basic algorithm that iterates through the outer table and inner table to check conditions.
    • Hash Join: Used exclusively for equi-joins (where the condition is an equality like r.a == s.a). It consists of a Build phase (creating a hash table from one relation) and a Probe phase (checking the other relation against the hash table).
    • Sort-Merge Join: Typically used for equi-joins; it sorts both tables by the join key and merges them. A 'band join' optimization can allow it to work with specific non-equi joins.