DOCSEARCHbyAlgolia
Submit Docs

Databend Documentation

website·Indexed 34 days ago

https://docs.databend.com

Comprehensive documentation for Databend, featuring a complete SQL reference for DDL, DML, and functions, user guides for Cloud setup and data loading, and developer resources including REST APIs, SDKs, and contribution guides for the open-source project.

Tokens
402K
Snippets
2.3K
Records
3.1K
Agent score
67%

What's inside Databend

  1. Overview of debezium-server-databend

    debezium-server-databend is a lightweight Change Data Capture (CDC) tool developed by Databend, based on the Debezium Engine. It captures real-time changes from relational databases and delivers them as event streams to be written into Databend. It allows for database change monitoring and transformation into consumable events without requiring heavy infrastructure like Kafka, Flink, or Spark.

  2. Overview of Databend Materialized Views

    A materialized view in Databend physically stores the result of a query. It is defined on a persistent FUSE table in the default catalog. Unlike logical views, materialized views can be explicitly refreshed to persist changes from the source table.

    Key behavior includes:

    • Read Fix: When source changes are unrefreshed, Databend unions persisted data with incremental source data at read time to ensure results are current rather than stale.
    • Change Tracking: Databend enables change tracking on the source table upon creation of the materialized view.

  3. Overview of Databend Cluster Keys

    Cluster keys provide automatic data organization for large tables to improve query performance. By grouping similar values into adjacent storage blocks (Parquet files) and creating block-to-value mappings, Databend can skip irrelevant blocks during query execution. This eliminates the need for manual maintenance, as Databend monitors DML operations and performs background clustering optimization automatically.

  4. Overview of Databend Audit Trail

    Databend provides an audit trail through system history tables that automatically capture database activities. This allows administrators to audit query execution, data access (object modifications), and authentication attempts for compliance and security monitoring.

  5. Introduction to Databend Cloud

    Databend Cloud is a fully managed version of Databend that removes the operational overhead of deployment, upgrades, and scaling. It is currently available in specific AWS regions and includes all features available in the Databend Enterprise Edition.

  6. Overview of Databend Fuse Engine

    The Fuse Engine is the default storage engine for Databend. It is designed for analytical workloads (OLAP) and data warehousing, utilizing a snapshot-based architecture that allows for time-travel queries and data recovery. It stores data in Parquet format on object storage or file systems, providing high compression and automatic indexing via bloom filters.

  7. Overview of Databend User-Defined Functions (UDFs)

    User-Defined Functions (UDFs) in Databend allow for custom operations when built-in functions are insufficient. Databend supports several types of UDFs based on the required return type and language:

    • Scalar Functions: Return a single value. Can be implemented via SQL expressions, Python, JavaScript, or WASM.
    • Aggregate Functions (UDAF): Return a single value based on a group of inputs. Implemented via JavaScript or Python runtimes.
    • Table Functions: Return a result set (table). Implemented using SQL queries.

  8. Overview of Databend Cloud Data Integration

    Databend Cloud Data Integration provides a visual, no-code interface to import, synchronize, or consume data from external systems. The system relies on two primary components:

    1. Data Sources: Reusable connection settings and credentials (e.g., AWS keys, MySQL passwords, Kafka brokers) used to access external systems. Data sources do not move data; they only store access information.
    2. Integration Tasks: The executable units that perform the actual data movement, such as imports, snapshots, continuous synchronization, or message consumption. They define the source, destination, runtime parameters, and scheduling.

    Note: Running Data Integration tasks incurs service hosting fees billed per second based on actual running time.

  9. Overview of Databend Self-Hosted

    Databend is an open-source, Rust-powered cloud data warehouse designed for deployment on your own infrastructure. It features a cloud-native, object-storage-first architecture, providing full control over data and the environment while ensuring memory safety and high performance.

  10. Overview of Databend Security and Reliability Features

    Databend implements a multi-layered security approach to ensure data integrity, compliance, and business continuity. Key security capabilities include:

    • Access Control: Role-based security and object ownership to manage user permissions.
    • Audit Trail: Tracking database activities for security monitoring, compliance, and performance analysis.
    • Network Policy: Restricting database connections to specific IP ranges.
    • Password Policy: Enforcing password complexity, rotation, and account lockout rules.
    • Data Protection Policies: Implementing row-level filtering and column-level masking for sensitive data.
    • Fail-Safe: Recovering accidentally deleted data from S3-compatible storage.
    • Recovery from Errors: Recovering from dropped databases, dropped tables, or incorrect data modifications.

  11. Overview of HTTP Streaming Load

    HTTP Streaming Load is a feature for self-hosted Databend (via the Query HTTP handler) that allows uploading a local file and loading it into a table in a single request, bypassing the need for a staging step. It is particularly useful for loading local files directly or streaming large files that should not be stored as single objects in object storage. It supports CSV, TSV, NDJSON, and Parquet formats.

  12. Overview of Amazon SQS (S3) event-driven ingestion

    The Amazon SQS (S3) integration task enables event-driven data ingestion. Instead of polling an S3 path, the system consumes ObjectCreated events sent from S3 to an SQS standard queue. When a new object is written to S3, S3 notifies SQS; the Platform then reads the SQS message via an IAM Role, parses the bucket and object key, and writes the data into a target table. Once the write is successful, the SQS message is deleted.