repository·main·Indexed 27 days ago
https://github.com/databendlabs/databendAn open-source enterprise data warehouse built in Rust for large-scale analytics, vector search, and full-text search. Optimized for AI workloads, it features Sandbox UDFs for agent orchestration and Git-like data branching. Documentation covers deployment via Docker (supporting linux/amd64 and linux/arm64), local Python installation using databend-driver, and comprehensive performance benchmarking for TPCH, Hits, internal queries, recluster quality, and security policies.
The databend-binaries package contains the core service and utility binaries for the Databend ecosystem:
databend-meta: The meta service binary responsible for managing metadata.databend-metabench: A utility used to run benchmarks specifically for the meta service.databend-metactl: A command-line tool used to dump data from a Sled database into JSON format.Databend Query is a distributed query engine. Its architecture is divided into several specialized modules:
ast provides the SQL parser; sql handles planning, binding, and optimization.expression is the core scalar expression framework (representation, type checking, evaluation); pipeline implements the execution pipeline and scheduling for physical operators.formats handles serialization/deserialization; storages contains table engines and storage-layer integrations (e.g., Fuse); datavalues is the legacy in-memory value representation.functions implements scalar and aggregate functions; script and script_udf_support provide script execution and UDF integrations.catalog defines abstractions like Catalog, Database, Table, and TableContext.service is the main databend-query service crate; management provides cluster and quota support; users handles access control (users, roles).config provides query-service configuration types; settings defines global and session settings.ee and ee_features contain enterprise-specific query functionality.The Proxy Engine MVP assumes that target tables are kept eventually consistent by an external mechanism (e.g., application dual-writes, asynchronous materialized views, or background sync tasks).
Warning: Because consistency depends on the external synchronization mechanism, strongly consistent business workloads should not use the MVP Proxy Engine as their primary read entry point. It is best suited for trace, log, and observability workloads where short delays in data visibility are acceptable.
The Semaphore crate provides a distributed semaphore implementation using the Databend meta-service for coordination. It allows you to manage access to limited resources across distributed systems with features like fair queueing (via sequence numbers), automatic lease management (TTL), and real-time updates via the meta-service watch API.
Semaphore: The primary entry point for semaphore operations.Permit: Manages the lifecycle of an acquired semaphore resource.PermitEntry: Represents a specific entry within the semaphore queue.PermitKey: Defines the key structure used for semaphore entries in the meta-service.databend-common-ast is a specialized module within the Databend project that provides the Abstract Syntax Tree (AST) infrastructure. It is used for parsing and interpreting SQL queries within the Databend ecosystem.The Optimizer IR is the foundation for query plan representation, transformation, and optimization in Databend. It is organized into several specialized systems:
expr/): Handles the representation of query plans via SExpr (Single Expression trees) and MExpr (Multiple Expression representations used within the Memo).memo.rs): A central data structure that manages equivalent expressions by organizing them into logical equivalence groups and maintaining a lookup table to prevent duplicates.group.rs): Represents sets of logically equivalent relational expressions, tracking their shared properties and optimization state.property/): Manages plan properties (e.g., RelationalProperty, PhysicalProperty) and provides enforcers to satisfy physical requirements.stats/): Provides tools for cost estimation, including column-level statistics, histograms, and selectivity estimation.Databend provides two primary binaries for interacting with the system:
query (databend-query): The core query service binary responsible for executing queries.table-meta-inspector (table-meta-inspector): A specialized tool used for decoding v3 table metadata.The shared replay data set is organized as follows:
tables/: SQL CREATE TABLE definitions.statistics/: YAML files containing table, column, and histogram statistics.cases/: YAML test case definitions.statistics_trace/: StatisticsTrace replay fixtures (SQL and JSON).results/: Generated test result files (raw, optimized, and physical plans).The recluster benchmark directory provides local tools for measuring data order quality during development and investigation. These tools are not intended for CI usage.
Available Scripts:
recluster_bench.py: Creates synthetic clustered tables, runs recluster rounds, and records clustering quality after each round.run_recluster_bench_with_bin.py: Orchestrates the full environment by starting databend-meta and a selected databend-query binary, running the benchmark, and cleaning up services.The src/query/sql/ directory (the databend-common-sql crate) is the core of Databend's SQL processing. It handles SQL binding, logical planning, optimizer behavior, and planner test infrastructure. Key sub-modules include:
src/planner/: Contains the binder, planner, optimizer, and plan representations. This is the primary location for SQL semantics and optimizer tasks.src/executor/: Contains SQL-side execution helpers that interface closely with planning output.src/evaluator/: Contains the evaluation logic used by the SQL crate.You can connect to Databend using bendsql or a MySQL client.
Using Default Root User:
If no user is specified, connect as root with no password.
Using a Custom Built-in User:
If you configured QUERY_DEFAULT_USER and QUERY_DEFAULT_PASSWORD, use those credentials to connect.
Note: When using docker run --net=host, the ports will be available on the host network directly.