DOCSEARCHbyAlgolia
Submit Docs

ByConity Documentation

website·Indexed 30 days ago

https://byconity.github.io/

Documentation for ByConity version 1.0.x, featuring a high-performance querying engine with ClickHouse-compatible patterns. Includes guides on SQL syntax, table engines like CnchMergeTree, data import/export via Hive and Kafka, deployment requirements, and comprehensive function references for hashing, arrays, and geospatial data.

Tokens
172.7K
Snippets
1.2K
Records
1.5K
Agent score
71%

What's inside ByConity

  1. Overview of ELT capabilities in ByConity

    ByConity supports Extract-Load-Transform (ELT) starting from version 0.2.0. Unlike traditional ETL, which relies on independent external systems for transformation, ByConity's ELT approach allows users to load minimally processed data into the cloud-native data warehouse and perform transformations during the analysis phase. This reduces maintenance costs and provides greater flexibility for analysts by minimizing the need for upfront data modeling.

  2. Overview of the CnchMergeTree Table Engine

    The CnchMergeTree table engine is the most commonly used engine in ByConity, utilizing a core concept similar to an LSM-Tree. It organizes data through a hierarchy of logical partitions, data parts (fragments), and granules.

    Key architectural components include:

    • Logical Partitioning: Data is divided into logical datasets based on a partition key. This allows the engine to prune unnecessary partitions during queries to improve performance.
    • Data Parts (Fragments): Within each partition, data is sorted by a sorting key. Each part contains a min/max index to accelerate selection.
    • Data Granules: The smallest indivisible unit for data queries and memory scanning. The default size is 8192 rows (configured via index_granularity). The first row of each granule is marked by the primary key value, creating a sparse index.
    • Background Merging: ByConity periodically merges DataParts within the same partition to maintain sorting order and optimize storage and query performance.

  3. Overview of ELT capabilities in ByConity

    Starting from version 0.2.0, ByConity supports Extract-Load-Transform (ELT) to allow users to perform data transformations directly within the cloud-native data warehouse. This eliminates the need for independent ETL systems, reducing maintenance costs and providing analysts with more flexibility by loading minimally processed data and performing transformations during the analysis phase.

  4. Overview of ELT capabilities in ByConity

    Starting from version 0.2.0, ByConity supports Extract-Load-Transform (ELT) as a cloud-native data warehouse. Unlike traditional ETL, which relies on independent external systems for transformation, ByConity's ELT approach loads minimally processed data into the warehouse and performs transformations during the analysis phase. This reduces maintenance costs associated with heterogeneous data systems and provides analysts with greater flexibility in data modeling.

  5. Overview of ByConity Data Warehouse

    ByConity is an open-source cloud-native data warehouse utilizing a compute-storage separation architecture. It is designed for elastic scaling, read-write separation, resource isolation, and strong data consistency. The system leverages several OLAP technologies to optimize performance, including column-based storage engines, MPP (Massively Parallel Processing) execution, vectorized execution, code generation, intelligent query optimization, indexing, and data compression.

  6. Overview of ByConity Resource Manager (RM)

    The Resource Manager (RM) is a core component of ByConity responsible for the unified management and scheduling of computational resources. Its primary goals are to enhance resource utilization, collect monitoring data from Virtual Warehouses (VWs) for workload-based scheduling, and provide elastic scaling through resource pooling. The RM manages a hierarchy of resources: Worker Nodes (Kubernetes Pods), Worker Groups (Kubernetes StatefulSets), and Virtual Warehouses.

  7. Overview of ByConity Cloud-Native Data Warehouse

    ByConity is an open-source cloud-native data warehouse designed for high-performance OLAP workloads. It implements a computing-storage separation architecture to enable elastic scaling and resource isolation. Key capabilities include read-write separation, tenant-level resource isolation, and strong data consistency for both read and write operations.

  8. Overview of ByConity SQL Data Warehouse

    ByConity is a distributed cloud-native SQL data warehouse engine optimized for interactive and Ad-Hoc queries. It supports querying multiple tables, seamless cluster expansion, and the unified aggregation of offline batch data and real-time data streams.

  9. Overview of ByConity ELT and BSP mode

    ByConity ELT enhances the ClickHouse high-performance computing framework by introducing 'bsp mode'. This mode is specifically designed to handle complex ETL tasks that typically struggle in standard ClickHouse environments. Key improvements include:

    • Task-level Fault Tolerance: Instead of retrying an entire query from the beginning upon failure, only the failed task is retried, reducing resource waste and improving SLA stability.
    • Improved Scalability: By increasing the degree of parallelism, memory occupation per unit of time is reduced, allowing queries to process data volumes that exceed the memory of a single machine.
    • Resource Management: It provides finer-grained scheduling to reduce resource contention and avoid 'Memory limit' errors common in high-concurrency OLAP scenarios.

  10. Overview of ByConity OLAP capabilities

    ByConity is an OLAP system designed for real-time data access, aggregation queries of large/wide tables, complex analysis of massive datasets, and multi-table associated queries. It leverages technologies including a column storage engine, MPP execution, intelligent query optimization, vectorized execution, Codegen, indexing, and data compression.

  11. Overview of ByConity architecture and features

    ByConity is an open-source cloud-native data warehouse designed to overcome the limitations of shared-nothing architectures (like ClickHouse) by implementing storage-computing separation.

    Key features include:

    • Storage-Computing Separation: Enables read-write separation and independent scaling of compute and storage resources.
    • Elastic Scaling: Supports real-time, on-demand expansion and contraction of computing resources.
    • Resource Isolation: Provides multi-tenant isolation to ensure tenants do not interfere with one another.
    • Strong Data Consistency: Ensures strong consistency for data read and write operations.
    • High Performance: Utilizes OLAP optimizations including column storage, vectorized execution, MPP (Massively Parallel Processing) execution, and query optimization.

  12. Overview of ByConity background tasks

    ByConity utilizes a storage and computing separation architecture where data is written at Part granularity. To maintain query performance and storage efficiency, the system employs three primary types of background tasks managed by the Server component:

    1. MergeMutate: Manages the merging of smaller Parts into larger ones to reduce file access and handles data modifications (Mutations) resulting from ALTER queries.
    2. GC (Garbage Collection): Identifies and deletes expired or eliminated Parts. To ensure consistency and avoid fragmented IO, ByConity does not delete Parts immediately after a merge; instead, it marks them for subsequent deletion by the GC task.
    3. CONSUMER (ConsumeManager): Handles real-time import consumption from Kafka (CnchKafka). It reads Kafka metadata, assigns partitions to consumer tasks, distributes them to worker nodes, and maintains heartbeats to ensure task health.