DOCSEARCHbyAlgolia
Submit Docs

CelerData Enterprise Documentation

website·Indexed 35 days ago

https://docs.celerdata.com/Serverless/docs/release_notes/

Technical documentation for CelerData Enterprise, featuring a comprehensive SQL reference including string, math, aggregate, and bitmap functions, as well as SQL statements for index management.

Tokens
913.1K
Snippets
4.5K
Records
6.4K
Agent score
52%

What's inside CelerData

  1. Overview of StarRocks Graceful Exit

    Graceful Exit (introduced in v3.3) allows for non-disruptive upgrades, restarts, and cluster scaling of StarRocks FE, BE, and CN nodes. It ensures that nodes stop accepting new tasks while allowing existing queries and load jobs to complete within a controlled time window. From v3.4 onwards, the mechanism is enabled by default and includes improved FE awareness of the BE/CN 'exiting' state to prevent premature query cancellation.

  2. Overview of StarRocks Data Cache

    Data Cache is used to accelerate queries on native and external tables by caching data in memory and on disk. It is enabled by default from v3.3.0 onwards. From v4.0, in-memory and disk caches are integrated into a single Data Cache system consisting of two components:

    1. Page Cache: An in-memory cache that stores decompressed data pages (native table data/index pages, external table footer info, and partial decompressed pages). It uses an LRU (Least Recently Used) eviction strategy.
    2. Block Cache: A disk-based cache that stores fixed-size data blocks from external tables or cloud-native tables in shared-data clusters to reduce remote I/O latency from HDFS or object storage.

  3. Overview of StarRocks AWS authentication methods

    StarRocks supports three methods to authenticate and integrate with AWS resources:

    1. Instance profile-based authentication: The cluster inherits privileges from the EC2 instance profile it runs on. This is a cluster-level access control solution where no isolation is required between different cluster users.
    2. Assumed role-based authentication: The cluster assumes a specific AWS IAM role to gain access to resources, providing more granular control.
    3. IAM user-based authentication: The cluster uses specific IAM user credentials to access AWS resources.

  4. Overview of StarRocks Query Feedback

    Query Feedback is a framework within the Cost-based Optimizer (CBO) available from StarRocks v3.4.0 onwards. It records execution statistics (such as InputRows and OutputRows) during query execution and reuses them in subsequent similar queries. This prevents 'bad plans'—such as incorrect join orders or inefficient broadcast joins—that occur when statistical information is outdated or inaccurate, which can otherwise lead to timeouts or system crashes.

  5. Overview of StarRocks Spark Load

    Spark Load is an asynchronous import method used for initial migrations and large data imports (up to TB level) into StarRocks. It leverages external Apache Spark resources to pre-process data, improving import performance and reducing compute load on the StarRocks cluster.

    Key Constraints:

    • Requires the INSERT privilege on the target table.
    • Cannot be used to load data into Primary Key tables.
    • Import jobs are created via the MySQL protocol and monitored using SHOW LOAD.

  6. Overview of N-gram bloom filter indexes

    The N-gram bloom filter index is a specialized Bloom filter index designed to accelerate LIKE queries and the execution of ngram_search and ngram_search_case_insensitive functions. Unlike a traditional Bloom filter that indexes the entire string, an N-gram bloom filter tokenizes strings into substrings of a fixed length (gram_num) and indexes each substring. For example, with gram_num = 4, the string 'Technical' is tokenized into 'Tech', 'echn', 'chni', 'hnic', 'nica', and 'ical'.

  7. Overview of Column Masking and Row Access Policies

    CelerData provides column masking and row access policies to protect sensitive data by dynamically filtering or masking data at query runtime based on the user's role. Unlike views, a single policy can be created once and applied to multiple tables.

    • Column Masking: Dynamically masks specific columns (e.g., showing only the last four digits of a phone number for analysts while showing plaintext for sales staff).
    • Row Access Policies: Filters which rows are returned based on conditions (e.g., restricting sales staff to only see data from their own region).

    CelerData achieves this by rewriting the query at runtime (similar to a temporary view); it does not encrypt data at rest or modify the underlying stored data.

  8. Overview of Hudi catalogs in StarRocks

    A Hudi catalog is an external catalog that allows StarRocks (v2.4+) to query and transform Apache Hudi data without ingestion. It supports both Copy On Write (COW) and Merge On Read (MOR) tables. The supported file format is Parquet with compression formats including SNAPPY, LZ4, ZSTD, GZIP, and NO_COMPRESSION.

    Supported storage and metastore combinations:

    • Storage: HDFS, AWS S3, Microsoft Azure Storage, Google GCS, or S3-compatible systems (e.g., MinIO).
    • Metastore: Hive Metastore (HMS) or AWS Glue.
    • Constraint: AWS Glue can only be used if AWS S3 is the storage system. All other storage systems must use HMS.

  9. Overview of Unified Catalogs in CelerData

    A unified catalog is an external catalog in CelerData Cloud Serverless that allows querying data from Apache Hive, Apache Iceberg, Apache Hudi, and Delta Lake without manual ingestion. It enables users to:

    • Directly query data from these sources.
    • Use INSERT INTO or asynchronous materialized views to load data into CelerData.
    • Create or drop Hive and Iceberg databases and tables via CelerData.

    Constraints:

    • A single unified catalog supports only one storage system and one metastore service.
    • Supported storage: AWS S3 (and others).
    • Supported metastores: Hive Metastore (HMS) or AWS Glue.
    • If AWS S3 is used as storage, either HMS or AWS Glue can be the metastore. For any other storage system, only HMS is supported.

  10. Overview of StarRocks generated columns

    Introduced in v3.1 (and supported in shared-data clusters from v3.5.0), generated columns allow StarRocks to precompute and store the results of complex expressions. This accelerates queries through query rewrites, where the Cost-Based Optimizer (CBO) automatically redirects queries containing the expression to read directly from the generated column.

    Key Considerations:

    • Performance: Computing expressions during data loading increases time and overhead. Evaluate the impact on loading performance before implementation.
    • Lifecycle: It is recommended to create generated columns at table creation rather than adding or modifying them later, as post-creation changes are more resource-intensive.

  11. Overview of CelerData Primary Key tables

    The Primary Key table is designed for scenarios requiring frequent real-time updates and deletes. Unlike Unique Key tables using Merge on Read (MoR), Primary Key tables use a Delete+Insert policy: update operations are converted into a delete of the old record and an insert of the new one. This allows the table to support predicate and index pushdown, delivering significantly higher query performance (3 to 10 times faster than MoR Unique Key tables) without requiring aggregate operations during queries.

  12. Overview of CelerData Cloud Serverless

    CelerData Cloud Serverless is a fully managed data lake analytics platform built on StarRocks. It utilizes a separated storage and compute architecture to provide cost-effective compute resources for analytical SQL workloads. It supports three primary use cases:

    1. Data Lake Analytics: Directly query data in open formats (Apache Hive, Apache Hudi, Apache Iceberg, and Apache Delta Lake) without migration.
    2. Query Acceleration: Use asynchronous materialized views to speed up data lake analytics workloads.
    3. Data Warehousing: Ingest data from external sources into CelerData for low-latency and high-concurrency analysis.