StarRocks Documentation

The StarRocks query cache enhances the performance of aggregate queries by storing intermediate results of local aggregations in memory. This avoids redundant disk access and computation for identical or similar queries, making it particularly effective for high-concurrency scenarios involving large, complex datasets.

Support Matrix:

• Shared-nothing clusters: Supported since v2.5
• Shared-data clusters: Supported since v3.4.0
• Single flat tables: Supported since v2.5
• Star schema (multiple joined tables): Supported since v3.0

Service accounts are independent identities used to facilitate access control for the PhoenixAI Cloud API and the PhoenixAI Cloud Terraform provider. Unlike application keys, service accounts do not depend on specific PhoenixAI members, meaning their access remains intact even if the member who created them is removed.

Constraints:

• Only members with the Member Management privilege can create service accounts and grant privileges.
• Service accounts cannot be used to log in to the PhoenixAI Cloud console; they are strictly for API and Terraform provider access.

Security integrations allow StarRocks to connect with external authentication systems (such as LDAP), eliminating the need to manually create users within StarRocks. When a user attempts to log in with an external ID, StarRocks authenticates them based on the authentication_chain configuration. Upon successful authentication, a virtual user is created for the session.

Important Constraints:

• Authorization: To enable external authorization when using security integrations, StarRocks must be integrated with Apache Ranger. Native StarRocks authorization is currently not supported for security integrations.
• Group Management: To access group information from the external system for user group creation and authorization, the Group Provider must be enabled.
• Connectivity: StarRocks does not perform a connectivity check during the creation of a security integration.

StarRocks allows blacklisting Backend (BE) and Compute Nodes (CN) to avoid frequent query failures caused by connection issues.

• BE Blacklisting: Supported since v3.3.0.
• CN Blacklisting: Supported since v4.0.
• Automatic Management: By default, StarRocks automatically adds nodes to the blacklist when connections are lost and removes them when connections are re-established. However, nodes added manually remain blacklisted until manually removed.
• Permissions: Only users with SYSTEM-level BLACKLIST permissions can use these features.
• Scope: Each Frontend (FE) node maintains its own independent BE and CN blacklist; they are not shared across FE nodes.

Spark Load is an asynchronous import method used to preprocess data imported via external Spark resources. It is primarily designed for initial migrations or importing massive datasets into StarRocks to improve performance and save cluster computing resources. Users create import tasks via the MySQL protocol and monitor results using the SHOW LOAD command.

Constraints:

• The user must have INSERT permissions on the target StarRocks table.
• Bucket columns in the target StarRocks table must NOT be of type DATE, DATETIME, or DECIMAL when using Spark Load.

Introduced in v3.3, Query Queue v2 replaces fixed thresholds (concurrency, memory, CPU) with a logical slot-based system. It estimates the BE resources required by each query and schedules them based on available slots.

In shared-nothing clusters, it is enabled via FE configuration (enable_query_queue_v2), which requires an FE restart. In shared-data clusters, it is configured per warehouse.

The AUTO_INCREMENT column attribute allows StarRocks to automatically assign unique integer IDs to new rows when values are not specified. This is useful for creating primary keys, optimizing join operations (using integers instead of strings like UUIDs), and improving the performance of distinct counts on high-cardinality columns.

Constraints:

• The data type of an AUTO_INCREMENT column must be BIGINT.
• Values start from 1 and increment by 1 for each new row.

Data Cache Warmup is an active process in StarRocks (introduced in v3.3) that proactively fetches desired data from remote storage into the local data cache. This differs from standard Data Cache, which is a passive process where data is cached only during active querying. Warmup is particularly useful for high-performance requirements like BI reports and PoC testing to avoid repeated remote data fetches.

Prerequisites and Constraints:

• The Data Cache feature must be enabled before using warmup.
• Storage Capacity: The cache disk must have significantly more capacity than the data being warmed up; otherwise, newer data will evict previously warmed data.
• Disk Stability: High volumes of new data writes or unexpected large 'cold' queries during the warmup process can evict warmed data from the cache.

StarRocks supports two types of vector indexing for approximate nearest neighbor search (ANNS) at the Segment file level. Each table supports only one vector index.

• IVFPQ (Inverted File with Product Quantization): Uses an inverted file to divide data into clusters and product quantization to compress high-dimensional vectors. It offers a high compression ratio (~1:0.15) but requires additional fine ranking, which can increase latency.
• HNSW (Hierarchical Navigable Small World): A graph-based algorithm that builds a hierarchical structure of navigable small world graphs. It has a lower compression ratio (~1:0.8) but provides precise ranking with lower computation costs and latency, though it requires more storage.