repository·main·Indexed 26 days ago
https://github.com/apache/calciteA dynamic data management framework providing SQL parsing, validation, and a customizable query optimizer. It enables the construction of query engines and adapters for various data sources—including Cassandra, Kafka, Druid, Elasticsearch, and MongoDB—allowing SQL queries without implementing storage primitives. Key features include SQL-to-algebra transformations, JDBC connectivity via Avatica, and support for User-Defined Functions (UDFs), aggregate functions, and window functions.
sqlsh for executing commands directly from your shell.Apache Calcite is a dynamic data management framework designed to provide the core components of a database management system without the storage primitives. It is used to build query engines, optimizers, and adapters for various data sources.
Key capabilities include:
Calcite implements the OpenGIS Simple Features Implementation Specification (version 1.2.1).
Supported Data Types:
GEOMETRYPOINT, LINESTRING, and POLYGONSpatial Functions:
ST_ (e.g., ST_Point, ST_DWithin, ST_Contains, ST_Buffer, ST_Envelope, ST_Rectangle).Calcite is an embedded library that runs within a host application's JVM. It does not expose its own network ports. The threat model assumes an attacker (e.g., a query author) can reach Calcite via a JDBC connection.
Attacker Capabilities:
model, parserFactory, schemaFactory, fun, typeSystem, dataSource, jdbcUrl).Attacker Limitations:
Schema adapters allow Calcite to present various data sources as tables within a schema. Available adapters include:
calcite-arrow)calcite-cassandra)example/csv)calcite-druid)calcite-elasticsearch)calcite-file)calcite-geode)calcite-innodb)calcite-core)calcite-mongodb)calcite-os)calcite-pig)calcite-redis)solr-sql)calcite-spark)calcite-splunk)mat-calcite-plugin)Relational algebra is the core representation of queries in Calcite. Every query is modeled as a tree of relational operators. Developers can interact with this layer in two ways:
Calcite uses Planner Rules to transform these expression trees using mathematical identities (e.g., pushing a filter into a join) to optimize the query. The optimization process is guided by a Cost Model, which helps the planner engine select alternative expressions that maintain the same semantics but reduce execution cost.
A lattice is a framework used for creating and populating materialized views and mapping user queries to them. It represents a star or snowflake schema where all relationships are many-to-one, originating from a central fact table.
Key characteristics of a lattice:
FROM clause defines the many-to-one relationship (e.g., FROM A, B implies a many-to-one relationship from A to B).GROUP BY). Measures are defined by associating a column with an aggregate function.["sales", "unit_sales"]).Calcite provides various educational resources including video recordings, slide decks, and summaries of technical talks. Key topics covered in past presentations include:
calcite-clj), Hazelcast In-Memory Data Grid, and Apache Beam.Calcite provides two complementary components for working with Pig:
The Apache Arrow adapter allows Calcite to read and process data in the Arrow format using SQL. It specifically supports reading files in the Arrow Feather format (typically with a .arrow suffix).
Note: This is an experimental feature; public API and usage patterns are subject to change.
.ibd files) directly without a running MySQL server. It uses the innodb-java-reader to access the files and can perform projections, filters, and sorts directly on the data files. It is index-aware and can leverage primary or secondary keys for lookups and push down conditions to the storage engine.