Presto Distributed SQL Query Engine

The Hive connector enables Presto to query data stored in a Hive data warehouse. It utilizes the data files (stored in HDFS or Amazon S3) and the metadata (stored in a metastore like MySQL), but it does not use HiveQL or the Hive execution environment (MapReduce/Tez).

Supported File Types

• ORC
• Parquet
• Avro
• RCFile
• SequenceFile
• JSON
• Text

PrestoDB is an open-source distributed SQL query engine designed for high-speed analytic queries against data sources ranging from gigabytes to petabytes.

Key Capabilities for Users:

• High Performance: Supports low-latency interactive workloads, ad-hoc exploratory analytics, and large-scale batch workloads.
• Scalability: Engineered to access and process massive amounts of data from data lakes.
• Connectivity: Features a highly customizable plugin infrastructure, including pluggable storage connectors, to adapt to changing infrastructure.
• User-Friendly Interface: Uses familiar ANSI-style SQL syntax and includes an optimizer designed to produce efficient query plans with sensible defaults.

Presto C++ (development name: Prestissimo) is a high-performance, drop-in replacement for Presto workers written in C++. It is built on the Velox library and uses the Proxygen C++ HTTP framework to implement the same RESTful endpoints as Java workers.

Key characteristics:

• No JVM Required: Because communication with the Java coordinator and between workers occurs via REST endpoints, Presto C++ workers do not use JNI and do not require a JVM on worker nodes.
• Sidecar Mode: A Presto C++ worker can be configured as a sidecar using the NativeSidecarPlugin. This allows the worker to provide additional information to the Java coordinator (such as session properties and functions) via additional REST endpoints. It is recommended to have at least one sidecar worker in a Presto C++ cluster.

Presto SQL Helpers provides inline SQL-invoked functions via two distinct plugin modules:

1. presto-sql-invoked-functions-plugin: Used for Java-based clusters and native-only clusters (those without a sidecar).
2. presto-native-sql-invoked-functions-plugin: Used specifically for native clusters that are sidecar-enabled.

Presto provides a REST API to interact with the cluster. The API is organized into several resource categories that allow you to manage and monitor the lifecycle of the cluster and its workloads. The primary resource types available are:

• Node: Manage and inspect cluster nodes.
• Query: Manage and monitor SQL queries (execution, status, and results).
• Stage: Inspect the execution stages of a query.
• Statement: Interact with specific statements within the engine.
• Task: Monitor individual tasks within a query stage.

Presto product tests use user-visible interfaces (such as presto-cli) to verify Presto's correctness through end-to-end execution. Unlike unit tests, these tests exercise the entire codebase.

Tests are executed using the Tempto harness. The environment typically consists of a Docker container running Hadoop in pseudo-distributed mode, with Presto running either in Docker containers (pseudo-distributed or distributed) or manually from an IDE like IntelliJ for debugging. Tests run in a separate JVM using scripts located in presto-product-tests/bin.

Several core Presto optimizations leverage HBO statistics when enabled:

• DetermineJoinDistributionType & DetermineSemiJoinDistributionType: Decides between broadcast or repartition joins using recorded data sizes from history. Falls back to cost model statistics if HBO is disabled or unavailable.
• ReorderJoins: Reorders join sequences based on input and output sizes recorded from history queries.
• PushPartialAggregationThroughExchange: Uses historical partial aggregation output sizes to decide on aggregation splitting (see configuration details).
• ScaledWriterRule: Uses historical write task counts to determine the starting number of tasks.
• RandomizeNullKeyInOuterJoin: Uses historical NULL key counts and ratios to mitigate skew (see configuration details).

The Presto Console is a built-in dashboard for monitoring cluster information, running SQL queries, query details, and resource groups. It consists of HTML pages and JavaScript files that interact with Presto APIs via the coordinator.

The source code is located in the presto-ui/src directory (referred to as the Presto Console root).

Directory Layout:

• Presto Console root: Contains static HTML pages.
• src/static: Static assets (favicon, logo, Presto CSS).
• src/static/dev: Contains the Query Viewer page and its JS file.
• src/static/vendor: Third-party JS/CSS libraries.
• src/: UI components and the webpack.config.js file.