repository·main·Indexed 22 days ago
https://github.com/apache/cloudberryApache Cloudberry (Incubating) is an open-source Massively Parallel Processing (MPP) database evolved from Greenplum, featuring a PostgreSQL kernel for data warehousing, large-scale analytics, and AI/ML workloads. Documentation covers the Interconnect module, including implementation of MotionIPCLayer and statistics extensions, as well as the PAX (Partition Attributes Across) storage method for improved OLAP performance via hybrid row-column storage, vectorized engines, and advanced data encoding.
stat_statements_parser is a component within the gp_stats_collector package. It provides a modified subset of the pg_stat_statements functionality (originally from PostgreSQL v9.4) specifically designed for generating query and plan IDs.In addition to the main repository, several ecosystem repositories provide specialized utilities and extensions:
The contrib/formatter_fixedwidth directory provides a custom formatter designed for reading and writing fixed-width data files within Apache Cloudberry.
Note on Writable External Tables:
While this formatter supports both reading and writing, testing writable external tables using this formatter requires a running gpfdist server. If you are performing regression tests or manual verification of writing capabilities, ensure a gpfdist instance is available. Standard tests in this directory focus on readable external tables to avoid this dependency.
PAX is organized into several functional layers and directories. The core logic resides in src/cpp/, which is subdivided into specific layers such as access, catalog, clustering, and storage. The storage implementation is further broken down into columns, filters, operators, ORC formats, and vectorized executors.
Key directory mapping:
src/api/: Public API for other modules.src/cpp/access/: Table access layer.src/cpp/catalog/: Catalog layer.src/cpp/storage/: Main storage implementation (includes columns, filters, operators, ORC, etc.).src/test/: Regression tests.sql/: SQL tests.PAX (Partition Attributes Across) is a database access method that combines the advantages of row storage (N-ary Storage Model) and column storage (Decomposition Storage Model). It is designed to improve query performance, particularly cache efficiency, in OLAP scenarios.
Key features include:
zstd and zlib.PAX uses two different methods for its catalog implementation:
HEAP to implement auxiliary tables, providing Multi-Version Concurrency Control (MVCC). This is the implementation covered in this documentation.JSON to store metadata with an independent MVCC implementation. While it offers better readability, DDL operations often result in poor performance.All auxiliary tables reside in the pg_ext_aux namespace, which is automatically created when the PAX plugin is initialized. The relevant access methods, namespaces, and auxiliary tables are inserted into Cloudberry during plugin initialization via the pax-cdbinit--1.0.sql script generated during compilation.
The Greenplum Partner Connector (GPPC) is a Greenplum Database extension that acts as a wrapper for C/C++ User-Defined Functions (UDFs).
Instead of writing standard Postgres C UDFs, you use the GPPC API. This allows your UDF to be portable across different Greenplum Database platforms and versions without requiring recompilation or modification. GPPC translates your GPPC-compliant UDF into a format compatible with the specific GPDB platform being used.
In a distributed system like Apache Cloudberry, deadlocks can occur at two levels:
To prevent global deadlocks, Apache Cloudberry currently employs a policy of holding an exclusive table lock for UPDATE and DELETE commands, which effectively disables concurrent updates to avoid the performance-heavy complexity of distributed cycle detection.
The DBMS_ALERT package allows a PL/pgSQL session to send alerts to multiple other PL/pgSQL sessions, facilitating 1:N communication.
REGISTER for a specific alert name (alerts are case-sensitive).SIGNAL with the alert name and a message. Note: Notifications are only sent upon committing the transaction. If the transaction is rolled back, the signal is discarded.WAITANY or WAITONE to block until an alert is received.REMOVE or REMOVEALL to unregister alerts when they are no longer needed to prevent memory issues and ensure future signals can be received.DBMS_ALERT and DBMS_PIPE share the same memory environment. Continuous signaling without receiving can lead to insufficient memory errors. If memory is low, call WAITANY or WAITONE to process accumulated messages.REMOVE or REMOVEALL, it may prevent other sessions from receiving signals for that same alert name.--- Sending side ---
CREATE FUNCTION send_dbms_alert_exe() RETURNS VOID AS $$
BEGIN
PERFORM DBMS_ALERT.SIGNAL('sample_alert','SIGNAL ALERT');
END;
$$ LANGUAGE plpgsql;
SELECT send_dbms_alert_exe();
--- Receiving side ---
CREATE FUNCTION receive_dbms_alert_exe() RETURNS VOID AS $$
DECLARE
alert_name TEXT := 'sample_alert';
alert_message TEXT;
alert_status INTEGER;
BEGIN
PERFORM DBMS_ALERT.REGISTER(alert_name);
SELECT message,status INTO alert_message,alert_status FROM DBMS_ALERT.WAITONE(alert_name,300);
RAISE NOTICE 'Message : %', alert_message;
RAISE NOTICE 'Status : %', alert_status;
PERFORM DBMS_ALERT.REMOVE(alert_name);
END;
$$ LANGUAGE plpgsql;
SELECT receive_dbms_alert_exe();When analyzing metrics from the GP Stats Collector, use the following notations to understand how values are calculated and scoped:
current_value - submit_value (the difference between the current state and the initial snapshot taken at submit).DIFF is not applicable).Query Dispatcher (on master) or an Execute (on segment).Data stored in ExtendProtocolData is allocated under the TopTransactionContext.
Lifecycle Rules: