DOCSEARCHbyAlgolia
Submit Docs

Neon Open-Source Serverless Postgres Platform

repository·main·Indexed 12 days ago

https://github.com/neondatabase/neon

An open-source serverless Postgres platform that decouples storage from compute to enable instant branching and scalable storage. Features include the compute_ctl tool for node initialization, a specialized Pageserver and Safekeeper architecture, and a local development environment via neon_local.

Tokens
180.7K
Snippets
445
Records
802
Agent score
98%

What's inside Neon

  1. Overview of postgres_ffi

    main

    The postgres_ffi module provides utilities for working with PostgreSQL on-disk file formats. It consists of Rust structs auto-generated from PostgreSQL header files via bindgen, along with Rust functions designed to read and manipulate these formats.

    Important Constraints:

    • Non-Portability: The PostgreSQL on-disk file format is not portable across different CPU architectures or operating systems.
    • Version Sensitivity: The format is subject to change with each major PostgreSQL version. This module provides version-specific support for PostgreSQL v14, v15, and v16.
    • Module Structure: Version-specific bindings are organized into postgres_ffi::v14, postgres_ffi::v15, and postgres_ffi::v16. Version-independent code is exported into the top-level postgres_ffi module.

  2. What is Neon?

    main
    Neon is an open-source serverless Postgres database platform. It features a decoupled architecture that separates storage from compute, substituting the traditional PostgreSQL storage layer with a distributed cluster of nodes.

  3. Overview of the proxy libraries

    main
    The libs/proxy directory contains libraries specifically designed for the Neon proxy. These libraries are a significant fork and refactoring of rust-postgres. They have been decoupled into their own set of libraries because the proxy requires unique, low-level access to the PostgreSQL protocol that differs from the standard rust-postgres API.

  4. Overview of the neon extension components

    main

    The neon extension is composed of two primary parts: a shared preload library and a set of SQL utility functions.

    Shared Preload Library (neon.so)

    The neon.so library handles core infrastructure tasks, including:

    • Storage Manager API: Manages storage and network communications with the remote page server.
    • walproposer: Implements the broadcast protocol between PostgreSQL and WAL safekeepers.
    • Control Plane Connector: Uses ProcessUtility_hook to capture updates to roles and databases, sending them to the control ProcessUtility_hook.
    • Remote Extension Server: Requests compute_ctl to download extension files.
    • file_cache: Provides a local file cache to temporarily store relation pages on the local file system for improved performance.
    • relsize_cache: Maintains a cache of relation sizes to optimize Neon performance.

    SQL Functions (neon--*.sql)

    These are utility functions used to expose Neon-specific information to the user and to facilitate metrics collection. By default, this extension is created in all databases within the cluster.

  5. Test PostgreSQL extensions for upgrades and regression

    main

    The docker-compose/ext-src/ directory contains PostgreSQL extensions used for two primary testing purposes:

    1. Extension Upgrade Testing: Verifying that extensions can be upgraded between different Neon Compute versions.
    2. Regular User Testing: Running regression tests simulating regular user behavior (primarily for cloud instances).

    Each extension is organized into its own directory (e.g., pg_vector-src/) and typically contains specific shell scripts to automate these tests.

  6. Use the tenant size model simulator

    main
    The tenant_size_model library is a simulator designed to calculate tenant size across various scenarios using the "Logical size + WAL" pricing method. It is primarily used to model how logical data size and Write-Ahead Log (WAL) activity impact tenant resource requirements and pricing.

  7. Overview of Consumption Tracking Metrics

    main

    Neon tracks consumption to support billing and cross-check AWS costs. Metrics are categorized by the service that collects them and the granularity of the measurement:

    Endpoint-level Metrics

    • CPU time: Measured as Wall clock seconds * current number of cores.
    • Traffic: In/out traffic measured at the proxy.

    Branch-level Metrics

    • Written size: Amount of data written (occupying disk bandwidth and crossing AZ boundaries). Collected per branch.

    Tenant-level Metrics

    • Synthetic storage size: The size exposed via pageserver's /v1/tenant/{}/size.
    • Real storage size: The size of the tenant directory on pageserver disks.
    • S3 storage size: The size of tenant data on S3.

  8. What is the storage-controller service?

    main

    The storage-controller is a service that manages the physical mapping of Tenants and Timelines to Pageservers and Safekeepers. It provides an abstraction layer for storage, allowing other system components to create or delete tenants and timelines without needing to know the specific underlying pageserver or safekeeper locations or orchestration rules.

    Key responsibilities include:

    • Managing storage sharding and shard splitting.
    • Scheduling tenant shards to pageservers.
    • Reconciling the observed state of shards with their intended state.
    • Handling pageserver failures and rescheduling affected shards.
    • Performing scheduling optimizations (e.g., moving shards to preferred Availability Zones or spreading shards of the same tenant across different pageservers).

  9. Overview of Neon Authentication

    main

    Neon uses JWT (JSON Web Tokens) for communication between most components, including Compute, Pageserver, Safekeeper, and the CLI.

    Key details:

    • Protocols: For HTTP connections, the Bearer authentication scheme is used. For PostgreSQL connections, the JWT is passed as the password.
    • Default State: Authentication is optional and disabled by default to facilitate debugging.
    • Security: Tokens are signed using the EdDSA algorithm (RFC 8037). Components validate tokens using a public key (PEM format). The private key is used for generation and should not be shared with components like Compute, Pageserver, or Safekeeper.
    • Limitations: There are currently no expiration or rotation schemes for tokens. Rotating keys requires restarting all components.

  10. What is the Storage Broker and how does it work?

    main

    The Storage Broker is a stateless pub-sub message broker built on tonic (gRPC). It solves two primary problems in the Neon architecture:

    1. Discovery: It allows safekeepers and pageservers to learn which nodes hold specific timelines and their current statuses.
    2. Scalability: It prevents $O(n^2)$ connection complexity between storage nodes by providing a centralized multiplexing point.

    Key Use Cases

    • Pageservers: Use the broker to identify the most advanced and healthy safekeeper to pull Write Ahead Log (WAL) data from.
    • Safekeepers: Use the broker to synchronize timeline state, including advancing remote_consistent_lsn and backup_lsn, and coordinating which node is responsible for offloading WAL to S3.

    Technical Implementation

    • Statelessness: Because it is stateless, fault tolerance is managed via Kubernetes (k8s) rather than built-in replication.
    • Messaging: The current primary message type is SafekeeperTimelineInfo. Safekeepers periodically push their timeline status to the broker, and other nodes subscribe to receive these updates.

  11. What is the Discrete Event SIMulator (desim)?

    main

    The Discrete Event SIMulator (desim) is a library designed for running simulations of distributed systems. It allows developers to test complex interactions between components (such as walproposer and safekeeper) in environments that closely mimic real-world scenarios.

    Key features include:

    • Thread-based Node Execution: Each node in the simulated system runs as a separate thread.
    • Virtual Simulation Time: Instead of relying on real-world wall-clock time, the simulator uses virtual time. This allows the simulation to 'fast-forward' through idle periods where nodes are sleeping or waiting, making it significantly faster than standard integration tests.
    • Determinism: The simulator is deterministic, ensuring that simulations can be reproduced.
    • Network Failure Injection: You can inject network failures to test system resilience without waiting for real-world timeouts to trigger.

  12. What is the Communicator in Neon

    main

    The Communicator (or "compute-pageserver communicator") is a background worker process that runs within the PostgreSQL server as part of the Neon extension.

    Its primary roles are:

    1. Current functionality: Providing an HTTP endpoint for metrics.
    2. Future functionality: Handling all communications between the compute nodes and the pageservers.