pgwire

repository·master·Indexed 21 days ago

https://github.com/sunng87/pgwire

A Rust library implementing the PostgreSQL Wire Protocol. It provides the building blocks to create PostgreSQL-compatible servers (backends) and clients or proxies (frontends). The library supports various protocol components including Startup, Simple Query, Extended Query, Copy, Replication, and Logical Replication, and includes support for SCRAM and OAuth/OIDC authentication.

Tokens
17.5K
Snippets
59
Records
74
Agent score
74%

What's inside pgwire

  1. Use the pgwire Client/Frontend API

    master

    The pgwire client/frontend API is currently under development. It is specifically designed for building components like PostgreSQL proxies that require full access to the wire protocol.

    Note: If you are looking for a general-purpose PostgreSQL driver for application development (rather than building a proxy or protocol-level tool), it is recommended to use rust-postgres instead.

  2. Understand the Postgres Wire Protocol parts

    master

    The Postgres Wire Protocol implemented by pgwire is a Layer-7 protocol consisting of six main parts:

    • Startup: The client-server handshake and authentication process.
    • Simple Query: A text-based protocol where queries are provided as strings and the server streams data in response.
    • Extended Query: A sub-protocol allowing queries to be cached on the server-side (prepared statements) and reused with new parameters. The response format is identical to Simple Query.
    • Copy: A sub-protocol used to stream data to and from PostgreSQL.
    • Replication: Protocol for streaming data.
    • Logical Replication: Protocol for logical data streaming.

    Note that the protocol has no inherent semantics about SQL; you can use any query language, data format, or natural language as long as the responses are encoded in the expected data row format with appropriate field descriptions (name, type, and format) in the header.

  3. Build a PostgreSQL compatible server with pgwire

    master

    To build a data service that is compatible with PostgreSQL clients (like psql), you must implement two core components in your server application:

    1. Startup Processor: Handles the initial client-server handshake and authentication.
    2. Query Processor: Handles incoming queries. There are two types:
      • Simple Query: Implement SimpleQueryHandler to provide basic compatibility with the psql command-line tool.
      • Extended Query: Implement ExtendedQueryHandler to support advanced features like prepared statements, binary encoding, and more sophisticated language drivers.

    pgwire acts as the protocol layer (similar to how hyper works for HTTP), allowing you to focus on your underlying data engine.

    // Conceptual implementation requirement:
    // 1. Implement a startup processor
    // 2. Implement SimpleQueryHandler for psql compatibility
    // 3. Implement ExtendedQueryHandler for prepared statements/binary support
  4. Run the libpq-client

    master

    The libpq-client can be executed either with a default connection string or by providing a custom PostgreSQL connection string as a command-line argument. Upon execution, the client connects to the server and runs SELECT version() to verify the connection.

    # Run with default connection string
    ./client
    
    # Run with custom connection string
    ./client "host=localhost port=5432 dbname=postgres"
  5. PgWireClient as a Stream and Sink

    master

    The PgWireClient is designed to work with the futures ecosystem:

    • Stream: It implements Stream<Item = Result<PgWireBackendMessage, PgWireError>>, allowing you to asynchronously iterate over messages sent by the PostgreSQL server.
    • Sink: It implements Sink<PgWireFrontendMessage>, allowing you to send protocol messages to the server using methods like send(), feed(), or flush().
  6. Use SessionExtensions for per-connection state

    master

    The SessionExtensions struct provides a type-safe, per-connection typed extension store. It allows different parts of your server (like authentication handlers or query handlers) to store and retrieve arbitrary state that is automatically cleaned up when the connection closes. It uses TypeId for lookups and supports interior mutability via Arc and RwLock.

    // Define a custom state type
    struct MyCustomState { pub count: i32 }
    
    // Inside a handler with access to ClientInfo
    let state = client.session_extensions().get_or_insert_with(|| {
        MyCustomState { count: 0 }
    });
    
    // Access or modify the state
    // Note: Since it's an Arc, you may need to wrap the inner data in a Mutex/RwLock
  7. Understand the OAuth authentication flow and SASL state

    master

    The OAuth authentication process follows the SASL protocol. The Oauth::process_oauth_message method handles the state transitions:

    1. SASLState::OauthStateInit: The server receives the client's initial response. It parses the message for the auth key (containing the Bearer token). If the token is missing or invalid, the server generates an error response containing the openid-configuration discovery document and the required scope (per RFC 7628).
    2. SASLState::OauthStateError: If the client responds to an error state, the server expects a single KVSEP (0x01) byte. Any other response results in an OAuthAuthenticationFailed error.
    3. Success: If validation succeeds, the state transitions to SASLState::Finished.

    Token Format: The implementation expects a case-insensitive Bearer scheme followed by a base64-like token (supporting characters: ALPHA, DIGIT, -, ., _, ~, +, /, and =).

  8. Implement a PostgreSQL-compatible server using handler traits

    master

    To build a full PostgreSQL server, you must implement the various handler traits for the protocol components. These handlers are aggregated by the PgWireServerHandlers trait. Implement PgWireServerHandlers on a single struct to serve a complete connection.

    Supported protocol components and their corresponding traits:

    • Startup & authentication: StartupHandler
    • Simple query: SimpleQueryHandler
    • Extended query: ExtendedQueryHandler
    • Copy: CopyHandler
    • Cancellation: CancelHandler
  9. Understand the pgwire layered API architecture

    master

    pgwire is designed as a layered library for building PostgreSQL-compatible servers and clients. It provides three distinct levels of abstraction that can be composed based on your needs:

    1. Protocol layer: Provides raw message definitions and codecs via the messages module. This is the lowest level and is available even without any features enabled.
    2. Message handler layer: Uses handler traits (e.g., StartupHandler, SimpleQueryHandler) to manage protocol state. You implement on_ prefixed methods for protocol bookkeeping, which then dispatch to do_ prefixed methods where you implement your specific query or authentication logic.
    3. High-level API layer: Provides ready-to-use components for common tasks, such as AuthSource for authentication, QueryParser/PortalStore for extended queries, and ConnectionManager for query cancellation.
  10. Use DecodeContext to track connection state during decoding

    master

    The DecodeContext struct is used during the decoding process to track the current state of the connection. This is critical for handling messages that depend on the connection phase, such as SSL/GSS negotiation or the initial startup sequence.

    Key fields include:

    • protocol_version: The ProtocolVersion currently in use.
    • awaiting_frontend_ssl: True if the connection is waiting for SSL negotiation.
    • awaiting_frontend_startup: True if the connection is waiting for a startup message.
    • awaiting_backend_ssl_response: True if the backend is waiting for an SSL response.
    • awaiting_backend_gss_response: True if the backend is waiting for a GSS encryption response.
    let mut ctx = DecodeContext::default();
    // ctx.awaiting_frontend_ssl is true by default
    // ctx.awaiting_frontend_startup is true by default
  11. The `MaybeTls` abstraction for unified stream handling

    master

    The MaybeTls enum is a wrapper that allows the server to treat plain TCP, Unix domain sockets, and TLS-encrypted streams uniformly. It implements AsyncRead and AsyncWrite, enabling the PgWireMessageServerCodec to operate on any of these underlying transport types without knowing if the connection is encrypted.

    #[non_exhaustive]
    pub enum MaybeTls {
        Plain(TcpStream),
        #[cfg(unix)]
        Unix(UnixStream),
        #[cfg(any(feature = "_ring", feature = "_aws-lc-rs"))]
        Tls(Box<TlsStream<TcpStream>>),
    }