Xata Documentation

repository·main·Indexed 21 days ago

https://github.com/xataio/xata

Xata is an open-source, cloud-native Postgres platform featuring Copy-on-Write (CoW) branching, scale-to-zero functionality, and auto-scaling. Built on CloudNativePG and OpenEBS, it provides a serverless driver for SQL access over HTTP/websockets and a control plane managed via REST APIs and a CLI. The platform supports high availability with read-replicas and automatic failover.

Tokens
6.7K
Snippets
29
Records
37
Agent score
76%

What's inside Xata

  1. What is Xata?

    main

    Xata is an open-source, cloud-native platform for self-hosting multiple Postgres instances on Kubernetes. It is designed to provide advanced database management features like fast branching and scale-to-zero functionality.

    Key Features

    • Fast Branching: Uses Copy-on-Write (CoW) at the storage level to 'copy' large amounts of data (TB) in seconds.
    • Scale-to-Zero: Automatically removes compute instances during inactivity and restores them upon new connections.
    • Auto-scaling: Automatically scales compute nodes and uses bin-packing for cost efficiency.
    • High Availability: Supports read-replicas and automatic failover.
    • Serverless Driver: Provides SQL access over HTTP/websockets.
    • Control Plane: Managed via REST APIs and a CLI with granular RBAC via API keys.

    Core Technologies

    Xata is built on top of:

    • CloudNativePG: A Postgres operator for Kubernetes handling HA, failover, and backups.
    • OpenEBS: A cloud-native storage project providing local and replicated storage engines.

    Architecture Components

    • SQL Gateway: Routes requests, handles IP filtering, and manages scale-to-zero wake-ups.
    • Branch Operator: Manages resources for database branches.
    • Control Plane: Comprised of clusters and projects services.
    • Auth Service: Based on Keycloak.
    • Scale-to-zero CNPG plugin: Hibernates branches during inactivity.
  2. Install and authenticate the Xata CLI

    main

    After deploying the platform, install the Xata CLI and authenticate using a local profile. For local development, use the following credentials:

    • Email: dev@xata.tech
    • Password: Xata1234!
    1. Install the CLI:
    curl -fsSL https://xata.io/install.sh | bash
    1. Authenticate to the local profile:
    xata auth login --profile local --issuer http://localhost:8080/realms/xata --api-url http://localhost:5001 --client-secret devsecret
    1. Switch to the local profile:
    xata auth switch local
    # Install CLI
    curl -fsSL https://xata.io/install.sh | bash
    
    # Authenticate to a local profile
    xata auth login --profile local --issuer http://localhost:8080/realms/xata --api-url http://localhost:5001 --client-secret devsecret
    
    # Set the profile
    xata auth switch local
  3. Run Xata locally using Kind and Tilt

    main

    To run a local development instance of Xata, you must have Docker, Kind, and Tilt installed. The process involves creating a Kubernetes cluster via Kind and then using Tilt to deploy the platform components.

    1. Create a Kind cluster:
    kind create cluster --wait 10m
    1. Deploy the platform using Tilt:
    tilt up

    Wait for all resources to become ready. Note that the initial deployment may take significant time due to image downloads.

    kind create cluster --wait 10m
    tilt up
  4. Create a project and a branch with the Xata CLI

    main

    Once authenticated, you can manage your Postgres instances using the CLI. You can create a new project (which includes a main branch) and subsequently create child branches for testing or development.

    1. Create a project and its main branch:
    xata project create --name my-project
    1. Create a child branch:
    xata branch create
    # Create project and main branch
    xata project create --name my-project
    
    # Create a child branch
    xata branch create
  5. Use the Xata CLI service entrypoint

    main

    The Xata CLI is structured around services. When running a service command, the CLI automatically handles configuration loading, service initialization, and graceful shutdown.

    Running the service command without arguments typically starts the service in its default mode. The CLI lifecycle for any service command follows this sequence:

    1. Read Config: Loads configuration settings.
    2. Init: Initializes the service.
    3. Run: Executes the specific command (e.g., run, setup, or version).
    4. Close: Performs cleanup and closes the service.
    # Example of the command structure (conceptual)
    xata <service-name> [command]
    
    Available commands for a service:
    - `<service-name>` (default): Starts the service
    - `run`: Executes the service run logic
    - `setup`: Executes the service setup logic
    - `version`: Displays the service version
  6. Use gen-scopes.go to generate authentication scope code

    main

    The gen-scopes.go tool is a CLI utility used to parse an OpenAPI specification (YAML) and generate Go code that maps API routes and HTTP methods to their required authentication scopes.

    It produces a file named scopes.gen.go containing two primary functions:

    1. GetScopes(method, path string) []string: Returns the required scopes for a specific HTTP method and path.
    2. GetAllScopes() []string: Returns a list of all unique scopes defined in the specification.

    Requirements for OpenAPI Scopes: All scopes defined in the security section of your OpenAPI spec must use one of two suffixes:

    • :read
    • :write

    If a scope does not end with one of these suffixes, the tool will fail with an error.

    Usage: Run the tool using go run and provide the path to your OpenAPI YAML file as the first argument.

    go run gen-scopes.go <openapi.yaml>
  7. Configure the serverless gateway server

    main

    The Config struct defines the operational parameters for the serverless gateway.

    FieldTypeDescription
    ListenAddressstringThe TCP address to listen on (e.g., ":8080").
    TLSCert*tls.CertificateAn optional TLS certificate for enabling HTTPS. If nil, the server runs over plain HTTP.
    type Config struct {
    	ListenAddress string
    	TLSCert       *tls.Certificate
    }
  8. Initialize a gRPC client connection with NewClient

    main

    Use NewClient to create a ClientConnection wrapper around a standard gRPC connection. This function automatically configures the connection with several production-ready defaults:

    • Insecure Credentials: Uses insecure.NewCredentials().
    • Increased Message Size: Sets the receive limit to DefaultMaxRecvMsgBytes (16 MiB) to avoid the standard 4 MiB gRPC limit.
    • Retry Policy: Implements a default retry policy for UNAVAILABLE status codes with up to 5 attempts and exponential backoff.
    • Observability: Automatically attaches logging interceptors and telemetry handlers via the provided o11y.O instance.

    You can pass additional grpc.DialOption arguments to further customize the connection.

    import (
    	"xata/internal/grpc"
    	"xata/internal/o11y"
    	"google.golang.org/grpc"
    )
    
    // Assuming 'o' is an initialized o11y.O instance
    conn, err := grpc.NewClient(o, "your-grpc-url:port", grpc.WithBlock())
    if err != nil {
    	// handle error
    }
    // Use conn.ClientConn to interact with your gRPC services
  9. Use the Client interface for analytics operations

    main

    The Client interface provides methods for tracking individual events and group-level events in Xata's analytics system. It supports tracking single events, tracking events associated with a specific group, registering new groups, and gracefully closing the client connection.

    // Example usage of the Client interface
    type Client interface {
    	Track(ctx context.Context, event events.Event)
    	TrackGroup(ctx context.Context, event events.Event)
    	RegisterGroup(ctx context.Context, groupType, groupKey string) error
    	Close(ctx context.Context) error
    }
  10. Reference the generated GetScopes and GetAllScopes functions

    main

    The gen-scopes.go tool generates a Go file (scopes.gen.go) in the spec package. This file provides the following API for looking up authentication scopes:

    func GetScopes(method, path string) []string

    Returns the required scopes for a given route and method. The method argument is case-insensitive (it is converted to uppercase internally). The path argument should match the route pattern used in the OpenAPI spec (e.g., /users/{id}).

    func GetAllScopes() []string

    Returns all unique scopes defined in the OpenAPI spec as a slice of strings.

    package spec
    
    // Example usage of generated functions:
    
    // Get scopes for a specific route
    scopes := GetScopes("GET", "/users/:id")
    
    // Get all available scopes
    allScopes := GetAllScopes()
  11. Initialize an OpenFeature client with NewClient

    main

    Use NewClient to create a new *Client instance. This function requires a clientName (string) and an openfeature.FeatureProvider. It automatically sets the global provider using openfeature.SetProviderAndWait before returning the client. If setting the provider fails, an error is returned.

    import (
    	"github.com/open-feature/go-sdk/openfeature"
    	"your-project/internal/openfeature/client"
    )
    
    // Assuming you have a provider implementation
    provider := myProvider
    client, err := client.NewClient("my-client", provider)
    if err != nil {
    	// handle error
    }
  12. Initialize a serverless gateway with NewServer

    main

    The NewServer function initializes a new Server instance for the serverless gateway. It configures the underlying Echo router with CORS support (allowing all origins), body size limits, and header normalization middleware. It also registers the core gateway handlers defined in the spec package.

    To use NewServer, you must provide:

    • An observability object (*o11y.O)
    • A session.BranchResolver for resolving branches
    • A session.ClusterDialer for cluster connections
    • *metrics.GatewayMetrics for telemetry
    • A session.IPFilter for IP-based access control
    • A Config struct containing the ListenAddress and an optional TLSCert.
    server, err := serverless.NewServer(
        o,
        resolver,
        dialer,
        gwMetrics,
        ipFilter,
        serverless.Config{
            ListenAddress: ":8080",
            TLSCert:       &cert,
        },
    )