DOCSEARCHbyAlgolia
Submit Docs

rudder-server Documentation

repository·master·Indexed 26 days ago

https://github.com/rudderlabs/rudder-server

Documentation for rudder-server, the core backend of RudderStack, an open-source Customer Data Platform (CDP). It covers deployment via Docker and Kubernetes, local development using the devtool for etcd management and event simulation, and the AsyncDestinationManager framework for building asynchronous data pipeline integrations.

Tokens
9.5K
Snippets
16
Records
76
Agent score
89%

What's inside rudder-server

  2. Overview of AsyncDestinationManager

    master

    AsyncDestinationManager is a framework within RudderStack's batch router designed to handle asynchronous data uploads to external destinations. It is used for integrations that require:

    • Bulk data uploads: Handling CSV or JSON files.
    • Asynchronous processing: Polling destination APIs for completion status.
    • Complex authentication: Managing OAuth or token-based auth.
    • File-based transfers: Using protocols like SFTP or cloud storage.

  4. Understand RudderStack Architecture

    master

    RudderStack is a standalone system that depends only on a PostgreSQL database.

    Key architectural components:

    • Backend: Written in Go.
    • Frontend/UI: Built with React.js.
    • Data Flow: Collects events from sources and routes them to destinations (tools/warehouses) or transforms them using a JavaScript-based framework.

  5. Configure environment variables for regulation-worker

    master

    Before running the regulation-worker, you must set the following environment variables to ensure correct connectivity and authentication:

    • CONFIG_BACKEND_URL: The URL for the configuration backend.
    • WORKSPACE_TOKEN: Required for single-tenant deployments (workspace secret).
    • WORKSPACE_NAMESPACE: Required for multi-tenant deployments (namespace secret).
    • DEST_TRANSFORM_URL: The transformer URL required to make downstream API calls to destinations of API type.

  6. Run a load test using wrk and post.lua

    master

    After generating your base64 encoded header, replace the placeholder <base_64_encoded_writekey_colon> in the post.lua script with your actual value. Then, execute the load test using the following command to target the batch endpoint:

    wrk -t100 -c100 -d120s -s post.lua http://localhost:8080/v1/batch

    Flag breakdown:

    • -t100: Use 100 threads.
    • -c100: Keep 100 HTTP connections open.
    • -d120s: Run the test for 120 seconds.
    • -s post.lua: Use the post.lua script for request logic.
    wrk -t100 -c100 -d120s -s post.lua http://localhost:8080/v1/batch

  7. Migrate existing destinations to AsyncDestinationManager

    master

    To migrate an existing destination to the AsyncDestinationManager framework, follow these steps:

    1. Identify the current upload pattern (sync vs async).
    2. Extract configuration parsing logic.
    3. Separate file generation from upload logic.
    4. Implement the AsyncDestinationManager interface.
    5. Add comprehensive tests.
    6. Update destination routing in the manager factory.

  9. Generate a Basic Authorization Header for wrk

    master

    To perform load testing with wrk against endpoints requiring Basic Authentication, you must generate a base64 encoded string of your write key followed by a colon. Use the following command to generate the required value:

    echo -n <your_write_key>: | openssl base64
    echo -n <your_write_key>: | openssl base64

  10. Onboard a new Async Destination integration

    master

    To add a new integration to the AsyncDestinationManager, follow these steps:

    1. Determine Integration Type

    Choose between Complex Async (polling required), Simple Async (fire-and-forget), or SFTP.

    2. Create Integration Directory

    Create a directory under asyncdestinationmanager/ with the following structure:

    asyncdestinationmanager/
└── your-destination/
    ├── manager.go          # Main manager implementation
    ├── types.go           # Data structures
    ├── your-destination.go # Core logic
    ├── utils.go           # Helper functions (optional)
    ├── testdata/          # Test files
    └── your-destination_test.go

    3. Implement Required Files

    • manager.go: Implement NewManager(logger logger.Logger, statsFactory stats.Stats, destination *backendconfig.DestinationT) (common.AsyncDestinationManager, error).
    • types.go: Define destination-specific configuration and response types.

    4. Implement Core Interface

    Implement either the full AsyncDestinationManager (Pattern A) or use SimpleAsyncDestinationManager (Pattern B) if the destination is fire-and-forget.

    5. Register in Manager Factory

    • Add the destination name to the asyncDestinations slice in utils.go.
    • Add a new case to the switch statement in newRegularManager within manager.go to return your new manager implementation.

  11. Install RudderStack via Docker, Kubernetes, or Local Setup

    master

    You can deploy RudderStack using the following methods:

    • Docker: For containerized deployment.
    • Kubernetes: Recommended for production environments. Use the official Helm charts for the most up-to-date images.
    • Developer machine setup: For local development and testing.

    After installation, verify your setup by sending test events to the server.

  12. Manage etcd state with the devtool

    master

    Since etcd is a critical component for rudder-server in normal mode, the devtool provides commands to manipulate its state for testing purposes. You can:

    • Switch between normal and degraded modes of rudder-server.
    • Change the workspace the server is responsible for.
    • List all etcd keys for debugging.
    • Use the --no-wait flag to avoid waiting for changes to be acknowledged.