DOCSEARCHbyAlgolia
Submit Docs

Qaynaq Documentation

website·Indexed 29 days ago

https://qaynaq.io

A data pipeline platform using a Coordinator & Worker model for horizontal scalability. Qaynaq allows users to build flows that move data from inputs through processors to outputs, featuring a built-in Model Context Protocol (MCP) server, RAG pipeline tools, and integrations with Kafka, PostgreSQL, and Google Workspace.

Tokens
44.6K
Snippets
149
Records
335
Agent score
53%

What's inside Qaynaq

  1. Overview of Qaynaq's built-in MCP server

    Qaynaq's coordinator includes a built-in Model Context Protocol (MCP) server located at the /mcp endpoint. Any flow configured with an MCP Tool input is automatically exposed as a tool that AI assistants can discover and call. Tools are synchronized every 30 seconds. The server is accessible at http://<coordinator-host>:<port>/mcp.

  2. Overview of Qaynaq Pipeline Processors

    Processors are components used within a pipeline to transform, validate, or route messages. They are executed in sequential order between the pipeline's input and output. Processors can be added during the creation or editing of a flow via the UI.

  3. Overview of Qaynaq data pipeline tool

    Qaynaq is a modern, open-source (Apache 2.0) data pipeline tool designed as an alternative to Airbyte and Fivetran. It uses a coordinator-worker architecture with Bento as the flow processing engine to provide at-least-once delivery guarantees. It is distributed as a single lightweight binary written in Go, requiring no Docker, JVM, or external dependencies.

  4. Overview of Qaynaq Connections

    Qaynaq Connections store OAuth credentials for third-party services. Once authorized, these connections can be attached to MCP server entries or any component requiring an access token. The coordinator handles token refreshing centrally, ensuring that worker nodes never see refresh tokens.

  5. Understand Qaynaq Coordinator and Worker architecture

    Qaynaq uses a Coordinator & Worker model for horizontal scalability.

    The Coordinator (Control Plane):

    • Serves the web UI and REST API (default port 8080).
    • Manages flow definitions (CRUD operations, start/stop).
    • Handles worker registration and health tracking via gRPC.
    • Balances workload distribution across workers.
    • Stores state in SQLite or PostgreSQL.
    • Exposes an MCP server at /mcp for AI assistants.
    • Only one coordinator exists per deployment.

    The Workers (Data Plane):

    • Register with the coordinator via gRPC on startup.
    • Execute the actual input, processing, and output pipelines of flows.
    • Are stateless; all configuration is provided by the coordinator.
    • Send periodic heartbeats to the coordinator.
    • Can be scaled horizontally by adding or removing worker instances.

  6. Understand the Qaynaq flow architecture

    A flow is the core building block in Qaynaq, defining a data pipeline that moves data from an input to an output through optional processing stages. The structure follows this sequence: Input → Processors → Output.

    • Input: The data source (e.g., Kafka, HTTP, database CDC).
    • Processors: Optional transformations applied to messages (e.g., mapping, validation, schema decoding).
    • Output: The data destination (e.g., database, Kafka, HTTP endpoint).

  7. Understand Qaynaq OAuth2 user roles

    When OAuth2 authentication is enabled, Qaynaq supports two primary roles to control access to the dashboard, API, and MCP server. If no role configuration is provided, all users passing the initial allowlist gate are treated as Admins.

    • Admin: Full access to the management dashboard, the API (/api/*), and the MCP server (/mcp).
    • MCP: Can complete the OAuth handshake from MCP clients (e.g., Claude Desktop, Cursor) and access the MCP server, but cannot access the dashboard or API. They are redirected to a page containing the MCP server URL and setup snippets.
    • No Role: Users who authenticate but do not match Admin or MCP criteria are sent to a 'no access' page.

    Note: Basic auth always treats the configured user as Admin. If AUTH_TYPE=none, all users are treated as Admin.

  8. Split byte streams into messages using the Lines scanner

    The Lines scanner is the default tool for processing line-oriented text such as log files, NDJSON, and plain text. It splits an incoming byte stream into individual messages based on line endings.

  9. Aggregate external MCP servers via the Qaynaq Proxy

    Qaynaq's /mcp endpoint can serve tools from your own native flows and external MCP servers simultaneously. This allows AI assistants (e.g., Claude Desktop, Cursor, Claude Code) to connect to a single endpoint instead of maintaining separate connections for every service. External servers are registered in the MCP Servers section and their tools appear within ~30 seconds, namespaced by the server name.

  10. Understand flow assignment and worker execution

    When a flow is set to 'active', the coordinator assigns it to an available worker. The worker is responsible for executing the pipeline. If a worker fails or goes offline, the coordinator can reassign the flow to a different available worker to ensure continuity.

  11. Decompress byte streams using the Decompress scanner

    The Decompress scanner decompresses a byte stream using a specified algorithm and delegates the resulting decompressed bytes to a child scanner. This allows for reading compressed source files such as .csv.gz, .json.zst, or .tar.bz2 by stacking the Decompress scanner on top of other scanners.

  12. Convert XML document streams to JSON using the XML Documents scanner

    The XML Documents scanner consumes a stream of XML documents and converts each individual document into a separate JSON message. It includes a configuration option to handle data type casting for numeric and boolean values.