Conveyor CI

Conveyor CI uses a conveyor-belt paradigm to manage CI/CD processes. The core components include:

• Resource: An internal object representing the item being acted upon (e.g., source code, an application, or a program).
• Drivers: Software components that perform specific actions on a Resource based on its state.
• Pipelines: Configuration objects that define the specific order of execution for drivers. Pipelines prevent the race conditions that occur when drivers execute simultaneously and allow for dependency management (e.g., ensuring a 'compile' driver finishes before an 'upload' driver starts).
• Pipeline Engine: A concurrent process in the core runtime that routes resources to drivers in the predefined order and manages the pipeline's state based on driver results.

Conveyor CI provides language-specific SDKs to simplify interactions with its API. The available SDKs are:

• Go: Stable release. Source code is available at github.com/open-ug/conveyor and documentation is hosted on Go Doc.
• Python: Currently in active development. Source code is located in the sdk/python directory of the main repository.
• Rust: Currently in active development. Source code is located in the sdk/rust directory of the main repository.

To implement a language-specific SDK for Conveyor CI, the following core components must be developed:

• Driver Manager (Orchestrator): Manages NATS/JetStream connections, creates consumers, and executes the 'Try-Catch-Ack' loop.
• Driver (User Logic): Defines resource filters and implements the domain-specific Reconcile logic.
• Driver Logger (Contextual Logger): Enforces structured logging and automatically attaches run_id, driver_name, and timestamps to logs.
• API Client (Stateless HTTP Client): Provides typed methods for REST API interactions regarding Secrets, Status updates, and Metadata.

Conveyor CI separates the control plane (orchestration) from the drivers (execution).

• Control Plane (Provided by Conveyor CI): Responsible for job lifecycle management, state persistence, queuing, retries, log aggregation, and APIs.
• SDK (Provided by Conveyor CI): Provides a typed interface for interacting with the engine.
• Drivers (Provided by the User): The custom code responsible for actually running the workloads.

This separation allows for infrastructure independence, custom security models, hardware flexibility, and multi-environment support (e.g., edge devices, air-gapped networks, or custom hardware).

Conveyor CI uses a driver-based architecture to remain infrastructure-agnostic. The core control plane manages job lifecycles, retries, and cancellations, but delegates the actual workload execution to user-defined drivers. This allows workloads to be executed across diverse environments including:

• Kubernetes
• Physical servers
• Virtual Machines (VMs)
• Edge devices
• Custom infrastructure

Drivers are responsible for executing the workload and streaming logs and status updates back to the Conveyor control plane.

Conveyor CI uses a modular, event-driven architecture where system state is stored as Resources and processed by Drivers. Instead of static pipelines, the system reacts to Resource mutations (creation, updates, or deletions). The architecture consists of three primary conceptual pillars:

• Resources: Store the current state of CI/CD processes.
• Resource Definitions: Define the schema and validation rules for Resources.
• Drivers: Watch for Resource changes and transform that state into actual CI/CD executions.

Conveyor CI utilizes an event-driven architecture based on NATS JetStream for low latency, replayability, and scalability. The system employs a language-agnostic runtime model where a shared conceptual runtime is implemented via language-specific SDKs.

Data Persistence:

• Events are stored in JetStream.
• Logs and Resources are stored by Conveyor CI.

Driver Lifecycle:

1. Initialization
2. Event subscription
3. Reconciliation
4. Logging and reporting
5. Termination or scale-out