DOCSEARCHbyAlgolia
Submit Docs

Conductor Workflow Engine

repository·main·Indexed 12 days ago

https://github.com/conductor-oss/conductor

An open-source, internet-scale agentic workflow engine designed to orchestrate microservices and AI agents. It provides durable execution and an AI module with integrations for LLM providers and vector databases, supporting Agent2Agent (A2A) patterns, Retrieval-Augmented Generation (RAG), and the Model Context Protocol (MCP).

Tokens
250.6K
Snippets
608
Records
947
Agent score
96%

What's inside Conductor

  1. Overview of Conductor E2E Test Suites

    main

    The E2E test suite is organized into packages covering different functional areas of the engine:

    PackageCoverage
    controlControl flow: DO_WHILE, SWITCH, SUB_WORKFLOW, DYNAMIC_FORK
    taskTask execution: AGENT lifecycle, WAIT, HTTP, concurrency limits, timeouts, and backoff
    workflowWorkflow management: Retry, restart, rerun, search, priority, and failure workflows
    processingData processing: GraalJS inline tasks, SET_VARIABLE, JSON_JQ
    eventEvent handling logic
    metadataCRUD operations for workflow/task definitions and event handler registration

    Note on Agent Lifecycle Tests: The io.conductor.e2e.task.AgentTaskTests suite is a deterministic suite that requires no LLM credentials. It covers complex scenarios like nested execution, callback/poll intervals, human input, and various cancellation types.

  2. Overview of Conductor AI Task Types

    main

    Conductor provides a variety of AI task types that can be integrated into workflows. These tasks range from Large Language Model (LLM) interactions to media generation and vector database operations.

    Core Task Categories:

    • LLM Interactions: Conversational chat, single-prompt completion, and embedding generation.
    • Media Generation: Creating images, audio (TTS), and video (async).
    • Vector Database Operations: Indexing text, storing pre-computed embeddings, and performing semantic searches.
    • MCP Integration: Listing and calling tools from Model Context Protocol (MCP) servers.
    • Document Generation: Converting Markdown to PDF.

  3. Overview of the Conductor AI Module

    main
    The Conductor AI module enables AI-powered workflows by providing built-in integrations with 13 popular LLM providers and vector databases. You can define AI tasks directly within your workflows, including chat, embeddings, image generation, audio synthesis, video generation, document generation, and tool calling, without needing to manage separate worker infrastructure.

  4. Overview of Conductor SDKs

    main

    Conductor provides a suite of open-source SDKs that allow you to build task workers and define workflows as code in various programming languages. Every SDK includes:

    • Task Polling: Automatically fetches tasks from the Conductor engine.
    • Workflow Management: Tools to manage and orchestrate workflows.
    • Full API Coverage: Access to the complete Conductor orchestration engine capabilities.

    By using an SDK, you can focus on business logic while the engine handles retries, state management, and orchestration.

  6. Key features and differentiators of Conductor

    main

    Conductor provides several key capabilities for distributed orchestration and AI agent workflows:

    Execution Model

    • Durable execution: Every step is persisted. Workflows survive crashes and restarts without losing state, featuring automatic retries with configurable policies.
    • Full replayability: You can restart any workflow from the beginning, rerun from a specific task, or retry just a failed step. This applies to completed, failed, or timed-out workflows.
    • Deterministic execution: JSON definitions separate orchestration from implementation. This ensures no side effects or hidden state, meaning every run produces the same task graph given the same inputs.

    AI and LLM Integration

    • Native LLM providers: Supports 14+ providers (including Anthropic, OpenAI, Gemini, Bedrock, Mistral, and Azure OpenAI) as system tasks without custom code.
    • MCP (Model Context Protocol) native integration: Connects AI agents to external tools and data sources using the open standard.
    • Built-in RAG support: Native integration with 3 vector databases: Pinecone, pgvector, and MongoDB Atlas.

    Ecosystem and Infrastructure

    • Polyglot Workers: Support for 7+ language SDKs (Java, Python, Go, JavaScript, C#, Clojure, Ruby, and Rust).
    • Event-driven triggers: Integration with 6+ message brokers (Kafka, NATS JetStream, SQS, AMQP, Azure Service Bus, etc.).
    • Flexible Persistence: Supports 5+ backends (PostgreSQL, MySQL, Redis, Cassandra, and SQLite).

  7. Rust SDK Overview and Requirements

    main

    The Conductor Rust SDK allows you to build type-safe workers and manage workflows using Rust. It supports both synchronous and asynchronous task execution.

    Requirements:

    • Rust version 1.75 and above (2021 edition).

    Key Capabilities:

    • Async Workers: Use async fn for I/O-bound tasks (API calls, database queries) to leverage high concurrency.
    • Sync Workers: Use regular fn for CPU-bound or blocking work.
    • Language Agnostic: You can mix Rust workers with workers written in other languages (Python, Java, Go, etc.) within a single workflow.

  8. Explore Conductor workflow orchestration recipes

    main

    The Conductor cookbook provides production-ready, copy-pasteable workflow orchestration recipes. These recipes include complete JSON workflow definitions and the necessary commands to register and execute them. Use these recipes to implement common patterns such as:

    • Microservice orchestration: HTTP service chains, conditional branching, and parallel HTTP calls using Fork/Join.
    • Dynamic parallelism: Dynamic forks (different tasks per branch), fan-out (same task across multiple branches), and parallel sub-workflows.
    • Wait and timer patterns: Fixed delays, scheduled execution, external signals, and human-in-the-loop approvals.
    • Task timeouts and retries: Exponential backoff (with cap and jitter), lease extensions for long-running workers, and hard SLAs using totalTimeoutSeconds.
    • Scheduled workflows: Cron-triggered execution, catchup mechanisms, and bounded time windows.
    • Event-driven patterns: Publishing to Kafka, NATS, RabbitMQ, or SQS, and using event handlers to trigger workflows or complete tasks.
    • AI & LLM orchestration: Chat completion, RAG pipelines, MCP agents with function calling, and image generation.
    • Dynamic workflows as code: Implementing workflows using Python for sequential chains, loops, and runtime-generated definitions.

  9. Conductor REST API Reference Overview

    main

    Conductor exposes a REST API for managing the entire workflow lifecycle. All endpoints are relative to the server's base URL (defaulting to http://localhost:8080/api/).

    API Sections

    SectionBase PathDescription
    Metadata/api/metadataRegister, update, validate, and delete workflow and task definitions
    Start Workflow/api/workflowStart workflows asynchronously, synchronously, or with dynamic definitions
    Workflow/api/workflowManage executions: get status, pause, resume, retry, restart, terminate, search
    Task/api/tasksPoll for tasks, update results, manage queues, view logs, search
    Bulk Operations/api/workflow/bulkPause, resume, restart, retry, terminate, or remove workflows in batch
    Event Handlers/api/eventCreate and manage event-driven workflow triggers
    Task Domains—Route tasks to specific worker pools at runtime

  10. Understand Conductor Schema usage

    main

    Conductor uses JSON Schemas to define the structure and validation rules for its core data models. Developers can use these schemas for:

    • Validation: Verifying workflow and task definitions before submission.
    • Documentation: Auto-generating API docs and client libraries.
    • IDE Support: Enabling autocomplete and real-time validation in editors.
    • Code Generation: Creating strongly-typed client code.
    • Contract Testing: Ensuring API responses match expected formats.

    All schemas conform to the JSON Schema Draft 07 specification.