DOCSEARCHbyAlgolia
Submit Docs

OpenWorkflow Documentation

repository·main·Indexed 23 days ago

https://github.com/openworkflowdev/openworkflow

A TypeScript framework for building durable, resumable workflows. OpenWorkflow allows developers to create long-running processes that survive crashes, deployments, or pauses using a step-based execution model via the `defineWorkflow` function and `step.run` for side effects. It includes a CLI (`@openworkflow/cli`) for project initialization, environment diagnostics with `doctor`, worker management, and a monitoring dashboard. Supported backends include PostgreSQL and SQLite.

Tokens
38.9K
Snippets
112
Records
227
Agent score
78%

What's inside OpenWorkflow

  1. Core features of OpenWorkflow

    main

    OpenWorkflow provides several key capabilities for managing long-running processes:

    • Durable: Workflows survive process restarts and server crashes. Completed steps are never repeated.
    • Resumable: After a crash or deployment, workflows automatically pick up from the last completed step.
    • Pausable: Workflows can sleep for durations ranging from seconds to months without consuming active resources.
    • Type Safe: Provides first-class TypeScript support for defining and consuming workflow inputs and outputs.

  2. Current capabilities in OpenWorkflow

    main

    The current npm release of OpenWorkflow includes the following core features and capabilities:

    • Backends: Support for PostgreSQL and SQLite.
    • CLI: Accessible via npx @openworkflow/cli.
    • Dashboard: Accessible via npx @openworkflow/cli dashboard.
    • Runtimes: Support for Bun.
    • Execution Model: Workers with concurrency control, parallel step execution, and step memoization & retries.
    • Workflow Control: Sleeping (pausing) workflows, workflow versioning, workflow cancellation, and graceful shutdown.
    • Reliability: Configurable retry policies and idempotency keys.
    • Observability: Prometheus /metrics endpoint.
    • Advanced Workflow Logic: Child workflows via step.runWorkflow and Signals via step.sendSignal and step.waitForSignal.

  3. Use llms.txt for AI agent documentation discovery

    main
    OpenWorkflow provides an /llms.txt file at the root of the documentation site. This file follows the emerging standard for providing AI agents with a structured map of the documentation, enabling tools to efficiently discover and retrieve plain text versions of relevant pages.

  4. How Signals work in OpenWorkflow

    main

    Signals enable workflows to communicate at runtime without polling. A workflow can pause and wait for a specific signal, and another workflow or external application code can send that signal to wake it up with attached data. This is ideal for waiting on non-timer events like human approvals, webhook callbacks, or coordination messages from other workflows.

    Important: Signals are not buffered. If a signal is sent before any workflow is actively waiting for it, the signal is lost.

  5. Namespace visibility and Dashboard behavior

    main

    Data Visibility

    Isolation is strict. A worker or client configured with a specific namespaceId will never see or pick up work from a different namespace.

    NamespaceCan See
    productionOnly production workflows
    stagingOnly staging workflows
    defaultOnly default workflows

    Dashboard

    The OpenWorkflow dashboard displays workflows for the namespace configured in your configuration file. To view workflows from a different namespace, you must update the namespaceId in your config and restart the dashboard.

  6. How the OpenWorkflow execution lifecycle works

    main

    OpenWorkflow uses a deterministic replay model to ensure reliability. When a workflow run is initiated, it follows these stages:

    1. Enqueue: Your application enqueues a run (e.g., using ow.runWorkflow(workflow.spec, input)), which creates a pending run in the database.
    2. Claim: A worker claims the run and marks its status as running.
    3. Replay: The worker replays the workflow code from the beginning.
    4. Checkpointing: During replay, completed steps return their cached outputs from the database. Only new steps are actually executed, and their results are persisted.
    5. Finalization: The run transitions to a terminal state: completed, failed, or canceled. If the workflow includes a step.sleep, the run stays in the running state while durably parked.

    This replay mechanism ensures that if a worker crashes, a replacement worker can resume the workflow from the last durable checkpoint without re-executing completed work.

  7. How workers execute workflows

    main

    Workers use a replay-based execution model to ensure reliability and statelessness:

    1. Claim: The worker atomically claims the workflow run from the database.
    2. Load history: The worker loads all completed step attempts.
    3. Replay: The worker executes the workflow function from the beginning.
    4. Memoization: Completed steps return cached results instantly from the database.
    5. Execute: New steps are executed and their results are stored.
    6. Complete: The workflow status is updated to completed or failed.

  8. Isolate environments using SQLite Namespaces

    main

    You can use the namespaceId option to isolate different environments (like development and testing) while using the same physical database file.

    const devBackend = BackendSqlite.connect("./backend.db", {
  namespaceId: "development",
});

const testBackend = BackendSqlite.connect("./backend.db", {
  namespaceId: "test",
});

  9. Core concepts of OpenWorkflow taxonomy

    main

    Understanding the relationship between the following entities is key to using OpenWorkflow:

    • Workflow: A deterministic, versioned, and resumable durable function that orchestrates multiple steps.
    • Workflow Run: A single execution instance of a workflow, managed as a state machine.
    • Step: A durable, memoized checkpoint within a workflow representing a unit of work (e.g., an API call).
    • Step Attempt: A record in the Backend representing the state and result of a single step attempt within a specific workflow run.
    • Worker: A long-running process in your infrastructure that polls the Backend, executes workflow code, and persists results.
    • Client: The SDK component used by your application to start and query workflow runs.
    • Backend: The pluggable persistence layer (e.g., Postgres or SQLite) that acts as the single source of truth and job queue.
    • Signal: A named, point-in-time message sent to workflows waiting on that signal name. Signals carry an optional JSON payload. If no workflow is waiting at the time of sending, the signal is not persisted.
    • availableAt: A timestamp on a workflow run that controls its visibility to workers (used for scheduling, heartbeating, and timers).
    • deadlineAt: An optional timestamp specifying when a workflow must complete; if reached, the run is marked as failed.

  11. How the OpenWorkflow execution flow works

    main

    OpenWorkflow follows a worker-driven model where the database (Backend) coordinates execution without a central orchestrator server. The lifecycle follows these steps:

    1. Workflow Registration: Workers automatically discover and register workflow code based on openworkflow.config.ts (defaulting to the openworkflow/ directory) upon startup.
    2. Workflow Invocation: The application uses the Client to create a new entry in the workflow_runs table with a pending status.
    3. Job Polling: A Worker polls the workflow_runs table for runs where availableAt is in the past and status is pending, running (parked or expired lease), or legacy sleeping. It uses an atomic FOR UPDATE SKIP LOCKED query to claim the run.
    4. Code Execution (Replay Loop): The Worker loads the history of completed step_attempts and executes the workflow code from the beginning, using the history to memoize results of previously completed steps.
    5. Step Processing: When a new step is encountered, the Worker creates a step_attempt (status running), executes the function, and updates it to completed upon success.
    6. State Update: The Worker updates the Backend with each step_attempt and updates the workflow_run status (e.g., to completed or running if parked for a wait).

  12. How workflow version matching works

    main

    The OpenWorkflow worker matches a workflow run to an implementation based on an exact match of both name and version.

    • If a run has a specific version (e.g., 1.0.0), the worker looks for an implementation with that exact version.
    • If a run has no version (null), it matches an implementation with null version.
    • If no match is found (e.g., the run is 3.0.0 but only 1.0.0 and 2.0.0 are registered), the worker records an error and reschedules the run as pending with a retry backoff. This allows time for a worker with the correct version to be deployed and pick up the run.