DOCSEARCHbyAlgolia
Submit Docs

Lobster Workflow Runtime

repository·main·Indexed 22 days ago

https://github.com/openclaw/lobster

An OpenClaw-native workflow engine for AI agents featuring typed, JSON-first pipelines, jobs, and approval gates. Lobster enables deterministic and resumable multi-step workflows to reduce token consumption. It includes a CLI for executing .lobster files, visualizing workflow structures via Mermaid, DOT, or ASCII graphs, and integrating with LLMs and OpenClaw agents through specialized pipeline stages like llm.invoke and openclaw.agent.

Tokens
16.5K
Snippets
42
Records
94
Agent score
76%

What's inside Lobster

  1. What is Lobster and how does it work?

    main

    Lobster is a workflow runtime for OpenClaw designed for safe, deterministic, and stateful automation. Unlike standard LLM-driven workflows that re-plan every step (which is expensive and risky), Lobster executes multi-step pipelines that follow a defined logic.

    Core Capabilities:

    • Deterministic Execution: Runs a predefined pipeline rather than relying on LLM re-planning at every step.
    • Human-in-the-loop Safety: Uses an approve primitive to create hard stops in the workflow, preventing irreversible side effects without explicit user consent.
    • Statefulness (Memory): Tracks cursors and checkpoints (e.g., "last processed email ID") so workflows can resume exactly where they left off.
    • Efficiency: Reduces token usage by replacing multiple LLM tool-call cycles with a single call to a Lobster workflow that returns structured results.

  2. Define steps in a Lobster workflow

    main

    Lobster workflows use a YAML-like syntax to define steps. There are three primary types of steps:

    • run: or command:: Executes deterministic shell or CLI commands.
    • pipeline:: Executes native Lobster stages (e.g., llm.invoke). These share the same args/env/results model as shell steps.
    • approval:: Creates a hard workflow gate between steps.

    Common step configurations include:

    • id: Unique identifier for the step.
    • stdin: Specifies input (e.g., $stepId.stdout or $stepId.json).
    • when / condition: Controls if a step runs based on previous results.
    • env: Environment variables for the step.
    • cwd: Current working directory.
    • retry, timeout_ms, on_error: Error handling and recovery.

    Example workflow:

    name: jacket-advice
args:
  location:
    default: Phoenix
steps:
  - id: fetch
    run: weather --json ${location}

  - id: confirm
    approval: Want jacket advice from the LLM?
    stdin: $fetch.json

  - id: advice
    pipeline: >
      llm.invoke --prompt "Given this weather data, should I wear a jacket?
      Be concise and return JSON."
    stdin: $fetch.json
    when: $confirm.approved
    name: jacket-advice
args:
  location:
    default: Phoenix
steps:
  - id: fetch
    run: weather --json ${location}

  - id: confirm
    approval: Want jacket advice from the LLM?
    stdin: $fetch.json

  - id: advice
    pipeline: >
      llm.invoke --prompt "Given this weather data, should I wear a jacket?
      Be concise and return JSON."
    stdin: $fetch.json
    when: $confirm.approved

  3. How Lobster integrates with OpenClaw

    main

    Lobster acts as the execution layer (the "hands") for OpenClaw (the "brain").

    The Workflow Flow:

    1. User expresses intent (e.g., "triage my email daily").
    2. OpenClaw understands the intent and chooses the appropriate Lobster workflow.
    3. Lobster executes the deterministic pipeline, calling existing OpenClaw tools (like gmail or trello).
    4. Lobster may halt at approve checkpoints, returning a structured result and a resume token to OpenClaw.
    5. OpenClaw presents the results and approval prompts to the user.
    6. User approves/rejects, and Lobster resumes execution.

  4. Invoke OpenClaw tools via shell `run:` steps

    main

    If you are using a shell run: step, you must use the Lobster shim executables to call OpenClaw tools. The preferred shim is openclaw.invoke (or its alias clawd.invoke).

    To use these, ensure OPENCLAW_URL is set (and OPENCLAW_TOKEN if auth is enabled).

    Example workflow step:

    steps:
  - id: greeting
    run: >
      openclaw.invoke --tool llm-task --action json --args-json '{"prompt":"Hello"}'
    openclaw.invoke --tool llm-task --action json --args-json '{"prompt":"Hello"}'

  5. Run Lobster workflows

    main

    Lobster workflows are defined in .lobster files and can be executed using the lobster run command. You can pass arguments to the workflow using the --args-json flag.

    # Run a workflow file
lobster run path/to/workflow.lobster

# Run a workflow with specific arguments
lobster run --file path/to/workflow.lobster --args-json '{"tag":"family"}'
    lobster run path/to/workflow.lobster
lobster run --file path/to/workflow.lobster --args-json '{"tag":"family"}'

  6. Quick start with Lobster

    main

    To get started with Lobster, ensure you have pnpm installed. From the project root, you can run the following commands to install dependencies, run tests, and verify the installation:

    pnpm install
pnpm test
pnpm lint
node ./bin/lobster.js --help
node ./bin/lobster.js doctor
# Run a sample pipeline
node ./bin/lobster.js "exec --json --shell 'echo [1,2,3]' | where '0>=0' | json"

    Note: bin/lobster.js prefers the compiled entrypoint in dist/ if it exists.

    pnpm install
pnpm test
pnpm lint
node ./bin/lobster.js --help
node ./bin/lobster.js doctor
node ./bin/lobster.js "exec --json --shell 'echo [1,2,3]' | where '0>=0' | json"

  7. Handle arguments safely in Lobster workflows

    main

    Lobster uses ${arg} substitution for raw string replacement in shell commands. This is unsafe for values containing quotes, $, backticks, or newlines.

    Best Practice: Use environment variables for complex data. Every resolved workflow argument is automatically exposed as an environment variable following the pattern LOBSTER_ARG_<NAME> (uppercased, non-alphanumeric characters replaced with _). The full arguments object is also available in LOBSTER_ARGS_JSON.

    Example of safe argument usage:

    args:
  text:
    default: ""
steps:
  - id: safe
    env:
      TEXT: "$LOBSTER_ARG_TEXT"
    command: |
      jq -n --arg text "$TEXT" '{"result": $text}'
    args:
  text:
    default: ""
steps:
  - id: safe
    env:
      TEXT: "$LOBSTER_ARG_TEXT"
    command: |
      jq -n --arg text "$TEXT" '{"result": $text}'

  8. Configure the state directory

    main

    The directory where state files are stored is determined by the following priority:

    1. ctx.stateDir: An explicit path provided in the context object.
    2. ctx.env.LOBSTER_STATE_DIR: An environment variable.
    3. Default: ~/.lobster/state (the user's home directory).

    State keys are converted into safe filenames by converting them to lowercase, replacing non-alphanumeric characters (except . _ and -) with underscores, and collapsing multiple underscores. The resulting file is stored as <safe_key>.json.

  9. Manage persistent state in a Lobster pipeline

    main

    Lobster provides two ways to manage persistent state: as pipeline stages (using .pipe()) or via direct asynchronous functions. State is stored as JSON files in a local directory.

    Using Pipeline Stages

    You can use stateGet(key) and stateSet(key) as stages within a Lobster pipeline.

    • stateGet(key): Drains the current input and yields the value associated with the key. If the key does not exist, it yields null.
    • stateSet(key): Collects all items from the input. If there is exactly one item, it saves that item; otherwise, it saves the entire array of items. It then passes the value through to the next stage.

    Using Direct Functions

    If you are not using a pipeline, you can use readState and writeState directly.

    • readState(key, ctx): Returns a Promise that resolves to the parsed JSON value or null if the file does not exist.
    • writeState(key, value, ctx): Returns a Promise that resolves when the value has been atomically written to disk.
    import { Lobster, stateGet, stateSet } from 'lobster-sdk';

// Read state
new Lobster()
  .pipe(stateGet('my-key'))
  .pipe(value => console.log(value));

// Write state
new Lobster()
  .pipe(() => ({ count: 42 }))
  .pipe(stateSet('my-key'));

  10. Run Lobster in tool mode

    main

    When using --mode tool, Lobster outputs a JSON envelope following a specific protocol. This is intended for non-interactive environments (like AI agents or automated systems). The envelope includes the status, output, and any requirements for approval or input.

    Example output for a tool:

    {
  "protocolVersion": 1,
  "ok": true,
  "status": "ok",
  "output": [...],
  "requiresApproval": null,
  "requiresInput": null
}

  11. Configure approval gates in workflows

    main

    Approval steps can enforce identity constraints to ensure only authorized users can proceed. Supported options include:

    • approval.required_approver (or requiredApprover): Requires an exact approver ID.
    • approval.require_different_approver (or requireDifferentApprover): Requires the approver ID to differ from the initiator.
    • approval.initiated_by (or initiatedBy): Sets the initiator ID for comparison.

    Environment variables for identity:

    • LOBSTER_APPROVAL_INITIATED_BY: Provides a default initiator ID at runtime.
    • LOBSTER_APPROVAL_APPROVED_BY: Used at resume/approval time for identity checks.

  12. Handle workflow pauses (Approval and Input requests)

    main

    When running a workflow via Lobster.run(), the process may halt if it encounters an approval_request or an input_request. The returned LobsterResult object provides the necessary information to resume the workflow later.

    • needs_approval: The workflow is waiting for a human to approve or deny an action. The requiresApproval object contains a resumeToken and the items pending approval.
    • needs_input: The workflow is waiting for human-provided data. The requiresInput object contains a resumeToken, a prompt, and a responseSchema (JSON Schema) describing the expected input.