Bamboo Engine

repository·master·Indexed 41 days ago

https://github.com/tencentblueking/bamboo-engine

An event-driven, general-purpose workflow engine for Python (version 2.11.4) designed to parse, execute, and schedule complex task flows. It supports advanced control capabilities including pausing, retrying, skipping nodes, and horizontal scalability. The engine is runtime-agnostic and requires an implementation of the EngineRuntimeInterface, such as the Django and Celery-based bamboo-pipeline.

Tokens
51.6K
Snippets
146
Records
182
Agent score
50%

What's inside Bamboo Engine

  1. What are LAZY variables and how do they work

    master

    LAZY variables (represented by the pipeline.core.data.var.LazyVariable class) are an extension of SPICE variables. They allow you to define relationships between variables and, crucially, perform custom logic on the results of variable references.

    While SPICE variables handle basic referencing, LAZY variables allow you to intercept the resolved value and apply further processing or transformations before the value is used in the workflow.

  2. How Rollback Modes (TOKEN vs ANY) work

    master

    The rollback behavior is determined by the mode parameter, which can be either TOKEN or ANY.

    TOKEN Mode

    In TOKEN mode, rollback follows the original execution path in reverse (if sub-processes exist, they are rolled back before the main process). It calls the specific rollback method on each node along the path.

    Constraints for TOKEN mode:

    • Task Status: Only tasks in RUNNING or ROLL_BACK_FAILED states can be rolled back. Finished or paused tasks cannot be rolled back.
    • Tokens: The start node and target node must share the same token. No other nodes with the same token can be currently RUNNING.
    • Start Node: Must be a terminal node of the currently running pipeline and must be in FINISHED or FAILED state.
    • Target Node: Must be in FINISHED state and must be a task-type node (gateway nodes are not supported).
    • Rollback Reservation: You can reserve a rollback for a RUNNING node; the rollback triggers automatically when that node finishes. Only one reservation is allowed per pipeline.

    ANY Mode

    In ANY mode, the pipeline jumps directly to the target node. It does not follow the path or call the nodes' rollback methods. Instead, it deletes the execution information of the nodes on the rollback path and resumes from the target location.

    Constraints for ANY mode:

    • Task Status: Only RUNNING tasks are eligible. Finished or paused tasks are not.
    • Running Nodes: No nodes in the current pipeline can be in a RUNNING state before the rollback begins.
    • Start Node: Must be the current terminal node of the pipeline and must be in FINISHED or FAILED state.
    • Target Node: Must be in FINISHED state and must be a task-type node.
    • Rollback Reservation: Similar to TOKEN mode, you can reserve a rollback for a RUNNING node. Only one reservation is allowed per pipeline.
  3. How bamboo-engine and runtimes work together

    master

    The bamboo-engine is the core engine that defines the basic engine model, workflow execution core, and the engine runtime interface. To use the engine in a project, you must use a runtime that implements the bamboo_engine.eri.interfaces.EngineRuntimeInterface.

    Currently, the available runtime is:

    • bamboo-pipeline: A runtime based on Django and Celery.

    Developers interact with the engine by providing a specific runtime instance to the engine's APIs.

  4. Understand the core concepts of Bamboo Engine

    master

    Bamboo Engine is an event-driven workflow engine based on directed graphs. The fundamental building blocks are:

    • Pipeline (流程): A directed graph that starts with a specific node and ends with a specific node.
    • Nodes (节点): The elements within a pipeline (excluding edges). A pipeline itself is considered a node.
    • Sequence Flow (顺序流): The directed edges that connect nodes in a pipeline.
    • ID: A unique identifier for every pipeline, node, and sequence flow in the system.
  5. Understand SPLICE variables and variable referencing

    master

    SPLICE variables allow you to declare relationships and references between variables, enabling complex data interactions within a workflow.

    Key Rules:

    • Use the syntax ${xxxx} to declare a reference to a variable in the data context.
    • In the data context, only variables whose keys are in the ${xxxx} format can be referenced by SPLICE variables.
    • SPLICE variables are lazy: they are not parsed when created. Instead, the engine resolves them just before a node (component) executes.
    • The engine performs recursive resolution: if param_1 references ${constant_1}, and ${constant_1} references ${constant_2}, the engine will resolve the entire chain to find the final value.

    This mechanism allows for direct references, chained references, and multi-variable references.

  6. Use different types of nodes in a pipeline

    master

    Nodes define the behavior and structure of your workflow. The primary types are:

    • Events (event): Control the lifecycle of a pipeline. A pipeline must have at least one StartEvent (to trigger execution) and one EndEvent (to signal completion).
    • Activities (activity): Represent specific tasks or business logic, such as making network requests, database operations, or executing commands. The behavior is defined by the user via Components (组件).
    • Gateways (gateway): Control the flow of execution:
      • ExclusiveGateway: Decides a single path for the workflow to follow.
      • ParallelGateway: Increases the number of concurrent execution processes.
      • ConditionalParallelGateway: Increases concurrent processes based on information in the current context.
      • ConvergeGateway: Reduces the number of concurrent execution processes.
    • Subprocesses (subprocess): A special node where one pipeline is embedded within another, allowing for workflow reuse and modularity.
  7. Connect workflow elements using the Flow Builder

    master

    Once elements are instantiated, they must be connected to define the execution path. The builder provides the following methods for establishing relationships between nodes:

    • extend: Chains the current element to the next one in a linear sequence.
    • connect: Establishes a connection between elements.
    • converge: Used to merge multiple incoming flows.
    • to: Specifies a target for a connection.
  8. Manage data exchange using Execution Data and Context

    master

    Bamboo Engine uses two distinct mechanisms to manage data, ensuring that components remain independent and reusable:

    Execution Data (执行数据)

    Each node has its own isolated execution data used to store its specific input and output. Nodes cannot directly access the execution data of other nodes.

    Context (上下文)

    To allow communication between nodes without breaking isolation, the pipeline provides a Context.

    • Purpose: Used for node communication and data passing across the entire pipeline execution.
    • Pattern: A node writes data to the context during its execution, and other nodes retrieve that data from the context when needed.

    Why use Context instead of direct access? Directly accessing other nodes' data or the global context within a component creates tight coupling. By using the Context pattern, components remain independent and can be reused in different pipeline structures regardless of their position.

  9. Use Django Signals as workflow lifecycle hooks

    master

    The pipeline runtime provides several Django signals that act as hooks throughout the workflow execution lifecycle.

    ⚠️ Warning: Because Django signals are a synchronous mechanism, do not execute long-running or time-consuming logic within a signal handler, as this will degrade the overall execution efficiency of the engine.

    You can use these signals to trigger custom logic when specific events occur, such as state changes or service executions.

  10. How bamboo-engine works with runtimes

    master

    bamboo-engine is a core workflow engine that defines the logic for parsing, executing, and scheduling tasks. It does not include a built-in runtime. To use it, you must provide a runtime that implements the bamboo_engine.eri.interfaces.EngineRuntimeInterface.

    A common runtime provided is bamboo-pipeline, which is based on Django and Celery.

  11. Construct workflow elements with the Flow Builder

    master

    The Flow Builder provides several types of elements to define the components of a workflow:

    Events

    • EmptyStartEvent: Represents the starting point of a workflow.
    • EmptyEndEvent: Represents a simple termination point.
    • ExecutableEndEvent: Represents an end point that may involve execution logic.

    Activities

    • ServiceActivity: Executes a specific component identified by component_code.
    • SubProcess: Represents a nested workflow within the main flow.

    Gateways

    • ParallelGateway: For parallel execution paths.
    • ExclusiveGateway: For branching logic where only one path is taken.
    • ConditionalParallelGateway: For conditional parallel execution.
    • ConvergeGateway: For merging multiple paths back into one.

    Data

    • Data, Var, NodeOutput, RewritableNodeOutput, Params, DataInput: Used to manage state and data flow between nodes.
  12. Understand the EngineAPIResult return type

    master

    All interactions with bamboo_engine.api return an EngineAPIResult object. This object provides a standardized way to check the success of an operation and retrieve associated data or error information.

    Fields:

    • result (bool): Indicates if the operation was successful.
    • message (str): Additional message; focus on this if result is False.
    • exc (Exception, optional): The exception object if an error occurred.
    • data (Any, optional): Associated data returned by the operation.
    class EngineAPIResult:
        def __init__(
            self,
            result: bool,
            message: str,
            exc: Optional[Exception] = None,
            data: Optional[Any] = None,
        ):
            self.result = result
            self.message = message
            self.exc = exc
            self.data = data