BlueKing Standard Operations (SOPS)

BlueKing Standard Operations (SOPS) is a lightweight SaaS product within the Tencent BlueKing ecosystem designed for task workflow orchestration and execution via a visual graphical interface.

It consists of two core services:

1. Workflow Orchestration Service: An API gateway service based on the BlueKing PaaS platform that integrates capabilities from various internal enterprise systems, allowing for automated scheduling across multiple systems.
2. Self-Service Service: Abstracts the differences between underlying systems, enabling operations personnel to delegate routine tasks (like business releases or changes) to product, development, or testing teams. It integrates deeply with the BlueKing PaaS platform to provide 'light applications' and 'functionalization' to increase self-service rates.

Technical Stack:

• Backend: Python with the Django framework.
• Frontend: Vue.js for pages and jQuery for standard plugin development (using a configuration-based pattern to simplify form development).
• Engine: Uses the bamboo-engine SDK for underlying workflow processing.

The goal of this plan is to expose both built-in and third-party plugins to BKFlow via the uniform_api v4.0.0. This is achieved using a 'Component Execution Shell (Scheme B)' that allows real operators and business contexts to execute tasks via synchronous, polling, or callback mechanisms.

Architecture Overview

• Directory Layer: Unifies built-in plugins (traversed via ComponentLibrary) and third-party plugins (PluginServiceApiClient) into a single v4 directory.
• Execution Layer: Located in gcloud/plugin_gateway/services/runner.py. It instantiates the component bound_service and drives execute/schedule directly without creating a PipelineModel engine instance.
• Context Parsing: Parses sops Project and writes operator information into parent_data.
• State Machine: Follows the lifecycle CREATED → RUNNING → (WAITING_CALLBACK) → SUCCEEDED/FAILED/CANCELLED.
• Infrastructure: Utilizes 3 independent Celery queues and existing callback bridges.

Tech Stack

• Django, DRF, Celery
• bamboo-pipeline (ComponentLibrary, Service, DataObject, FancyDict)
• API Gateway
• pytest / Django TestCase

The 'Stuck Process Governance Console' is a new administrative feature for bk-sops located under /admin/. It allows super-administrators to associate DiagnosticCase objects (from bamboo-engine) back to specific tasks, templates, businesses, and nodes, providing status management and the ability to run diagnostic or recovery actions.

Architecture Overview

• Scope: Pure bk-sops side changes; no modifications to bamboo-engine.
• Backend: Adds a reverse mapping layer, a status writing layer, and three new views in gcloud/contrib/admin. Existing list/detail views are enhanced.
• Frontend: Uses an independent Django template with native JavaScript (fetching JSON endpoints) rather than integrating into the main Vue SPA.
• Resilience: All references to pipeline.contrib.diagnostics must be wrapped in try/except ImportError to ensure the system degrades gracefully with readable errors if the engine package is missing.

Key Constraints

• Access Control: Restricted to platform super-administrators via @check_is_superuser() and @iam_intercept() (using AdminViewViewInterceptor for reads and AdminEditViewInterceptor for writes).
• Status Management: DiagnosticCase statuses are limited to three states: open (STATUS_OPEN), resolved (STATUS_RESOLVED), and ignored (STATUS_IGNORED).
• Data Integrity: Due to a unique_together constraint on (root_pipeline_id, node_id, stuck_type, status), status updates must follow a "merge + delete duplicates" pattern rather than a direct update(status=...) call.
• Data Mapping: Reverse mapping uses TaskFlowInstance.objects.filter(pipeline_instance__instance_id__in=ids) where pipeline_instance.instance_id == root_pipeline_id.

The current implementation of the Plugin Gateway has specific functional boundaries. Use this to determine if the gateway meets your current requirements.

Supported Features

• Third-party Plugin Discovery: Discovery is currently limited to third-party plugins only.
• Synchronous Scheduling: Supports synchronous scheduling for third-party plugins.
• Execution Management: Includes execution registration, idempotency, source governance, status/detail retrieval, cancellation, and callback bridging.

Unsupported/Planned Features

• Built-in Plugins: Directory exposure and execution scheduling for built-in plugins are not yet implemented.
• Asynchronous Modes: Asynchronous execution modes (such as polling or callback-based recovery for RUNNING states) are not yet implemented.
• Consistency: Consistency between categories and plugins lists regarding builtin status is pending.

The bk-sops codebase is organized into four primary architectural layers: the development framework, the pipeline engine, the business logic layer, and the frontend presentation layer.

1. Framework Layer (framework)

Built as a secondary encapsulation of the Django framework, this layer provides foundational SaaS operations and PaaS configurations.

• config: Environment-specific configurations (local, test, production) and routing.
• blueapps: Core modules for the new development framework, including unified login, authentication, middleware, and common functions.
• packages: The BlueKing API Gateway SDK, providing access to APIs from the Configuration Platform, Job Platform, etc.

2. Pipeline Engine Layer (pipeline, pipeline_web, pipeline_plugins)

A self-developed workflow engine framework responsible for task orchestration and execution.

• pipeline: Contains the core engine logic. Key sub-directories include:
  • core: Core elements defined following BPMN 2.0 specifications (e.g., Activity, gateways, events, Data objects).
  • engine: Runtime execution logic and task state management.
  • component_framework: The plugin framework and plugin definitions.
  • validators: Data validation (e.g., circular structure detection).
• pipeline_web: The frontend data adaptation layer that supports workflow data generated by the frontend canvas.
• pipeline_plugins: The official standard operations plugin library and global custom variables.

3. Business Logic Layer (gcloud)

A business adaptation layer built on top of the pipeline engine. It handles business permissions, template management, and task management.

• core: Core business logic, permission control, and business homepages.
• tasktmpl3: Workflow template management.
• taskflow3: Task management.
• webservice3: Data resource API management.
• apigw: External-facing API modules.

4. Frontend Layer (web)

Contains frontend resources and static assets.

• frontend: Includes the workflow orchestration desktop module, implemented using Vue.
• static: Contains frontend definitions for plugin components and variables. These are located within their respective module directories.
• templates: Pages required for the homepage and Django admin.