DOCSEARCHbyAlgolia
Submit Docs

BlueKing Job (bk-job)

repository·master·Indexed 21 days ago

https://github.com/tencentblueking/bk-job

A high-concurrency operations script management system for automated task orchestration, scheduled cron jobs, and secure file distribution across server OS environments. It includes a V3 API for script management and credential handling, as well as an AI Agent framework for O&M scenarios featuring reusable rules and skills for log troubleshooting, APM trace analysis, and task duration analysis.

Tokens
209.8K
Snippets
479
Records
739
Agent score
73%

What's inside bk-job

  1. Overview of BK-JOB Multi-tenant Migration Tool

    master

    The BK-JOB Multi-tenant Migration Tool is used to migrate business data from older BK-JOB versions (3.11.x) to a 3.12.x version that has multi-tenancy enabled (tenant.enabled: true).

    Key Migration Principles:

    • The target environment database starts empty.
    • Data IDs from the source environment are preserved exactly in the target environment.
    • Global Resources are migrated once to the target environment.
    • Business Resources are migrated individually per business.

    When NOT to use this tool: If the target 3.12.x environment has multi-tenancy disabled (tenant.enabled: false), you can perform an in-place upgrade from 3.11.x without this tool.

  2. Overview of Job V3 API Resources

    master

    The Job Platform (Job) provides atomic capabilities to other systems via APIs, allowing for script execution, file distribution, scheduled tasks, and workflow scheduling. Each job can act as an atomic node for automation in surrounding systems.

    Available API resource categories include:

    • Script Management: Create, delete, update, and version control for both business and public scripts.
    • Job Execution & Monitoring: Execute job plans, query job instance lists, check execution status, and retrieve execution logs (by host or bulk).
    • Scheduled Tasks (Cron): Create, update, delete, and query Cron jobs.
    • File & Data Operations: Quick file transfers, SQL execution, file source management, and configuration file distribution.
    • Security & Credentials: High-risk script detection, dangerous rule management, and credential creation/updates.
    • Workflow & Instance Control: Perform actions on executed job instances (e.g., terminating) and manage individual step instances (e.g., retry, ignore errors).

  3. Overview of BlueKing Job (bk-job)

    master

    BlueKing Job (bk-job) is an operations script management system designed for high-concurrency task processing. It serves as an atomic operation platform for server OS management, providing capabilities for script execution, file distribution, and scheduled tasks.

    Key functional areas include:

    • Quick Execution: One-time, temporary operational entries.
    • Task Orchestration: Combining repetitive operations into manageable "Jobs" using templates and execution schemes.
    • Scheduled Execution: Native Linux-style Cron periodic or one-time execution plans.
    • Script Management: Cloud-based version control for scripts to support orchestration and external system scheduling.
    • Account Management: Managing OS-level execution accounts (e.g., root for Linux, administrator for Windows).
    • File Source Management: An extensible plugin system to pull files from various sources like Object Storage, FTP, or Samba for distribution.
    • Security: High-risk command detection using regular expressions to intercept unsafe operations, even when triggered via API.

  4. Overview of BK-JOB capabilities

    master

    BK-JOB is an operations script management and execution system designed for handling multiple tasks simultaneously. It functions as an automated cross-system dispatching system where individual tasks act as nodes that can be integrated into automated workflows or offered to peripheral platforms via APIs.

    Key capabilities include:

    • Quick Execution: Versatile, temporary access for immediate task runs.
    • Task Arrangement: Converting repetitive operations into manageable "jobs".
    • Scheduled Execution: Support for periodic (Cron) or one-off execution plans.
    • Script Management: Cloud-based management of script versions for better collaboration and security.
    • Account Management: Managing execution accounts for server OS (e.g., Linux root, Windows administrator).
    • Notification: Sending execution result notifications based on task status.
    • File Source Management: Using plugins to retrieve and transfer files from various file systems (Object Storage, FTP, Samba, etc.).
    • Operation Analysis: Providing platform statistics and insights for administrators.

  5. Overview of bk-job microservices

    master

    The bk-job project is composed of several specialized microservices that work together to provide script management, job orchestration, and execution capabilities. When deploying or integrating with the system, you should be aware of the following service components:

    • job-config: The configuration center for managing system settings.
    • job-gateway: The API gateway for handling incoming requests.
    • job-manage: The management service for job orchestration and lifecycle.
    • job-crontab: The service responsible for handling scheduled (Cron) tasks.
    • job-execute: The execution engine that runs the actual scripts/jobs.
    • job-logsvr: The logging service for capturing and storing job output.
    • job-backup: The service for importing and exporting data.
    • job-analysis: The statistical analysis service for job performance and metrics.

  6. Overview of the BK-JOB Multi-tenant Migration Tool

    master

    The BK-JOB multi-tenant migration tool is designed to move data from a source environment (sourceDb) to a target environment (targetDb) using an intermediate temporary database (tmpDb).

    Design Principles:

    • Read-only Source: No write operations are performed on the sourceDb.
    • Isolated Transformation: All data transformations (e.g., adding tenant_id, replacing app_id) occur within the tmpDb.
    • Clean Target Writes: Only fully processed and validated data is written to the targetDb.

    Architecture Flow: sourceDb (Read-only) $\rightarrow$ tmpDb (Read/Write transformations) $\rightarrow$ targetDb (Final Write)

  7. Overview of BlueKing Job V3 API

    master

    BlueKing Job (Job) is an operations management system designed for basic maintenance tasks such as script execution, file distribution, and scheduled tasks (Cron). It also provides workflow orchestration capabilities, allowing users to assemble individual tasks into automated job flows. Each job can act as an atomic node that can be integrated into higher-level systems or platforms via APIs to achieve scheduling automation.

    The V3 API provides a wide range of resources including script management, credential handling, high-risk command detection, and job instance monitoring.

  8. BK-JOB Design Philosophy and Core Capabilities

    master

    BK-JOB is designed around five core pillars to support large-scale, secure, and flexible job execution:

    • Batch Execution: Capability to dispatch multiple scripts or files simultaneously across multiple devices.
    • Security Control: Mechanisms to verify operator permissions and intercept scripts based on their security levels.
    • Resource Reuse: Abstraction of scripts, tasks, and objectives to allow for reuse across different workflows.
    • Upper-level Application Support: Provides environmental services such as record keeping, indexing, statistical analysis, and process arrangement based on execution content.
    • Flexible File Source Management: Uses a scalable architecture that allows users to access multiple heterogeneous data sources through standardized access points.

  9. Understand the BK-JOB Architecture

    master

    BlueKing Job Platform (bk-job) is a foundational atomic platform within the BlueKing ecosystem. It leverages the 'File Pipeline' and 'Command Pipeline' of the BlueKing Control Platform (GSE) to provide capabilities for job orchestration, script execution, and file distribution.

    The architecture is divided into three main layers:

    1. FrontEnd: A Vue.js-based MVVM framework built with Webpack 4.
    2. BackEnd: A microservices architecture based on Java, SpringBoot, and SpringCloud.
    3. Storage & Middleware: A combination of relational databases, NoSQL stores, and message queues to handle configuration, logs, and task scheduling.

  10. Understand the bk-job directory structure

    master

    The skill is organized as follows:

    PathPurpose
    scripts/job_apigw_client.pyThe main executable script for API calls and validation.
    config.yamlDeployment configuration for Gateway and Job Platform base URLs.
    reference/manuals/Task-specific documentation (progressive disclosure).
    reference/apidocs/Field-level API documentation for the Gateway.
    memory/businesses/Optional business memory files used to pre-fill context.

    All paths within the scripts are relative to the skill root directory.

  11. Understand the BK-JOB code structure

    master

    The Blueking Platform (BK-JOB) repository is organized into a frontend (Vue.js) and a backend (Java microservices) architecture. The core logic resides in the src directory, while scripts contains operational tools and support-files contains configuration templates and migration files.

    High-Level Directory Map

    • src/frontend: Vue.js application code.
    • src/backend: Java microservice implementations.
    • scripts: Operational scripts for launching, shutting down, and restarting backend microservices.
    • support-files: SQL migrations, permission change migrations, and configuration templates.
    • versionLogs: Version history and generated frontend resources.
    |- bk-job
  |- docs
  |- scripts
  |- src
    |- backend
    |- frontend
  |- support-files
  |- versionLogs

  12. Navigate the bk-job Skill Manuals

    master

    The bk-job skill manuals are organized using a progressive disclosure pattern. Instead of loading all details at once, you should refer to specific manuals based on your current task requirements. The main entry point for all capabilities is SKILL.md.

    Use the following index to select the appropriate manual for your task:

    ManualUse Case
    environment-and-auth.mdConfiguring gateway addresses, tokens, and resource scopes (bk_scope)
    listing-and-token-efficient-analysis.mdHandling large lists (defaulting to 20 items), using jq or local scripts for filtering to save tokens
    confirmation-and-output-protocol.mdManaging plan-execute confirmation gates (G1–G4), summary formats, and output specifications
    business-memory.mdManaging business/business set memory, paths, and the memory/businesses/ loading/writing rules
    cron-tasks-and-last-execution.mdSearching for cron tasks and checking cron-last-run execution/logs
    job-plans-create-and-cron.mdUsing template-search, template-detail, plan-create, cron-save, and cron-update-status
    job-plans-search-and-execute.mdSearching for job plans, viewing details, and executing plans via plan-execute
    job-instance-status.mdChecking the status of a specific job_instance_id
    troubleshooting-and-status-codes.mdResolving authentication failures, missing history, or interpreting status codes

    Note: For field-level gateway specifications, always refer to the official API documentation in the ../apidocs/ directory. These manuals focus on script behavior, combined workflows, and operational precautions rather than duplicating the OpenAPI specification.