DOCSEARCHbyAlgolia
Submit Docs

Halo-Job Backend Manual

website·Indexed 29 days ago

https://zorroe.github.io/

Backend documentation for Halo-Job, detailing the system's core architecture, the separation of Admin and Executor components, and both synchronous and asynchronous task dispatching paths. Includes technical insights into design decisions, database schemas, and source code entry points.

Tokens
15.1K
Snippets
69
Records
97
Agent score
75%

What's inside Halo-Job

  1. Understand the Halo-Job task execution lifecycle

    The Halo-Job execution flow follows a specific sequence: Task Saving $\rightarrow$ Configuration Normalization $\rightarrow$ Next Execution Time Calculation $\rightarrow$ Scheduling Scan $\rightarrow$ Dependency Validation $\rightarrow$ Instance Creation $\rightarrow$ Routing $\rightarrow$ Attempt Creation $\rightarrow$ Dispatch $\rightarrow$ Result Aggregation $\rightarrow$ Alerting/Logging/Auditing.

  2. Halo-Job Admin architecture and design

    The Halo-Job Admin centralizes scheduling decisions, leaving the business systems (Executors) to focus solely on execution. It uses a layered architecture (Controller -> Service -> Mapper/Strategy) to separate business rules from data access and HTTP calls. To ensure high availability and prevent duplicate execution in multi-node clusters, it relies on Redis locks for mutual exclusion rather than JVM memory.

  3. Distinguish between Job Instances and Job Attempts

    Halo-Job separates the concept of a trigger from the actual execution attempts to support retries and sharding:

    • job_instance: Represents a single trigger event of a task (the 'run instance').
    • job_attempt: Represents a specific attempt to execute that instance on a target, a specific shard, or a retry attempt.

    Examples:

    • 1 target, 0 retries $\rightarrow$ 1 instance, 1 attempt.
    • 3 shard targets, 0 retries $\rightarrow$ 1 instance, 3 attempts.
    • 1 target, 1 retry after failure $\rightarrow$ 1 instance, 2 attempts.

  4. Synchronize MySQL data to Elasticsearch using Canal

    Canal is a tool based on MySQL binary log parsing that provides incremental data subscription and consumption. To sync MySQL data to Elasticsearch, you need to set up a pipeline consisting of Canal-server (which acts as a MySQL slave to receive binlogs) and Canal-adapter (which consumes data from the server and syncs it to ES).

  5. Ancient Poetry Website Project Overview

    This is a decoupled ancient poetry website featuring search capabilities and browsing by dynasty. The project is built using the following technology stack:

    • Frontend: Nuxt3
    • Backend: Spring Boot
    • Database: MySQL
    • Search Engine: Elasticsearch
    Frontend Repository: https://github.com/zorroe/ancient-poetry
Backend Repository: https://github.com/zorroe/poetry-backend

  6. Halo-Job Database Schema Overview

    The system separates data into three states to ensure that modifying task definitions does not overwrite historical execution records.

    1. Definition State (Static Configuration):

    • job_info: Task definitions (includes next_execute_time, trigger_type, route_strategy, block_strategy).
    • executor_info: Executor registration and heartbeat status (heartbeat_time).
    • executor_handler: Handler metadata reported by executors.
    • platform_user: Users, roles, and group permissions.

    2. Runtime State (Execution Data):

    • job_instance: A single trigger instance (status).
    • job_attempt: Specific dispatch attempts for an instance (attempt_no, shard_index).
    • job_execution_log: Historical execution logs.
    • job_flow_instance: DAG runtime records.

    3. Governance State (Management & Audit):

    • audit_log: Operation auditing.
    • alert_event, alert_rule, alert_channel, alert_notification: Alerting system.
    • job_dependency: Task dependency relations.
    • job_flow, job_flow_node, job_flow_edge: DAG definitions.
    • job_publish_history: Configuration snapshots.
    • job_reroute_record: Manual reroute logs.

  7. Halo-Job Architecture Components

    The system consists of the following key components:

    • halo-job-admin: The scheduling center. Handles task management, scheduling scans, trigger parsing, misfire handling, routing, instance/attempt persistence, result aggregation, permissions, auditing, alerts, and DAGs.
    • halo-job-core: The executor SDK. Scans for @HaloJob methods, handles automatic registration, heartbeats, exposes execution interfaces, and maintains thread contexts.
    • halo-job-spring-boot-starter: A thin wrapper around halo-job-core that uses Spring Boot auto-configuration to import ExecutorAutoConfig.
    • MySQL: Persists task definitions, executors, instances, attempts, logs, dependencies, DAGs, alerts, and audit history.
    • Redis: Provides distributed locks for the scheduling scanner to prevent multiple Admin nodes from triggering the same task.

  8. Halo-Job System Overview

    Halo-Job is a lightweight distributed task scheduling system designed to decouple scheduling decisions from business execution. It allows administrators to centrally create, start, stop, and query tasks, which are then dispatched to remote executors.

    Core Responsibility Split:

    • Admin: Determines "when to run, who to send it to, and how to record it."
    • Executor: Determines "how to run the task locally."

  9. Install a local JAR file to the Maven repository

    Use the mvn install:install-file command to manually add a third-party JAR file to your local Maven repository. This allows you to manage the dependency via a standard Maven coordinate (groupId, artifactId, version) in your pom.xml without needing a remote repository.
    mvn install:install-file \
  -Dfile=<path-to-your-jar-file> \
  -DgroupId=<group-id> \
  -DartifactId=<artifact-id> \
  -Dversion=<version> \
  -Dpackaging=jar

  10. Revert code to a specific commit using git reset

    To roll back your local branch to a previous state, use git reset --hard. This operation is destructive and will overwrite local changes. After resetting locally, you must force-push the changes to the remote repository to synchronize the history.
    # Option 1: Revert to the immediately preceding version
git reset --hard HEAD^

# Option 2: Revert to n commits ago (e.g., n=3 for 3 commits back)
git reset --hard HEAD~n

# Option 3: Revert to a specific commit SHA (Recommended)
git reset --hard <commit_sha>

# After any of the above, force push to the remote repository
git push origin HEAD --force

  11. Prepare Tomato and Beef Soup (2 servings)

    A recipe for a clear tomato and beef soup using a rice cooker. The beef can be beef ribs, brisket, or chuck. The soup is seasoned with white pepper and a small amount of light soy sauce for color.

    Ingredients

    • Beef ribs, brisket, or chuck (500g/1 jin)
    • Carrot (1 piece)
    • Tomato (1 piece)
    • Green onion, ginger, star anise (moderate amount), white pepper

    Instructions

    1. Cut beef into chunks and blanch with yellow wine.
    2. Cut tomato into large chunks and carrot into rolling cuts.
    3. Cut green onion into rolling cuts and slice the ginger.
    4. Add boiled water to a pot with the blanched beef, tomato, carrot, green onion, ginger, and star anise (do not add too much).
    5. Add a small amount of salt, yellow wine, and a small amount of light soy sauce for coloring.
    6. Add white pepper and a small amount of vegetable oil.
    7. Use the 'Soup' mode on a rice cooker: 2 hours for beef ribs, 1.5 hours for brisket (time can be extended).
    8. Once finished, garnish with chopped green onion and cilantro. Adjust salt if too bland.

    Dipping Sauce (Optional)

    • Mix light soy sauce, vinegar, and sesame oil in a ratio of 1 : 1 : 1/6. Add chopped green onion.

  12. Key Source Code Entry Points for Halo-Job

    For developers looking to explore the implementation or extend the system, the following files are the primary entry points:
    - Admin Application: halo-job-admin/src/main/java/com/zorroe/cloud/job/admin/HaloJobAdminApplication.java
- Admin Configuration: halo-job-admin/src/main/resources/application.yaml
- Executor Auto-Config: halo-job-core/src/main/java/com/zorroe/cloud/job/core/config/ExecutorAutoConfig.java
- Database Schema: sql/halo-job-full.sql