DOCSEARCHbyAlgolia
Submit Docs

dcron

repository·master·Indexed 19 days ago

https://github.com/libi/dcron

A lightweight distributed job scheduler library for Go that uses Redis or etcd to synchronize service states and task assignments. It provides load-balanced task execution without relying on synchronized system clocks, ensuring that tasks within a service group are distributed evenly and executed uniquely across nodes.

Tokens
6.2K
Snippets
26
Records
36
Agent score
66%

What's inside dcron

  1. Understand ServiceName and Task Groups

    master

    The ServiceName is a critical concept in Dcron used to define the boundary of task allocation and scheduling.

    • Task Grouping: Multiple nodes configured with the same ServiceName are considered part of the same task group.
    • Distribution: Tasks within a group are distributed evenly across all nodes in that group.
    • Uniqueness: Dcron ensures that the same task within the same service group is only started as a single running instance and is not executed repeatedly across different nodes.

  2. Understand the serviceName concept

    master

    The serviceName defines a boundary for task allocation and scheduling.

    Nodes that share the same serviceName are considered part of the same task group. Within a single task group, tasks are distributed evenly across all participating nodes, and Dcron ensures that no task is executed more than once across the group.

  3. Get Started with Dcron

    master

    To use Dcron, you need to initialize a driver (such as redisdriver), create a Dcron instance with a ServiceName, add tasks using cron-language, and then start the scheduler.

    Note that ServiceName defines the task unit; multiple nodes using the same ServiceName form a task group where tasks are distributed evenly and executed uniquely.

    Steps:

    1. Initialize the driver (e.g., redisdriver).
    2. Create the Dcron instance using NewDcron(serviceName, driver).
    3. Add tasks using AddFunc(taskName, cronExpr, func) where taskName is the unique primary key.
    4. Start the scheduler using Start() (non-blocking) or Run() (blocking).
    // 1. Setup driver and dcron
redisCli := redis.NewClient(&redis.Options{
  Addr: DefaultRedisAddr,
})
drv := redisdriver.NewDriver(redisCli)
dcron := NewDcron("server1", drv)

// 2. Add a task
dcron.AddFunc("test1", "*/3 * * * *", func() {
  fmt.Println("execute test1 task", time.Now().Format("15:04:05"))
})

// 3. Start the scheduler
dcron.Run()

  4. Stop dcron example instances

    master

    You can manage the lifecycle of the running example processes using the following scripts:

    Stop all instances

    To terminate all running example processes at once, use:

    ./killexamples.sh

    Stop a specific instance

    To terminate a single instance by its ID, use:

    ./kill-instance.sh $sub_id

    (Replace $sub_id with the actual ID of the process you wish to stop, e.g., ./kill-instance.sh 2).

    # Stop all
./killexamples.sh

# Stop 1 instance
./kill-instance.sh 2

  6. Configure Dcron with Options

    master

    Dcron is built on top of github.com/robfig/cron. Any configuration parameters passed as arguments after the driver in NewDcron are passed directly to the underlying cron engine. For example, to enable second-level precision in cron expressions, use cron.WithSeconds().

    You can also use NewDcronWithOption for more advanced configurations like log output. For a full list of available options, refer to the project's option.go file.

    // Example: Configuring second-level cron expressions
dcron := NewDcron("server1", drv, cron.WithSeconds())

  7. How NodePool states work

    master

    The NodePool manages cluster stability through two distinct states:

    1. NodePoolStateSteady: The current list of nodes matches the previous update. In this state, the hash ring is stable, and the node is permitted to run jobs via CheckJobAvailable.
    2. NodePoolStateUpgrade: The node list has changed (nodes added or removed). During this state, the node is in transition, the hash ring is being rebuilt, and CheckJobAvailable will return ErrNodePoolIsUpgrading. This prevents jobs from running on nodes that might no longer be the correct owners according to the new hash ring.

    When Start() is called, the pool stays in the Upgrade state until the first consistent view of the cluster is established.

  8. Configure Dcron using functional options

    master

    Dcron uses the functional options pattern for initialization. You can pass multiple Option functions to the Dcron constructor to configure loggers, timing, cron behavior, and recovery mechanisms.

    Common configuration categories include:

    • Logging: Set a global logger for both Dcron and the underlying cron engine using WithLogger.
    • Cron Engine Tuning: Configure time locations, second-level precision, custom schedule parsers, or job wrappers (chains) using CronOption... functions.
    • Cluster & Node Behavior: Adjust node update durations, hash replica counts, or enable cluster stability features with WithClusterStable to rerun recent jobs after upgrades.
    • Local Execution: Use RunningLocally() to indicate the instance is not part of a distributed cluster.
    // Example of applying multiple options
dcron, err := NewDcron(
    WithLogger(myLogger),
    WithNodeUpdateDuration(5 * time.Minute),
    CronOptionSeconds(),
    WithClusterStable(1 * time.Hour),
    RunningLocally(),
)

  9. Run the stablejob example

    master

    The stablejob example demonstrates how to use DCRON to execute bash commands as stable jobs. To run this example, you must have redis, docker, and docker-compose installed on your system.

    Prerequisites

    1. Start Redis: Ensure a Redis instance is running, as it is required for job storage.
    2. Store Jobs: Run the provided tools to register the jobs in the system:
      go run ../tools/tools.go

    Execution Steps

    Run the following commands from the dcron source root directory:

    1. Build the Docker image:

      docker build -f examples/stablejob/Dockerfile -t stable-job-example .

    2. Start the environment:

      docker-compose -f examples/stablejob/stablejob-compose.yml up -d

    3. Scale the jobs (to test scaling behavior):

      docker-compose -f examples/stablejob/stablejob-compose.yml scale stablejob=5

    4. Inspect logs:

      docker logs --details [containername]
    # in dcron srcRoot dir
# build image
docker build -f examples/stablejob/Dockerfile -t stable-job-example .
# run image
docker-compose -f examples/stablejob/stablejob-compose.yml up -d
# scale up
docker-compose -f examples/stablejob/stablejob-compose.yml scale stablejob=5
# get log
docker logs --details [containername]

  10. Run a single dcron instance via run-instance.sh

    master

    To launch a specific single instance of the example, use run-instance.sh. It accepts two arguments:

    1. $sub_id_for_this_process: A unique identifier for this specific process.
    2. $number_of_cronjob: The number of cron jobs for this instance.

    Example: To run instance ID 6 with 10 cron jobs:

    ./run-instance.sh 6 10

  11. Run multiple dcron instances via run.sh

    master

    The run.sh script allows you to launch multiple processes of the example application simultaneously. It accepts two arguments:

    1. $number_of_process: The total number of processes to spawn.
    2. $number_of_cronjob: The number of cron jobs per process.

    Example: To run 5 processes, each with 10 cron jobs:

    ./run.sh 5 10

  12. Extend Dcron with Custom Drivers

    master
    Dcron supports custom storage backends via the DriverV2 interface. If you need to use a storage mechanism other than Redis or etcd, you can implement this interface to manage node data and synchronization.