Tron Batch Scheduling System

Tron is a centralized batch scheduling system designed to manage complex job dependencies and multiple machines from a single daemon. The system is composed of four primary command-line tools:

• trond: The central daemon responsible for scheduling, executing jobs, and maintaining state. It provides an HTTP interface for other tools.
• tronview: Used to inspect job states and view output.
• tronctl: Used to control jobs (start, stop, enable, disable, etc.).
• tronfig: Used to modify Tron's configuration dynamically while the daemon is running.

Tron is a centralized system designed for managing periodic batch processes across a cluster. It is intended as a more robust alternative to cron or fcron for users managing complex workflows that span multiple computers.

Note: Tron is currently undergoing a transformation from an SSH-based execution engine to a system compatible with Kubernetes. It is primarily developed to meet Yelp's specific needs.

Nodes can be grouped into node_pools. When a job is assigned to a node_pool instead of a specific node, Tron selects a random node from that pool to execute the job's actions.

If a job defines a cleanup_action, that action is guaranteed to run on the same node where the primary job actions were executed.

Configuration Example

node_pools:
    - name: pool
      nodes: [node1, node2]

jobs:
  "job2":
    node: pool
    schedule: "cron * * * * *"
    actions:
      "pool_action":
        command: "ls /; sleep 1"
    cleanup_action:
      command: "echo 'all done'"

A job is a periodic event consisting of a name, a target node or node pool, a list of actions, a schedule, and an optional cleanup action.

Success/Failure Criteria:

• A job succeeds if all its actions exit with status 0.
• A job fails if any action exits with a non-zero status.

Jobs are defined as a dictionary where the job name is the key. The name is used for identification in tronview and tronctl.

jobs:
  "foo":
    "schedule": ...
    "command": ...
    "actions":
      "run_first":
        "command": ...

Tron organizes work through a hierarchy of Nodes, Jobs, and Actions:

• Nodes: Remote machines that trond accesses via public key SSH to execute work.
• Jobs: Units of work assigned to a specific node or a node_pool. Jobs run on a defined schedule (using cron syntax).
• Actions: The actual commands executed within a job. Actions can have dependencies on other actions within the same job.

Configuration Example

In your YAML configuration, you define nodes and then map jobs to them:

nodes:
  - name: node1
    hostname: 'batch1'
  - name: node2
    hostname: 'batch2'

jobs:
  "job0":
    node: node1
    schedule: "cron * * * * *"
    actions:
      "batch1action":
        command: "sleep 3; echo asdfasdf"
  "job1":
    node: node2
    schedule: "cron * * * * *"
    actions:
      "batch2action":
        command: "cat big.txt; sleep 10"

All contributions must adhere to the following standards:

• PEP8 Compliance: All code must be PEP8 compliant.
• Linting: Code should pass pyflakes without warnings.
• Test Coverage: All new code must include full test coverage. Bug fixes must include a test case that reproduces the reported issue.

To prevent jobs from running at incorrect times during DST transitions (e.g., running an hour early or late), you should explicitly set the time_zone configuration variable.

Example: time_zone: US/Pacific

Behavior during transitions:

• Fall back (Repeated time): If a job is scheduled at a time that occurs twice (e.g., 1 AM), it runs on the first occurrence.
• Spring forward (Non-existent time): If a job is scheduled at a time that does not exist (e.g., 2 AM), it runs an hour later in the "new" time (e.g., 3 AM). Tron ensures the job runs as soon as is correct and no sooner.

You can use command context variables within the command attribute of an action. These variables are inserted at runtime. The context is populated by Tron's built-in variables and custom variables defined in the command_context configuration key.

For example, using {runid} will append a unique run identifier to the command output (e.g., command_context_demo.1, command_context_demo.2).

jobs:
 "command_context_demo":
   node: local
   schedule: "1st monday in june"
   actions:
     "print_run_id":
       command: "echo {runid}"

As of v0.3.3, logging is not configured in the YAML configuration file.

By default, Tron uses Python's standard logging with a rotating file handler that rotates daily. The default log directory is /var/log/tron/tron.log.

To use a custom logging configuration, pass the -l flag to trond:

trond -l /path/to/logging.conf

Special Log Files:

• twisted: Logs from the Twisted framework.
• tron.api.www.access: API access logs (INFO level) in Apache combined format.

Cleanup actions run after a job succeeds or fails. They are used to destroy shared resources (like intermediate files or EMR job flows) regardless of the job's outcome.

Rules:

• There is only one cleanup action per job.
• It has no name or requirements list.
• You can access the job's final state using the command context variable cleanup_job_status, which evaluates to either SUCCESS or FAILURE.

cleanup_action:
  command: "python -m mrjob.tools.emr.job_flow_pool --terminate MY_POOL"

Execute the project's test suite using the make command.

$ make test

The tronview command-line tool is used to display the status of scheduled Tron jobs. You can use it to view a high-level overview of all configured jobs or drill down into specific job runs and actions.

Usage Patterns

• Show all configured jobs: Run tronview without arguments.
• Show details for a job: Provide the job name: tronview <job_name>.
• Show details for a specific run: Provide the job run ID (e.g., job_name.run_number): tronview <job_run_id>.
• Show details for a specific action run: Provide the action run ID (e.g., job_name.run_number.action_name): tronview <action_run_id>.

# Show all jobs
tronview

# Show details for a specific job
tronview my_job

# Show details for a specific run
tronview my_job.0

# Show details for a specific action run
tronview my_job.0.my_action