Digdag Documentation

Digdag is a tool designed to build, run, schedule, and monitor complex task pipelines. It uses Directed Acyclic Graphs (DAG) as its underlying infrastructure to handle dependency resolution, ensuring that tasks execute in the correct order or in parallel where possible.

Common use cases include:

• Replacing cron jobs
• IT operations automation
• Data analytics batch jobs
• Machine learning pipelines

The Command Executor allows you to run operators like sh>, py>, and rb> in different environments instead of just locally. This enables you to switch the execution environment without changing your task definitions.

Digdag uses a fallback mechanism for selecting the executor:

1. AWS ECS (Elastic Container Service): The default executor if valid configuration is provided.
2. Docker: Fallback if ECS configuration is missing or invalid.
3. Local: Fallback if no Docker configuration is provided.

Note: Kubernetes support is currently under development.

When running multiple servers that share the same underlay database, you can use specific flags to designate the role of each server:

• REST API Only: Use --disable-local-agent to prevent this server from executing tasks. This is useful for servers intended only for the REST API.
• Task Execution Only: Use --disable-executor-loop to prevent this server from updating task states in the database. Note: At least one server sharing the database must have the executor loop enabled.
• Disable Schedules: Use --disable-scheduler to disable the schedule executor on this specific server without modifying workflow files.

The retry parameter controls whether Digdag retries ephemeral errors.

• Default behavior: true for GET, HEAD, OPTIONS, or TRACE methods; false for others.
• Error handling: Client 4xx errors are not retried, except for 408 Request Timeout and 429 Too Many Requests.

Warning: Enabling retries can cause duplicate requests to the target endpoint. Only enable retries if the target operation is idempotent.

The redshift> operator can use a status table to ensure operations are idempotent (preventing duplicate records if a task is retried). This is controlled by the strict_transaction option.

• strict_transaction: true (default): The operator creates and uses a status table in the database. If the query that created the status table completed more than 24 hours ago, the operator cleans up the table.
• strict_transaction: false: Disables the status table mechanism. Use this if your database user does not have permissions to create tables.

Related Configuration:

• status_table: Prefix for the status table name (default: __digdag_status).
• status_table_schema: Schema for the status table (defaults to the value of schema).
• status_table_cleanup: Period after which the status table is cleaned up (default: 24h).

Digdag separates the intent to run a workflow from the actual execution to make debugging and monitoring easier.

• Session: A plan to run a workflow that is expected to complete successfully. A session is identified by its session_time.
• Attempt: An actual execution of a session. If a session fails, you can perform multiple attempts by retrying the failed workflow.

Workflow for debugging:

1. Monitor sessions to ensure all planned executions are 'green' (successful).
2. If a session fails, inspect its attempts and check the logs to debug the problem.
3. Fix the issue (e.g., by uploading a new revision) and start a new attempt within that session.

Digdag allows you to organize tasks into groups to manage complexity and provide a hierarchical view of your workflow.

Execution Logic:

• A task starts if there are no dependent siblings, or if all of its dependent siblings have successfully completed.
• When a parent task of a group runs, it executes its children.
• A parent task completes successfully only when all of its children complete successfully.
• If a child task fails, both that child and its parent are marked as failed.
• The entire workflow execution finishes when the root task completes or fails.

Grouping is also used for parameter passing: a parent task can export variables to its children (similar to the UNIX export command) or generate children tasks dynamically at runtime based on previous results.

The Workspace is the directory where a task is executed. Digdag extracts the project files into this directory and changes the working directory there before execution.

Security and Isolation:

• Plugins are restricted from accessing directories above the workspace. This ensures projects remain self-contained and do not depend on the shared server environment.
• Exception: The scripting operator (e.g., sh>) can access parent directories.
• Recommendation: To ensure isolation and avoid environment dependencies, it is recommended to run scripts using the docker: option.

Most Digdag migrations are executed within a transaction, allowing you to fix errors and retry. However, starting from version 0.9.36, Digdag introduced non-transactional migrations. These are used for DDL (Data Definition Language) operations that cannot run within a transaction.

Warning: If a non-transactional migration fails, the database may be left in a partially migrated state. You might need to manually fix the schema before retrying the migration.