DOCSEARCHbyAlgolia
Submit Docs

doit Automation Tool

repository·master·Indexed 24 days ago

https://github.com/pydoit/doit

doit is a task management and automation tool written in pure Python. Similar to make, it allows users to define tasks using Python code instead of a DSL, enabling complex logic and integration with the Python ecosystem. It features a CLI for running, listing, and cleaning tasks, as well as tools for dependency tracking, parallel execution, and shell tab completion.

Tokens
32.2K
Snippets
61
Records
222
Agent score
84%

What's inside doit

  1. Core features of doit

    master

    Python-Native Task Definition

    • Pure Python: Tasks are Python dicts. You can use any library, generate tasks programmatically, and debug with pdb.
    • Flexible Actions: You can run shell commands, call Python functions, or mix both in a single task.
    • Self-documenting: Uses docstrings to provide task descriptions via doit list.

    Incremental Builds & Pipelines

    • Dependency Tracking: Tracks file dependencies and targets (using MD5 or timestamps) to skip unnecessary work.
    • Flexible Up-to-date Checks: Supports computed dependencies (calc_dep), custom checkers, and result-based checks.
    • Pipelines: Allows passing results between tasks without intermediate files and supports dynamic task creation via yield.

    Advanced Capabilities

    • Extensibility: Plugin architecture for custom commands, storage backends, task loaders, and output reporters.
    • Parallel Execution: Supports threaded or multi-process execution.
    • Visualization: DAG visualization via graphviz.

  2. Use doit for high-level build control and automation

    master

    Beyond simple compilation, doit is effective for high-level build orchestration and development workflows. Common use cases include:

    • Tool Management: Downloading and installing necessary development tools.
    • Multi-toolchain Orchestration: Calling build systems like CMake for multiple different compiler toolchains.
    • Code Analysis: Driving various static analysis and linting tools.
    • Packaging: Preparing software for deployment (e.g., creating ISO archives, RPMs, or resource packs).
    • Dependency Management: Checking if toolchain dependencies are outdated and automating updates.
    • Resource Pipelines: Managing complex transformations of assets (e.g., converting textures, audio, or 3D objects through multiple compilation and encryption steps).
    • Content Generation: Automating the generation of HTML, RSS feeds, and indexes for static site generators.
    • Data Pipelines: Fetching remote data, converting formats, and pushing to search engines like Elasticsearch.

  3. Optimize processing time with incremental builds

    master
    To avoid redundant work, define tasks with their input and output files. doit will automatically detect if the input and output are already in sync. If the files haven't changed, doit will skip the task, completing the check in a fraction of a second instead of re-running the entire process.

  4. Control task execution with the uptodate API

    master

    The uptodate API allows you to define custom logic to determine whether a task should be executed. This is particularly useful for tasks involving remote resources where you want to avoid unnecessary work if the remote state hasn't changed.

    For example, when checking remote files, you can create a task that inspects headers like ETag or Last-Modified. If the header indicates no change, you can set uptodate to True to prevent the task from re-running.

  5. Use positional arguments in tasks

    master

    Tasks can accept positional arguments from the command line, similar to standard Unix commands. These arguments must appear after any optional (flagged) arguments.

    Warning: If a task accepts positional arguments, you cannot pass other tasks after it in the command line. For example, doit task1 pos1 task2 will interpret task2 as a positional argument for task1 rather than a separate task to run.

  6. Use delayed task creation with `doit.create_after`

    master

    By default, doit completes the task-loading phase (finding task_ functions and creating metadata) before starting the task-execution phase.

    If you cannot know all tasks until some tasks have already executed, use delayed task creation. By decorating a task-creator function with @doit.create_after(executed='task_name'), the evaluation of that function is delayed until after the specified task has finished executing.

  7. Use setup and teardown tasks

    master

    Setup Tasks

    Use setup_dep (or similar setup mechanisms) to define tasks that prepare the environment.

    • A setup-task is executed only if the main task is determined to be not up-to-date.
    • Setup-tasks are normal tasks that follow standard doit behavior.
    • Note: A task_dep is checked before determining if the main task is up-to-date, whereas a setup-task is executed after that check if the task needs to run.

    Teardown Actions

    Tasks can define teardown actions. These are executed after all task actions have finished, in the reverse order of the tasks that were executed.

  8. Configure task names and sub-tasks

    master

    Task Naming

    By default, a task's name is the name of its creator function (minus the task_ prefix). You can override this using the basename key in the task dictionary.

    Sub-tasks

    To create multiple related tasks, a task-creator can yield dictionaries. Each sub-task must have a unique name field. This is often formatted as parent_task:sub_task_name.

    To ensure a sub-task exists even if it has no specific logic, you can yield a sub-task with name: None to set attributes like doc or watch for the parent.

  9. Why choose doit over Make, Scons, or Invoke

    master

    Based on real-world implementations (like MetalK8s), doit offers several advantages over other automation tools:

    • vs. Make: doit is easier to maintain, evolve, and debug as complexity grows. It is Python-based, allowing developers to use professional testing frameworks, linters, and code analyzers on the build system itself.
    • vs. Scons/Waf: While Scons and Waf focus on hiding compiler complexity, doit excels at running arbitrary shell commands and providing a flexible high-level interface.
    • vs. Invoke: Unlike Invoke, doit features a robust dependency tracking system. Invoke may re-execute tasks even if dependencies are unchanged, whereas doit only executes tasks when necessary.

    Key doit features for complex builds:

    • Easy external command invocation: Seamlessly mix shell commands with Python logic.
    • Customizable execution: Use the uptodate API to define custom logic for determining if a task needs to run.
    • Inspection tools: Use doit info to inspect task dependencies.
    • Automation: Use doit auto to automatically replay tasks based on dependency changes.
    • Output: Supports JSON output for integration with other tools.

  10. Extend doit using the plugin system

    master

    The doit plugin system uses Python entry points. Plugins do not require a specific interface; they simply need to implement the API of the component they are extending (e.g., a Command, Reporter, or Loader).

    Plugins can be enabled in two ways:

    1. Local plugins: Configured directly in your project's configuration files (doit.cfg or pyproject.toml).
    2. Installed plugins: Plugins distributed via setuptools that provide an entry point are automatically enabled upon installation.

  11. Pass parameters to actions

    master

    Actions can receive parameters from three sources:

    1. kwargs: Explicitly defined in the action's definition.
    2. Task Metadata: Automatically calculated values like dependencies, changed, targets, and task.
    3. Computed Values: Values calculated by other tasks (via getargs).

    Keywords with Task Metadata

    doit automatically calculates these values for every task:

    • dependencies: list of file_dep.
    • changed: list of file_dep that changed since the last successful execution.
    • targets: list of targets.
    • task: (Python actions only) The actual Task object instance.

    Using Metadata in cmd-action

    For cmd-action strings, you can use implicit keyword substitution via Python formatting. The keyword value is a string of all respective filenames separated by a space.

    Control the formatting style via DOIT_CONFIG['action_string_formatting']:

    • 'old': Uses % (default in version 0.32).
    • 'new': Uses {}.
    • 'both': Supports both.

    Note: If using the string form of cmd-action, you must escape control characters by doubling them (e.g., %% for % or {{}} for {}). If using the list form (e.g., ['echo', '{}']), escaping is not required.

    Using Metadata in python-action

    For python-action, simply add the keyword as a parameter in your function. dependencies, changed, and targets are passed as lists of strings.

  12. Use result_dep to track non-file dependencies

    master

    When a task's dependency is not a file (e.g., a database value, an API response, or a version number from a VCS), use result_dep to track the output of another task. doit will monitor the result of the specified task and only execute the dependent task if that result changes.

    • For python-actions: The result is the value returned by the function (string or dict).
    • For cmd-actions: The result is the combined stdout and stderr output.
    • Group-tasks: result_dep also works with group-tasks by checking that all subtask results and the set of subtasks themselves remain unchanged.