targets

The targets R package is a pipeline toolkit designed for computationally intensive, reproducible research. It functions as a dynamic, function-oriented alternative to tools like GNU Make, specifically optimized for the R language.

Core Capabilities:

• Dependency Management: Uses static code analysis to detect relationships between computational tasks and constructs a Directed Acyclic Graph (DAG).
• Efficiency: Automatically skips tasks that are already synchronized with their upstream dependencies, reducing runtime during iterative development.
• Reproducibility: Provides tangible evidence of reproducibility by ensuring all results are consistent with their underlying data and code.
• Scalability: Supports high-performance computing (HPC) through implicit parallel computing and optional cloud storage integration.

To run pipelines at scale, targets uses its internal DAG to identify which targets can run concurrently. As soon as a target's dependencies are met, it is deployed to an available parallel worker.

targets leverages two primary frameworks for workload submission:

• clustermq: Used for persistent workers, suitable for submitting workloads to resource managers on shared computing clusters.
• future: Used for transient workers, capable of submitting workloads to multiple cores on a single machine or to various computing clusters.

This architecture allows targets to work across single machines and popular resource managers on shared computing clusters.

The tar_watch() app provides a visual interface to monitor the status and progress of a targets pipeline. It offers several different views depending on your needs:

• summary: Provides an overall runtime progress summary. The "time" column indicates when a target last started, finished, errored, or canceled, and the "since" column shows the elapsed time since that event.
• branches: Similar to the summary view, but specifically includes progress information for dynamic branching.
• progress: Displays a large, searchable table containing detailed progress information and metadata.
• graph: Renders the tar_visnetwork() dependency graph to visualize the pipeline structure.

If the graph view is slow to refresh, you can toggle the outdated setting, optimize your _targets.R file, or switch to a different view.

To build a new pipeline, follow these steps:

1. Write R functions: Define the logic for your data preparation, analysis, and summarization. Save these in R scripts, ideally within a folder named "R/" in your project.
2. Initialize the pipeline: Call use_targets() to generate the necessary key files, most importantly the _targets.R file which configures and defines the pipeline.
3. Configure _targets.R: Open the generated _targets.R file and follow the comments to define your specific sequence of targets.
4. Execute and inspect:
   • Use tar_visnetwork() to visualize the pipeline structure.
   • Use tar_make() to run the pipeline.
   • Use tar_read() to access the resulting R objects.

# High-level workflow summary
use_targets()      # Initialize
tar_visnetwork()   # Visualize
tar_make()         # Run
tar_read(target)   # Read results

You can install targets from CRAN (stable release) or via GitHub/rOpenSci (development versions). If you plan to use targets with crew for distributed computing, ensure you install crew version 0.4.0 or higher first.

# Recommended if using crew for distributed computing
install.packages("crew")

# Install the stable release from CRAN
install.packages("targets")

# Install the development version from GitHub
pak::pkg_install("ropensci/targets")

# Install the development version from rOpenSci
install.packages("targets", repos = "https://dev.ropensci.org")

The targets package is a Make-like pipeline tool for statistics and data science in R. It coordinates computationally demanding analysis projects by managing a workflow of "targets".

Core Concepts:

• Pipeline: A computational workflow (e.g., forecasting, simulations, genomics) consisting of multiple tasks.
• Targets: The individual tasks in a pipeline. Each target runs a user-defined R function and returns an R object.
• Efficiency: The package skips costly runtime for tasks that are already up to date. If the current output matches the upstream code and data, the pipeline is considered up to date.
• Abstraction: It abstracts files as R objects and orchestrates computation with implicit parallel computing.

To build a new pipeline, follow these steps:

1. Write R functions: Define the logic for your tasks in R scripts (ideally stored in a R/ folder in your project).
2. Initialize the pipeline: Call use_targets() to generate the necessary configuration files, most importantly the _targets.R file.
3. Configure _targets.R: Open the generated _targets.R file and follow the comments to define your specific pipeline targets.
4. Verify and Run:
   • Use tar_visnetwork() to visualize the pipeline structure.
   • Use tar_make() to execute the pipeline.
   • Use tar_read() to access the resulting R objects.

# Typical workflow steps
use_targets()
# (Edit _targets.R to define targets)
tar_visnetwork()
tar_make()
tar_read(target_name)

When using the graph view in tar_watch(), you can customize the dependency graph using these settings:

• targets_only: When enabled, the graph shows only targets. When disabled, it also includes functions and other global objects.
• outdated: Toggles color-coding of nodes based on whether they are up to date. Note: This may impact performance in pipelines with an enormous number of targets.
• label: Allows you to append metadata to node names, such as target size, runtime, or the number of branches, based on recorded information.
• level_separation: Controls the width/spacing of the graph levels.

You can control how the tar_watch() app updates its data using the following settings:

• Refresh once: A button to manually force all displays in the app to refresh immediately.
• Refresh periodically: A toggle to enable or disable automatic refreshing every few seconds.
• Refresh seconds: A setting to define the interval (in seconds) at which the app refreshes when periodic refreshing is enabled.

The tar_watch() function launches a built-in Shiny app that allows you to visualize the progress of your pipeline while it is running.

If you are building a custom application, you can use these as Shiny modules:

• tar_watch_ui()
• tar_watch_server()

The following functions are the primary interface for managing and interacting with a targets pipeline:

• use_targets(): Writes key files, including the vital _targets.R file which configures and defines the pipeline.
• tar_visnetwork(): Checks the pipeline by providing a visual network representation.
• tar_make(): Runs the pipeline.
• tar_read(): Reads output objects from the pipeline.