redun

redun is designed for multi-domain scientific pipelines, focusing on task parallelism, caching, and data provenance.

Key distinctions:

• Task vs. Data Parallelism: redun provides expressive task parallelism, but it does not perform fine-grained data parallelism (for which you should use tools like Spark or Dask).
• Integration: redun does not use 'dirty tricks' like complex static analysis, making it safe to combine with frameworks like pyspark, pytorch, or Dask.
• Compute: redun does not provide its own compute cluster; it orchestrates existing systems like cloud provider batch services or Spark clusters.

redun is suitable for various domains including:

• Bioinformatics
• Cheminformatics
• Web or API data extraction
• General data science

redun uses memoization to cache task results. When a task is called with the same arguments and the same code (hash), redun reuses the cached result instead of re-executing.

Key behaviors:

• Argument Changes: If task arguments change, redun detects the change and re-executes only the affected tasks and their downstream dependencies.
• Code Changes: redun hashes the task implementation. If you update the code inside a @task(), redun detects the new hash and triggers re-execution.
• Fast-Forwarding: If a program is re-run without changes, redun 'fast-forwards' through the execution by fetching all results from the cache without running any task bodies.

You can limit the parallelism of tasks by defining resource limits in the [limits] section.

1. Define limits in redun.ini:

[limits]
db = 10
web_api = 5

2. Assign limits to tasks in Python:

• Single resource: @task(limits=["db"]) consumes 1 unit of db.
• Multiple resources: @task(limits=["db", "web_api"]) consumes 1 unit of each.
• Multiple units of one resource: @task(limits={"db": 2}) consumes 2 units of db.

@task(limits={"db": 2})
def read_from_database(args):
    # Consumes 2 units of 'db'
    ...

The AliasExecutor allows multiple executor names to resolve to the same underlying implementation. This is useful for resource management; for example, if you have multiple tasks that conceptually need different executors but you want them to share a single process pool to limit concurrency.

To implement this, define a single base executor and then create multiple alias executors that point to it using the target key.

; Define the actual resource-constrained executor
[executors.single_worker]
type = local
mode = process
max_workers = 1

; Create aliases for other names to point to the single worker
[executors.process]
type = alias
target = single_worker

[executors.default]
type = alias
target = single_worker

[executors.foo_exec]
type = alias
target = single_worker

Federated tasks allow you to invoke tasks located in different Python repositories without having the code locally. This is typically achieved by publishing the remote repository as a Docker image.

Configuration

1. federated_tasks: Define in your config file to map task names to their import paths and associated executors (e.g., a Docker-based executor).
2. federated_imports: Allows remote repositories to provide their own configuration files containing additional federated tasks and executors, which can be shared via a shared file system.

REST-based Proxy

For fire-and-forget job launching (useful for UIs) or managing AWS role switches, use:

• redun.federated_tasks.rest_federated_task
• redun.federated_tasks.launch_federated_task

If you use non-deterministic functions (like random.random()) inside a @task, you must be aware of how Redun's caching affects them:

• With Caching (Default): The first generated value will be cached. Subsequent calls with the same inputs will return the exact same cached value, effectively making the 'random' function deterministic.
• Without Caching: If the cache is disabled or limited in scope, Redun will run the task implementation every time, allowing for new random values to be generated and recorded.

@task
def rand():
    return random.random()

The Console includes a Python REPL for deep inspection of job arguments and results. When you press r on a Job screen, the REPL starts with local variables from that job (such as job, args, kwargs, and result) already injected into the environment.

Redun uses SQLAlchemy to lazily load these values from the database. Values are deserialized using pickle; if a class is not currently importable, it will default to a pickle preview (mock-like object).

Built-in functions for inspection:

• query(): Provides access to the SQLAlchemy ORM for navigating Call Graph models in the redun database.
• console(): Provides an easy way to navigate to the screen of a specific model (e.g., console(Execution)).

Redun uses metaprogramming to implement an asynchronous functional programming language on top of Python. This allows you to use Python syntax while benefiting from distributed computing features. The mapping between Python and Redun concepts is as follows:

Redun's context allows you to define arguments in the redun.ini file and access them in tasks using get_context. This avoids passing the same argument through every intermediate task in a workflow.

1. Define context in redun.ini:

[scheduler]
context = {"my_tool": {"ratio": 1.618}}

2. Access in a task:

from redun import task, get_context

@task
def my_tool(value: int, ratio: float = get_context("my_tool.ratio")) -> str:
    return f"Ran my_tool with argument: {ratio}"