future

When evaluating R expressions asynchronously (parallel) or via lazy evaluation, future must identify and pass 'global' (free) objects to the evaluator.

• Automatic Identification: future uses the globals package to perform static code inspection to identify global variables.
• Package Globals: If a global is defined within a package, future ensures that the corresponding package is attached during evaluation rather than exporting the variable itself. This saves memory and bandwidth.
• Manual Specification: Static analysis can fail in complex corner cases. If globals are missing (causing runtime errors) or incorrectly identified, you can manually specify them using the globals argument in future creation functions.

You can specify globals as a character vector of names or as a named list of name-value pairs.

# Manual specification examples:
# Using names
globals = c("a", "slow_sum")

# Using name-value pairs
globals = list(a = 42, slow_sum = my_sum)

When using future-based packages (like future.apply, future.batchtools, or future.callr), you must ensure that the global objects used in your future expressions do not contain non-exportable references.

A common error is Error: Detected a non-exportable reference ('externalptr') in one of the globals (<unknown>) used in the future expression. This typically occurs when an object contains an externalptr (external pointer), such as a database connection, a file connection, or a processx_connection. These objects cannot be serialized and sent to worker nodes.

To fix this, ensure that any objects passed into functions like future_by(), flapply(), or future_lapply() are composed of standard R data types that can be safely exported to parallel workers.

# Example of what causes the error:
# If 'conn' is a database connection (externalptr), this will fail:
plan(multisession)
future_lapply(1:10, function(i) use_connection(conn))

# Error message:
# Error: Detected a non-exportable reference ('externalptr') in one of the globals (<unknown>) used in the future expression

A flat topology is when all futures are created in the same environment. A nested topology is when one future creates other futures internally.

By default, nested futures use sequential evaluation to prevent an accidental explosion of background processes (e.g., in recursive calls).

To define a specific hierarchy of backends, pass a list to plan(). For example, plan(list(multisession, sequential)) will run the first level of futures in parallel, but any futures created inside those workers will run sequentially.

To allow nested futures to also run in parallel, you must use tweak() to explicitly specify the number of workers at each level of the hierarchy.

# Explicitly defining a nested topology: 
# Level 1: Multisession, Level 2: Sequential
plan(list(multisession, sequential))

# Allowing nested multisession workers by tweaking
plan(list(tweak(multisession, workers = 2), tweak(multisession, workers = 2)))

The Future API is designed so that code remains agnostic of the underlying backend. Whether running sequentially or on a remote cluster, the following behaviors are enforced:

1. Local Environment Evaluation: All evaluations are performed within a local environment (using local({ expr })). This ensures that assignments made inside a future do not affect the calling environment.
2. Global Variable Identification: When a future is constructed, global variables are identified and exported to the evaluating process. To prevent accidental exporting of massive objects, there is a built-in size threshold (configurable via future.options).
3. Single Evaluation: Future expressions are evaluated only once. Once the value or error is collected, it is cached for all subsequent requests.

# Demonstrating that assignments inside a future do not affect the calling environment
plan(sequential)
> a <- 1
> x %<-% {
+     a <- 2
+     2 * a
+ }
> x
[1] 4
> a
[1] 1

The Future API is designed to encapsulate differences between strategies so that code behaves consistently regardless of whether it is running locally or on a remote cluster. This allows developers to prototype with sequential evaluation and switch to asynchronous processing later without changing the core logic.

All futures follow these rules:

1. Local Environment Evaluation: Expressions are evaluated using local({ expr }), meaning assignments inside a future do not affect the calling environment.
2. Global Variable Handling: When a future is created, global variables are identified. For asynchronous evaluation, they are exported to the worker process. For lazy = TRUE sequential futures, they are "frozen" (cloned to a local environment). There is a built-in size threshold for globals to prevent accidental large exports (configurable via future.options).
3. Single Evaluation: A future expression is evaluated only once. Once the value or error is collected, it is available for all subsequent requests.

plan(sequential)
a <- 1
x %<-% {
    a <- 2
    2 * a
}
x
[1] 4
a
[1] 1

The future package provides two ways to create futures: an implicit style and an explicit style. Both styles behave identically in terms of execution, but they differ in syntax and code readability.

Implicit Style

This style is most similar to standard R assignment. You replace the standard assignment operator <- with %<-%. This creates a future and a promise to its value.

Explicit Style

This style makes it very clear to anyone reading the code that a future is being used, which reduces the risk of accidentally assuming code is running sequentially. It uses two distinct steps: creating the future and then retrieving its value.

• f <- future({ expr }): Creates the future.
• v <- value(f): Retrieves the value (blocks if the future is not yet resolved).

Note: When using futures, output (like cat()) is typically relayed after the future is resolved (when the value is queried), rather than during the evaluation of the expression.

# Implicit style
library(future)
v %<-% {
  cat("Hello world!\n")
  3.14
}

# Explicit style
library(future)
f <- future({
  cat("Hello world!\n")
  3.14
})
v <- value(f)