ForwardDiff.jl

ForwardDiff implements forward-mode automatic differentiation (AD) to compute various derivatives of native Julia functions or any callable object.

When to use ForwardDiff:

• It is highly accurate and generally faster than non-AD algorithms.
• It is an excellent choice for functions mapping a scalar to a vector (e.g., g(y::Real) -> Vector).
• For functions mapping a vector to a scalar (e.g., f(x::Vector) -> Scalar), ForwardDiff is a good choice if the input vector x is not too large, as the implementation is simpler than reverse-mode AD.

using ForwardDiff

# Example: Gradient and Hessian of a scalar-valued function
f(x::Vector) = sin(x[1]) + prod(x[2:end])
x = vcat(pi/4, 2:4)

g = ForwardDiff.gradient(f, x)
H = ForwardDiff.hessian(f, x)

# Example: Derivative of a vector-valued function
g_vec(y::Real) = [sin(y), cos(y), tan(y)]
d = ForwardDiff.derivative(g_vec, pi/4)

# Example: Jacobian using an anonymous function
J = ForwardDiff.jacobian(x) do x
    [sin(x[1]), prod(x[2:end])]
end

ForwardDiff.jl implements forward-mode automatic differentiation (AD) to compute derivatives, gradients, Jacobians, Hessians, and higher-order derivatives of native Julia functions or any callable object.

While reverse-mode AD is typically preferred for functions mapping a vector to a scalar, ForwardDiff is a highly efficient and simple choice when the input vector x is not excessively large. Conversely, ForwardDiff is the optimal choice for functions mapping a scalar to a vector.

Key functions include:

• ForwardDiff.derivative(f, x): Computes the derivative of a function.
• ForwardDiff.gradient(f, x): Computes the gradient of a scalar-valued function.
• ForwardDiff.jacobian(f, x): Computes the Jacobian matrix.
• ForwardDiff.hessian(f, x): Computes the Hessian matrix.

using ForwardDiff

# Example 1: Scalar-valued function (Gradient and Hessian)
f(x::Vector) = sin(x[1]) + prod(x[2:end]);
x = vcat(pi/4, 2:4)

g = ForwardDiff.gradient(f, x)
H = ForwardDiff.hessian(f, x)

# Example 2: Vector-valued function (Derivative)
g(y::Real) = [sin(y), cos(y), tan(y)]
d = ForwardDiff.derivative(g, pi/4)

# Example 3: Jacobian using an anonymous function
J = ForwardDiff.jacobian(x) do x
    [sin(x[1]), prod(x[2:end])]
end

ForwardDiff uses ForwardDiff.AbstractConfig types to manage internal state like chunk size, work buffers, and perturbation seeds. While the API allocates these automatically, you can improve performance and reduce memory usage by preallocating them yourself.

Configuration Types

• ForwardDiff.DerivativeConfig
• ForwardDiff.GradientConfig
• ForwardDiff.JacobianConfig
• ForwardDiff.HessianConfig

Key Usage Rules

1. Chunk Size: You can explicitly provide a chunk size N to the constructors. It is highly recommended to specify this manually when possible.
2. Function Specificity: Configurations constructed for a specific function f cannot be reused for different functions, but they can be reused to differentiate f at different input values.
3. Generic Configurations: To create a configuration that can be reused for any function, pass nothing as the function argument. Note that this reduces ForwardDiff's ability to prevent perturbation confusion.

ForwardDiff uses an implementation of dual numbers to perform forward-mode automatic differentiation. The core mechanism relies on the Dual type, which pairs a real value with its partial derivatives.

The Dual Type

A Dual number is represented as:

• value: The original real-valued component.
• partials: An N-dimensional vector of partial derivatives stored in a Partials type.

Mathematically, a dual number is expressed as $a + \sum_{i=1}^N b_i \epsilon_i$, where $a$ is the value and $b_i$ are the partials. When an elementary function (like sin) is called on a Dual number, ForwardDiff overloads the function to compute both the function value and the derivative using the chain rule.

Higher-Order Differentiation

ForwardDiff supports higher-order derivatives (like Hessians) through hyper-dual numbers. This is achieved by nesting Dual types. For example:

• A second-order hyper-dual number has the type Dual{T, Dual{S, V, M}, N}.
• A third-order hyper-dual number has the type Dual{T, Dual{S, Dual{R, V, K}, M}, N}.

# Example of how a function like sin is overloaded for Dual numbers
Base.sin(d::Dual{T}) where {T} = Dual{T}(sin(value(d)), cos(value(d)) * partials(d))

# Example of forming a second-order hyper-dual number
Dual(Dual(x, one(eltype(x))), one(eltype(x)))

The ForwardDiff API abstracts the underlying Dual number implementation, allowing users to compute gradients, Jacobians, and Hessians without manually managing dual numbers.

The Seeding Process

To compute a derivative, the API performs the following steps:

1. Seeding: It takes the input vector $\vec{x}$ and seeds it with Dual numbers by adding infinitesimal $\epsilon$ terms (e.g., $\vec{x}_{\epsilon} = [x_1 + \epsilon_1, ..., x_N + \epsilon_N]^T$).
2. Evaluation: It passes these seeded values into the target function $f$.
3. Extraction: It extracts the derivative information from the resulting Dual number.

Chunking

For performance, ForwardDiff does not always seed the entire input vector at once. Instead, it processes the input in chunks. For example, if an input vector has size 4 and the chunk size is 2, the API will make two separate calls to the function $f$, each handling a subset of the partial derivatives, and then combine the results.