numdifftools

numdifftools is a Python suite for automatic numerical differentiation of functions in one or more variables. It uses adaptive finite differences combined with Richardson extrapolation to maximize accuracy.

Users can configure several aspects of the differentiation process, including:

• The order of the method or extrapolation.
• The difference type: complex-step, central, forward, or backward differences.

The numdifftools library provides a suite of tools for automatic numerical differentiation using adaptive finite differences and Richardson extrapolation. The following methods are available:

• Derivative: Computes derivatives of order 1 through 10 on any scalar function.
• directionaldiff: Computes the directional derivative of a function of $n$ variables.
• Gradient: Computes the gradient vector of a scalar function of one or more variables.
• Jacobian: Computes the Jacobian matrix of a vector-valued function of one or more variables.
• Hessian: Computes the Hessian matrix of all 2nd partial derivatives of a scalar function of one or more variables.
• Hessdiag: Computes only the diagonal elements of the Hessian matrix.

All methods also produce error estimates for the results.

The complex-step derivative method is an alternative to standard central difference rules.

Advantages:

• It avoids the problem of round-off error caused by subtraction when using very small steps, as it relies on the imaginary part of the function evaluation.
• For $n > 1$ (derivatives of order 2 or higher), it has a truncation error of $O(\delta^4)$. For $n=1$, the truncation error is $O(\delta^2)$.

Limitations/Requirements:

• The function must be analytic.
• The function must support complex numbers.
• It cannot be used with non-analytic functions such as abs, max, or min.

Because of these requirements, the central method is the default in Numdifftools.

Richardson extrapolation is a technique used to improve the accuracy of finite difference approximations. It works by generating multiple estimates of a derivative at different step sizes ($\delta, \delta/2, \delta/4$, etc.) and extrapolating the results to a zero step size.

By assuming the error follows a predictable pattern (e.g., linear or quadratic in $\delta$), the method cancels out lower-order error terms. numdifftools.Derivative uses multiple-term Richardson extrapolants for all derivatives up to any practical order, allowing for high accuracy even with relatively large step sizes.

Numdifftools provides an interface to AlgoPy (Algorithmic Differentiation in Python) via numdifftools.nd_algopy. This allows for evaluating higher-order derivatives using forward and reverse modes of Algorithmic Differentiation (AD).

import numdifftools.nd_algopy as nda
import numpy as np

# 1st derivative using AlgoPy
fd = nda.Derivative(np.exp)

# Jacobian using reverse mode
xdata = np.reshape(np.arange(0, 1, 0.1), (-1, 1))
ydata = 1 + 2 * np.exp(0.75 * xdata)
fun = lambda c: (c[0] + c[1] * np.exp(c[2] * xdata) - ydata)**2
Jfun = nda.Jacobian(fun, method='reverse')

To achieve higher-order approximations for derivatives, Numdifftools utilizes odd and even transformations of the target function $f(x)$ around a point $x_0$.

• Odd transformation ($f_{odd}$): Used to generate higher-order approximations for odd-order derivatives (like the 1st, 3rd, or 5th derivative). It effectively cancels out even-order terms in the Taylor series.
• Even transformation ($f_{even}$): Used to generate higher-order approximations for even-order derivatives (like the 2nd, 4th, or 6th derivative). It cancels out odd-order terms.

By working with these transformed functions, the library can construct high-order differentiation rules that remain accurate across different scales.

When using numdifftools.Derivative with full_output=True, you can access an error_estimate. This is calculated by treating the Richardson extrapolation as a polynomial curve fit in the step size parameter $\delta$. The uncertainty is derived using statistical methodology (specifically a Student's t-distribution) applied to the error of the constant term in the polynomial model.

numdifftools automatically selects the optimal step size spacing that minimizes this predicted error, balancing between approximation error (from large steps) and floating-point noise (from small steps).

import numdifftools as nd
from numpy import exp

f = nd.Derivative(exp, full_output=True)
result = f(1)

print(f"Estimate: {result.estimate}")
print(f"Error Estimate: {result.error_estimate}")
print(f"Final Step Size: {result.final_step}")

The numdifftools.Derivative class treats the Richardson extrapolation step as a polynomial curve fit in the step size parameter $\delta$. The desired derivative is the constant term of this polynomial.

To provide a statistical uncertainty estimate, the library uses a three-term Richardson-like extrapolant evaluated at four distinct values of $\delta$. This provides an estimate of the standard error of the constant term, which is then converted into a two-sided 95% confidence interval using the Student's t-distribution.

Numdifftools is designed for the numerical differentiation of analytically supplied functions. It is distinct from symbolic differentiation (e.g., SymPy) and differentiation of data points (e.g., curve fitting or splines).

The core of the library is the Derivative class, which uses adaptive differentiation schemes. Instead of fixed spacing, it generates a sequence of sample points following a log spacing away from the point of interest. This allows the library to use a single differentiation rule that applies at any scale by simply adjusting a scale factor.

The complex-step derivative is an alternative method available in numdifftools.Derivative. It approximates the derivative by replacing the step $\delta$ with a complex step $ih$.

Advantages:

• It avoids the round-off errors typically associated with small steps in finite difference methods because it does not involve subtraction.
• For $n > 1$ (higher-order derivatives), it has a truncation error of $O(\delta^4)$ for both odd and even orders.
• For $n = 1$ (the first derivative), the truncation error is $O(\delta^2)$.

Requirements and Limitations:

• Analytic Functions Only: The function being differentiated must be analytic and must support complex number inputs.
• Unsupported Functions: It will not work with non-analytic functions such as abs, max, or min.
• Default Method: Because of these requirements, the central method is the default in Numdifftools.