deepdow

DeepDow is a framework designed for portfolio optimization using end-to-end deep learning. Unlike traditional two-stage optimization methods, DeepDow focuses on constructing networks that perform weight allocation in a single forward pass.

Instead of separating predictive modeling (estimating returns and covariance) from optimization (solving a convex problem), DeepDow merges these steps. The network takes raw features (such as returns and volumes) as input and directly outputs asset allocations. This allows for:

• Using a single loss function for the entire process.
• Turning hyperparameters (like risk aversion coefficients) into trainable weights.
• Extracting features specifically optimized for allocation rather than just prediction.

The deepdow framework provides several capabilities for research and implementation:

• Differentiable Layers: All layers are built on torch and are fully differentiable.
• Convex Optimization: Integrates cvxpylayers for differentiable convex optimization.
• Portfolio Algorithms: Implements clustering-based portfolio allocation algorithms.
• Data Loading: Provides multiple strategies including RigidDataLoader and FlexibleDataLoader.
• Experiment Tracking: Integrates with mlflow and tensorboard via callbacks.
• Loss Functions: Provides various loss functions such as Sharpe ratio and maximum drawdown.
• Hardware Support: Supports both CPU and GPU.

deepdow (read as "wow") is a Python framework that merges market forecasting with portfolio optimization into a single, fully differentiable pipeline.

Instead of treating forecasting (e.g., LSTM, GARCH) and optimization (e.g., convex optimization) as separate steps, deepdow constructs a pipeline of layers where the final layer performs the weight allocation and preceding layers act as feature extractors. Because the entire network is built on torch, the parameters can be optimized end-to-end using gradient descent.

Key Constraints:

• Buy and Hold: It is designed for finding allocations to be held over a specific horizon, not for active trading strategies. Consequently, frequent transaction costs are not the primary focus.
• Not Reinforcement Learning: While layers can be reused in RL applications, deepdow itself is not a reinforcement learning framework.

To prepare data for training, deepdow uses a rolling window method to decompose a large time series tensor into three disjoint subtensors at any given time step (representing "now"):

• x (Features): Represents past and present knowledge. Shape: (n_channels, lookback, n_assets).
• g (Gap): Represents information in the immediate future that cannot be used for investment decisions. Shape: (n_channels, gap, n_assets).
• y (Labels): Represents the future evolution of the market. Shape: (n_channels, horizon, n_assets).

By rolling this window across the time dimension, you generate a collection of feature tensors (x1, x2, ...) and corresponding label tensors (y1, y2, ...).

Callbacks allow you to execute code at specific lifecycle stages of the training loop. To create a custom callback, inherit from deepdow.callbacks.Callback and override one or more of the following methods:

• on_batch_begin: At the start of each batch.
• on_batch_end: At the end of each batch.
• on_epoch_begin: At the start of each epoch.
• on_epoch_end: At the end of each epoch.
• on_train_begin: At the start of the training.
• on_train_end: At the end of the training.
• on_train_interrupt: When training is interrupted.

Each method receives a metadata dictionary containing the most recent values of relevant variables. Additionally, the callback instance can access the active Run instance via the .run attribute, which is injected during the launch process.

In deepdow, benchmarks are algorithms that map an input feature tensor x to a weights tensor. They are categorized into two types:

1. Networks: Benchmarks with learnable parameters. Their allocation algorithm must be a differentiable forward pass implemented using torch functions, modules, or deepdow.layers.
2. Simple Benchmarks: Benchmarks without learnable parameters. These serve as baselines and do not change predictions over different epochs. Because they are not trainable, their allocation algorithms do not need to be differentiable; you can cast the input torch.Tensor to a numpy array and use external libraries like scipy.

Use Simple Benchmarks to establish a performance baseline before training a network.

The goal of a deepdow network is to take the feature tensor x as input and return a single weight allocation vector w of shape (n_assets,).

Key constraints and behaviors:

• Sum to One: The weights must satisfy $\sum_{i} w_{i} = 1$.
• Portfolio Construction: The weights w represent a portfolio that is purchased immediately and held for the duration of the horizon time steps.
• Pipeline: The neural network $F$ with parameters $\theta$ maps $x \to w$.

In deepdow, financial time series are represented as a 3D tensor with three dimensions:

1. time: The temporal dimension (e.g., daily frequency).
2. asset: The dimension representing different financial instruments (e.g., multiple stocks).
3. indicator/channel: The dimension representing different features or data streams (e.g., open price returns, close price returns, and volumes).

A tensor shape is denoted as (n_channels, n_timesteps, n_assets). For example, a shape of (3, 10, 6) implies 3 channels, 10 timesteps, and 6 assets.

The Resample layer is a metallocator that applies parametric bootstrapping to reduce noise in $\boldsymbol{\mu}$ and $\boldsymbol{\Sigma}$ estimates. It wraps a base allocator and performs the following:

1. Samples n_portfolios * n_draws new vectors from $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$.
2. Runs the base allocator for each pair of sampled $\boldsymbol{\mu}_i$ and $\boldsymbol{\Sigma}_i$.
3. Averages the resulting allocations to produce the final weight vector.

Supported Base Allocators:

• AnalyticalMarkowitz
• NCO
• NumericalMarkowitz

When using deepdow, keep the following architectural assumptions in mind:

• Contiguous Time: The time dimension is assumed to be contiguous with a single, consistent frequency (e.g., daily).
• Hold Strategy: Predicted weights w are treated as an investment that is held constant over the specified horizon time steps.