FinRL: Financial Reinforcement Learning Framework

FinRL is an open-source framework designed for financial reinforcement learning. It is intended to help users transition into quantitative finance by developing stock trading strategies using Deep Reinforcement Learning (DRL).

Key capabilities include:

• Algorithm Support: Provides fine-tuned DRL algorithms such as DQN, DDPG, PPO, SAC, A2C, and TD3.
• Problem Solving: Addresses dynamic decision-making in trading, specifically deciding where to trade, at what price, and what quantity.
• Core Advantages: Offers portfolio scalability and market model independence by learning through interactions with unknown environments.
• Scope: Supports various markets, state-of-the-art (SOTA) DRL algorithms, benchmarks for quantitative finance tasks, and live trading capabilities.

The PortfolioOptimizationEnv (POE) is a reinforcement learning environment designed to simulate market effects on a portfolio that is periodically rebalanced.

At every timestep $t$, the agent determines a portfolio vector $W_{t}$, which specifies the percentage of money invested in each stock. The environment then uses user-provided data to simulate the new portfolio value at timestep $t+1$.

FinRL Imitation Learning implements a multi-stage machine learning workflow designed to analyze financial big data by first imitating expert strategies (such as alpha factors or smart investors) and then refining those results using Reinforcement Learning (RL).

This approach is used to initialize deep neural networks with human-level performance (via imitation learning) before allowing the RL agent to learn through trial and error to potentially surpass that performance.

The reward $r_{t}$ is calculated using the logarithmic return of the portfolio value:

$$r_{t} = \ln(V_{t}/V_{t-1})$$

Where:

• $V_{t}$ is the value of the portfolio at time $t$.
• $V_{t-1}$ is the value of the portfolio at the previous timestep.

Behavior: The reward is positive if the portfolio value increases and negative if the portfolio value decreases due to rebalancing or market movement.

In FinRL, multiple stock trading is modeled as a Markov Decision Process (MDP) where the goal is to maximize the portfolio value.

Core Components:

• State: Observations from the environment, typically technical indicators (MACD, RSI, CCI, ADX) calculated from Open-High-Low-Close prices and volume.
• Action: The agent outputs a trading signal for each stock. In a discrete space, this might be {-k, ..., 0, ..., k} where k is the number of shares. A value of 1 represents a buy signal, -1 a sell signal, and 0 holding.
• Reward: The change in portfolio value between time steps: $r = v_{t+1} - v_t$, where $v$ is the total portfolio value (balance + market value of stocks).
• Environment: The set of stocks being traded (e.g., Dow 30 constituents).

FinRL provides access to several fine-tuned Deep Reinforcement Learning (DRL) algorithms through three primary backend libraries: ElegantRL, Stable Baselines 3, and RLlib.

Users can choose from the following supported algorithms depending on their task requirements (discrete vs. continuous action spaces):

• DQN (Deep Q-Network)
• DDPG (Deep Deterministic Policy Gradient)
• Multi-Agent DDPG
• PPO (Proximal Policy Optimization)
• SAC (Soft Actor-Critic)
• A2C (Advantage Actor-Critic)
• TD3 (Twin Delayed DDPG)

FinRL also allows for the design of custom algorithms, such as Adaptive DDPG or ensemble methods, by adapting these existing implementations.

To handle market volatility, you can calculate a turbulence threshold based on historical data. A common approach is to use a specific quantile (e.g., 99%) of the in-sample turbulence index. If the current turbulence index exceeds this threshold, the environment assumes the market is volatile.

insample_turbulence = dow_30[(dow_30.datadate<'2019-01-01') & (dow_30.datadate>='2009-01-01')]
insample_turbulence = insample_turbulence.drop_duplicates(subset=['datadate'])

FinRL is organized into three core layers that facilitate the reinforcement learning workflow for financial tasks:

1. Market Environments (meta/): Provides the simulation environment where agents interact with market data.
2. DRL Agents (agents/): Contains implementations of Deep Reinforcement Learning algorithms (e.g., via stablebaseline3, elegantrl, or rllib).
3. Financial Applications (applications/): Specific trading tasks such as stock_trading, cryptocurrency_trading, portfolio_allocation, and high_frequency_trading.

The standard workflow follows a train-test-trade pipeline using train.py, test.py, and trade.py.

Raw OHLCV data must be converted into a 'state' suitable for Reinforcement Learning agents. This involves two primary steps:

1. Feature Engineering (Technical Indicators): Adding indicators like MACD (Moving Average Convergence/Divergence) or RSI (Relative Strength Index) to capture market trends and momentum.
2. Risk Management (Turbulence Index): Adding a turbulence index to measure extreme asset price fluctuations. This helps the agent account for market volatility and risk-aversion during periods of financial crisis.

Key classes for these tasks include FeatureEngineer and data_split from finrl.meta.preprocessor.preprocessors.

Users should choose between the original FinRL repository and the newer FinRL-X (FinRL-Trading) based on their use case:

Recommendation: Use FinRL-X / FinRL-Trading for modern, production-oriented, or deployment-aware quantitative trading systems.