Cross-Market State Fusion

The project implements a Proximal Policy Optimization (PPO) agent designed to trade 15-minute binary prediction markets on Polymarket. The agent uses a single shared policy to trade 4 concurrent crypto markets (BTC, ETH, SOL, XRP) simultaneously.

Key Setup Details:

• Markets: 4 concurrent crypto markets.
• Data Sources: Fuses live data from Binance Futures (fast signal) and Polymarket CLOB (slow execution venue).
• Architecture: Uses a temporal architecture and an 18-dimensional state space.
• Goal: Learn profitable trading patterns from sparse PnL rewards by observing price discovery in the fast market (Binance) before it reflects in the slow market (Polymarket).

The agent employs a single neural network to manage multiple assets simultaneously. This approach requires the agent to:

• Allocate attention across all active markets.
• Learn asset-specific patterns while sharing weights across the network.
• Handle asynchronous expirations and refreshes for each market.

This design encourages the learning of generalizable crypto patterns rather than overfitting to a single specific market.

The agent utilizes a 'fast signal source + slow execution venue' pattern to exploit cross-exchange lag. It fuses two real-time WebSocket (WSS) streams:

1. Binance Futures WSS (Fast Market): Provides price returns (1m, 5m, 10m), volatility, order flow, CVD, and large trades. This represents institutional flow and sub-second price discovery.
2. Polymarket CLOB WSS (Slow Market): Provides bid/ask spread and orderbook imbalance. This represents the execution venue where prices often lag Binance by seconds.

By observing the movement in the Binance 'fast' market, the agent can attempt to bet on Polymarket before the orderbook adjusts to the new price.

Binance Futures WSS  → Price returns (1m, 5m, 10m), volatility, order flow, CVD, large trades
Polymarket CLOB WSS  → Bid/ask spread, orderbook imbalance

The current recommended approach (Phase 4+) for training uses share-based PnL rather than probability-based PnL. This better reflects the actual economics of binary markets, where low-probability entries provide higher proportional returns.

Rewards are sparse: the agent only receives a reward signal when a position closes, rather than receiving feedback at every step.

Calculation Logic:

1. Calculate the number of shares acquired: shares = dollars / entry_price
2. Calculate the PnL: pnl = (exit_price - entry_price) × shares

The project is organized into a main engine, a dashboard, strategy implementations, and data helpers:

• run.py: Main trading engine.
• dashboard.py: Real-time web dashboard.
• strategies/: Contains base.py (abstractions), rl_mlx.py (PPO implementation), and various baselines (momentum.py, mean_revert.py, fade_spike.py).
• helpers/: Handles data ingestion via polymarket_api.py, binance_wss.py, binance_futures.py, and orderbook_wss.py.

The training evolution shows two distinct approaches to reward signals:

Phase 1: Shaped Rewards (Dense)

• Method: Rewards were given at every step (not just on close).
• Components:
  1. Unrealized PnL delta (scaled down by 0.1).
  2. Transaction cost penalty.
• Outcome: This approach failed and backfired.

Phase 4+: Sparse Rewards

• Method: The agent only receives a reward when a position closes.
• Components: Share-based PnL.
• Outcome: Significant improvement in ROI (4.5x improvement compared to previous phases).

In binary markets, using a share-based PnL reward signal is more effective than a probability-based signal. Share-based rewards capture the asymmetric payoff structure where lower entry probabilities yield proportionally larger returns for the same price move.

Comparison:

• Probability-based (Less effective): pnl = (exit_price - entry_price) * dollars
• Share-based (More effective): shares = dollars / entry_price followed by pnl = (exit_price - entry_price) * shares

# Old (Phases 1-3): probability-based
pnl = (exit_price - entry_price) * dollars

# New (Phase 4): share-based
shares = dollars / entry_price
pnl = (exit_price - entry_price) * shares

When designing reward functions for Reinforcement Learning (RL) in trading, avoid 'shaping rewards' (micro-bonuses) that are similar in magnitude to the actual PnL signal. If bonuses for momentum, position size, or transaction costs are too large, the agent may learn to 'game' the reward function—optimizing for the bonuses rather than actual profitability.

Signs of Reward Shaping Failure:

• Entropy Collapse: The policy becomes nearly deterministic (e.g., dropping from 1.09 to 0.36).
• Win Rate Divergence: A significant gap between the Buffer win rate (% of experiences with reward > 0, including bonuses) and the Cumulative win rate (% of closed trades that were actually profitable).

To monitor the agent's performance and market data in real-time, run the dashboard in a separate terminal session. You can specify the port using the --port flag.

python dashboard.py --port 5001

To prevent agents from gaming the reward function, use a sparse reward signal that only triggers upon position closure. This ensures the agent optimizes for realized profit rather than unrealized PnL deltas or micro-bonuses.

Implementation Pattern:

1. Set reward to 0 for all steps except when a position closes.
2. Store pending rewards in a dictionary keyed by the contract ID (cid).
3. When a position closes, calculate the PnL and pop it from the pending rewards to be used as the step reward.

# Reward is 0 for all steps EXCEPT position close
def _compute_step_reward(self, cid, state, action, pos):
    return self.pending_rewards.pop(cid, 0.0)

# pending_rewards is set when position closes:
# pnl = (exit_prob - entry_prob) * size
self.pending_rewards[cid] = pnl

To set up the environment, navigate to the experiment directory, create a virtual environment, and install the required dependencies. This project requires mlx for on-device training on Apple Silicon.

Requirements:

• mlx>=0.5.0
• websockets>=12.0
• flask>=3.0.0
• flask-socketio>=5.3.0
• numpy>=1.24.0
• requests>=2.31.0

cd experiments/03_polymarket
python -m venv venv
source venv/bin/activate
pip install mlx websockets flask flask-socketio numpy requests