RL4CO

Beyond the quickstarts, the examples/ directory is organized into specialized folders for different use cases:

• Modeling: Located in modeling/, these examples focus on model architecture and inference.
• Datasets: Located in datasets/, these examples demonstrate how to use custom datasets for training and evaluation.
• Advanced Topics: Located in advanced/, these include complex configurations (such as Hydra tutorials) and advanced techniques.
• Other: Located in other/, containing miscellaneous examples.

The rl4co.envs.eda module provides environments for Electronic Design Automation (EDA) tasks. These environments are structured around specific placement problems, typically requiring a generator to create problem instances and an environment to manage the reinforcement learning task.

Supported problem types include:

• Decap Placement Problem (DPP)
• Multi-port Decap Placement Problem (mDPP)

RL4CO is a unified framework for Reinforcement Learning (RL) applied to Combinatorial Optimization (CO). The library is structured into three primary components that work together to facilitate reproducible research:

1. Environments: Represent the problem as a Markov Decision Process (MDP). These are built on top of TorchRL.
2. Policies: The neural networks responsible for solving the CO problems by mapping states to actions. These are built on PyTorch.
3. RL Algorithms (Models): The training processes used to optimize the policies. These are built on PyTorch Lightning.

RL4CO modularizes the process of transforming raw features into an embedding space via parametrized functions $\phi_\omega$. You can control these components independently or let the library select them automatically by passing the env_name to the policy.

Supported embedding types include:

• Node Embeddings $\phi_n$: Transforms $m_n$ node features from the feature space to the embedding space: $[B, N, m_n] \rightarrow [B, N, h]$.
• Edge Embeddings $\phi_e$: Transforms $m_e$ edge features from the feature space to the embedding space: $[B, E, m_e] \rightarrow [B, E, h]$.
• Context Embeddings $\phi_c$: Captures contextual information by transforming $m_c$ context features from the current decoding step $s_t$ to the embedding space: $[B, m_c] \rightarrow [B, h]$.

RL algorithms in rl4co are implemented as RL4COLitModule classes, which inherit from PyTorch Lightning's pl.LightningModule. This architecture provides several built-in capabilities for training:

• Standard Lightning Lifecycle: Full support for train_step, val_step, and test_step methods.
• Automatic Logging: Integration with logging services (e.g., Wandb) via the log_metrics method.
• Optimizer Management: Automatic configuration through configure_optimizers.
• RL-Specific Callbacks: Specialized hooks like on_train_epoch_end designed for reinforcement learning workflows.

An RL algorithm acts as the bridge that takes an Environment (containing problem instances) and a Policy to optimize the policy parameters $\theta$.

The library supports two primary modes of reinforcement learning:

1. Inductive RL: The primary training phase where the model learns patterns from a training dataset to generalize to new, unseen problem instances. This amortizes the cost of inference.
2. Transductive RL (Test-time optimization): A phase where parameters are optimized specifically for target instances during the testing/inference stage.

A typical workflow involves training a policy $\pi$ using Inductive RL, followed by applying Transductive RL for fine-tuning on specific instances at test time.

Non-autoregressive policies generate all elements of a solution simultaneously or in parallel, rather than sequentially. The base classes available are:

• models.common.constructive.nonautoregressive.encoder: For encoding the input problem state.
• models.common.constructive.nonautoregressive.decoder: For generating the solution components in parallel.
• models.common.constructive.nonautoregressive.policy: The top-level policy class for non-autoregressive architectures.