Raptor Foundation Policy

To use the Raptor foundation policy with your own simulator, install the foundation-policy package. The policy expects observations in the FLU (Forward, Left, Up) axis convention.

Observation Vector Requirements: Construct a numpy array containing the following elements in order:

1. Position: Absolute position in meters (or relative offset to target trajectory).
2. Orientation: Flattened (row-major) $3\times3$ rotation matrix.
3. Linear Velocity: Linear velocity in m/s (or relative offset to target trajectory).
4. Angular Velocity: Angular velocity in the body frame in rad/s.
5. Previous Action: The action from the previous timestep (using the same normalization).

Action Output Requirements:

• The policy returns actions in batches by default. Use policy.evaluate_step(observation)[0] to get a single action.
• Motor conventions: [front-right, back-right, back-left, front-left].
• Motor commands are normalized in the range [-1, 1].
• To convert to RPM: rpm(action) = (max_rpm - min_rpm) * (action + 1)/2 + min_rpm.

Timing: The policy is trained for 100 Hz control. While it can handle other frequencies, 100 Hz (10 ms dt) is the nominal value for best results.

pip install foundation-policy==1.0.1

from foundation_policy import Raptor
import numpy as np

policy = Raptor()
policy.reset()

for simulation_step in range(1000):
    # Construct observation: [pos, rot, lin_vel, ang_vel, prev_action]
    observation = np.array([[*sim.position, *R(sim.orientation).flatten(), *sim.linear_velocity, *sim.angular_velocity, *sim.action]])
    
    # Policy works on batches; take the first element
    action = policy.evaluate_step(observation)[0]
    
    simulation.step(action) # simulation dt=10 ms

Once inside the Docker container, follow these steps to build and run the training pipeline. The pipeline includes sampling dynamics parameters, pre-training, and post-training.

Optimization Tips:

• Use -j$(nproc) during the CMake build to use all available CPU cores.
• For foundation_policy_pre_training, you can add -P $(nproc) to run multiple teacher training processes in parallel to save time.
• The foundation_policy_post_training step uses teacher checkpoints from foundation-policy-v1-data by default.

# Build the training targets
mkdir build
MKL_ROOT=/opt/intel/oneapi/mkl/latest cmake -S /rl-tools -B /build -DCMAKE_BUILD_TYPE=Release -DRL_TOOLS_BACKEND_ENABLE_MKL=ON -DRL_TOOLS_ENABLE_TARGETS=ON -DRL_TOOLS_EXPERIMENTAL=ON -DRL_TOOLS_ENABLE_HDF5=ON -DRL_TOOLS_ENABLE_JSON=ON -DRL_TOOLS_ENABLE_TENSORBOARD=ON
cmake --build /build --target foundation_policy_pre_training_sample_dynamics_parameters --target foundation_policy_pre_training --target foundation_policy_post_training -j$(nproc)

# Execute training
cd /rl-tools
export MKL_NUM_THREADS=1
export RL_TOOLS_EXTRACK_EXPERIMENT=$(date '+%Y-%m-%d_%H-%M-%S')

# 1. Sample dynamics parameters
/build/src/foundation_policy/foundation_policy_pre_training_sample_dynamics_parameters

# 2. Pre-train teachers (parallelized)
seq 0 999 | xargs -I {} /build/src/foundation_policy/foundation_policy_pre_training ./src/foundation_policy/dynamics_parameters/{}.json

# 3. Post-training
/build/src/foundation_policy/foundation_policy_post_training

To ensure reproducibility, you can use the provided Docker image. This involves cloning the repository, initializing submodules, and assembling the foundation policy data.

Note: The Docker image rltools/rltools:ubuntu24.04_mkl_gcc_base is available on Docker Hub and will be automatically downloaded when running docker run.

# 1. Clone and initialize submodules
git clone https://github.com/rl-tools/raptor.git
cd raptor
git submodule update --init rl-tools data
cd rl-tools
git submodule update --init --recursive -- external/highfive external/json external/tensorboard
cd ..

# 2. Assemble and extract foundation policy data
cat data/foundation-policy-v1-data.tar.gz.part_* > data/foundation-policy-v1-data.tar.gz
cd rl-tools
tar -xvf ../data/foundation-policy-v1-data.tar.gz
cd ..

# 3. Run the Docker container
docker run -it --rm -v $(pwd)/rl-tools:/rl-tools -v data:/data rltools/rltools:ubuntu24.04_mkl_gcc_base

For deployment on hardware like PX4, Betaflight, Crazyflie, or M5StampFly, use the provided adapters found in the embedded_platforms directory of the repository.

Key Configuration for High-Frequency Control: When running the policy at frequencies higher than the training frequency (100 Hz), you must configure the RL_TOOLS_INFERENCE_APPLICATIONS_L2F_CONFIG struct correctly:

• CONTROL_INTERVAL_NATIVE_NS: Must be set to 10 * 1000 * 1000 (10 ms) to match the 100 Hz training frequency.
• CONTROL_INTERVAL_INTERMEDIATE_NS: The interval at which you call the inference function (e.g., for 500 Hz control, set this to 2 ms).
• FORCE_SYNC_NATIVE: Determines how often the policy's native state is advanced. For example, if running at 400 Hz with a 100 Hz native interval, set FORCE_SYNC_NATIVE = 4.

Inference API: Use the rl_tools_inference_applications_l2f_control function to perform inference within your control loop.

// Example configuration snippet for high-frequency inference
struct RL_TOOLS_INFERENCE_APPLICATIONS_L2F_CONFIG{
    // ... other settings ...
    static constexpr TI CONTROL_INTERVAL_INTERMEDIATE_NS = 2 * 1000 * 1000; // 500Hz inference
    static constexpr TI CONTROL_INTERVAL_NATIVE_NS = 10 * 1000 * 1000;    // 100Hz native state
    static constexpr bool FORCE_SYNC_INTERMEDIATE = true;
    static constexpr TI FORCE_SYNC_NATIVE = 4; // Advance native state every 4 intermediate steps
    // ...
};

// Inference call
auto executor_status = rl_tools_inference_applications_l2f_control(current_time * 1000, &observation, &action);