When optimizing parameters for reinforcement learning (RL), the environment is often noisy. Instead of manually averaging evaluations over multiple episodes, you should allow the optimizer to handle re-evaluations. TBPSA (and its variants like NaiveTBPSA) is a strong candidate for these noisy, parameter-tuning tasks because it is based on population-control mechanisms.
To implement this, use the ask() and tell() pattern. This allows for asynchronous execution where you can request multiple parameter sets (ask()), evaluate them (e.g., running a simulation), and then report the results back to the optimizer (tell()).
import nevergrad as ng
import numpy as np
def simulate_and_return_test_error_with_rl(x, noisy=True):
return np.linalg.norm([int(50. * abs(x_ - 0.2)) for x_ in x]) + noisy * len(x) * np.random.normal()
budget = 1200
# TBPSA is recommended for noisy RL parameter optimization
optim = ng.optimizers.registry["TBPSA"](parametrization=300, budget=budget)
for u in range(budget // 3):
# Ask for parameters
x1 = optim.ask()
x2 = optim.ask()
x3 = optim.ask()
# Evaluate (these can be parallelized)
y1 = simulate_and_return_test_error_with_rl(*x1.args)
y2 = simulate_and_return_test_error_with_rl(*x2.args)
y3 = simulate_and_return_test_error_with_rl(*x3.args)
# Tell the results back to the optimizer
optim.tell(x1, y1)
optim.tell(x2, y2)
optim.tell(x3, y3)
recommendation = optim.recommend()
print("Best parameters:", recommendation.args)
