simulo.RLTrainer

Reinforcement learning trainer using skrl.

Supports PPO and other RL algorithms through the skrl library.

Example:

task = CartpoleTask()
env = simulo.LearningEnv(task, num_envs=4096)

trainer = simulo.RLTrainer(
    env=env,
    algorithm="PPO",
    log_dir="logs/cartpole",
)
trainer.train(max_iterations=1000)
trainer.save("checkpoints/cartpole_final.pt")  # skrl checkpoint (for evaluate/load)
trainer.export_policy("checkpoints/cartpole_policy.pt")  # TorchScript for load_policy_network / RLPlayer

Attributes:

algorithm — RL algorithm name (e.g., “PPO”)
agent — The skrl agent instance
runner — The skrl runner instance

Real usage, from the shipped cartpole example’s train.py. The job builds an environment and a trainer, trains, and returns the statistics. A train job saves its best and latest checkpoints automatically, so there is no save call:

env = simulo.LearningEnv(
    task=CartpoleTask(),
    num_envs=num_envs,
    device="cuda",
    dt=1.0 / 120.0,
    physics_steps_per_action=2,
    env_spacing=4.0,
    headless=True,
    seed=42,
)
trainer = simulo.RLTrainer(env=env, algorithm="PPO", device="cuda", seed=42)
stats = trainer.train(max_iterations=max_iterations)
trainer.close()  # close the trainer before the environment
env.close()
return {"num_envs": num_envs, **stats}

simulo.RLTrainer(
    env: LearningEnv,
    algorithm: str = 'PPO',
    device: str = 'cuda',
    seed: int = 42,
    log_dir: Optional[str] = None,
    ml_framework: str = 'torch',
    agent_cfg: Optional[Dict[str, Any]] = None,
    debug_on_anomaly: Optional[DebugOnAnomaly] = None,
)

train()

RLTrainer.train(
    max_iterations: int = 1000,
    *,
    checkpoint_dir: Optional[str] = None,
    checkpoint_every: Optional[int] = None,
    keep_last: Optional[int] = None,
) -> Dict[str, Any]

Run the training loop.

Train jobs automatically save a latest checkpoint and retain the best checkpoint by mean episode reward. To continue training, submit a new job from a checkpoint, for example with simulo run train.py --from policy_x:best.

Raises:

ValueError — if checkpoint_every or keep_last are provided without a checkpoint_dir (or SIMULO_CHECKPOINT_DIR env var), since these kwargs declare intent that cannot be honored silently.

save()

RLTrainer.save(path: str) -> None

Save trainer checkpoint.

Args:

path — Path to save checkpoint

load()

RLTrainer.load(path: str) -> None

Load trainer checkpoint.

Args:

path — Path to checkpoint

export_policy()

RLTrainer.export_policy(path: str) -> str

Export the trained policy’s deterministic action head as a TorchScript .pt.

The exported module maps a batch of observations (N, obs_dim) to the mean action (N, act_dim) of the Gaussian policy — the deterministic inference action. log_std_parameter is a separate learned parameter used only for action sampling and is deliberately NOT part of the deterministic path. The file is a standalone TorchScript program loadable by simulo.core.load_policy_network (.pt → JIT) and by RLPlayer(policy=...) — no skrl and no trainer are required at inference time.

The mean-head export is only correct when the agent applies no observation transform and no action transform outside the network itself. Today’s _setup guarantees that (state_preprocessor=None and GaussianMixin(clip_actions=False)); this method still VERIFIES both on the live agent and refuses to export a silently-wrong policy if a future configuration enables either.

Args:

path — Output path for the TorchScript file (parent directories are created).

Returns:

The path the module was saved to.

Raises:

RuntimeError — if called before training/loading (no live agent), or after close().
NotImplementedError — if a state preprocessor or action clipping is active (the export would silently drop that transform), or if the policy model carries no exportable net action head.

close()

RLTrainer.close() -> None

Close the trainer and cleanup resources.

This should be called before closing the environment to ensure proper cleanup of skrl resources.