simulo.RLTrainer

Reinforcement learning trainer using skrl.

Supports PPO and other RL algorithms through the skrl library.

Example:

task = CartpoleTask()
env = simulo.LearningEnv(task, num_envs=4096)

trainer = simulo.RLTrainer(
    env=env,
    algorithm="PPO",
    log_dir="logs/cartpole",
)
trainer.train(max_iterations=1000)
trainer.save("checkpoints/cartpole_final.pt")  # skrl checkpoint (for evaluate/load)
trainer.export_policy("checkpoints/cartpole_policy.pt")  # TorchScript for Policy.load / RLPlayer

Attributes:

algorithm — RL algorithm name (e.g., “PPO”)
agent — The skrl agent instance
runner — The skrl runner instance

Real usage, from the shipped cartpole_eval app’s train and evaluate jobs:

env = simulo.LearningEnv(task=CartpoleEvalTask(), num_envs=512, device="cuda", seed=42)
trainer = simulo.RLTrainer(env=env, algorithm="PPO", device="cuda", seed=42)

stats = trainer.train(max_iterations=20)
checkpoint = f"{checkpoints.path}/cartpole_eval_final.pt"
policy_path = f"{checkpoints.path}/cartpole_eval_policy.pt"
trainer.save(checkpoint)
trainer.export_policy(policy_path)  # TorchScript, for RLPlayer / simulo.Policy.load

# elsewhere, evaluating a saved checkpoint over several rounds:
metrics = trainer.evaluate(checkpoint=checkpoint, num_episodes=10)

trainer.close()  # close the trainer before the environment
env.close()

simulo.RLTrainer(
    env: LearningEnv,
    algorithm: str = 'PPO',
    device: str = 'cuda',
    seed: int = 42,
    log_dir: Optional[str] = None,
    ml_framework: str = 'torch',
    agent_cfg: Optional[Dict[str, Any]] = None,
    debug_on_anomaly: Optional[DebugOnAnomaly] = None,
)

train()

RLTrainer.train(
    max_iterations: int = 1000,
    *,
    checkpoint_dir: Optional[str] = None,
    checkpoint_every: Optional[int] = None,
    keep_last: Optional[int] = None,
    resume: Optional[str] = None,
) -> Dict[str, Any]

Run the training loop, optionally with periodic resumable checkpoints.

Each None keyword-only param falls back to the runner’s execution envelope (SIMULO_CHECKPOINT_DIR / _EVERY / _KEEP_LAST / SIMULO_RESUME) so a job body’s trainer.train(...) never changes; no checkpoint_dir (env absent too) -> EXACTLY the old single-shot path. Checkpointing adds resumed_from_iteration, checkpoints_written, checkpoint_dir and best_reward to the returned stats, and automatically keeps a best-so-far checkpoint (best.pt + best.json) beside latest.pt, updated whenever a chunk’s mean episode reward strictly improves.

Raises:

ValueError — if checkpoint_every, keep_last, or resume are provided without a checkpoint_dir (or SIMULO_CHECKPOINT_DIR env var), since these kwargs declare intent that cannot be honored silently.

evaluate()

RLTrainer.evaluate(checkpoint: Optional[str] = None, num_episodes: int = 10) -> Dict[str, Any]

Evaluate the trained policy.

Args:

checkpoint — Path to checkpoint (uses current model if None)
num_episodes — Number of episodes to evaluate

Returns:

Dictionary with ‘mean_reward’, ‘std_reward’, ‘episodes’

save()

RLTrainer.save(path: str) -> None

Save trainer checkpoint.

Args:

path — Path to save checkpoint

load()

RLTrainer.load(path: str) -> None

Load trainer checkpoint.

Args:

path — Path to checkpoint

export_policy()

RLTrainer.export_policy(path: str) -> str

Export the trained policy’s deterministic action head as a TorchScript .pt.

The exported module maps a batch of observations (N, obs_dim) to the mean action (N, act_dim) of the Gaussian policy — the deterministic inference action. log_std_parameter is a separate learned parameter used only for action sampling and is deliberately NOT part of the deterministic path. The file is a standalone TorchScript program loadable by simulo.core.Policy.load (.pt → JIT) and by RLPlayer(checkpoint=...) — no skrl and no trainer are required at inference time.

The mean-head export is only correct when the agent applies no observation transform and no action transform outside the network itself. Today’s _setup guarantees that (state_preprocessor=None and GaussianMixin(clip_actions=False)); this method still VERIFIES both on the live agent and refuses to export a silently-wrong policy if a future configuration enables either.

Args:

path — Output path for the TorchScript file (parent directories are created).

Returns:

The path the module was saved to.

Raises:

RuntimeError — if called before training/loading (no live agent), or after close().
NotImplementedError — if a state preprocessor or action clipping is active (the export would silently drop that transform), or if the policy model carries no exportable net action head.

close()

RLTrainer.close() -> None

Close the trainer and cleanup resources.

This should be called before closing the environment to ensure proper cleanup of skrl resources.