Tasks & the RL Loop

A simulo.Task defines a learning problem: observations, rewards, termination conditions, action application, and reset logic. A Task is independent of the training algorithm — the same Task can be trained, evaluated, and played back through different learning workflows. See Scene, Robot & World for what simulo.Robot, simulo.Terrain, and simulo.World mean and how per_environment works.

Authoring a Task

from __future__ import annotations

import simulo

app = simulo.App("balance")
cartpole = simulo.Asset.from_registry("simulo/robot/cartpole:v1")

with app.runtime.imports():
    import torch  # only resolved on the GPU runtime — see Runtime, Images & Dependencies


class BalanceTask(simulo.Task):
    observation_dim = 4
    action_dim = 1

    def build(self, scene: simulo.Scene) -> None:
        scene.add(simulo.Terrain.plane(name="ground"), at="/", per_environment=False)
        self.robot = simulo.Robot(
            asset=cartpole,
            initial_pose=simulo.Pose.identity(),
        )
        scene.add(self.robot, at="/World/Robot")

    def on_start(self, env: simulo.LearningEnv) -> None:
        self._cart_idx = self.robot.find_joints("slider_to_cart")

    def get_observations(self) -> "torch.Tensor":
        ...

    def get_rewards(self) -> "torch.Tensor":
        ...

    def get_dones(self) -> tuple:
        ...

    def apply_actions(self, actions: "torch.Tensor") -> None:
        self.robot.set_joint_effort_target(actions, joint_ids=self._cart_idx)

    def reset_idx(self, env_ids: "torch.Tensor") -> None:
        self.robot.reset(env_ids)

A few things make this authoring style work on a machine with no GPU and no torch installed:

from __future__ import annotations turns every annotation into a string — never evaluated at import — so -> "torch.Tensor" doesn’t require torch to be importable on your machine.
class BalanceTask(simulo.Task) at module level works because on your machine simulo.Task is a lean, torch-free base class; the real GPU-runtime training base class with the same name lives in the cloud runtime. The class is definable and type-checked either way — only the cloud worker resolves it to the heavy implementation.
simulo.Scene, simulo.Robot, simulo.Asset.from_registry(...), simulo.Pose are all part of the same lazily-resolved surface: safe to use inside build() because build() runs on the cloud worker, never on your machine.
The one module-level heavy import — torch — is written inside with app.runtime.imports():, deferred on your machine and real on the worker. See Runtimes.

The Task lifecycle

Method	Called	Purpose
`build(scene)`	Once, at environment construction	Declare the scene: terrain, lights, the robot, and any props or sensors.
`on_start(env)`	Once, after the runtime is live	Resolve joint indices and any other state that depends on the built scene.
`get_observations()`	Every step	Return the policy’s observation tensor.
`get_rewards()`	Every step	Return the per-environment reward tensor.
`get_dones()`	Every step	Return `(terminated, truncated)` tensors.
`apply_actions(actions)`	Every step	Apply the policy’s action to the robot.
`reset_idx(env_ids)`	On reset	Reset the given environment indices to their initial state.

Vectorized training: `LearningEnv`, `RLTrainer`, `RLPlayer`

simulo.LearningEnv wraps a Task into a vectorized interface suitable for large-scale learning: it manages parallel environments, episode tracking, physics stepping, and device placement.

env = simulo.LearningEnv(
    task=BalanceTask(),
    num_envs=4096,
    device="cuda",
    dt=1.0 / 120.0,
    physics_steps_per_action=2,
    env_spacing=4.0,
    headless=True,
    seed=42,
)

simulo.RLTrainer orchestrates the actual learning workflow:

trainer = simulo.RLTrainer(env=env, algorithm="PPO", device="cuda", seed=42)
stats = trainer.train(max_iterations=200)

trainer.save(f"{vol.path}/checkpoint.pt")        # resumable training checkpoint
trainer.export_policy(f"{vol.path}/policy.pt")   # standalone TorchScript policy

.train(max_iterations=...) runs the training loop and returns summary stats.
.save(path) writes a trainer-specific checkpoint you can later reload with .evaluate(checkpoint=...).
.export_policy(path) exports the trained policy’s deterministic mean-action head as a standalone TorchScript file — the file you hand to RLPlayer for trainer-free inference.

simulo.RLPlayer runs a trained policy for evaluation, visualization, or recording — no trainer, no training algorithm state:

player = simulo.RLPlayer(env=env, checkpoint=f"{vol.path}/policy.pt", device="cuda")
stats = player.play(num_steps=200)

RLPlayer.play() also accepts record=simulo.RecordConfig(...) to capture the rollout to an MCAP flight recording — see Recordings & Live Viewstream.

For a scripted simulation with no task, reward, or trainer at all, see Scenarios.