Callbacks and anomaly capture

A callback is optional behavior you attach to a job with callbacks=. It is recorded with the job, and Simulo applies it on the GPU machine. You don’t need a callback for checkpoints: a type="train" job saves them automatically.

The callback that does something today is anomaly capture.

Anomaly capture: `DebugOnAnomaly`

When training goes wrong, the useful evidence is what happened just before. simulo.callbacks.DebugOnAnomaly watches a training run and, when something looks broken, writes a short MCAP recording of the steps leading up to it. The job keeps training; the capture is a side effect.

debug_cb = simulo.callbacks.DebugOnAnomaly(window_steps=64, max_captures=1)


@app.job(type="train", system=simulo.SystemType.TIER_1, callbacks=[debug_cb])
def train(num_envs: int = 256, max_iterations: int = 12) -> dict:
    env = simulo.LearningEnv(task=MyTask(), num_envs=num_envs, device="cuda")
    trainer = simulo.RLTrainer(env=env, algorithm="PPO", device="cuda")
    ...

Declaring the callback is enough. RLTrainer picks it up on the GPU machine, so the job body doesn’t pass it anywhere.

What it watches

Signal	Checked	Meaning
`non_finite_reward`	Every checked step, and at each checkpoint save	A reward became NaN or infinite.
`observation_explosion`	Every checked step	An observation became NaN or infinite, or its magnitude exceeded `observation_limit`.
`non_finite_loss`	At each checkpoint save	A training loss became NaN or infinite.
`reward_collapse`	At each checkpoint save	The mean episode reward fell by at least `collapse_drop_fraction` of the best so far, for `collapse_patience` saves in a row.

check_observations, check_losses, and check_reward_collapse turn off their signals and are all on by default. non_finite_reward is always checked while the callback is enabled. check_every_n_steps sets how often the per-step checks run.

What a capture holds

A capture is an MCAP file with the anomaly signal and the last window_steps checked steps of observations, rewards, and terminations for the environments in env_indices (by default, environment 0). After max_captures captures (default 1), the monitor stops, so a noisy run can’t fill the disk.

video="<camera name>" adds that camera’s frames for the first watched environment, so you can see what the simulation looked like. It has two costs: the environment must be built with cameras enabled, which slows every training step, and the frames are held in memory, up to window_steps raw images.

Find a capture

When the job ends completed, its captures are uploaded with its other outputs. A job that fails, times out, or is cancelled does not upload its captures.

simulo jobs          # the OUTPUTS column shows, for example, ⚠ 1 anomaly (non_finite_reward)
simulo result        # stderr reports the anomaly output
simulo recordings    # downloads the capture
simulo outputs       # lists it with the kind anomaly_capture

A run that captures nothing still ends completed. Open a capture the same way as any recording; see Recordings & live view. The Cartpole anomaly example injects a fault on purpose so you can watch the whole path.