Record and inspect MCAP

A training job records MCAP when you turn on anomaly capture. Simulo then watches the run, and when something goes wrong it saves a short recording of the steps just before, like a flight recorder. Training carries on. You download the recording and open it in Foxglove or Lichtblick.

A play job records MCAP too, on every run, not just when something goes wrong, with its own topics for the policy’s actions and video: see Play a policy and record it. Downloading a recording (step 3 below) works the same way for both; the topics in step 4 are specific to an anomaly capture.

1. Turn on anomaly capture

Declare simulo.callbacks.DebugOnAnomaly on your training job. In the starter’s train.py:

import simulo

from task import StarterTask, app

anomaly_capture = simulo.callbacks.DebugOnAnomaly(window_steps=64, max_captures=1)


@app.job(
    type="train",
    system=simulo.SystemType.TIER_1,
    timeout=10 * 60,
    callbacks=[anomaly_capture],
)
def train(num_envs: int = 16, max_iterations: int = 2) -> dict:
    ...  # the job body stays the same

You don’t pass anything to RLTrainer. Declaring the callback on the job is enough, and Simulo turns the monitor on when the job runs.

The monitor looks for these problems:

Kind	What it means
non_finite_reward	A reward became NaN or infinite.
observation_explosion	An observation became NaN, infinite, or larger than observation_limit (default 1e6).
non_finite_loss	A training loss became NaN or infinite.
reward_collapse	Mean episode reward dropped by at least half from the best seen so far, at two checks in a row.

window_steps is how many steps before the problem the recording keeps (default 256). max_captures is how many recordings a job may write before the monitor stops (default 1). env_indices picks which environments to record in full (default the first one).

Add video (optional)

To see what the scene looked like, add a camera to your task’s scene and name it with video="<camera name>". The job’s LearningEnv also needs enable_cameras=True. Rendering a camera slows every training step, whether or not anything goes wrong. The Cartpole Anomaly example shows a complete app with a side camera.

2. Train and see the capture

Submit as usual:

simulo run train.py --detach
simulo logs --follow

When the monitor catches a problem, simulo jobs shows it in the OUTPUTS column, for example ⚠ 1 anomaly (non_finite_reward). simulo result prints the same warning. The job still ends completed when training finishes normally.

3. Download the recording

simulo recordings

With no job ID, the command uses your most recent job you submitted (train, play, preview or eval). It downloads the recording, reads it back to check it is a valid MCAP file, and prints where it saved it:

verified anomaly_non_finite_reward_<timestamp>_<id>.mcap — <n> messages
Open in Lichtblick: /home/you/anomaly_non_finite_reward_<timestamp>_<id>.mcap

If the file can’t be read back as MCAP, the command prints a warning and keeps the file.

For a specific job or several recordings:

simulo recordings <job-id>                      # a specific job
simulo recordings <job-id> --all -o ./recordings/   # every recording, into a folder
simulo recordings <job-id> <name> -o ./capture.mcap  # one recording by name

A job with more than one recording needs a name or --all. Without them, the command lists the names. simulo outputs shows the same recordings with the kind anomaly_capture.

4. Open it

Foxglove

Choose File → Open local file… and pick the .mcap file. Add a Raw Messages panel on /debug/anomaly to read what was caught and when. If you added video, add an Image panel on /debug/window/video_foxglove to watch the steps before the problem.

Lichtblick

Same flow as Foxglove: File → Open local file…, then the same panels. Lichtblick is a community fork of Foxglove’s open-source viewer and reads the same MCAP files.

A capture holds these topics:

Topic	What it holds
/debug/anomaly	The problem that was caught, the monitor’s settings, and context.
/debug/window/step	One message per recorded step before the problem: observations, rewards and terminations for the watched environments.
/debug/window/video and /debug/window/video_foxglove	H.264 video of those steps, when you set video=.
/debug/window/camera_info	Camera calibration for the video panel, when you set video=.

Observation and reward values are stored as JSON text in fields ending in _json, so a NaN or infinity survives exactly as it was.

Read it from Python

The mcap package ships with simulo, so you can read a recording without installing anything else:

from mcap.reader import make_reader

with open("capture.mcap", "rb") as f:
    reader = make_reader(f)
    for schema, channel, message in reader.iter_messages(topics=["/debug/anomaly"]):
        print(channel.topic, message.data[:200])

Next

Use Watch a run live to watch training as it happens.