Callbacks

simulo.callbacks declare job-lifecycle behavior on an @app.job — periodic checkpointing, volume commits, anomaly capture — without your job body having to implement any of the plumbing itself.

@app.job(
    system=simulo.SystemType.TIER_1,
    timeout=8 * 60 * 60,
    retries=2,
    callbacks=[simulo.callbacks.ResumableCheckpoint(every=50)],
)
def train(num_envs: int = 4096, max_iterations: int = 200) -> dict:
    ...

A callback is a declaration: it’s serialized into the package manifest at submit. What happens at execution differs per callback today: ResumableCheckpoint and DebugOnAnomaly are honored by the platform runner from the declaration alone, while CommitVolume is recorded-only — its declaration rides the manifest, but the managed runner does not act on it yet (each section below says what that means in practice). Invalid configuration (e.g. a negative every) raises immediately when you construct the callback — on your machine, before packaging — never as a surprise failure on the worker.

ResumableCheckpoint

Declares periodic, resumable checkpointing for a training job:

simulo.callbacks.ResumableCheckpoint(every=50, keep_last=None)

• every — checkpoint cadence in trainer iterations.
• keep_last — additionally retain the last N numbered checkpoint copies alongside the latest one.

With this callback, a retried or preempted job resumes from its own latest checkpoint automatically — this is crash recovery for a single job’s own in-flight progress, not a way to continue a different job’s training. To continue from an earlier, completed job’s checkpoint, use simulo run ... --from <job-id> instead — see Jobs Lifecycle & the Org Workspace.

CommitVolume

Commits a volume’s contents at checkpoint and/or job-end boundaries:

simulo.callbacks.CommitVolume(my_volume, commit_on_checkpoint=True, commit_on_end=True)

Use this alongside ResumableCheckpoint when you want a volume’s state durably persisted at the same cadence your training checkpoints are.

Note

CommitVolume is recorded-only today: the declaration is serialized into the package manifest, but the platform runner does not yet act on it — volume contents are still made durable at process exit. Honoring it at checkpoint boundaries is a tracked follow-up.

DebugOnAnomaly

Declares anomaly detection and bounded MCAP debug-capture for a job — the platform’s flight-recorder equivalent of a black box:

simulo.callbacks.DebugOnAnomaly(
    window_steps=256,
    env_indices=(0,),
    video="side_cam",
    max_captures=1,
)

DebugOnAnomaly watches for several distinct failure modes rather than a single threshold:

• non-finite training loss, or reward collapse (a sudden, sustained drop), checked at trainer chunk boundaries;
• non-finite rewards, and non-finite or out-of-envelope observations, checked at every environment step (or every N steps).

On detection, it captures a bounded window of pre-anomaly evidence — observations, rewards, terminations, and (if video= names a camera) that camera’s frames for the watched environments — into a debug MCAP file, then disarms after max_captures. The job completes normally; the capture path (if any) is returned in the job’s result.

Note

Unlike CommitVolume, the @app.job(callbacks=[...]) declaration for DebugOnAnomaly is honored automatically — the platform runner constructs the anomaly monitor from the manifest entry on its own, with no job-body wiring required. Captures land in an executor-owned directory in that case (the manifest can never choose where they’re written). A job that wants a custom location instead opts in explicitly: simulo.RLTrainer(..., debug_on_anomaly=cb.to_runtime_config(output_dir=...)) — shown end-to-end in Cartpole Anomaly.

Caution

Enabling video= forces the render pipeline on for the whole training run (one camera’s render cost, independent of --num-envs) — the same trade-off as recording rollout video anywhere else in Simulo. Omit video= to capture numeric evidence only, render-free.

See Recordings & Live Viewstream for how the resulting debug MCAP relates to a regular rollout recording.