simulo.callbacks

Callbacks are passed to @app.job(callbacks=[...]). Each one declares intent — its configuration is recorded into the package manifest at submit time. What happens at execution differs per callback today: the platform honors ResumableCheckpoint (periodic, resumable checkpointing is wired into the runner), while CommitVolume and DebugOnAnomaly are recorded-only — their declarations ride the manifest but the managed runner does not act on them yet. DebugOnAnomaly captures today when the job body wires it into the trainer itself via to_runtime_config().

Job lifecycle callbacks (simulo.callbacks).

Each callback is a structural JobCallbackProtocol implementation. In the thin client the lifecycle hooks are intentionally no-ops — a callback DECLARES intent and serialises its configuration into the package manifest via manifest_entry(); the platform honors it at execution. Today the backend runner honors ResumableCheckpoint (it exports the SIMULO_CHECKPOINT_* / SIMULO_RESUME execution envelope that RLTrainer.train picks up for periodic, resumable checkpointing) and DebugOnAnomaly (it exports the SIMULO_ANOMALY_DEBUG / SIMULO_ANOMALY_DEBUG_DIR execution envelope that RLTrainer reads when constructed without an explicit debug_on_anomaly); CommitVolume is still recorded-only, resolved by the cloud worker in a follow-up.

DebugOnAnomaly here shares one field vocabulary with the runtime config simulo.interfaces.runtime.DebugOnAnomaly that RLTrainer consumes; DebugOnAnomaly.to_runtime_config() and DebugOnAnomaly.from_manifest_config() are the explicit declaration → execution mapping. from_manifest_config delegates to the contract’s own simulo.interfaces.runtime.DebugOnAnomaly.from_config() classmethod rather than re-implementing the config → dataclass mapping. Torch-free by construction: this module imports only the (torch-free) simulo-interfaces contract, never simulo-backend.

simulo.callbacks.ResumableCheckpoint

Declare periodic, resumable checkpointing for a job.

every is the checkpoint cadence in trainer iterations (the platform picks a default when None); keep_last additionally keeps the last N numbered iter_<n>.pt copies beside latest.pt.

Raises:

• ValueError — if every or keep_last are provided and < 1 (caught at authoring time, before packaging, so the error appears on the author’s machine rather than on the remote worker).

Real usage, from the shipped cartpole training app — periodic checkpointing declared on the job itself:

@app.job(
    system=simulo.SystemType.TIER_1,
    timeout=8 * 60 * 60,
    retries=2,
    callbacks=[simulo.callbacks.ResumableCheckpoint(every=50)],
)
def train_cartpole(num_envs: int = 4096, max_iterations: int = 200) -> dict[str, Any]:
    ...

Retries are then safe: a retried or preempted run resumes from the latest checkpoint instead of restarting.

simulo.callbacks.ResumableCheckpoint(
    *,
    every: Optional[int] = None,
    keep_last: Optional[int] = None,
)

simulo.callbacks.CommitVolume

Commit a volume’s contents at checkpoint / job-end boundaries.

Attach to a job whose body writes into volume to make those writes durable at well-defined points instead of only at process exit.

Args:

• volume — The simulo.Volume to commit.
• commit_on_checkpoint — Also commit every time the job checkpoints.
• commit_on_end — Commit when the job ends (success or failure).

Execution resolution: recorded-only today — the declaration rides the package manifest; the cloud worker honors it in a follow-up.

simulo.callbacks.CommitVolume(
    volume: Volume,
    *,
    commit_on_checkpoint: bool = True,
    commit_on_end: bool = True,
)

simulo.callbacks.DebugOnAnomaly

Declare anomaly detection + MCAP debug-session capture for a job.

Mirrors the runtime config simulo.interfaces.runtime.DebugOnAnomaly field-for-field — same names, same defaults, same bounds validation (the runtime config IS the validator, so an invalid declaration raises ValueError on the author’s machine, before packaging). See that class for the full field/signal semantics; in brief, each anomaly kind maps to a distinct failure mode rather than a single threshold:

• non-finite loss (check_losses) and reward collapse (check_reward_collapse with collapse_drop_fraction / collapse_patience / collapse_baseline_floor) at trainer chunk boundaries;
• non-finite rewards, and non-finite or out-of-envelope (observation_limit) observations, at env steps (check_observations, every check_every_n_steps).

On detection, a bounded MCAP debug session (window_steps of pre-anomaly evidence for env_indices, at most max_captures files) is captured.

video="<camera name>" additionally buffers that camera’s frames for the first watched env and writes them into the same debug MCAP (/debug/window/video + a foxglove.CompressedVideo sibling) so the capture shows what the sim looked like before the anomaly. Two honest costs, documented on the runtime config’s docstring in full: the camera forces the render pipeline on during training (the env must enable cameras), and the frame ring buffer holds up to window_steps raw RGB frames in host memory (window_steps x H x W x 3 bytes).

The one deliberate field difference from the runtime config: output_dir is NOT declarable. Where debug outputs land is decided by the execution environment (exactly like checkpoint directories) and supplied at resolution time via to_runtime_config() / from_manifest_config().

Execution resolution: honored by the backend runner. When the selected job declares this callback, the runner exports the SIMULO_ANOMALY_DEBUG / SIMULO_ANOMALY_DEBUG_DIR execution envelope before the job runs, and an RLTrainer constructed without an explicit debug_on_anomaly argument arms its anomaly monitor from it — declaring the callback is enough; no hand-wiring in the job body. Precedence, strongest first: an explicit RLTrainer(debug_on_anomaly=...) argument (including an explicit enabled=False) > an operator --env override of the envelope > a package runtime_env-supplied SIMULO_ANOMALY_DEBUG (the config key is user-tunable; SIMULO_ANOMALY_DEBUG_DIR is runner-reserved, so a package can never choose where captures land) > this declaration > nothing. The mapping helpers remain the explicit declaration → execution mapping for custom executors: RLTrainer(..., debug_on_anomaly=cb.to_runtime_config()), or from a parsed manifest entry, DebugOnAnomaly.from_manifest_config(entry["config"], output_dir=...).

Raises:

• ValueError — on out-of-bounds configuration (caught at authoring time, before packaging, so the error appears on the author’s machine rather than on the remote worker).

Real usage, from the shipped cartpole_anomaly app — one callback object, declared on the job and honored automatically by the platform runner (the job body never wires it into RLTrainer by hand):

debug_cb = simulo.callbacks.DebugOnAnomaly(
    window_steps=64, check_every_n_steps=1, max_captures=1, video="side_cam"
)


@app.job(system=simulo.SystemType.TIER_1, timeout=1 * 60 * 60, callbacks=[debug_cb])
def train_with_fault(num_envs: int = 256, max_iterations: int = 12, fault_step: int = 185) -> dict[str, Any]:
    ...

A job that wants a custom capture location instead passes RLTrainer(..., debug_on_anomaly=debug_cb.to_runtime_config(output_dir=...)) explicitly.

simulo.callbacks.DebugOnAnomaly(
    *,
    enabled: bool = True,
    window_steps: int = 256,
    env_indices: Sequence[int] = (0,),
    check_observations: bool = True,
    observation_limit: Optional[float] = 1000000.0,
    check_every_n_steps: int = 1,
    video: Optional[str] = None,
    check_losses: bool = True,
    check_reward_collapse: bool = True,
    collapse_drop_fraction: float = 0.5,
    collapse_patience: int = 2,
    collapse_baseline_floor: float = 1e-06,
    max_captures: int = 1,
)

to_runtime_config()

DebugOnAnomaly.to_runtime_config(*, output_dir: Optional[str] = None) -> _RuntimeDebugOnAnomaly

The simulo.interfaces.runtime.DebugOnAnomaly this declaration maps onto.

output_dir is the executor-supplied output directory; None keeps the runtime config’s default (runs/debug).

from_manifest_config()

DebugOnAnomaly.from_manifest_config(
    config: Mapping[str, Any],
    *,
    output_dir: Optional[str] = None,
) -> _RuntimeDebugOnAnomaly

classmethod

Construct the runtime config from a manifest config payload (executor side).

Delegates to the contract’s own simulo.interfaces.runtime.DebugOnAnomaly.from_config() (the canonical declaration → execution mapping, added alongside it). Reads exactly the keys _manifest_config() writes; unknown keys are ignored (forward compatibility with newer clients) and value validation is the runtime config’s own __post_init__. A manifest-supplied output_dir is ignored too — the output directory is executor-owned and passed only via the output_dir keyword.