Continue or resume

Simulo has three related pieces that are easy to conflate because they all touch checkpoints. They operate at different scopes, and only one of them is something you invoke yourself from the CLI.

--from (CLI, cross-job)

simulo run app.py --max-iterations 900 --from <job-id>

Continues training in a brand-new job, seeded from a different, already-finished job’s uploaded checkpoint. This is the one you reach for explicitly, from the command line, when you want to keep training a policy you already have. Requires being logged in — a logged-out simulo run refuses it outright. Full detail: Continue training.

resume= (in-job retry)

@app.job(retries=2)   # resume defaults to "auto" — every scaffolded template relies on this
def train(...): ...

Governs one narrow thing: if this same job’s worker dies mid-run and the platform retries it (up to retries times), the retry picks up from where that job’s own progress left off instead of starting over. It never reaches across jobs — it has nothing to do with --from, and there is no CLI flag for it; it’s a keyword on the @app.job(...) decorator itself, defaulting to "auto".

ResumableCheckpoint (the callback)

from simulo.callbacks import ResumableCheckpoint

@app.job(callbacks=[ResumableCheckpoint(every=50, keep_last=None)])
def train(...): ...

Declares periodic checkpointing — every 50 trainer iterations by default in the scaffolded templates. This is the mechanism that makes both of the above possible: without periodic checkpoints there is nothing for resume= to pick up mid-job, and nothing for a later --from to seed a new job from. keep_last=N additionally keeps the last N numbered checkpoints alongside the latest one.

The one-sentence version

ResumableCheckpoint declares that checkpoints happen. resume= decides whether this job’s own retry picks one up. --from is you, explicitly, telling a brand-new job to start from a different job’s checkpoint.

Why the split matters

Before this was split out, resubmitting the same simulo run command used to silently resume a previous run’s checkpoint — including skipping training entirely if that checkpoint already covered the requested iterations. That implicit behavior is gone. Every simulo run (no --from) is now unconditionally a fresh run, and continuation is something you always ask for by name — either at the code level (resume=, scoped to one job’s own retries) or at the CLI level (--from <job-ref>, scoped to you deliberately picking a source job).

Tip

A cancelled job is still a valid --from source — cancelling a run never discards the progress it made before you cancelled it.

Next

Continue training walks through --from end to end, including the :best/:latest checkpoint suffix and what happens when you ask for fewer iterations than the seed already has.