Continue training
Every simulo run is a fresh run. Run the same command twice and you
train twice from scratch — there is no implicit resume, ever. To keep
training a policy you already have, say so explicitly:
simulo run app.py --max-iterations 900 --from job_swift-falcon-3nqk8nContinuing from selected job (best.pt, best checkpoint)Following job <new-job-id> (--detach to submit without waiting)...That first line deliberately does not name the source job — it is a
confirmation that seeding happened, not of which job it came from. As with
any cloud submit, simulo run then follows the new job’s logs to
completion by default (--detach to skip waiting). To confirm which job
you continued from, check simulo jobs’ CONTINUED FROM column, or
simulo result, which announces
continued from job job_swift-falcon-3nqk8n (best.pt) alongside the
result — that line does carry the identifier.
How --from works
Section titled “How --from works”JOB_REFis a finished job’s exact friendly id (job_<adjective>-<noun>-<suffix>) — friendly ids are matched by exact equality only, never by prefix. An unambiguous prefix (4+ characters, resolved like a Git short SHA) is accepted only for the legacy UUID form some jobs from before friendly identifiers still carry — see Observe.- The optional suffix picks the checkpoint:
:best(the default) or:latest.simulo run app.py --from job_swift-falcon-3nqk8n:latestseeds from the last policy instead of the best-scoring one. - The seed is pulled from platform storage, not from whatever disk the original job happened to run on — continuing from a job is deterministic regardless of which worker executes the new one.
--fromrequires being logged in. Logged out,simulo runrefuses--fromoutright rather than silently training fresh — the checkpoint it seeds from lives in platform storage, so there must be a signed-in cloud session to resolve it against.
--max-iterations becomes a total, not an increment
Section titled “--max-iterations becomes a total, not an increment”With --from, the iteration count you pass is the total the new job
should reach — not an additional amount on top of the seed. Continuing a
600-iteration job with --max-iterations 900 trains the 300 iterations it
doesn’t have yet, not another 900.
Ask for an iteration count the seed already covers, and the job refuses rather than silently completing having trained nothing:
The seed checkpoint already covers 600 iterations, which is >= the requestedmax_iterations=600 — there is nothing to train. Raise --max-iterations above600 to continue training the seeded policy, or run fresh (without --from).Two related but different things
Section titled “Two related but different things”--from continues training across jobs, from a finished job’s
checkpoint. It’s unrelated to the resume= option on @app.job(...), which
only governs what happens if this same job’s worker dies mid-run and the
platform retries it — it never reaches across jobs. See Continue or
resume for the full distinction, including the
ResumableCheckpoint callback that makes either one possible in the first
place.