Skip to content

Continue training

Every simulo run is a fresh run. Run the same command twice and you train twice from scratch — there is no implicit resume, ever. To keep training a policy you already have, say so explicitly:

Terminal window
simulo run app.py --max-iterations 900 --from job_swift-falcon-3nqk8n
Continuing from selected job (best.pt, best checkpoint)
Following job <new-job-id> (--detach to submit without waiting)...

That first line deliberately does not name the source job — it is a confirmation that seeding happened, not of which job it came from. As with any cloud submit, simulo run then follows the new job’s logs to completion by default (--detach to skip waiting). To confirm which job you continued from, check simulo jobsCONTINUED FROM column, or simulo result, which announces continued from job job_swift-falcon-3nqk8n (best.pt) alongside the result — that line does carry the identifier.

  • JOB_REF is a finished job’s exact friendly id (job_<adjective>-<noun>-<suffix>) — friendly ids are matched by exact equality only, never by prefix. An unambiguous prefix (4+ characters, resolved like a Git short SHA) is accepted only for the legacy UUID form some jobs from before friendly identifiers still carry — see Observe.
  • The optional suffix picks the checkpoint: :best (the default) or :latest. simulo run app.py --from job_swift-falcon-3nqk8n:latest seeds from the last policy instead of the best-scoring one.
  • The seed is pulled from platform storage, not from whatever disk the original job happened to run on — continuing from a job is deterministic regardless of which worker executes the new one.
  • --from requires being logged in. Logged out, simulo run refuses --from outright rather than silently training fresh — the checkpoint it seeds from lives in platform storage, so there must be a signed-in cloud session to resolve it against.

--max-iterations becomes a total, not an increment

Section titled “--max-iterations becomes a total, not an increment”

With --from, the iteration count you pass is the total the new job should reach — not an additional amount on top of the seed. Continuing a 600-iteration job with --max-iterations 900 trains the 300 iterations it doesn’t have yet, not another 900.

Ask for an iteration count the seed already covers, and the job refuses rather than silently completing having trained nothing:

The seed checkpoint already covers 600 iterations, which is >= the requested
max_iterations=600 — there is nothing to train. Raise --max-iterations above
600 to continue training the seeded policy, or run fresh (without --from).

--from continues training across jobs, from a finished job’s checkpoint. It’s unrelated to the resume= option on @app.job(...), which only governs what happens if this same job’s worker dies mid-run and the platform retries it — it never reaches across jobs. See Continue or resume for the full distinction, including the ResumableCheckpoint callback that makes either one possible in the first place.