Jobs Lifecycle & the Org Workspace

A Job is one execution of a package — the unit the CLI, and your organization’s workspace, are organized around. A Job is distinct from the package it runs: the package is an immutable, content-addressed source archive; a job is one execution of it, and always references its package by id.

Status

A job moves through a small set of statuses over its life:

queued → running → completed | failed | cancelled

• queued — accepted, waiting for capacity.
• running — executing on a worker.
• completed — finished successfully; its result and any models or recordings it published are ready.
• failed — execution stopped due to an error. User-facing job records and CLI output include the latest curated terminal reason code and detail.
• cancelled — stopped by request before it finished.

Working with jobs from the CLI

simulo jobs                               # recent jobs you submitted
simulo jobs --org                         # recent jobs across your organization
simulo jobs --json                        # all matching job records as JSON
simulo logs <job-id> --follow             # stream logs live; exits 0/1 with the job
simulo result <job-id>                    # the job's result JSON
simulo cancel <job-id>                    # stop a running or queued job
simulo models <job-id> [name|model-id]    # list models, or download one
simulo outputs <job-id> [name|output-id] # list outputs, or download one
simulo recordings <job-id> [name]         # list recording choices, or download

Bounded list views

Human-readable lists are kept short because tables with hundreds of rows are hard to scan and can flood your terminal. The limits apply to the terminal view, not to the records or files stored for the job.

Command	What the human-readable view shows	Default	Maximum	Retrieve everything
simulo jobs	A jobs table	20 rows	250 rows; --all shows the widest table	Use --json without --limit
simulo models <job-id>	A model table when neither a name nor --all is given	20 rows	250 rows	Use --json without --limit, or --all to download every model
simulo outputs <job-id>	An outputs table when neither a name nor --all is given	20 rows	250 rows	Use --json without --limit, or --all to download every output
simulo recordings <job-id>	Recording choices when the job has several and neither a name nor --all is given	20 rows	250 rows	Use --all to download every recording; this command has no JSON list

For all four commands, --limit N accepts values from 1 through 250. An explicit --limit also bounds JSON output, so leave it off when you need the complete JSON list. simulo jobs --all --json is also complete, even if a --limit is present.

simulo logs and simulo result accept an optional job id and default to the most recent job, announcing what they resolved to. --follow waits for a job that doesn’t exist yet, so you can arm a log tail before the job starts.

simulo jobs adds a REASON column when the page includes failed jobs. simulo logs prints the terminal reason/detail after its header, including for a fast failure that emitted no logs. A followed run exits non-zero when the job fails, so scripts do not mistake a terminal failure for success.

A job id prefix works everywhere a job id is taken, but only for the legacy UUID form. Today’s friendly ids (job_<adjective>-<noun>-<suffix>) are matched by exact equality only — a friendly id must be given in full. An unambiguous prefix of the legacy UUID form some jobs from before friendly identifiers still carry (4 or more characters, scoped to your full id — for simulo logs, simulo cancel, simulo models, and --from alike. An ambiguous prefix errors out and lists the matching candidates rather than guessing.

How long a job’s logs are kept

A job’s output is streamed live while it runs and stored when it finishes. What is stored is the only copy — there is no second copy to recover, in any environment. Production expires that stored copy on a schedule; staging and local development do not remove it automatically:

Environment	Job logs are kept for	After that
Production	90 days	the log is permanently unavailable
Staging	no limit	nothing is removed automatically
Local development	no limit	nothing is removed automatically

A job’s status, result and outputs summary are records rather than files, and are never removed by log retention — in any environment.

Its downloadable files are a different question, and the answer depends on where the job ran:

Environment	Models and recordings
Production	kept; only logs expire, at 90 days
Staging	kept; nothing is removed automatically
Local development	kept; nothing is removed automatically

“Only the log expires” is therefore a statement about production — staging and local development have no automatic expiry for logs or files at all. That is not a durability guarantee: nothing here is backed up, and staging remains subject to deliberate cleanup by the team operating it, outside this schedule.

Reading an expired log is an explicit answer, not an empty one — simulo logs says the log is no longer available and exits non-zero, so a script cannot read it as a run that produced no output. To keep a run’s output beyond the window, save it while the job is recent:

simulo logs <job-id> --from-start > run.log

Note

Two conditions are reported separately from an ordinary expiry, each with its own wording. Simulo could not save a finished job’s log at all — rare, and permanent. Or the log could not be read right now — a temporary storage problem, where retrying works and nothing has been lost.

Continuing training: --from

Cloud submits are always fresh — running the same simulo run command twice creates two independent jobs and trains twice from scratch. To continue from an earlier job’s (or model’s) checkpoint instead:

simulo run app.py --from <job-id>[:best|:latest] --max-iterations 400

• <job-id> accepts a full friendly id, or an unambiguous prefix of the legacy UUID form some jobs from before friendly identifiers still carry.
• The optional :best / :latest suffix selects which of the source job’s checkpoints to seed from (default: best).
• --max-iterations is read as the total iteration count for the new job, not an increment on top of the seed.

Outputs: models and recordings

Jobs list only the models they actually create. A training job that uses ResumableCheckpoint can produce checkpoints; a job without checkpoints can have no models to list.

Run simulo models <job-id> with no model name to list models. Add a model name or model id to download one, or pass --all to download every model.

simulo outputs <job-id> brings the job’s downloadable models and recordings into one list. Add an output name or Output ID to download one, or pass --all to download every output.

Any job may also produce MCAP rollout recordings — as many as it was configured to record. With a single recording, a bare simulo recordings <job-id> downloads it. With several, the command lists the available names; name one to download it, or pass --all to download every recording. For any single download, -o PATH chooses the output file; with --all, it chooses the output directory. See Recordings & Live Viewstream.

These outputs are all Simulo-side: a downloaded checkpoint is a PyTorch file that still needs Simulo’s runtime to mean anything. Crossing off the platform is a separate step — simulo export converts a trained policy into a portable ONNX bundle, validates the conversion against the original, and ships a verifier that reproduces Simulo’s own inference on a machine with no Simulo installed. See Export a policy to ONNX.

Who can see and read a job

Every job belongs to a project, which belongs to an organization. By default, listing jobs and reading a job’s status, logs, models, outputs, and recordings are scoped to the jobs you yourself submitted — a teammate’s job simply doesn’t appear, and looking one up directly by id looks exactly like looking up an id that was never used.

Pass --org to simulo jobs to list jobs from your organization instead of just your own — the listing is labeled with who submitted each job. It still shows 20 rows by default and follows the same 250-row table ceiling described above. This is an explicit choice you make each time; the default scope is always your own jobs.

Cancelling is the one exception: simulo cancel can always stop any job in your organization, regardless of who submitted it — every active member has equal standing to cancel a run, the same as before.

Note

A job outside your active organization looks exactly like a job that doesn’t exist — Simulo returns “not found,” never a response that would confirm the job exists under someone else’s organization. The same is true of another member’s job when you haven’t passed --org.