Job lifecycle

A submitted job waits in queued until a GPU machine picks it up, runs in running, and then ends in exactly one terminal status. A terminal status never changes, and neither do the job’s logs, result, and checkpoints once it has one.

Status	Meaning
queued	Accepted and waiting for a GPU machine. Waiting here usually means a machine is starting for the job.
running	A GPU machine is running the job.
completed	The job body returned. Its result and outputs are available.
failed	The job did not finish successfully. The reason says why.
cancelled	Stopped because someone ran simulo cancel.
timed_out	Stopped because it ran past its time limit.

A running job can go back to queued if the GPU machine running it stops responding. The job then runs again, a limited number of times, and after that it ends failed with the reason lease_expired_max_attempts. This is separate from retries= on @app.job, which reruns the job body on the same machine after it fails and picks up from the job’s own latest checkpoint when one exists.

Why a job ended the way it did

For a failed, cancelled, or timed_out job, simulo jobs adds a REASON column with a short code. simulo jobs --json includes the code as status_reason and a sentence as status_detail, and simulo logs has the full story. The reasons you are most likely to see:

Reason	Status	What happened
job_failed	failed	Your job’s code raised an error or exited with a non-zero status.
no_checkpoint_saved	failed	A training job finished without saving a checkpoint (“no checkpoint was saved”).
job_refused	failed	The job could not proceed, for example a --from checkpoint that already covers the requested iterations.
memory_limit_exceeded	failed	The job used more memory than its GPU machine allows.
timeout	failed	The GPU machine’s own one-hour limit on a single run stopped the job.
org_runtime_limit_exceeded	timed_out	The job reached its time limit and was stopped.
cancelled	cancelled	Someone ran simulo cancel on a running job.
cancelled_by_user	cancelled	Someone ran simulo cancel on a queued job, or on a running job whose GPU machine had stopped responding.
lease_expired_max_attempts	failed	The GPU machine stopped responding on every attempt.

A training job must save a checkpoint

A train job exists to produce a policy, so Simulo checks what it actually produced. If a training job’s code returns normally but no checkpoint was saved, the job is stored as failed with the reason no_checkpoint_saved, never as completed with nothing to show. In practice this means the job body never ran RLTrainer.train(), which is what saves checkpoints. It also happens if the checkpoints could not be uploaded. See Policies and checkpoints.

This check applies only to a job that would otherwise be completed. A cancelled or timed-out training job keeps its status even if it saved nothing.

Timeouts

Every job has a time limit. It is the lower of the job’s own timeout= and your organization’s runtime limit, which is one hour unless your organization has a different one. The clock starts when a GPU machine picks the job up, so time spent queued and the machine starting up does not count. If the job has to run again on another machine, it keeps its original deadline.

@app.job(type="train", system=simulo.SystemType.TIER_1, timeout=5 * 60)
def train(num_envs: int = 16, max_iterations: int = 100_000) -> dict: ...

When the limit is reached, Simulo stops the job and it ends timed_out, so a hung job can never sit running forever. Checkpoints the job saved before that are normally kept in its policy. Separately, each GPU machine stops any single run after one hour unless the platform operator has raised that ceiling; a job stopped that way ends failed with the reason timeout. timeout= must be a positive whole number of seconds; anything else is refused when the file is imported.

Cancelling a job

simulo cancel JOB_ID
simulo cancel            # your most recent job you submitted (train, play, preview or eval), after asking you to confirm

A queued job is cancelled at once. For a running job, the request reaches the GPU machine on its next check-in. The machine stops the job, saves any checkpoints it had written to its policy, and the job ends cancelled. simulo cancel waits for that confirmation; press Ctrl-C to stop waiting without withdrawing the cancel. If the job finishes on its own while it is being stopped, it keeps the outcome it actually reached.

Recordings from a cancelled job are not kept. Its checkpoints are, and you can continue from them with --from.

Policies and checkpoints

A training job creates one policy. Its best and latest checkpoints are published before the job is marked terminal, so a finished job never points to a checkpoint that is still uploading. After that, the policy is sealed: no checkpoint can be added or replaced.

simulo policy list
simulo policy get policy_brisk-heron-4f8k2m:best
simulo run train.py --max-iterations <N> --from policy_brisk-heron-4f8k2m:best
simulo export policy_brisk-heron-4f8k2m:best

Checkpoint references name a policy. Job IDs name everything else: logs, results, recordings, outputs, the live view, and cancelling. Policies and checkpoints explains how best and latest are chosen.

Watch a job

simulo jobs                      # your recent jobs: type, status, policy
simulo logs JOB_ID --follow      # stream the log until the job ends
simulo result JOB_ID             # the dict a completed job returned
simulo recordings JOB_ID         # download the job's MCAP recordings
simulo outputs JOB_ID            # list checkpoints and recordings together

Leave out JOB_ID and each command uses your most recent job you submitted (train, play, preview or eval) and names it first, so you can type commands one after another without copying IDs.

simulo jobs is a table with one row per job. Its first column is the job type, and its POLICY column says which policy a training job made (made policy_...) or which checkpoint a job used (uses policy_...:best). It shows 20 rows by default; --limit N and --all show more, and --json prints the records instead.

Your jobs and your organization’s

simulo jobs lists the jobs you submitted. Add --org to list every job in your organization. The Console shows the same choice as Owner: Mine / Organization on the Jobs and Policies pages.

Anyone in your organization can read a teammate’s job by its ID with simulo logs, simulo result, simulo recordings, and simulo outputs. A job in another organization looks exactly like a job that does not exist.