Observe jobs and policies

Commands that take a job ID use your most recent job you submitted (train, play, preview or eval) when you leave the ID out, and print which job they picked. So right after simulo run, plain simulo logs --follow and simulo result act on the job you just submitted. Export jobs are skipped, so right after simulo export these commands still use your most recent submitted train, play, preview or eval job; name the export job by ID if you want it.

List your jobs

simulo jobs

Each row shows the job’s type, its ID, its name (the job function’s name, such as train), and its status: queued, running, completed, failed, cancelled, or timed_out. The POLICY column shows the policy a training job made, or the checkpoint it started from. Columns for tags, outputs, and failure reasons appear when any listed job has them.

simulo jobs --org            # jobs from everyone in your organization
simulo jobs --type train     # only training jobs
simulo jobs --tag baseline   # only jobs with this tag; repeat to require more

Follow the logs

simulo logs --follow
simulo logs JOB_ID --follow

--follow shows the last 50 lines, then streams new output until the job ends, and exits with an error code if the job failed. Use --from-start to see the whole log. Pressing Ctrl-C stops following; the job keeps running.

Simulo removes usernames, home directories, and credentials from the logs it stores, so some paths show ***REDACTED***.

Read the result

simulo result JOB_ID

This prints the JSON the job function returned. It works once the job has completed.

Find and download policies

simulo policy list
simulo policy get policy_brisk-heron-4f8k2m:best

simulo policy list shows the policies you trained, newest first, with their status, checkpoints, best reward, and tags. Add --org to include your organization’s policies and --tag NAME to filter by tag. Anyone in your organization can download or use a policy by its ID.

simulo policy get needs the checkpoint kind, best or latest. It writes policy_brisk-heron-4f8k2m-best.pt in the current directory, or the path you give with -o PATH, and checks the file’s SHA-256 digest before writing it.

A policy’s checkpoints appear when its job ends. A job that’s cancelled or reaches its time limit normally still uploads the checkpoints it had saved. A training job that ends without saving any checkpoint is marked failed with the reason no_checkpoint_saved (“no checkpoint was saved”).

If you pass a job ID by mistake, the CLI suggests the policy ID:

job_brisk-heron-4f8k2m is a job; did you mean policy_brisk-heron-4f8k2m:best?

simulo policy inspect POLICY_ID shows one policy in full: the task and robot it was trained on, the job that trained it, what it started from, its tags, and every checkpoint with its iteration and reward. Add --json for the full record. See Policies and checkpoints.

Use the Console

Simulo Console shows the same jobs and policies in your browser. Sign in with the same account you use for simulo login.

• Jobs lists your jobs with their type, tags, and status. Switch Owner between Mine and Organization, and filter by type or tag. Each job’s menu opens its logs, result, outputs, and live view, and cancels it.
• Policies lists your policies with their checkpoints, best reward, and tags, and downloads the Best or Latest checkpoint.

Watch the 3D scene live

Submit with --viewstream, then open the live view in your browser:

simulo run train.py --viewstream --detach
simulo view

Training runs roughly 4 to 5 times slower while it streams. See Watch a run live.

Recordings and other outputs

simulo outputs JOB_ID lists the checkpoints and recordings a job published, and simulo recordings JOB_ID downloads its recordings. See Retrieve job outputs.

Cancel a job

simulo cancel JOB_ID

A queued job is cancelled straight away. For a running job, the command asks the GPU machine to stop the job and waits until it has. Without a job ID, simulo cancel picks your most recent job you submitted (train, play, preview or eval) and asks you to confirm first. Cancelled is final: the job never runs again.

Pressing Ctrl-C while following logs only stops following. It does not cancel the job.