Evaluate your policy

Training gives you a checkpoint. Evaluation tells you how often that checkpoint completes the task you care about and shows you failures to inspect.

Open the success rule

Open eval.py in the app made by simulo create. It has two parts:

• an eval job that runs fixed starting episodes with a checkpoint;
• one @app.success rule that decides whether each completed episode succeeded.

The starter’s rule requires both checks to pass: the cartpole lasted to the time limit, and its pole finished within 10 degrees of upright. Change this rule when you change what success means for your task.

Run it against your checkpoint

Use the policy ID printed by your training job. If you used the starter’s 2-iteration training command, the policy is barely trained and will likely succeed in few or none of the 100 episodes. That is expected. Evaluation is how you find that out, and a near-zero report says Watch the worst episode; check training progress under NEXT.

To get a policy worth evaluating, run the longer training command from the Quickstart and use the new policy ID it prints:

simulo run train.py --num-envs 512 --max-iterations 600

The starter’s job stops after 10 minutes (timeout=10 * 60 in train.py); raise that before you start a long run.

Then evaluate that policy’s best checkpoint:

simulo run eval.py --policy policy_brisk-heron-4f8k2m:best

When the job completes, the report shows the result and the count for each check. Here is a sample report for a better-trained cartpole policy:

RESULT
  87 of 100 episodes succeeded. Likely range: 79% to 93%.
  Each check:  lasted to the time limit 91 of 100 · pole within 10° of upright 87 of 100
  Among 13 failures: 9 missed both checks, 4 missed only "pole within 10° of upright".

This is a sample report. Your policy’s numbers will differ. Read the success count together with the likely range, because 100 episodes estimate the true success rate rather than fixing it at one exact number. The per-check counts show which part of the rule fails most often.

Download a failure

Farther down, WENT WRONG lists representative failures and the command for each recording. One row in the sample report is:

  64        1.1 s      pole fell             58             simulo recordings job_calm-otter-9p3x7d episode-64
            lasted to the time limit: no · pole within 10° of upright: no

Run the command from the row:

simulo recordings job_calm-otter-9p3x7d episode-64

The download includes the MCAP recording and ready-made Lichtblick and Foxglove layouts. Open either layout to inspect the recorded robot, timeline, and data for that failure.

Next

• Evaluation explains the report, recordings, exit codes, and where evaluation fits beside preview and play.

• Evaluate a trained policy has the full report and shows how to adapt the rule and gate a pipeline.

• Continuing training with --from creates a new policy with new checkpoints. Evaluate the new checkpoint, then run it as the candidate and compare it with the checkpoint you continued from:

  simulo run eval.py \
    --policy policy_quiet-lynx-2b7h5c:best \
    --compare policy_brisk-heron-4f8k2m:best

  Both run on the same episodes, and the report says whether the candidate did better, did worse, or showed no clear difference.