Evaluation

An evaluation runs a trained checkpoint through fixed starting episodes. At the end of each episode, the success rule in eval.py decides whether the policy succeeded. The result is a success count and a likely range for the policy’s true success rate, not one score to trust on its own.

Run the starter’s evaluation after training:

simulo run eval.py --policy policy_brisk-heron-4f8k2m:best

Training reward tells the trainer which behavior to improve. The best checkpoint is the one that had the highest mean episode reward during training. Your success rule asks a different question: did the policy complete the task you care about?

A high training reward does not tell you how often that task succeeds. A policy can earn more reward while still failing your success rule, so evaluate a checkpoint before deciding whether it is ready to use.

The success rule is yours

An eval file has one @app.job(type="eval") job and exactly one @app.success rule. The rule runs on the final step of each episode and returns one boolean per environment.

Judge success from the task’s own state, such as a generated command or the robot’s measured state. Never use the observation values as the success verdict.

Use these helpers to say what success means:

Helper	What it tells the rule
task.survived()	The episode reached its real time limit without a named end condition.
task.ended_by(name)	The named condition from get_dones() ended the episode.
task.check(name, value)	Records each environment’s true or false value for the report and returns value unchanged.

For example, the starter counts an episode only when the cartpole lasted to the time limit and its pole finished within 10 degrees of upright. Give each check a name where “yes” means good. Judging success explains the helpers and their rules.

Read the report

Unless task.py changed since training, the report starts with RESULT: how many episodes succeeded, a likely range, the count for each named check, and groups of failures that missed the same checks. If the task changed, a notice comes first. WHAT WAS TESTED follows RESULT and records the checkpoint, task, episodes, action mode, and success rule. The range matters because the episodes are a sample. A result such as 87 of 100 estimates the policy’s success rate; it does not prove that the rate is exactly 87 percent.

WENT WRONG lists representative failed episodes, how each one ended, its total reward, and every named check as yes or no. A recorded episode shows a command to download its recording; a row without one says not recorded. The worst failure of each kind comes first. NEXT suggests what to inspect or run next.

Every completed evaluation also provides two report files:

• report.json holds the machine-readable report.
• episodes.csv has one row for every checkpoint and episode, including its seed, outcome, ending reason, reward, checks, and recording name. A regular evaluation has one row per episode; a comparison has two.

Watch recorded failures

An evaluation records video for a limited set of representative failures. If every episode succeeds, it records up to three of the lowest-reward successes instead. Download a failed episode with the command shown in its WENT WRONG row:

simulo recordings job_calm-otter-9p3x7d episode-64

The download contains an MCAP recording and ready-made layouts for Lichtblick and Foxglove. The layouts open with the recorded robot, timeline, and data panels arranged for that episode. If filming fails, the evaluation still keeps its report and success result, and the report explains the recording problem.

Compare checkpoints on the same episodes

Use --compare to judge a candidate against a checkpoint you already trust:

simulo run eval.py \
  --policy policy_quiet-lynx-2b7h5c:best \
  --compare policy_brisk-heron-4f8k2m:best

Both checkpoints run on the same episode numbers, starting states, and success rule. The verdict says the candidate did better, did worse, or showed no clear difference. No clear difference means this sample did not establish a winner; it does not mean the checkpoints are equal.

The comparison uses the task.py in the app you submit today. If that file changed since either checkpoint was trained, the report says so. An older checkpoint’s count can therefore differ from an earlier evaluation that used an older task.

Gate a pipeline

Two flags turn the report into a pipeline gate:

• --require-success 0.9 exits 1 when fewer than 90 percent of the episodes succeeded.
• --fail-if-worse requires --compare and exits 1 when the candidate did worse than the comparison checkpoint.

The exit codes are:

Exit code	Meaning
0	The evaluation produced a valid report and every requested gate passed.
1	A requested success-rate or comparison gate failed.
2	The CLI refused the gate flags before submitting, the job failed, timed out, or was cancelled, or the job returned no valid evaluation report or comparison report.

The CLI refuses --fail-if-worse without --compare. Both gates need the CLI to wait for the result, so they are also refused with --detach.

What it helps you troubleshoot

A low success rate tells you something is wrong. The report, the per-episode episodes.csv and the recordings help you find what:

• Reward: failed episodes that still earn high total reward mean the reward pays for something other than success.
• Success rule: a recording that looks like a success while the report counts a failure means the rule judges the wrong thing.
• Observations: an observation column that never changes across episodes, or that disagrees with the robot’s real state, points to observation code. A moderate success rate does not rule this out.
• Actions: a policy that sees the right state but barely moves the robot points to action scaling or wiring.

Troubleshoot a policy with evaluation walks through each case on a real task, from the report to the fix. To find scene problems before you train, use Troubleshoot a scene with preview.

Where evaluation fits

Preview the task, train a policy, then use this loop:

1. Evaluate a saved checkpoint.
2. If the result is weak, continue training from it with --from.
3. Evaluate the new checkpoint.
4. Use --compare to judge the new checkpoint against the one you continued from.
5. If the new result is still weak, continue training from the new checkpoint and repeat. Once the result meets your requirement, play or export it.

• Preview checks the robot and task before training. It uses settle, joint, random-action, and training-loop checks, and makes no policy.
• Evaluation measures a trained checkpoint over many fixed episodes with the success rule you wrote.
• Play runs a checkpoint so you can watch or record it. It does not produce an evaluation success rate.

In the Simulo Console, an eval job’s page shows the result, likely range, checks, failures, NEXT advice, and report downloads. A policy page has an Evaluations list linking to its eval jobs and an Evaluate card that copies the command for a selected checkpoint.

See Evaluate a trained policy for the complete eval.py, a full report, failure analysis, comparison output, and pipeline examples.