Evaluate a trained policy

An evaluation runs a trained checkpoint on fixed starting episodes and tells you how often it met the success rule you wrote. Use it after training to find a failure worth inspecting or to make a pipeline stop below a required rate. You need a policy from a finished training job. simulo policy list shows yours.

Words

• A policy is the trained network that chooses actions for your task.
• A checkpoint is a saved version of that policy during training.
• An episode is one attempt from its starting state until it ends.
• Success is the boolean result of your @app.success rule at that end.
• Total reward is the reward accumulated across an episode. It can help diagnose learning, but it is not the success verdict.

Judge success from the task’s own state, such as a generated command or the robot’s measured state. Never use the observation values as the success verdict.

1. Write eval.py

Put eval.py beside task.py. simulo create writes this example for you to adapt. It has one eval job and exactly one @app.success function. The job takes its checkpoint from the command line, so the same rule can judge every checkpoint you train.

"""Evaluate a Cartpole policy against the task's success rule."""

from __future__ import annotations

import math

import simulo

from task import StarterTask, app


@app.job(type="eval", system=simulo.SystemType.TIER_1, timeout=60 * 60)
def evaluate(
    policy: simulo.PolicyCheckpoint,
    episodes: int = 100,
    num_envs: int = 100,
    actions: str = "best",
):
    return simulo.evaluate(
        StarterTask(),
        policy,
        episodes=episodes,
        num_envs=num_envs,
        actions=actions,
    )


@app.success
def balanced(task):
    """Count an episode as successful when the pole stays upright to the time limit."""
    pole_angle = task.robot.state.joint_positions[:, task.robot.find_joints("cart_to_pole")[0]]
    lasted = task.check("lasted to the time limit", task.survived())
    upright = task.check("pole within 10° of upright", pole_angle.abs() < math.radians(10))
    return lasted & upright

If your task class has another name, use that name in the import and the job. A project made by simulo create already calls it StarterTask. The report’s Task line shows that class name. find_joints("cart_to_pole") looks up the joint name in the robot file. joint_positions holds one row per environment, so this rule judges the angle where each episode ended, not where it started.

Leave task unannotated. The type-checking version of simulo.Task does not expose the evaluation helpers, although the running task does. task.py must return terminated as a dictionary of named end conditions from get_dones(); a task that returns one tensor fails on its first step. See Episode end conditions and Judging success.

episodes, num_envs, and actions are ordinary job parameters. For example, --episodes 400 uses more episodes. An action is one set of controls the policy chooses. --actions sampled keeps the random variation used while training for diagnosis; the default best uses the policy’s best-guess action. When your task trains wider than the default 4 m, add env_spacing: float = 4.0 to evaluate() and pass env_spacing=env_spacing to simulo.evaluate(). The value is the distance in metres between environment origins. Then --env-spacing 7.0 evaluates a task trained 7 metres apart with the same spacing.

2. Run the evaluation

Name the policy checkpoint when you submit the file:

simulo run eval.py --policy policy_brisk-heron-4f8k2m:best

The command prints the eval job ID and follows its logs. When the job completes, it prints the report. The report starts like this:

RESULT
  87 of 100 episodes succeeded. Likely range: 79% to 93%.
  Each check:  lasted to the time limit 91 of 100 · pole within 10° of upright 87 of 100
  Among 13 failures: 9 missed both checks, 4 missed only "pole within 10° of upright".

WHAT WAS TESTED
  Policy     policy_brisk-heron-4f8k2m:best, saved at iteration 18 (training reward 71.8)
  Task       StarterTask in task.py, unchanged since the policy was trained
  Episodes   0 to 99, the same starting positions every time
  Actions    the policy's best-guess action, without training's random variation
  Success    balanced() in eval.py, with checks: lasted to the time limit, pole within 10° of upright

WENT WRONG  (13 failures; 10 recorded: the worst of each kind first, then the lowest total reward)
  Episode   Ended at   Why                   Total reward   Watch it
  64        1.1 s      pole fell             58             simulo recordings job_calm-otter-9p3x7d episode-64
            lasted to the time limit: no · pole within 10° of upright: no
  31        5.0 s      time limit            371            simulo recordings job_calm-otter-9p3x7d episode-31
            lasted to the time limit: yes · pole within 10° of upright: no
  5         1.4 s      pole fell             77             simulo recordings job_calm-otter-9p3x7d episode-5
            lasted to the time limit: no · pole within 10° of upright: no
  19        1.9 s      cart left the track   95             simulo recordings job_calm-otter-9p3x7d episode-19
            lasted to the time limit: no · pole within 10° of upright: no
  27        2.0 s      pole fell             104            simulo recordings job_calm-otter-9p3x7d episode-27
            lasted to the time limit: no · pole within 10° of upright: no
  12        2.4 s      pole fell             131            simulo recordings job_calm-otter-9p3x7d episode-12
            lasted to the time limit: no · pole within 10° of upright: no
  40        2.7 s      pole fell             150            simulo recordings job_calm-otter-9p3x7d episode-40
            lasted to the time limit: no · pole within 10° of upright: no
  53        3.1 s      cart left the track   188            simulo recordings job_calm-otter-9p3x7d episode-53
            lasted to the time limit: no · pole within 10° of upright: no
  71        3.3 s      pole fell             203            simulo recordings job_calm-otter-9p3x7d episode-71
            lasted to the time limit: no · pole within 10° of upright: no
  90        3.8 s      pole fell             240            simulo recordings job_calm-otter-9p3x7d episode-90
            lasted to the time limit: no · pole within 10° of upright: no
  88        5.0 s      time limit            402            not recorded
            lasted to the time limit: yes · pole within 10° of upright: no
  47        5.0 s      time limit            409            not recorded
            lasted to the time limit: yes · pole within 10° of upright: no
  76        5.0 s      time limit            415            not recorded
            lasted to the time limit: yes · pole within 10° of upright: no
  Every episode: simulo outputs job_calm-otter-9p3x7d episodes.csv

NEXT
  Watch the worst failure

“Among 13 failures” groups failed episodes by exactly which checks they missed. Here, 9 missed both checks, while 4 reached the time limit but missed the angle check. When more than one success-rule check was recorded and the largest group misses one check, fix that check first. When it misses several, watch the worst failure first.

One hundred episodes are a sample, so 87% is an estimate. The likely range, 79% to 93%, is where the true success rate probably falls, like a poll’s margin of error. At 100 episodes, this range is about 14 points wide, so 87% in one run and 82% in another do not establish that either checkpoint is better. --episodes 400 roughly halves the range. WHAT WAS TESTED records the checkpoint, task, episode numbers, action mode, and rule. The full machine-readable report is also the job result:

simulo result job_calm-otter-9p3x7d

Download episodes.csv when you need one row for every episode. Its columns include the episode number and seed, checkpoint, success result, ending reason and time, total reward, any simulation problem, each named check, the first observation, and recording name:

simulo outputs job_calm-otter-9p3x7d episodes.csv

3. Inspect a recorded failure

WENT WRONG lists at most 13 failed episodes and records video for at most 10. The kinds are the groups of failures that missed the same checks in RESULT’s Among ... failures line. The Why column shows how each episode ended. It records the worst failure of each kind first, then the failures with the lowest total reward.

WENT WRONG  (13 failures; 10 recorded: the worst of each kind first, then the lowest total reward)
  Episode   Ended at   Why                   Total reward   Watch it
  64        1.1 s      pole fell             58             simulo recordings job_calm-otter-9p3x7d episode-64
            lasted to the time limit: no · pole within 10° of upright: no

Download the selected recording with the command in that row:

simulo recordings job_calm-otter-9p3x7d episode-64

Sometimes recordings cannot be made, for example when the filming step runs out of memory. The evaluation still completes with its report and success rate.

When the report contains a recording problem, the CLI report’s RECORDINGS line, report.json’s recording_problem, and the Console report panel explain why, such as Recordings could not be made because the filming step ran out of memory. When only some recordings failed, it says Some recordings could not be made because the filming step ran out of memory.

Open the downloaded .mcap in Lichtblick or Foxglove. See Record and inspect MCAP. NEXT in the report points to a likely next action:

Successes in 100 episodes	NEXT points you to	What it usually means
0 to 3	Watch the worst episode and check the training log.	The policy may not have learned, or the rule may be stricter than intended.
4 to 16	Watch the worst failure.	It learned something but still fails often.
17 to 83	Run more episodes and watch the worst failure.	The likely range is too wide for a firm comparison.
84 to 99	Watch the worst failure.	The remaining failures are likely specific situations.
100	Watch the lowest-reward successes and add --require-success.	Check that the rule’s success looks right.

4. Require a minimum success rate in a pipeline

--require-success compares the observed success rate with a fraction from 0 to 1. It exits 1 below that rate, 0 when the rate meets it, and 2 when the evaluation fails, times out, is cancelled, or returns no valid report.

In a pipeline, tag the training run, then find the policy that training created:

set -e
run="ci-${GITHUB_RUN_ID}"
simulo run train.py --tag "$run"
policy="$(simulo policy list --tag "$run" --json | jq -er '.[0].policy_id')"
simulo run eval.py --policy "${policy}:best" --require-success 0.9
./publish.sh "${policy}:best"

jq -e stops the pipeline when no tagged policy exists. The evaluation’s gate line is either PASSED: 91 of 100 succeeded, at least 90% required. or FAILED: 89 of 100 succeeded, at least 90% required.. Exit 2 means there is no verdict. Check simulo logs JOB_ID, then run the evaluation again.

set -e makes publish.sh run only when the observed rate is at least 90%. A pass can still show that the likely range crosses 90%; in that case the report suggests a larger episode count. --require-success needs the command to wait for the result, so it is refused with --detach.

5. In the Console

An eval job’s page has an evaluation report panel with the result, likely range, checks, failures, NEXT advice, and report downloads. On the policy page, the Evaluations list links to those jobs and the Evaluate card copies the eval command for the checkpoint you choose.

6. Compare two checkpoints

Use --compare when you have a candidate checkpoint and a checkpoint you already trust. Simulo runs both on the same episode numbers with the same success rule. The comparison therefore answers whether the candidate did better, did worse, or has no clear difference:

simulo run eval.py \
  --policy policy_quiet-lynx-2b7h5c:best \
  --compare policy_brisk-heron-4f8k2m:best

--compare runs both checkpoints on today’s task.py, on the same episodes. If task.py changed since the older checkpoint was trained, the report’s first line says, for example, task.py has changed since policy_brisk-heron-4f8k2m was trained.

The older checkpoint’s comparison count is its score on today’s task, so it can differ from its earlier evaluation. For example, it can be 48 of 100 on its own task and 0 of 100 in a comparison after the action scale changed.

The comparison report starts with the verdict:

RESULT
  policy_quiet-lynx-2b7h5c:best did better than policy_brisk-heron-4f8k2m:best on these 100 episodes.
  quiet-lynx: 97 of 100 (likely range 91% to 99%). brisk-heron: 87 of 100 (79% to 93%).
  12 episodes changed outcome: 11 that failed with brisk-heron succeed with quiet-lynx, 1 the other way.
  A split this uneven is unlikely by chance (p = 0.0063).

When the comparison checkpoint’s task changed, RESULT also qualifies that checkpoint’s count directly: The 0 of 100 for brisk-heron was scored on the task.py used for this evaluation, which changed since this checkpoint was trained.

Only episodes whose success outcome changed decide the verdict. No clear difference means this evaluation did not establish that either checkpoint is better. It does not mean they are equal. Increase the episode count, for example with --episodes 400, when you need a firmer result. You can also compare two saved versions of one policy, such as --compare policy_brisk-heron-4f8k2m:latest.

7. Gate a regression in a pipeline

--fail-if-worse requires --compare. It exits 1 when the checkpoint after --policy did worse than the checkpoint after --compare. It exits 0 and reports that no regression was detected when the comparison is valid and every other requested gate also passes. A missing comparison report exits 2, and a simultaneous --require-success failure exits 1. A pass is weaker than proof that no regression exists, because the sample may not distinguish the two checkpoints. The flag is refused with --detach, because the CLI must wait for the evaluation result.

Save this as ci-eval.sh beside your training script:

#!/usr/bin/env bash
set -euo pipefail

simulo run eval.py \
  --policy policy_quiet-lynx-2b7h5c:best \
  --compare policy_brisk-heron-4f8k2m:best \
  --require-success 0.9 \
  --fail-if-worse

set -euo pipefail stops the script when the candidate is below the required success rate or did worse than the comparison checkpoint. Put this job before your pipeline’s publication step so either failure prevents a release.