Troubleshoot a policy with evaluation

Start with the first place where the report and the behavior disagree. A high reward with failed episodes points to the reward. An observation that does not match the starting state points to observation code. Large actions with little movement point to action scaling. A recording that looks successful while the report says it failed points to the success rule.

This guide assumes you already ran an evaluation as described in Evaluate a trained policy.

1. Read the report to find the cause

Read RESULT, WENT WRONG, and NEXT in that order. Then use episodes.csv or a recording to check the report’s strongest lead.

Start with RESULT

The first line gives the number of successful episodes and a likely range for the true success rate. The next line gives a count for every named check in your success rule:

Each check:  lasted to the time limit 91 of 100 · pole within 10 degrees of upright 87 of 100

The failure-groups line counts the exact combinations of checks that failed:

Among 13 failures: 9 missed both checks, 4 missed only "pole within 10 degrees of upright".

Use this line to separate broad failures from one suspicious check. If most failed episodes miss several checks, inspect the worst episode. If most miss one check, inspect that check and a recording together. A success-rule bug can look like a policy bug until you watch the recording.

WHAT WAS TESTED records the checkpoint, today’s task.py, the episode numbers, the action mode, and the success rule. Check this section before you compare a new result with an older one.

Use every NEXT line

The report tests the conditions below in order and prints the first three that match. It can then add the last-checkpoint comparison offer as a fourth line. The episode number, counts, check name, file name, policy name, and commands change to match your evaluation.

What the report says	When it appears	What to do
Watch the first flagged episode (episode 12); check the task's reward and observation code	At least one episode has an invalid reward or observation. 12 is the first affected episode in this example.	Inspect that episode first. Check the reward and observation values at the first bad step.
Watch the worst episode; check training progress	The top of the likely range is below 10%.	Check the training log to see whether the policy learned anything, then watch the lowest-reward recorded episode.
Failures earn as much reward as successes: either the reward pays for something other than success, or the success rule judges something else; watch a failure with a high total reward	There are successes and failures, and the median failed episode earns at least as much total reward as the median successful episode.	Compare the reward breakdown with the success checks. One of them values the wrong outcome.
The largest group of failures (73 of 73) missed only "pole within 10 degrees of upright": fix that one	The rule has more than one check, and the largest failure group misses exactly one. The counts and check name come from your report.	Inspect that check beside a recording. Confirm that it reads the state it names.
Run more episodes	The likely range is wider than 15 percentage points, or it crosses the rate supplied to --require-success.	Rerun with --episodes 400. The printed command keeps your policy and comparison checkpoint.
Watch the worst failure	At least one episode failed, the policy is not in the under-10% case, and recording did not fail.	Download the first recorded failure in WENT WRONG and inspect the behavior that led to its ending.
Watch the successes with the lowest total reward; add --require-success to your pipeline	Every episode succeeded.	Check that low-reward successes really satisfy your intended outcome, then gate later runs.
Try the last checkpoint	You evaluated a :best checkpoint without --compare, and the same policy has a :latest checkpoint.	Run the printed command to compare :best and :latest on the same episodes.

NEXT is a lead, not a diagnosis. For example, a very low success rate can make the training-progress line appear before the report has enough successful episodes to compare their rewards with failed episodes.

Read WENT WRONG

Each row gives you five facts:

Column	What it tells you
Episode	The numbered starting state. The same number starts from the same state on another evaluation.
Ended at	How long the episode lasted. A cluster of short episodes usually points to control, observation, or termination behavior.
Why	The named end condition, time limit, or cut short by the task.
Total reward	What the reward function paid over the episode. Compare this with success rather than treating it as the verdict.
Watch it	The exact simulo recordings command, or not recorded.

The line under each row shows yes or no for every success check. The list puts a low-reward example of each failure kind first, followed by more failures with low total reward.

Check every episode in episodes.csv

Download the table when the report’s shortlist is not enough:

simulo outputs JOB_ID episodes.csv

The useful columns are:

• episode and seed, which identify the repeatable starting state.
• checkpoint, which separates rows when the evaluation used --compare.
• success, the final result of the success rule.
• ended_by and ended_at_s, which show why and when the episode stopped.
• total_reward, which lets you compare what the reward paid with what the success rule accepted.
• sim_problem, which names an invalid reward or observation when one occurred.
• One check:<name> column for every named success check.
• first_observation, a JSON list containing the policy’s input at the start of the episode.
• recording, the recording name when that episode was recorded.

Without --compare, the file has one row per episode. With --compare, it has one row per checkpoint per episode. Each episode number appears twice, and the checkpoint column tells you which run produced the row.

Look for a value that stays fixed when the starting states differ, a check that disagrees with visible behavior, or failed episodes that earn as much as successful ones.

Inspect the recording

Run the command from WENT WRONG, then open the downloaded MCAP file as described in Record and inspect MCAP. These topics connect the policy’s input and output to what happened:

Topic	What to inspect
/policy/observation	The values the policy received at each step.
/policy/action	The values the policy chose at each step.
/robot/state	Joint positions, joint velocities, and the robot pose that resulted.
/reward	The total reward for each step and its named breakdown when available.
/episode/end	The ending reason, success result, and named check results.

Compare topics at the same step. An action can be large while the joint barely moves. An observation can say the pole is centered while the robot state shows it leaning. A reward component can grow as the visible behavior gets worse.

Confirm the fix

Retrain when you change the reward, observations, or action scaling. Then compare the corrected policy with the original on the same episode numbers:

simulo run eval.py \
  --policy policy_corrected:best \
  --compare policy_original:best

--compare scores both checkpoints on today’s task.py. The older checkpoint’s count can therefore differ from its own earlier evaluation. A comparison does not reproduce the older checkpoint’s original task. The report makes this visible with a notice such as:

task.py has changed since policy_original was trained

After the comparison, run the corrected checkpoint with a required rate:

simulo run eval.py --policy policy_corrected:best --require-success 0.90

--require-success exits 1 when the observed success rate is below the requested fraction and 0 when it meets the fraction. It exits 2 when the job fails, times out, is cancelled, or returns no valid report.

2. Fix four common Cartpole failures

Each story below starts with one wrong line. The measured numbers and output come from real 100-episode evaluations and are trimmed with .... Use the same sequence on your task: read the report, inspect the most specific evidence, change one line, and evaluate again.

The stories use a Cartpole task with the class CartpoleTask, the reward scale rew_scale_pole_pos, and the check pole within 10 degrees of upright; a project made by simulo create uses StarterTask and labels the equivalent check pole within 10° of upright, so match the idea rather than copying these names.

In /robot/state, joint_positions and joint_velocities are unnamed lists in the robot’s joint order. For this Cartpole, index 0 is slider_to_cart (cart position, metres) and index 1 is cart_to_pole (pole angle, radians). For your own robot, get the order from Robot.get_joint_names().

Push is 1,000 times too weak

What you see. The report shows that every episode fails, misses both checks, and ends when the pole falls. It shows that and how the policy fails. It does not tell you why.

EVAL  job_tough-marsh-87ez70
Policy:  policy_durable-shrub-ve774w:best (iteration 200)
...
[eval] 0 of 100 episodes succeeded. Likely range: 0% to 4%.

RESULT
  0 of 100 episodes succeeded. Likely range: 0% to 4%.
  Each check:  lasted to the time limit 0 of 100 · pole within 10 degrees of upright 0 of 100
  Among 100 failures: 100 missed both checks.
...
WENT WRONG  (100 failures; 10 recorded: the worst of each kind first, then the lowest total reward)
  Episode   Ended at   Why         Total reward   Watch it
  35        0.4 s      pole fell   -8.7           simulo recordings job_tough-marsh-87ez70 episode-35
            lasted to the time limit: no · pole within 10 degrees of upright: no
...

Download episodes.csv. Its first observation changes with each episode’s starting tilt, so the policy does see the pole:

episode,seed,checkpoint,success,ended_by,ended_at_s,total_reward,sim_problem,check:lasted to the time limit,check:pole within 10 degrees of upright,first_observation,recording
0,353308032,policy_durable-shrub-ve774w:best,false,pole fell,0.85,24.716178,,false,false,"[0.10393571853637695,0.0,0.0,0.0]",
...
35,2773284534,policy_durable-shrub-ve774w:best,false,pole fell,0.366667,-8.707303,,false,false,"[0.7812597751617432,0.0,0.0,0.0]",episode-35
...
76,4186014423,policy_durable-shrub-ve774w:best,false,pole fell,0.366667,-8.653161,,false,false,"[-0.7793431878089905,0.0,0.0,0.0]",episode-76
...
99,2953433426,policy_durable-shrub-ve774w:best,false,pole fell,0.55,3.727753,,false,false,"[0.37417981028556824,0.0,0.0,0.0]",

What you inspect. Open episode-35.mcap in Lichtblick or Foxglove and compare /policy/action with joint_positions[0] (the cart position) in /robot/state. What you will see is an action range from -6.71813 to -1.87917, while the cart moves from 0.000359488 m to a maximum of 0.02859 m. The policy pushes, but the cart moves only about 3 cm.

Fix the line. 0.1 supplies one-thousandth of the intended force:

action_scale = 0.1  # [N]
action_scale = 100.0  # [N]

Retrain after changing the action scale. The corrected policy succeeds in 100 of 100 episodes:

EVAL  job_new-phrase-3fnwgj
Policy:  policy_funky-amplitude-9mrfh3:best (iteration 200), compared with policy_durable-shrub-ve774w:best (iteration 200)
...
task.py has changed since policy_durable-shrub-ve774w was trained

RESULT
  policy_funky-amplitude-9mrfh3:best did better than policy_durable-shrub-ve774w:best on these 100 episodes.
  funky-amplitude: 100 of 100 (likely range 96% to 100%). durable-shrub: 0 of 100 (0% to 4%).
  100 episodes changed outcome: 100 that failed with durable-shrub succeed with funky-amplitude, 0 the other way.
  A split this uneven is unlikely by chance (p < 0.001).
...

The comparison’s old-checkpoint score belongs to today’s corrected task.py. Use the first report above for the old checkpoint’s result on the task it was trained with.

Observation misses the pole

What you see. The report shows that every episode fails, misses both checks, and ends when the pole falls. It shows that and how the policy fails. It does not tell you why. In episodes.csv, the first_observation field records the policy’s input at the start of the episode, and it is the same even though the starting pole tilt changes from episode to episode:

EVAL  job_miniature-proposal-db7dwt
Policy:  policy_primordial-pear-59sa8p:best (iteration 150)
...
RESULT
  0 of 100 episodes succeeded. Likely range: 0% to 4%.
  Each check:  lasted to the time limit 0 of 100 · pole within 10 degrees of upright 0 of 100
  Among 100 failures: 100 missed both checks.
...
WENT WRONG  (100 failures; 10 recorded: the worst of each kind first, then the lowest total reward)
  Episode   Ended at   Why         Total reward   Watch it
  76        0.4 s      pole fell   -8.1           simulo recordings job_miniature-proposal-db7dwt episode-76
            lasted to the time limit: no · pole within 10 degrees of upright: no
...

episode,seed,checkpoint,success,ended_by,ended_at_s,total_reward,sim_problem,check:lasted to the time limit,check:pole within 10 degrees of upright,first_observation,recording
0,353308032,policy_primordial-pear-59sa8p:best,false,pole fell,0.933333,29.350584,,false,false,"[0.0,0.0,0.0,0.0]",
...
35,2773284534,policy_primordial-pear-59sa8p:best,false,pole fell,0.366667,-6.907871,,false,false,"[0.0,0.0,0.0,0.0]",episode-35
...
76,4186014423,policy_primordial-pear-59sa8p:best,false,pole fell,0.366667,-8.139185,,false,false,"[0.0,0.0,0.0,0.0]",episode-76
...
99,2953433426,policy_primordial-pear-59sa8p:best,false,pole fell,0.583333,5.704028,,false,false,"[0.0,0.0,0.0,0.0]",

What you inspect. Open episode-76.mcap and compare /policy/observation with /robot/state. At the first recorded step, the observation is [-0.00431489, -0.335723, -0.00431489, -0.335723]. At the same step, joint_positions[0] (the cart position) is -0.00431489 m and joint_positions[1] (the pole angle) is -0.786295 rad. The policy receives the cart’s position and speed in both pairs. The input therefore disagrees with the visible pole angle.

A moderate success rate does not rule out an observation bug. A policy can succeed from some starting states while still missing a state value it needs.

Fix the line. Put the pole’s position and speed in the first two values returned by get_observations():

return torch.cat((cart_pos, cart_vel, cart_pos, cart_vel), dim=-1)
return torch.cat((pole_pos, pole_vel, cart_pos, cart_vel), dim=-1)

Retrain after changing the observation. The corrected policy succeeds in 100 of 100 episodes:

EVAL  job_fortunate-glazing-8jbwgz
Policy:  policy_bipedal-palette-jvd0kt:best (iteration 150), compared with policy_primordial-pear-59sa8p:best (iteration 150)
...
RESULT
  policy_bipedal-palette-jvd0kt:best did better than policy_primordial-pear-59sa8p:best on these 100 episodes.
  bipedal-palette: 100 of 100 (likely range 96% to 100%). primordial-pear: 0 of 100 (0% to 4%).
  100 episodes changed outcome: 100 that failed with primordial-pear succeed with bipedal-palette, 0 the other way.
  A split this uneven is unlikely by chance (p < 0.001).
...

Reward pays for leaning

What you see. Failed episodes last under about a second and earn several hundred total reward. The first recorded failure earns 392.2 in 0.4 s. WHAT WAS TESTED shows a training reward of 503.3, while a policy that balances for the full five seconds earns roughly one point per step: a five-second episode is about 300 steps, so about 300 in total.

EVAL  job_unknown-combustion-z4m7z0
Policy:  policy_favorite-ambient-1dndhp:best (iteration 200)
...
[eval] 0 of 100 episodes succeeded. Likely range: 0% to 4%.

RESULT
  0 of 100 episodes succeeded. Likely range: 0% to 4%.
  Each check:  lasted to the time limit 0 of 100 · pole within 10 degrees of upright 0 of 100
  Among 100 failures: 100 missed both checks.
...
WHAT WAS TESTED
  Policy     policy_favorite-ambient-1dndhp:best, saved at iteration 200 (training reward 503.3)
  Task       CartpoleTask in task.py, unchanged since the policy was trained
  Episodes   0 to 99, the same starting positions every time
  Actions    the policy's best-guess action, without training's random variation
  Success    balanced() in eval.py, with checks: lasted to the time limit, pole within 10 degrees of upright
...
WENT WRONG  (100 failures; 10 recorded: the worst of each kind first, then the lowest total reward)
  Episode   Ended at   Why         Total reward   Watch it
  35        0.4 s      pole fell   392.2          simulo recordings job_unknown-combustion-z4m7z0 episode-35
            lasted to the time limit: no · pole within 10 degrees of upright: no
  68        0.4 s      pole fell   403.7          simulo recordings job_unknown-combustion-z4m7z0 episode-68
            lasted to the time limit: no · pole within 10 degrees of upright: no
...

What you inspect. Open episode-35.mcap and compare /reward with joint_positions[1] (the pole angle) in /robot/state. What you will see is the pole angle moving from 0.81643 rad to 1.58074 rad while the per-step total reward rises from 7.62845 to 22.8225. The reward grows as the pole leans farther from upright.

Fix the line. A positive squared-angle scale rewards a larger pole angle. Make it a penalty:

rew_scale_pole_pos = 10.0
rew_scale_pole_pos = -1.0

Retrain after changing the reward. The corrected policy succeeds in 100 of 100 episodes:

EVAL  job_beveled-cockatiel-40zber
Policy:  policy_full-pier-2gdgwn:best (iteration 200), compared with policy_favorite-ambient-1dndhp:best (iteration 200)
...
RESULT
  policy_full-pier-2gdgwn:best did better than policy_favorite-ambient-1dndhp:best on these 100 episodes.
  full-pier: 100 of 100 (likely range 96% to 100%). favorite-ambient: 0 of 100 (0% to 4%).
  100 episodes changed outcome: 100 that failed with favorite-ambient succeed with full-pier, 0 the other way.
  A split this uneven is unlikely by chance (p < 0.001).
...

Success rule checks the cart

What you see. Every failed episode lasts the full five seconds and misses only the upright check. The report points to that one check:

EVAL  job_topaz-attache-1z4s0v
Policy:  policy_arcane-scan-234195:best (iteration 200)
...
[eval] 0 of 100 episodes succeeded. Likely range: 0% to 4%.

RESULT
  0 of 100 episodes succeeded. Likely range: 0% to 4%.
  Each check:  lasted to the time limit 100 of 100 · pole within 10 degrees of upright 0 of 100
  Among 100 failures: 100 missed only "pole within 10 degrees of upright".
...
WENT WRONG  (100 failures; 10 recorded: the worst of each kind first, then the lowest total reward)
  Episode   Ended at   Why          Total reward   Watch it
  35        5.0 s      time limit   286.6          simulo recordings job_topaz-attache-1z4s0v episode-35
            lasted to the time limit: yes · pole within 10 degrees of upright: no
...
NEXT
  Watch the worst episode; check training progress:   simulo logs job_arcane-scan-234195
  The largest group of failures (100 of 100) missed only "pole within 10 degrees of upright": fix that one
  Try the last checkpoint:                            simulo run eval.py --policy policy_arcane-scan-234195:best --compare policy_arcane-scan-234195:latest

What you inspect. Open episode-35.mcap and compare /robot/state with /episode/end. What you will see at the end is joint_positions[1] (the pole angle) at 0.00780324 rad, about 0.45 degrees from upright, while joint_positions[0] (the cart position) is 1.68463 m. The episode-end check says the pole is not within 10 degrees. The visible pole and the check disagree because the rule reads slider_to_cart.

A success-rule bug looks like a policy bug until you watch a recording.

Fix the line. Make the upright check read the pole joint:

:, task.robot.find_joints("slider_to_cart")[0]
:, task.robot.find_joints("cart_to_pole")[0]

This change only corrects how the evaluation judges the episode. Do not retrain or compare against a second policy. Evaluate the same checkpoint again:

EVAL  job_dichotomic-number-zmby0m
Policy:  policy_arcane-scan-234195:best (iteration 200)
...
[eval] 100 of 100 episodes succeeded. Likely range: 96% to 100%.

RESULT
  100 of 100 episodes succeeded. Likely range: 96% to 100%.
  Each check:  lasted to the time limit 100 of 100 · pole within 10 degrees of upright 100 of 100
...

3. Match a symptom to the next check

What you see	Likely cause	Where to look
Very low success, short episodes, and higher reward than a full successful episode	The reward pays for behavior the success rule rejects.	Compare total_reward in episodes.csv with /reward and /robot/state in a failed recording.
first_observation is identical across episodes that start with different pole tilts	Observation code repeats the cart values where the pole values belong.	Compare first_observation and /policy/observation with the pole angle in the first /robot/state.
The policy produces actions, but the cart moves only a few centimetres before the pole falls	The action scale is too weak or the action is applied to the wrong joint.	Compare /policy/action with cart movement in /robot/state, then inspect the action scale and target joint.
Every failure misses one check while recordings look successful	The success check reads the wrong state.	Compare the check:<name> column and /episode/end with the final /robot/state.
Failed and successful episodes earn the same reward	The reward and success rule judge different outcomes.	Compare reward medians in episodes.csv, then inspect a high-reward failure’s /reward breakdown and checks.
The old checkpoint’s comparison count differs from its own earlier evaluation	--compare scored it on today’s task.py.	Read the task-change notice and WHAT WAS TESTED. Keep the earlier standalone result as the score on the earlier task.