Skip to content
Simulo

Train cartpole

This guide takes the simulo create starter from an empty folder to a trained policy. The starter balances a pole on a cart, using the global catalog robot simulo/robot/cartpole:v1.

Before you start, install the client and sign in with simulo login. See Getting Started.

1. Create the app

Section titled “1. Create the app”
Terminal window
simulo create cartpole_demo
cd cartpole_demo
Created app in cartpole_demo/ (eval.py + play.py + preview.py + task.py + train.py + .simuloignore).
Next steps:
  cd cartpole_demo
  simulo run preview.py
  simulo run train.py
  simulo run eval.py --policy policy_<job>:best  # evaluate the trained policy
  simulo run play.py --policy policy_<job>:best  # after training

The folder holds six files:

File What it holds
task.py The app (simulo.App("cartpole_demo")) and StarterTask: the scene, observations, actions, reward, and when an episode ends.
preview.py One preview job that checks the robot inside StarterTask and makes no policy.
train.py One training job, @app.job(type="train"), that imports the task and runs PPO with simulo.RLTrainer.
eval.py One evaluation job and one @app.success rule that judge a trained checkpoint.
play.py One play job, @app.job(type="play"), that runs a trained checkpoint back to watch or record it.
.simuloignore Files to leave out of the uploaded package, such as local checkpoints and virtual environments.

Each job file holds exactly one job. simulo run refuses a file with no job, a job with no type=, or more than one job, before anything uploads.

2. Preview, then run a quick training job

Section titled “2. Preview, then run a quick training job”
Terminal window
simulo run preview.py
simulo run train.py

The preview checks the robot inside the task before training and makes no policy. Read every finding and fix any failed check before training. The preview guide shows the full report, its exit codes, and how to inspect the recording. Then, with no training flags, the starter trains 16 environments for 2 iterations, which is enough to see the whole loop work. The training command packages the app, uploads it, prints the job type and IDs, then follows the job’s logs:

TRAIN  job_brisk-heron-4f8k2m
Policy:  policy_brisk-heron-4f8k2m

Your IDs differ. The policy ID is always the job ID with policy_ in place of job_. Every training job makes exactly one policy.

When the job finishes, the CLI lists the checkpoints the policy holds and the next commands you can run:

Checkpoints:  best (iteration <n>)  latest (iteration <n>)
Next: simulo policy get policy_brisk-heron-4f8k2m:best
      simulo run eval.py --policy policy_brisk-heron-4f8k2m:best
      simulo run train.py --max-iterations <2 x latest iteration> --from policy_brisk-heron-4f8k2m:best
      simulo export policy_brisk-heron-4f8k2m:best

3. Train for real

Section titled “3. Train for real”

The job function’s parameters become flags. The starter’s train(num_envs: int = 16, max_iterations: int = 2) accepts --num-envs and --max-iterations. Run simulo run train.py --help to list them.

Terminal window
simulo run train.py --num-envs 512 --max-iterations 600 --tag baseline --detach
simulo logs --follow
  • --tag baseline labels the job, and its policy shows the same tag. Repeat --tag for more labels. Tags use lowercase letters, digits, ., _ and -, and never change after you submit.
  • --detach returns as soon as the job is submitted. simulo logs --follow then streams the logs of your most recent job you submitted (train, play, preview or eval).

The starter declares timeout=10 * 60, so a job stops after ten minutes. Raise timeout= in train.py before a long run.

As the job trains, the log shows each checkpoint it writes:

[RLTrainer] Checkpoint written at iteration <n>/600
[RLTrainer] Best checkpoint saved at iteration <n> (reward=<value>)

You never add checkpoint code. A training job saves checkpoints on its own and publishes them to its policy when it ends. A training job that ends without saving any checkpoint is marked failed.

4. Check the result

Section titled “4. Check the result”
Terminal window
simulo jobs
simulo result
simulo policy list
  • simulo jobs lists your jobs with their type and status. Add --type train or --tag baseline to filter.
  • simulo result prints the job’s return value. For the starter it includes num_envs, iterations and best_reward.
  • simulo policy list shows the new policy and its checkpoints.

A policy normally has two checkpoints:

  • best: the checkpoint with the highest mean episode reward seen during training. It is often captured before the last iteration.
  • latest: the most recent checkpoint.

5. Play it back

Section titled “5. Play it back”
Terminal window
simulo run play.py --policy policy_brisk-heron-4f8k2m:best
PLAY  job_swift-otter-8h4q2t
Policy:  policy_brisk-heron-4f8k2m:best (iteration 600)
...
[RLPlayer] Completed: 3 episodes, <steps> steps
Next: simulo recordings job_swift-otter-8h4q2t
Terminal window
simulo recordings job_swift-otter-8h4q2t

play.py adds one overhead camera just for playing, so the recording holds the policy’s actions and that camera’s video, and training stays headless. See Play a policy and record it for what happens when the task no longer matches the checkpoint.

What the starter task does

Section titled “What the starter task does”

StarterTask in task.py has the shape every task follows:

  • Observation (4 values): pole angle, pole angular velocity, cart position, cart velocity.
  • Action (1 value): a force on the cart, scaled by action_scale.
  • Reward: +1 for each step, or -1 on the step where the pole falls or the cart leaves its track, minus small penalties for pole angle, cart speed and pole speed.
  • End of episode: the cart moves more than 3 m from center, the pole falls past 90 degrees, or 5 seconds pass.

To train a different robot, replace the catalog asset and this task logic. Bring your own robot shows how.

Next

Section titled “Next”