Simulate and train

The design in MuJoCo

The model is generated by sim/ouro_model.py and written to sim/ouro.xml. Build it with python ouro_model.py. It compiles to 23 position coordinates, 22 degrees of freedom, 8 actuators and 349 g.

Conventions. Metres, kilograms, radians. x runs along the body from tail to head, y is across (all eight servo axes), z is up. Every segment's body frame sits on its rear joint axis. A positive joint angle curls the body upward, toward its back.

Bodies. Nine segments, s0 (tail) to s8 (head), in one nested chain. s0 carries a free joint, so the robot can fall over sideways in sim just as it can on a desk.

The electronics are modelled as boxes at the positions and sizes the mechanical design gives them (see Mechanical, where each board mounts), so they collide in sim where they would on the robot.

Geometry per segment. One box 37 × 76 × 16 mm for the link, and one cylinder of radius 9 mm and length 80 mm along y at its front joint axis (the tread cheeks sit on the front of the link that holds the servo; the tail has a second set at its rear). The cylinder is the O-ring tread and is the only thing that touches the floor. Pitch between joint axes is 48 mm. Closed, the nine links form a 9-gon with 40° at every corner and a circumradius of 70 mm.

Joints. Each of s1 to s8 has two hinges on the same axis:

• j1 to j8: the servo. Range −1.20 to +1.50 rad (−69° to +86°). With up to 4° of backlash a link can reach +90°, where a printed end stop on the real part takes over; the limits are stiff in sim to match. At +92° the electronics on the middle links would touch. Damping 0.012 N·m·s/rad, armature 1.5e-4 kg·m², dry friction 0.006 N·m.
• slop1 to slop8: an unactuated hinge with a hard range of ±2° (randomised 0.5° to 4° per joint). This is the gear backlash: the link can rattle freely inside that band around wherever the servo is holding.

Actuators. Eight position actuators on j1 to j8, kp = 1.0 N·m/rad, force limited to ±0.15 N·m (1.5 kg·cm, a derated MG90S at 5 V). The low kp is deliberate. A bare servo stalls about 2° off target; the servo plus the 1 mm printed leaf spring in each link takes about 8°. That number comes from the leaf-spring design: k = E·b·t³ / (12·l) with E ≈ 2.5 GPa, b = 66 mm, t = 1.0 mm, l = 11 mm gives 1.25 N·m/rad, and in series with the servo's own stiffness about 1.0.

The magnet latch. Two connect equality constraints join the tail tip to the head tip at y = ±30 mm, which together make a free hinge across the width. They start inactive. The environment switches them on when the two latch points come within the capture radius (7 to 12 mm), and off again when the pull-out force along the tail's axis stays above the magnet strength (1.2 to 2.0 N) for a few tens of milliseconds. The pull is low-passed over 30 ms because magnets are a short-range well and a 2 ms knock does not separate them. Only pull-out releases it, because on the real part the pin sits in a notch that carries push and side loads in plastic.

Sensors. Accelerometer and gyro at imu_head (on s8) and imu_mid (on s4), in the same axes as the real boards when mounted with X toward the head and Z out of the back.

Contacts. Neighbouring segments do not collide with each other, and neither do head and tail (they are neighbours once latched). Everything else collides. Floor friction 0.9 nominal.

Physics. 2 ms timestep, implicitfast integrator, elliptic friction cones. The policy runs every 10 steps (50 Hz).

Making the simulated servos as bad as real ones

A policy trained on perfect servos fails on cheap ones. So in OuroEnv._apply_servo and OuroEnv._set_params, every commanded angle goes through, in order:

1. Latency: a queue of 1 to 3 control steps (20 to 60 ms) for radio, USB and the servo's 20 ms PWM frame.
2. Slew limit: the internal target moves at 5 to 10 rad/s (0.10 to 0.21 s per 60°).
3. Offset: a fixed error of up to ±4° per joint, for horn-spline indexing and trim that calibration did not remove.
4. Deadband: inside 0.3° to 1.5° of target the servo stops correcting.
5. Soft, weak actuator: kp scaled 0.6 to 1.6, stall torque scaled 0.7 to 1.25.
6. Backlash: the free slop hinge, 0.5° to 4° per joint.

The simulated agent, like the real one, commands a position and never knows where the joint actually went.

The RL program

Tasks

The included policies cover roll and bite. crawl is set up and left for you: the hand-written wave already moves 19 cm in 6 s, and that is the baseline to beat.

Observation: only what the real robot can send

One frame is 22 numbers, and three frames are stacked (66 numbers):

No joint angles, no positions, no velocities from the simulator. Noise and per-episode bias are added to both IMUs and the current.

Action

Eight numbers in [−1, 1], mapped to servo targets as centre + scale × a:

Reward

roll, per 20 ms step:

4.0 × clip(forward speed, −0.4, 0.25 m/s)      # go forward, but there is no prize above 25 cm/s
− 0.30 × current in amps                       # stop the servos fighting
− 0.02 × Σ(a − a_prev)²                        # smoothness, saves gears
− 1.0 × (latch pull ÷ magnet strength)²        # sim-only signal: do not strain the bite
− 40 and episode ends if the wheel bursts open
− 3 and episode ends if it falls on its side

bite, per step:

20 × (decrease in head-to-tail gap, metres)
+ 10 once, when the magnets snap
+ 1 for every step it remains a wheel
− 0.15 × current − 0.01 × Σ(a − a_prev)²

crawl: 6 × clip(speed, ±0.3) − 0.03 × current − 0.02 × smoothness, −2 and end if it latches.

The latch-strain term uses a quantity the real robot cannot measure. That is allowed: rewards are computed in the simulator, and only observations have to exist on the robot.

Reset

Every reset draws a new robot: masses ±15 %, servo stiffness, stall torque, backlash, offsets, deadband, slew rate, latency, floor friction 0.5 to 1.1, floor tilt ±2°, magnet strength and capture radius, IMU bias. roll starts the wheel at a random rotation; bite and crawl start flat with ±3° of pose noise. The model then settles for 0.2 s.

Train it yourself

Libraries: mujoco, gymnasium, stable-baselines3 (PPO, PyTorch on CPU), numpy, tensorboard, pyserial, pillow.

python -m venv ouro-env && source ouro-env/bin/activate
pip install mujoco gymnasium "stable-baselines3>=2.3" tensorboard pyserial pillow
cd sim
python ouro_model.py            # 1. build the model
python test_gaits.py            # 2. hand-written gaits: proves the model and the latch work
python train.py roll            # 3. PPO, all cores but two (about 12 min on 10 cores)
python train.py bite --steps 900000
tensorboard --logdir runs       # 4. watch it
python evaluate.py roll 100     # 5. score over 100 new random builds
python demo.py                  # 6. the whole act, saved to ../visual/

PPO settings (in train.py): 9 parallel environments, 8190 steps per update, batch 512, 6 epochs, γ = 0.98, λ = 0.95, learning rate 3e-4, entropy 0.002, two hidden layers of 128, initial log-std −0.7. Checkpoints every 500k steps in runs/ckpt.

If you are learning RL with this project, the two pieces to write yourself are the reward function and the observation frame: empty _frame() and the reward block of step() in ouro_env.py, keep test_gaits.py passing, and train.

How to tell training is working

train.py prints one line per update:

  steps      time   dist     amps   latched   len

• roll: dist climbs from 0 toward about 2.5 m per 8 s episode, latched (fraction of episodes that end still a wheel) climbs toward 1, and amps falls toward the standing-wheel baseline. If dist rises but latched stays low, the policy has learned to throw itself forward and burst (see the lessons below).
• bite: latched climbs from about 0.1 past 0.9.
• A flat dist near zero after 500k steps means the reward or the observation is broken, not that it needs longer.

What to expect

Trained on an Apple M1 Max laptop, 10 cores, CPU only:

The full act (demo.py: crawl, bite, 6 s of rolling, startle) on 20 random builds: bitten 20, still a wheel after rolling 20, mean roll 2.22 m, opened by the startle 17. 17 of 20 complete all of it. The three misses are the hand-written startle not opening the latch from the pose the roll left it in; started from a resting wheel it opens 30 of 30. Small changes to the model (collision shapes, end stops, electronics boxes) leave these scores unchanged, so the behaviours do not hang on fine details. The stress tests below push them much further.

Four lessons built into the design

Each of these showed up while developing the robot, and each one changed the design. They are worth knowing before you change anything.

1. Stiff joints fight the loop. With stiff joints (kp = 4), random actions drew 5 A because the closed loop fought itself. The leaf spring in every link exists to fix this.
2. A policy will cheat if the reward lets it. One policy learned to throw itself forward and burst the latch after about 4 s, and a larger burst penalty helped only slowly.
3. The model was the problem, not the reward. The latch released on any 2 ms force spike when a knuckle landed. Real magnets are a short-range well, so the pull is now low-passed over 30 ms. Rolling then loads the latch with about 1 N at most.
4. That sets the magnet strength. Stronger than 1 N so rolling holds, weaker than the 2.1 N the servos can reliably pull so the startle opens it. Hence 1.2 to 2.0 N, and the pull-off test in the build.

Running the trained policy on the real robot

The policy never leaves the computer. host/run_policy.py loads sim/policies/roll.zip, and 50 times a second it reads the newest telemetry packet, builds the same 22-number frame as OuroEnv._frame, stacks three, calls policy.predict, converts the action to eight angles and then to eight pulse widths using calibration.json, and sends them through the dongle. It stops and lets the servos go limp if telemetry is older than 0.2 s or the current sits at the sensor limit for 2 s.

cd host
python calibrate.py PORT              # once per robot
python sysid.py PORT                  # once per robot
python run_policy.py PORT roll --gain 0.4 --log run1.npz

--gain scales the action range. Start at 0.4, watch, raise toward 1.0.

Why RL works with servos that do not go where they are told

The policy does not learn "set joint 3 to 40°". It learns "when the head's gravity vector looks like this, the gyro reads that and the current is this, push these servos this way". The IMUs measure what the body did, whatever the gears did. Three signals carry it:

1. Gravity direction at two points on the body. A direct measurement of shape and rotation that does not pass through any servo.
2. Gyro rate. It shows within 20 ms whether the last command produced motion, so lag and stall are visible.
3. Rail current. Position error turns into torque and torque into current, so current is a cheap reading of "the body is not where the servos want it".

Because every training episode has different slop, offsets and lag, the policy cannot rely on any one servo behaving. It ends up closing the loop through the IMUs.

Sim to real

There is no reinforcement learning on the physical robot in the base project. A wheel that bursts open 2,000 times while exploring would wear out $22 of servos.

How hard you can push it

A policy that only works on the exact robot it was trained on is no use to you, because your robot will not be that robot. python sim/stress.py makes one thing worse than anything in the training range and runs 30 new randomised builds under it. Each row below is one of those conditions. "Servo load" is the average torque as a share of stall torque while rolling.

What this means for your build:

• Cheap, sloppy parts are fine. Two-thirds-strength servos, 40 % extra weight, a slippery desk and 100 ms of lag all still roll and bite. The first thing to fail is the bite with half-strength servos (19 of 30), because it lifts both ends off the desk and needs the torque. If your robot rolls but struggles to bite, suspect weak or worn servos first.
• It is hard to knock over. It takes a sideways shove of nearly half its own weight to tip it, and even then only 8 times in 30. A 5° sideways slope is fine. Magnet strength anywhere from 1 N to 3 N makes no difference to rolling.
• The servos run warm whatever you do. Rolling holds every servo at about 45 % of stall, and the worst joint above half. A policy rewarded for going slowly and drawing less current rolled at 14 cm/s instead of 38, and its servo load was no lower, so it is not in the repo. Letting two servos go limp while it is a wheel cut standing current only from 1.0 A to 0.8 A. The load comes from holding a nine-sided shape against gravity with soft joints, and no policy makes it go away. Plan for short runs, let the servos cool between them, and treat servos as consumables. That is why the parts list has two spares.

Joint limits: does anything swing into the electronics?

python sim/limits.py. The electronics on the middle links touch at +92°. The printed end stop is at +90°. This is the largest true joint angle (servo plus backlash) seen on any of 30 random builds, per behaviour:

Every normal behaviour stays clear of the stops. The startle lands on the end stops at every joint, which is what they are for. One thing to watch: the stop limits the servo's rotation, but the next link's leaf spring can still flex a few degrees beyond it under load. If you change the electronics layout or the link shape, keep that margin in mind and rerun check_fit.py and limits.py.