Teachers for hard against soft floor
Before a camera can learn from the sole, the sole has to label the steps. Three teachers label every footstep of the Booster K1 as hard or soft floor after the step: one reads only the sole, one only the joints and IMU, one both. Every step is labelled by a model that never trained on its episode.
In short
- The task is almost solved by every teacher. Sole 0.997, joints and IMU 0.990, both together 0.995, on 22,773 steps. Chance is 0.500.
- The sole is slightly better than the joints. +0.007 on all steps, real by the rule used here, but both error rates are already below 1 %.
- Reading both does not help over the sole alone.
- Standing and stopping is the only weak spot. On steps commanded below 0.05 m/s, the joints teacher drops to 0.899, the sole to 0.975.
- The consequence: the three label sets agree with truth on 99.0 to 99.7 % of steps, so a vision head trained on any of them cannot show which teacher is better on this task.
What was done
- Data. Every walking episode of G113, an ordinary flat-foot walking policy, in the step-1 data on hard floor (240), soft floor (240), and strips that start on hard floor and walk onto soft (60): 22,805 steps. 22,773 are scored; 32 transition steps have no class and are labelled but not scored.
- A step is a touchdown before any fall, with its stance window from touchdown to lift-off, 5 to 30 frames at 50 Hz. Truth is the floor class under the landing foot.
- Teachers. The small temporal convolutional network of the step-1 work, on one step's stance window. The sole teacher reads only 16 coarse sole features: four pressure blocks, four shear vectors, centre of pressure, total force and contact area. The joints teacher reads joint positions and velocities and the IMU. The third reads both.
- Cross-fitting. Five folds by episode. The fold-k model trains on training episodes outside fold k and stops early on validation episodes outside fold k; it then labels the episodes of fold k. No model ever sees a test-split episode, and those hold stiffness and friction ranges kept out of training. Three seeds.
- Settings were chosen on the validation split only: sole learning rate 0.002, dropout 0.1; joints and both 0.004 and 0.3; width 96 for all. A sole teacher trained on shuffled labels is the control.
- Chance. Soft floors are 0.462 of the scored steps. A random guess at the class prior scores 0.500, a constant majority answer 0.350.
How well each teacher labels
| Teacher | All steps | Test split | Single-floor episodes | Strips |
|---|---|---|---|---|
| Sole (coarse) | 0.997 ± 0.001 [0.996, 0.997] | 0.997 ± 0.001 [0.996, 0.999] | 0.997 ± 0.001 [0.996, 0.998] | 0.995 ± 0.001 [0.992, 0.997] |
| Joints + IMU | 0.990 ± 0.001 [0.988, 0.992] | 0.993 ± 0.001 [0.990, 0.995] | 0.990 ± 0.001 [0.989, 0.992] | 0.987 ± 0.002 [0.980, 0.992] |
| Sole + joints + IMU | 0.995 ± 0.001 [0.994, 0.996] | 0.997 ± 0.001 [0.996, 0.999] | 0.995 ± 0.001 [0.994, 0.996] | 0.991 ± 0.002 [0.987, 0.995] |
| Sole, shuffled labels | 0.377 ± 0.045 [0.357, 0.397] | 0.365 ± 0.039 [0.326, 0.400] | 0.378 ± 0.042 [0.354, 0.397] | 0.363 ± 0.072 [0.338, 0.384] |
| Scored steps | 22,773 | 5,016 | 20,301 | 2,472 |
Reading the table. Macro-F1 against truth, mean ± sd over three seeds, with a 95 % interval from resampling episodes. All three teachers are right on about 99 % of steps or more, everywhere. Accuracy is the same as F1 to three decimals (0.997, 0.990, 0.995); per class, the sole scores 0.997 on hard and 0.996 on soft. On the test split, with stiffness and friction values no model trained on (soft stiffness 8.3e3 to 1.6e4 against 1.1e4 to 1.1e5 in training; friction up to 1.30 against 1.10), all three keep 0.993 to 0.997.
The shuffled control sits at or below chance (seeds 0.429, 0.349, 0.353): it collapses toward one class, as the constant majority answer (0.350) does. Its accuracy is 0.515. The independent re-check reproduced the main scores but did not verify the stiffness and friction ranges of the test split.
By walking speed
| Teacher | Stand / stop, below 0.05 m/s | 0.05 to 0.30 | 0.30 to 0.40 | 0.40 and above |
|---|---|---|---|---|
| Sole (coarse) | 0.975 ± 0.003 | 0.996 ± 0.001 | 0.999 ± 0.001 | 0.999 ± 0.000 |
| Joints + IMU | 0.899 ± 0.011 | 0.992 ± 0.001 | 0.998 ± 0.000 | 0.998 ± 0.000 |
| Sole + joints + IMU | 0.952 ± 0.007 | 0.996 ± 0.001 | 0.998 ± 0.000 | 0.999 ± 0.001 |
| Sole, shuffled labels | 0.367 ± 0.009 | 0.390 ± 0.034 | 0.358 ± 0.044 | 0.388 ± 0.067 |
| Scored steps | 1,447 | 7,177 | 9,039 | 5,110 |
Reading the table. Speed is the commanded forward speed at touchdown. Above 0.05 m/s every teacher scores 0.992 to 0.999. When the robot stands or stops, the joints barely move and the joints teacher falls to 0.899; the sole still feels the floor directly and keeps 0.975. This is the only subset where the teachers differ, and the fixed-patch results use it to try to separate them.
Is the sole really better?
Differences in seed-mean macro-F1 with a paired 95 % interval from resampling episodes. A difference counts as real only if the interval excludes 0 and the difference is larger than twice the seed sd.
| Comparison | Steps | Difference | 95 % interval | Real? |
|---|---|---|---|---|
| Sole minus joints | all | +0.0067 | [+0.0053, +0.0083] | yes |
| Both minus sole | all | −0.0020 | [−0.0027, −0.0014] | no |
| Both minus joints | all | +0.0048 | [+0.0034, +0.0062] | yes |
| Sole minus joints | test split | +0.0045 | [+0.0022, +0.0069] | yes |
| Both minus sole | test split | −0.0003 | [−0.0014, +0.0008] | no |
| Both minus joints | test split | +0.0042 | [+0.0019, +0.0069] | yes |
| Sole minus joints | strips | +0.0080 | [+0.0027, +0.0147] | yes |
| Both minus sole | strips | −0.0032 | [−0.0062, −0.0004] | no |
| Both minus joints | strips | +0.0047 | [−0.0001, +0.0103] | no |
Reading the table. The sole beats the joints by a small margin that passes the rule everywhere: +0.007 [+0.005, +0.008] on all steps, +0.005 [+0.002, +0.007] on the test split. Both error rates are below 1 %, so for labelling the gap does not matter. Reading both sensors does not help over the sole: on all steps both minus sole is −0.002, with an interval that excludes 0 but a size within twice the seed sd, so it does not count.
A shifted sole readout
The same teacher weights, applied to a second, shifted model of how the simulated sole turns contact into readings. Two sets: 159 of these episodes replayed bit-identically for the earlier vision experiment (cross-fitted as above), and 32 separate G113 runs on hard and soft floor that no model saw.
| Teacher | Replays, usual readout | Replays, shifted | Separate runs, usual | Separate runs, shifted |
|---|---|---|---|---|
| Sole (coarse) | 0.997 ± 0.001 | 0.997 ± 0.000 | 1.000 ± 0.000 | 1.000 ± 0.000 |
| Joints + IMU | 0.989 ± 0.001 | 0.989 ± 0.001 | 0.997 ± 0.000 | 0.997 ± 0.000 |
| Sole + joints + IMU | 0.996 ± 0.001 | 0.995 ± 0.001 | 1.000 ± 0.000 | 1.000 ± 0.000 |
| Sole, shuffled labels | 0.375 ± 0.042 | 0.367 ± 0.033 | 0.356 ± 0.000 | 0.356 ± 0.001 |
| Scored steps | 6,761 | 6,761 | 1,317 | 1,317 |
Reading the table. The coarse sole teacher loses nothing under the shifted readout: 0.997 to 0.997 on the replays (+0.0002 [−0.0004, +0.0008]), 1.000 to 1.000 on the separate runs. The coarse features were chosen for this reason; step 1 showed that the fine pressure map does not survive a change of sensor model.
The independent re-check did not recompute the shifted readout; only the original check covers it.
At a floor change
On strip episodes: how many steps after the boundary each teacher needs to name the new floor (0 = on the boundary step itself). "Never" means not before the next boundary. 60 strips, one boundary each, always hard to soft; pooled over three seeds.
| Teacher | Cases | 0 | 1 | 2 | 3 | 4+ | Never | Median | 90th pct | Mean |
|---|---|---|---|---|---|---|---|---|---|---|
| Sole (coarse) | 180 | 179 | 1 | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 0.01 |
| Joints + IMU | 180 | 178 | 0 | 0 | 2 | 0 | 0 | 0.0 | 0.0 | 0.03 |
| Sole + joints + IMU | 180 | 180 | 0 | 0 | 0 | 0 | 0 | 0.0 | 0.0 | 0.00 |
| Sole, shuffled labels | 180 | 16 | 0 | 2 | 2 | 6 | 154 | 0.0 | 23.5 | 4.96 |
Reading the table. The teachers name the new floor on the boundary step in 178 to 180 of 180 cases. That is expected: each teacher reads one step's own stance window and keeps no memory, so nothing carries over from the old floor. The shuffled control is censored in 154 of 180 cases, so its median of 0 describes only the 26 cases it got right.
What it means and its limits
- An easy task. Hard floor (stiffness 7.7e8) and soft floor (1e4 to 1e5) differ by about four orders of magnitude. In the six-class experiment, by contrast, the hard grippy floor alone scored 0.76.
- Consequence for vision. The sole, joints and both-together label sets agree with truth on 99.0 to 99.7 % of steps. A vision head trained on any of them can differ from one trained on truth by at most about 1 % label noise, so the vision comparison cannot separate the teachers on this task. It did not: see the fixed-patch results.
- Window end from the simulator. The stance window ends at the noise-free contact signal of the simulator, as in the earlier experiment. A real robot has to detect lift-off itself.
- Deceptive floors. The teachers read no image, so a deceptive look cannot change their labels; their scores on deceptive episodes depend only on the physics and are on the results page.
- Independent re-check. It recomputed the main, speed and shuffled scores and the differences from the stored predictions and found them the same, and found no teacher that labelled an episode it had trained on or that had seen a test episode. The floor-change table rests on the original check only.
Scores are macro-F1 per step against simulator truth, cross-fitted, mean ± sd over three seeds. Related: dataset of experiment 2, fixed-patch results, re-check of experiment 2, teachers of the six-class experiment, step 1, second simulation run.