SO101 Manipulation and SmolVLA Experiments
Policy training and evaluation on the LeRobot SO101 platform
Before a manipulation policy can be judged, the pipeline feeding it has to be proven correct — so the first result here is that replayed actions reproduce their demonstrations.
Problem
Most reported manipulation failures are not policy failures. They are camera ordering, normalisation, action scaling or dataset construction failures wearing a policy's name.
The goal of this phase was to make that class of explanation unavailable before drawing any conclusion about the policy itself.
What I built
Hardware, training and the evaluation scaffolding around both.
- SO101 hardware setup with a two-camera configuration
- Leader and follower system bring-up
- SmolVLA training and dataset validation
- MuJoCo evaluation work
- Action replay verification through the evaluator
- Camera pipeline and image consistency verification
- State and action normalisation verification
- Policy chunking investigation
- Real-robot deployment preparation
Ruling out the wiring before blaming the policy
Recorded actions were replayed through the evaluator and reproduced successful demonstrations, with joint trajectories agreeing very closely. That single check invalidates a whole family of explanations for a future failure.
Image consistency and state/action normalisation were verified separately, so a mismatch in either can be excluded rather than argued about.
Ruling out the wiring, in order
Each step removes a class of explanation for a future failure. Running them in this order means that when a policy does underperform, the reason is not one of these.
Record demonstrations
Leader arm teleoperates the follower; two cameras record what the policy will see.
Validate the dataset
Malformed episodes — fix before training, not afterReplay the recorded actions through the evaluator
Demonstrations reproduce, joint trajectories agree closely — the pipeline is soundThey do not — the fault is in the harness, not the policyVerify the camera pipeline
Image consistency between what was recorded and what the policy is fed.
Verify state and action normalisation
So a scaling mismatch cannot masquerade as a policy failure.
Train SmolVLA, evaluate in MuJoCo
Only now does a policy number mean anything.
Real-robot deployment
In preparation, not done.
What is verified, and what is not
Stated plainly because this project is early, and an active-research page that implies more than it has is worse than one that admits where it is.
| Item | State |
|---|---|
| Hardware bring-up, leader and follower | Working |
| Two-camera setup | Working |
| Dataset validation | Verified |
| Action replay reproduces demonstrations | Verified — joint trajectories agree closely |
| Image consistency | Verified |
| State/action normalisation | Verified |
| Policy chunking behaviour | Under investigation |
| SmolVLA task performance | No claim made |
| Real-robot deployment | In preparation |
Limitations and failure modes
- No policy performance claim is made yet. The validated work is infrastructural.
- Evaluation is in MuJoCo; real-robot deployment is in preparation, not done.
- Active research — nothing here should be read as a final result.
Gallery
Not yet captured
- SO101 platform, two-camera setup
- Camera views as the policy sees them
- SmolVLA rollout
Attribution
- Based on external research
- LeRobot, the SO101 platform and SmolVLA are upstream work.
- Built by me
- Hardware bring-up, the two-camera configuration, training runs, dataset validation and the evaluation and verification tooling.
- Experiment performed by me
- Action replay verification, camera and normalisation checks, and the policy chunking investigation.
Related
- Ripple
An always-on site engineer for Nav2 robots