What EMG Taught Me About Vision

"Trying to understand perception by studying only neurons is like trying to understand bird flight by studying only feathers: it just cannot be done."

— David Marr, Vision (1982)

Try something. Close your eyes and touch the tip of your nose.

You got there. You did not need to see your hand, and you were not guessing. Your body reports its own configuration through a sense most people never think about, called proprioception. Muscle spindles and tendon organs, wired straight to the spinal cord, running the whole time, in the dark, behind your back, under a blanket.

Evolution had the option of building one excellent sensor. It built several mediocre ones that fail at different times.

I did not appreciate that until I had spent a year on wrist EMG.

what I thought the project was

When I started CycloFormer I assumed the story was architecture. Sixteen electrodes, a cyclic symmetry, a Transformer that should not care where channel 0 landed. Tidy problem, tidy prior.

That part worked and it is still the part in the paper. It is not the part I think about.

The part I think about is that sensors fail differently, and the difference is more informative than the average.

two ways to be wrong

Vision is extraordinary when the hand is visible. It reads geometry off the image directly. Look at a clean frame and you can see why a monocular hand model does well, because you are doing the same task with the same evidence.

Then the hand does the thing we actually built all this for. It closes around a mug. Fingertips go behind the palm. Contact happens. And the evidence starts leaving the image, one finger at a time.

sEMG is close to the opposite. It never sees the hand. It reads a noisy electrical trace upstream of the motion, which sounds worse and usually is. But when a finger disappears behind an object, the muscle signal does not disappear with it. The forearm is still contracting. Nothing occluded it.

We measured this properly, which needed a paired corpus that did not exist, so we built one. Aggregate numbers: HaMeR 8.30 mm, CycloFormer 12.63 mm. Vision wins overall and it is not close.

Then we started hiding fingertips. Vision degrades steadily. EMG stays flat. Past two occluded fingertips, EMG is the more reliable of the two.

That result reorganised how I think about the question "which modality is better". It is the wrong question. The answer was never a leaderboard row. It is a failure mode, and you pick a sensor by which failures you can live with.

the symmetry stopped being decoration

I mentioned the rotation problem. People do not put the band on at the same angle twice, so identical forearm activity can arrive as a cyclic shift of the channel axis.

At first this was a nice property to exploit. Later it stopped feeling optional. If a nuisance is exact, making the model rediscover it from augmentation is just paying twice. Build it in and the model spends its capacity on the task instead of on something we already knew.

the scaling curve, briefly

The other uncomfortable result was that bigger models stopped helping. The loss curve bent back up, and I lost about a week to looking for a bug that was not there.

emg2pose is large for EMG and small for scaling laws. We were not compute-limited. We were data-limited, and the fitted law says the field needs on the order of 34× more session-hours to reach the vision target we compared against.

I do not read that as a prophecy. I read it as a scale check, and a fairly humbling one.

what I actually changed my mind about

Before this, I would have called sEMG a weaker alternative to vision. A replacement candidate, behind on the benchmark, catching up.

I do not think that framing survives the occlusion curve.

The version I believe now is the one your arm already implements. Two channels. Different failure modes. Neither sufficient. You use vision where vision is strong and you use the biosignal where the visual channel collapses, and you stay honest about what the biosignal costs, which is data, careful sensor geometry, and a model that respects how the signal was sampled.

Marr's line is about levels of analysis, not sensors. But the reflex is the same one. Study the feather long enough and you can convince yourself you understand flight. Study one modality long enough and you can convince yourself it is the whole problem.