When You Wear It in a Different Spot: Symmetry as an Inductive Bias

"A motivation for dropout comes from a theory of the role of sex in evolution."

— Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov, Dropout (JMLR, 2014)

I read that sentence in my second year and it would not leave me alone.

Dropout is one of the most used tricks in deep learning. Its stated motivation is sex. Genes that only function alongside one fixed set of partners are brittle. Genes that function alongside a random set are robust. So you delete half the network at random and force the units to stop leaning on each other.

There is a second version of the same story, and I like it more. A newborn has far more synapses than an adult does. The brain does not build connections as it learns. It removes them. Dense and clumsy first. Sparse and good later.

Since then I look for this on purpose. When I am stuck, I ask what in nature already has my problem. Something usually does. It has also had a few million years to work on it, which is longer than my deadline.

For CycloFormer, the answer was a jellyfish.

the problem

A wrist EMG band is a ring of 16 electrodes reading the electrical signature of forearm muscles. Wear it twice, get the same prediction. That is the whole ask. It is not what happens.

The reason is boring. People rotate the band when they put it on. The electrode over your flexor digitorum superficialis is channel 0 today and channel 7 tomorrow. Your arm has not changed. The indexing has.

I spent a while treating this as noise to be beaten down. Then I noticed what the ring actually is.

A jellyfish has no front. Neither does a sea anemone or a starfish. Radially symmetric animals have no head, no left, no first tentacle. You can spin one and nothing about it has changed, because there was never a canonical starting point to lose. Biologists have a name for this. We have a name for it too. It is a cyclic group.

Sixteen electrodes in a circle are the same object. There is no channel 0. There is only a channel we decided to call 0, which is a fact about our labelling and not about the arm.

augmentation versus construction

The emg2pose baselines handle the rotation by augmentation. Spin the channel axis randomly during training. Hope the network figures it out.

It works, in the sense that test loss goes down. It is also fragile in the way augmentation is always fragile. You are asking a network to learn something you already knew and could have handed it for free.

CycloFormer hands it over. The model is exactly ℤ₁₆-invariant: for any cyclic rotation σ_k of the channels, f(σ_k · x) = f(x). Not approximately. Not after enough epochs. By construction.

Three pieces do it, each ℤ₁₆-equivariant on its own:

1. A channel-shared TDS-CNN front end. The same convolutional weights process every channel. Identical weights commute with any channel permutation, rotation included.
2. Circular rotary position embeddings. Standard RoPE encodes absolute position. CRoPE encodes cyclic distance on the ring, so attention depends only on the offset between two channels mod 16.
3. A permutation-invariant attention pool. One learned query, softmax-weighted sum over the 16 channel tokens. Permuting the inputs to a sum does not change the sum.

Equivariant, equivariant, invariant. The composition is invariant. The proof is a paragraph. The implementation is a file.

did it help

Yes, and the shape of the help is the interesting part.

On the easy User split, each component contributes under 0.2°. Nearly nothing. On the harder Stage split, the three together contribute about +5°. The bias earns its keep exactly where the model has to extrapolate, and is almost invisible where it does not.

A parameter-matched baseline using rotation augmentation instead of built-in invariance does not close that gap. In our runs it sometimes made things slightly worse. My guess is that memorising an extra invariance costs capacity the model would rather spend on the actual task.

why here

Equivariant architectures are not new. SE(3)-Transformers for point clouds, E(2)-CNNs for images, ℤ_n-CNNs for crystallography. Most of them target modalities where the symmetry sits in the world and you can see it.

Biosignals are different, and this took me a while to say properly. The symmetry is not in the world the model observes. It is in how we chose to sample the world. The sensor array brings its own group structure to the party, and respecting it costs essentially no parameters.

That makes EMG a good test of an old claim. If hand-designed structure buys anything in the era of scale, it should buy the most where data is scarce and the symmetry is exact. EMG is both.

The jellyfish figured this out first. We just wrote it down.