Scaling Laws Aren't Scaling Laws When You're Out of Data
A plant does not grow faster because you gave it more of the nutrient it already had enough of.
"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."
— Richard Sutton, The Bitter Lesson (2019)
I believe this. I also spent three months watching it fail in front of me.
Here is what I expected. Grow the model, loss goes down. That is the reflex you pick up from every language and vision scaling plot you have ever seen. Kaplan and Chinchilla write it down cleanly: L ∝ N^(-α), smooth, decreasing in N.
We tried to fit that on emg2pose, the largest public surface-EMG hand-pose dataset. 370 session-hours. 193 people. It is a lot of EMG.
The curve refused.
Validation loss went down as the model grew, bottomed out somewhere between 4M and 17M parameters, and then climbed back up. A U. Not a slope.
My first assumption was a bug in my training loop. I looked for it for about a week. There wasn't one.
the nutrient
The thing in nature that already had my problem turned out to be a field of wheat.
In 1828 Carl Sprengel wrote down what everyone now calls Liebig's law of the minimum, because Liebig popularised it and history is unfair about credit. The law says growth is set by the scarcest input, not by the total. Double the nitrogen on a field that is short of phosphorus and you get nothing back. The plant was never nitrogen-limited.
The picture people use is a barrel built from staves of unequal height. Water fills to the shortest stave. The tall ones are decoration.
Parameters were my tall stave. Data was my short one.
Standard scaling laws describe the resolution-limited regime, where the model is the bottleneck and more of it genuinely helps. We were in the other regime. Around the interpolation threshold the model was just large enough to fit the training set exactly and had nothing left over for generalising. This is the classical side of double descent, characterised in linear, kernel and random-feature settings by Hastie, Mei–Montanari and Bahri et al. At N ≈ D, validation loss spikes. The spike flattens as D grows.
Chinchilla cannot represent that curve. Not because it was fit badly. Because it is monotone in N by construction, and our curve is not.
the patch
We wanted one closed form that fits every data fraction. Chinchilla when data is plentiful, a U when it isn't. So we added a term:
L(N, D) = ε∞ + a · N^(-α) + b · D^(-β) + c · (N/D)^γ
The first three are Hoffmann et al. exactly: irreducible loss, finite-N bias, finite-D bias. The fourth is the variance term. It dominates when N is large relative to D, produces the upturn, and disappears in the data-rich limit. γ sets how hard overfitting bites.
Fit by robust regression on a 5×8 grid from 1.49M to 85M parameters and 20% to 100% of the data, it explains 98.8% of the variance, with 0.13 mm RMS residual. A cell we held out entirely (85M, full data) came in within 0.21 mm. Setting c = 0 and refitting gives R² = 0.906 and misses every U.
The exponents were the part I did not expect to care about. β = 0.42 on data. α = 0.06 on parameters. Data is worth roughly an order of magnitude more here. Doubling the corpus shrinks its term by about 25%. Doubling the model shrinks its term by about 4%.
That is Sprengel again, in a different notation.
what it buys
Once the law is fit, you can ask it for a budget instead of a vibe.
To reach the 75th-percentile error of HaMeR on monocular vision, 9.15 mm, the fit projects roughly 34× more session-hours than emg2pose currently holds, at a compute-optimal model near 2.2B parameters. That is a number someone can push back on. "More data would be good" is not.
I do not think this form is permanent. It is a description of a field that is currently short on data. As the datasets grow, the variance term shrinks and Chinchilla starts fitting again, and this whole post ages into a footnote.
The lesson I would keep is smaller than the equation. Sutton is right that general methods win. He is not saying they win at every dataset size you happen to have. If you are on the wrong side of the interpolation threshold, the honest move is to put that in the loss function rather than to keep buying parameters and hoping.