What Is Overfitting?


5 min read≈ 1,200 words3 questions · up to 3 points
What this is about
From 1940 to 2000, one football game before each American presidential election matched the outcome every single time. You learn why such a perfect record is weak evidence, why a famous physicist was not impressed by a theory that fit his measurements, and why huge AI models seem to break the rule. You also learn three questions to ask before you trust a rule built from your own experience.
In a nutshell· skip ahead
Overfitting happens when a model or a rule is fitted so closely to past cases that it copies their accidents along with the real pattern. It then looks excellent on the cases it was built from and predicts new cases badly. The risk is greatest when there are few cases and many ways to adjust the rule.
On 30 October 2000, Steve Hirdt of the Elias Sports Bureau, a sports statistics company, was looking for an election fact for a football broadcast. He found that Washington’s team, then called the Redskins, had matched every presidential election since 1940. When the team won its last home game before the vote, the party in the White House stayed there, and when it lost, that party was voted out. About a week later the 2000 election fitted this “Redskins Rule” too. Should a rule with such a record make you more sure about the next election?
What is overfitting?
Overfitting happens when a rule or a model is fitted so closely to past cases that it copies their accidents along with the real pattern. A model here is any rule that turns past cases into predictions, from a line on a chart to a belief such as “this kind of person makes a good friend”. An overfitted model describes the old cases very well and predicts new ones badly, and the Redskins Rule shows why in four steps.
- Every record mixes pattern and chance. Elections have real causes, such as the economy or a war, but whether a football team wins in late October is not one of them. Any match between them is an accident in the data, which statisticians call noise.
- Rules can be chosen to fit the noise. Someone had to pick the team, the game, what counts as a match and what it predicts. If the game result were a coin toss with no link to elections, the chance that one such rule matches all 16 elections from 1940 to 2000 would be 1 in 65,536. But sports records offer a huge number of such patterns, and among 65,536 unrelated ones you should expect about one perfect match by chance alone.
- Only new cases can test it. After 2000 the rule had to predict instead of describe. On 31 October 2004 Washington lost to Green Bay 28 to 14, which meant the incumbent should lose, and George W. Bush won the election two days later.
- Patching makes it worse. Hirdt changed the rule: because Al Gore had won more votes in 2000 but lost the election, the meaning of the game was reversed. The patched rule fitted every election again, but only because it had one more setting adjusted to the past. In 2012 Washington lost to Carolina, which under both versions meant the incumbent should lose, and Barack Obama won a second term.
Why does a perfect fit to the past prove so little?
The reason is that a perfect fit is easy to get when a rule has enough settings that can be changed. Mathematicians call such a setting a parameter. Through any 10 points on a chart, each at a different time, a smooth curve with 10 parameters can pass exactly. Such a perfect fit cannot tell you whether the curve has found the real trend or only followed the noise.
In the spring of 1953 the young physicist Freeman Dyson travelled to Chicago to show Enrico Fermi a theory whose numbers agreed well with Fermi’s measurements. Fermi asked how many arbitrary parameters the calculation used, and when Dyson said four, Fermi answered with a remark from the mathematician John von Neumann.
with four parameters I can fit an elephant, and with five I can make him wiggle his trunk.
Dyson’s team finished their paper and then gave up the approach, and fifty years later he wrote that Fermi had been right. In 2010 three scientists took the joke literally and drew an elephant’s outline with four complex numbers, which are really eight ordinary numbers, and a fifth one that makes its trunk move.
What overfitting teaches you
Before you trust a rule from experience, ask how many cases it is based on, how many other rules you could have found in the same cases, and whether it has ever predicted a case it was not built from. Overfitting is most likely when the cases are few and the possible rules are many.
- Small samples. One great meal at a new restaurant or one bad week with a new colleague contains a lot of chance. A firm opinion from one case fits that case and little else, and the next visit often looks more ordinary, which What Is Regression to the Mean? explains.
- Rules that fit every past case. “My team wins when I wear this shirt” or “all my best hires had studied abroad” may match every case you remember. If you could have noticed many other patterns, the match may be noise.
- Health and money. Suppose you note 20 possible migraine triggers every day, each present on about half of all days. After five migraines, the chance that at least one trigger was present before all five is close to one in two, even if none of them matters. An investment strategy tested on past prices has the same problem, because it can be tuned until its past profits look perfect.
Professionals protect themselves by keeping some data out of sight. In machine learning a model learns from a training set and is judged on a separate test set. In a 1974 paper the statistician Mervyn Stone set out cross-validation, which repeats this by holding back a different part of the data each time. The same idea is applied to everyday decisions in “Algorithms to Live By” by Brian Christian and Tom Griffiths: Summary and Review, where Christian and Griffiths argue that thinking longer about thin evidence can make a decision worse.
Is a simpler rule always better?
No, and that is the most common misreading. A rule that is too simple makes the opposite error, underfitting: it misses a real pattern and is wrong on old and new cases alike. With plenty of data a detailed rule can be the right one, so overfitting means too much adjustment compared with the evidence, not too much detail in general.
The strongest objection comes from modern AI. Some very large neural networks fit their training data exactly and still predict new cases well. In 2019 Mikhail Belkin and three colleagues called this “double descent”. As a model grows, its error on new data first falls, then rises as it starts to overfit, and then falls again once the model is large enough to fit every training example exactly. The lesson survives, though, because those models are trusted only after they are tested on data they have never seen.
Learn more
Overfitting has no single originator. The worry is as old as fitting curves to measurements, and von Neumann’s elephant shows that physicists joked about it in the early 1950s. Statisticians turned it into methods: Stone’s cross-validation in 1974 and, around the same time, Hirotugu Akaike’s measure that charges a price for every parameter in a model.
The closest relative of overfitting is the rule of preferring the explanation with fewer assumptions, explained in What Is Occam’s Razor?. To weigh a single surprising result against what you knew before, read Bayes’ Theorem, Explained.
Quiz
3 questions · up to 3 points
Open a question and pick one answer. Only your first answer counts: +1 if it is right, −1 if it is wrong.
What is overfitting?
Show answer
Fitting a rule so closely to past cases that it also copies their accidents and predicts new cases badly. An overfitted rule describes the old cases very well because it has learned their noise, and noise does not repeat in new cases.
Tom writes down 20 possible triggers every day. After his first five migraines he finds that all five came on days when he drank coffee after 4 p.m. What is the best next step?
Show answer
Treat the coffee rule as a guess and see whether it predicts his next migraines. With 20 candidate triggers and only five migraines, a perfect match can easily appear by chance. Only new cases can show whether the rule holds.
Which statement about overfitting matches the article?
Show answer
A complex model is not bad in itself, and the real test is how well it predicts cases it has not seen. Complex models work when there is enough data, and some very large networks fit their training data perfectly and still predict well. Performance on new cases decides.
Your points are saved only in this browser for now. Accounts that keep them for good are on the way.
Continue learning: "Algorithms to Live By" by Brian Christian and Tom Griffiths: Summary and Review →The takeaway
Overfitting means fitting a model or rule so closely to past cases that it also copies their accidents, so it describes the past well and predicts the future badly. The Redskins Rule matched sixteen presidential elections in a row by chance and failed in 2004 and again in 2012, even after it was patched. More complexity is not always the problem, since some very large AI models fit their data perfectly and still predict well. So judge a rule only by cases it was not built from.
References
- Overfitting. Google for Developers, Machine Learning Crash Course
A clear short lesson on overfitting, training and test data, with a chart that shows a model getting better on its training data and worse on new data.
- A meeting with Enrico Fermi. Freeman Dyson, Nature 427, 297, 2004
Dyson's one-page account of the 1953 meeting in Chicago where Fermi asked about his four arbitrary parameters and quoted von Neumann's elephant.
- Drawing an elephant with four complex parameters. Jürgen Mayer, Khaled Khairy and Jonathon Howard, American Journal of Physics 78 (6), 648, 2010
The two-page paper that really drew von Neumann's elephant, with a fifth parameter that makes the trunk wiggle.
- Reconciling modern machine-learning practice and the classical bias–variance trade-off. Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal, PNAS 116 (32), 15849–15854, 2019
The paper that described the 'double descent' curve, in which very large models fit their data exactly and still predict well.
- The Redskins Rule and the Election. Mark Jones, Boundary Stones, WETA, 2012
A short local-history article on the rule and its first failure in 2004, when Washington lost to Green Bay 28 to 14.





