Context Windows Are Limited by Atoms, Not Bits
There is a popular narrative in tech right now: AI progress is exponential, context windows will grow to infinity, and all vertical AI products will soon be replaced by general-purpose AI that can use all the context of your entire business. This implies that the big players like Anthropic, OpenAI, and Google, with their general-purpose agents like Claude Cowork, ChatGPT, or Gemini, will subsume all software.
[... 828 words]Can humans make AI any better? Another great Welch Labs video about the bitter lesson, contextualizing the recent Dwarkesh interview with Rich Sutton.
Many people misunderstand the bitter lesson. It's not that some solutions are "past" it while others aren't. It's a ladder of increasingly general approaches to the same problem, with a persistent tradeoff between heuristics that perform well now versus general methods that win in the long term, given enough compute.