Generative AI models are hitting a technological wall, with improvements from scaling up large language models slowing down, according to reports and experts. Gary Marcus, a cognitive scientist and AI skeptic, warns that once the industry acknowledges these shortcomings, a financial crash could follow.
"The economics are likely to be grim," Marcus wrote on his Substack. "Sky high valuation of companies like OpenAI and Microsoft are largely based on the notion that LLMs will, with continued scaling, become artificial general intelligence." He added, "As I have always warned, that's just a fantasy."
The warning follows a report from The Information that OpenAI researchers found its upcoming flagship model, code-named Orion, showed noticeably less improvement over GPT-4 than GPT-4 did over GPT-3. In areas like coding, there may be no improvements at all. Ilya Sutskever, founder of Safe Superintelligence and former chief science officer at OpenAI, told Reuters that improvements from scaling up AI models have plateaued.
Training runs for large models can cost tens of millions of dollars, require hundreds of AI chips, and take months to complete, according to Reuters. Tech companies have also run out of freely available data to train their models, having practically scraped the entire surface web.
"LLMs such as they are, will become a commodity; price wars will keep revenue low. Given the cost of chips, profits will be elusive," Marcus predicted. "When everyone realizes this, the financial bubble may burst quickly."
OpenAI researchers are developing ways to surmount the scaling problem, such as training models to "think" or "reason" in a similar way to humans, capabilities previewed in its o1 model. One technique, called "test-time compute," has an AI model explore multiple possibilities for complex problems and then choose the most promising one, instead of jumping to a conclusion.
Whether such work will lead to significant AI improvements remains to be seen. As it stands, the AI industry continues to have a profitability problem, and as economic markets are rarely patient, there could be another AI winter to come if these improvements aren't made fast.
Editorial Notes