But the model, known internally as Orion, did not hit the company's desired performance, according to two people familiar with the matter, who spoke on condition of anonymity to discuss company matters. For example, Orion fell short when trying to answer coding questions that it hadn't been trained on, the people said. Overall, Orion is so far not considered to be as big a step up from OpenAI's existing models as GPT-4 was from GPT-3.5, the system that originally powered the company's flagship chatbot, the people said.
OpenAI isn't alone in hitting stumbling blocks recently. After years of pushing out increasingly sophisticated AI products at a breakneck pace, three of the leading AI companies are now seeing diminishing returns from their costly efforts to build newer models. At Google, an upcoming iteration of its Gemini software is not living up to internal expectations, according to three people with knowledge of the matter. Anthropic, meanwhile, has seen the timetable slip for the release of its long-awaited Claude model called 3.5 Opus.
The companies are facing several challenges. It's become increasingly difficult to find new, untapped sources of high-quality, human-made training data that can be used to build more advanced AI systems. Orion's unsatisfactory coding performance was due in part to the lack of sufficient coding data to train on, two people said. At the same time, even modest improvements may not be enough to justify the tremendous costs associated with building and operating new models, or to live up to the expectations that come with branding a product as a major upgrade.
In a statement, a Google DeepMind spokesman said the company is "pleased with the progress we're seeing on Gemini and we'll share more when we're ready." OpenAI declined to comment. Anthropic declined to comment, but referred to a five-hour podcast featuring CEO Dario Amodei that was released on Monday.
"People call them scaling laws. That's a misnomer," he said on the podcast. "They're not laws of the universe. They're empirical regularities. I am going to bet in favour of them continuing, but I'm not certain of that."
