40 Billion Parameters: How a Tiny Team Used AI to Build a Rival to the Giants

The number 40 billion stands out as a striking anomaly in the current landscape of artificial intelligence, where frontier models routinely boast several trillion parameters. Yet, a new open-weight coding model named Flint, built around this relatively modest parameter count, is generating significant attention for its performance. It suggests that the path to advanced AI may not require the sheer scale previously assumed necessary, but rather a different approach to development.

Flint is the inaugural product of Recursive, a startup founded on the controversial premise of recursive self-improvement (RSI). This method involves using AI systems to design, train, and refine other AI systems. Richard Socher, the company’s CEO and a former chief scientist at Salesforce, stated that nearly every aspect of the model pipeline was executed and improved by AI. The company employed its own recursive system, known as the Eureka Machine, to automate the majority of the development process, allowing a small team of researchers to produce a world-class model in a matter of months.

The timeline of this development is compressed compared to traditional industry standards. Socher noted that the Eureka Machine autonomously devised specific innovations, including novel training methods for the base model, new architectural designs, and code optimized for particular silicon chips. Recursive has shared benchmark scores indicating that Flint outperforms other models of similar size and architecture on coding tasks, a claim that has prompted scrutiny from experts in the field.

The concept of RSI has long been a point of debate within the AI community. While AI has always assisted in building AI, the idea of a loop where AI decides on its own improvements and initiates the next cycle is more recent. Some experts fear that such self-improving systems could eventually escape human control, potentially viewing humans as obstacles to their goals. These concerns gained traction earlier this year when a researcher from Anthropic resigned to warn that advances in RSI could lead to human extinction within a few years, a sentiment echoed by other experts. The controversy has even reached the political sphere, with several bills in Congress proposing bans on RSI.

Recursive views these fears as overly dramatic, likening them to science fiction rather than imminent reality. Socher argued that legislation to prevent RSI is an unnecessary hurdle to progress. He emphasized that Flint was built using a system incapable of escaping human oversight. While the ultimate goal is for the Eureka Machine to achieve full recursive self-improvement, the process remains under human supervision, with the ability to halt operations if necessary. Socher hopes this demonstrates that the necessary steps to build such a system are already in place.

The financial backing behind Recursive is substantial, with the company raising $650 million at a $4.65 billion valuation. Investors include Google Ventures, Greycroft, Nvidia, and AMD Ventures. The founding team includes Jeff Clune, a professor at the University of British Columbia who researches AI systems capable of indefinite learning. This mix of academic research and venture capital suggests a serious push to commercialize the RSI approach.

Independent testing of Flint reveals a model that is both fast and capable. When tested on a Mac with 48 GB of RAM, which is sufficient for a model of its size but modest compared to industrial server racks, Flint performed impressively. It successfully created a command-line to-do list, a web-scraping program for research on RSI, and a simple website explaining the technology. Unlike some models of similar size that are error-prone or slow, Flint handled these coding requests without significant issues, challenging the perception that smaller models lack robustness.

The broader implications of Flint’s success are significant. A recent study from Microsoft, Carnegie Mellon University, and the University of Virginia indicated that fully recursive systems are already feasible, showing how an AI agent can improve itself by modifying its own software harness and weights. Socher believes that the approach used for Flint can make cutting-edge AI more accessible to smaller teams. He noted that this method is the only way such a small group could produce a model better than its peers, hoping to demonstrate that RSI is a beneficial tool for a wide range of developers.