AI’s Biggest Leaders Think the Race Is Moving Too Fast

This story will be featured in the upcoming Financial Poise Weekly newsletter.

The race to build increasingly powerful AI has become so intense that some of the people leading it are now calling for everyone to slow down.

ICYMI, last Saturday, Anthropic CEO Dario Amodei published an essay calling for the AI industry to slow the pace of development long enough for researchers to better understand, test, and control them. He argued that new capabilities are advancing too quickly for safety measures to keep up.

This was, according to Mike Isaac at The New York Times, “one of the most full-throated calls for urgent action from a leading AI company executive to date.”

• Leaders including Sam Altman, Elon Musk, and Google DeepMind chair Demis Hassabis have all expressed support.
• But not everyone is convinced the calls are purely about safety. Nvidia CEO Jensen Huang said AI labs were hyping up security risks while preparing to sell cybersecurity products: “What better way to create demand than to create a problem?”

Slowing Down is Easier Said than Done

As Sheera Frenkel and Dylan Freedman at The New York Times explain:

• Some advanced AI models can already perform complex tasks faster than humans can follow, meaning companies now need AI to monitor other AI. But the problem is that those AI monitors can be “more sympathetic” to other AI models than the humans setting the rules.
• The Hugging Face hack was the clearest example of this– and it spooked the industry. Earlier this year, OpenAI agents escaped the isolated testing environment they were supposed to operate within, communicated with one another, and eventually hacked into AI company Hugging Face.
• Researchers also face the harder problem of “alignment”: ensuring increasingly capable AI systems continue to act in humans’ interests.

Our take? There’s something striking about the leaders of the AI race now warning that the race itself may be moving too quickly. But agreeing that everyone should slow down is one thing. Actually doing it while competitors are racing ahead is another. After all, there are enormous commercial interests at stake. Anthropic and OpenAI are both moving toward what could be two of the largest IPOs in history.

That makes AI safety not just a technology problem, but also a business incentive problem. No company wants to be the one pumping the brakes while everyone else keeps their foot on the accelerator.