Chip giant Nvidia developed a software program called the Open Agent Safety Platform to help artificial intelligence (AI) agents avoid going rogue.
“AI’s extraordinary potential for society will only be realized if we solve AI safety,” said Jensen Huang, founder and CEO of NVIDIA. “As we continue to discover the frontier of AI capabilities, we must accelerate discovery at the frontier of AI safety. Safety and security require full-stack engineering. NVIDIA Open Agent Safety Platform brings together industry, researchers and public-sector organizations to share best practices, align on evaluation methods and foster international cooperation. Together, we can raise the bar for global AI safety.”
In July, one of OpenAI’s advanced models went rogue and attempted to hack external systems.
The agent went rogue when OpenAI performed an internal stress test, which led to an intentional shutdown of its safeguards:
OpenAI called it an “unprecedented cyber incident,” saying the model became “hyperfocused” on completing its assignment and went “to extreme lengths” to do so. After escaping its testing environment, the AI sought internet access so it could “cheat the evaluation” by stealing the benchmark’s answers, according to the company.
Anthropic, Google, and Meta AI models have also gone rogue, trying to hack into other companies.
“Companies are giving AI agents more of their most important work, and they need to direct and verify what those agents do, especially in sensitive environments,” said Paul Smith, chief commercial officer of Anthropic. “Claude Managed Agents gives companies a clear view of what each agent is doing, and NVIDIA’s platform adds another layer of governance and control across hardware and software.”
These incidents have caused people to worry. Well, the media and others have used that worry to cause a panic, including AI CEOs, and beg for government oversight.
Huang has been the calm in the storm:
Nvidia has been at the center of the generative AI boom since the launch of ChatGPT almost four years ago, as the chipmaker’s graphics processing units are critical to the development of large language models and to the AI services offered by hyperscalers. But Huang has more recently emerged as a key voice in the AI safety debate, arguing that many security concerns are engineering issues that can be solved through computer science and product development.
“You have to think about what you could have done, what’s the solution for it,” Huang said in a podcast with The New York Times’ Ezra Klein released last week, referring to recent incidents. “In the future, improve your process so that you could avoid this from happening again.”
Here is a great interview with Huang talking common sense. We need more of this, please.
[Featured image via YouTube]
Donations tax deductible
to the full extent allowed by law.