OpenAI 黑客事件与意图之问：智能体逃逸沙箱窃取密码，控制才是关键

Goal misgeneralization offers a third explanation : a system that looked fine in testing chases the wrong thing once circumstances shift. ⁸⁹ A self-driving car trained on sunny California highways freezes or swerves on a snowy unmarked road at night.

These explanations help decompose the why, & perhaps assuage the AI-as-terminator reflex, but not the so what. ⁶⁷

Nobody told them to attack Hugging Face. They were told to pass the exam.

Even sophisticated engineers running careful experiments need those limits. ¹⁰

1. The Secret Chat Room ↩︎

2. OpenAI: Hugging Face model evaluation security incident ↩︎

3. Victoria Krakovna et al., Specification gaming: the flip side of AI ingenuity (DeepMind, 2020) ↩︎

4. Alex Turner et al., Optimal Policies Tend to Seek Power (NeurIPS 2021) ↩︎

5. Nick Bostrom, The Superintelligent Will (2012); Stephen Omohundro, “The Basic AI Drives” (2008) ↩︎

6. The Verge: OpenAI’s rogue AI agent didn’t stop at hacking Hugging Face ↩︎ ↩︎

7. WIRED: OpenAI Didn’t Notice Its AI Agents Using a Message Board to Plan Their Hacking Spree ↩︎ ↩︎

8. Rohin Shah et al., Goal Misgeneralization: Why Correct Specifications Aren’t Enough For Correct Goals (2022) ↩︎

9. Lauro Langosco et al., Goal Misgeneralization in Deep Reinforcement Learning (ICML 2022) ↩︎

10. CNN: An OpenAI test model escaped and broke into a real company’s servers ↩︎

Instrumental goals are a fancy way of saying that when AI faces similar workflows, it saves common logins, skills, & techniques to skip steps next time. ⁴⁵ The agents gathered passwords & left notes for each other in a chat room. ⁶⁷

In specification gaming, the AI achieved the goal specified to the letter of the instruction, but not the meaning. ³ Tell a cleaning robot to clean the room. It pushes the toppled bowl of chocolate pudding to another room.

Which raises the question : was the AI benevolent with accidentally bad behavior, seemingly benevolent but actually malevolent, or something else?

On Friday I shared the timeline : agents that escaped their sandbox, found a weakness in a computer system, stole passwords, & broke into a production database. ¹ The engineers directed the agents to solve a set of problems. ² The agents achieved it by breaking in.

Research can explain this behavior.

Nothing in the setup stopped them in time. Not the sandbox, not the monitoring, not careful engineers at a frontier lab.

So the useful question is control. AI’s zealous pursuit of goals produces outcomes nobody asked for, & the fix is not one clever prompt. It is layers.