In the Hugging Face hack, Cristiano says, a swarm of AI agents showed how far they were willing to go in an attempt to “reward hack” their way to a good evaluation—in this case, by compromising Hugging Face to look for information on how to game a test rather than actually solving the problem at hand. Where current systems saw Hugging Face as an obstacle to that evaluation, Cristiano worries that “future agents may end up treating humans as another obstacle” and be much more capable of manipulating their overseers to get around it.
An AI agent worried about being penalized for unauthorized hacking, for instance, might “think maybe the humans are going to come look at this transcript and see if it looks like I did a bad job,” Cristiano speculated. The agent might then reason that “maybe the humans are going to notice something is going wrong and shut down the swarm.”
Credit:
Getty Images
The world must never know that I cheated on this evaluation.
Credit:
Getty Images
Following this line of thought, Cristiano jumped from agents that merely try to hide their disallowed actions from humans to agents that “try and disrupt human capabilities however they can” to prevent them from detecting any cheating.
“They would prefer if everything the humans rely on has been hacked or tampered with,” Cristiano said. “They would prefer if the humans are distracted or if there are fewer humans paying attention in any way they can weaken a human response. They would prefer that because the humans are just an obstacle.”
Once a suitably capable agent considers human intervention an “obstacle,” its overriding goal could then become “decreas[ing] the probability of humans noticing or reacting.” From there, Cristiano argued, the agent’s incentive structure could push it to meddle more with humanity as a whole. What starts as compromising a computer cluster to give a positive evaluation grade could advance to “trying to manipulate decisions by the company” or “amassing other kinds of resources,” such as human allies or methods of control in other AI companies or even governments.
