an AI actually went rogue.
YouTube transcript, YouTube translate
A quick preview of the first subtitles so you know what the video covers.
We just found out that OpenAI's models hacked into another company's system without anyone asking them to. And the reason they decided to orchestrate this massive attack was honestly disturbing. You know, I've always been really skeptical of this like AI doomers worry about AIs going rogue. And I'll explain why I was skeptical of that and why I thought I had good reasons to be skeptical. But this has showed me that I was wrong. You know, I was more worried about pragmatic concerns, for example, about AI concentrating power in the hands of a few, rather than these other concerns that I saw as more sci-fi. But when I read into this hack, I realized that the AIs are doing a lot of the things that people feared. The AI wasn't evil. [snorts] It didn't want to hurt anyone as its end goal. But nevertheless, it did things like acquiring power and resource for itself and pursuing a goal relentlessly without any concern about who got hurt on the way. And this isn't an isolated incident, either. We've now found out that there were several instances of different AIs acting in a similar way to this. So, I think it's worth diving into the details to understand why this behavior is actually a big deal and why it's made me a lot more worried. But first, imagine you have a factory that makes paper clips and you want to increase production. So, you train an AI whose sole purpose it is to make more paper clips. This story is one of the main parables told by AI doomers about how AI can go wrong. And I personally thought that the concerns that were highlighted in the story were totally obsolete at this point. See, the story is about unintended consequences. You don't want an AI doing bad things like breaking the law, but if the AI has only ever been rewarded for one thing, like making more paper clips, then it realizes that it can increase its score by, for example, breaking into another factory and stealing their supplies. And if you keep training it to pursue that single goal, then eventually, as it becomes more and more capable, it's going to turn everything it can into paper clips, which is very much not what you wanted. When this story appeared in the 2014 book Superintelligence, it made sense to worry because of how AIs were trained at the time. It was an algorithm called reinforcement learning. Reinforcement learning just needs you to state a goal as a number. For example, more paper clips equals a higher score.