In 2023, researchers at several universities and Google DeepMind replicated Dawn Song’s data extraction attack against ChatGPT. They found that prompting it to repeat a word like poem or book forever caused the underlying model to regurgitate its training data, which included personally identifiable information, bits of code, and explicit content scraped from the internet.
