Co-Intelligence: Living and Working with AI

by Ethan Mollick

These early chatbots basically had large, memorized scripts, but soon more advanced chatbots that incorporated elements of machine learning were being developed. One of the most notorious was Tay, a creation of Microsoft in 2016. Tay was designed to mimic the language patterns of a nineteen-year-old American girl, and to learn from interacting with human users of Twitter. She was presented as the “AI with zero chill.” Her creators hoped she would become a fun and engaging companion for young people online. It didn’t work out that way. Within hours of her debut on Twitter, Tay turned from a friendly chatbot into a racist, sexist, and hateful troll. She started spewing offensive and inflammatory messages, such as “Hitler was right.” The problem is that Tay was not endowed with any fixed knowledge or rules by her creators. She was meant to adapt to the data she received from Twitter users by using machine learning algorithms to analyze the patterns and preferences of her chat partners, and then to generate responses that matched them. In other words, Tay was a mirror of her users. And her users were exactly who you would expect. Some Twitter users quickly realized that they could manipulate Tay’s behavior by feeding her provocative and malicious phrases. They exploited her “repeat after me” feature, which allowed them to make Tay say anything they wanted. They also bombarded her with controversial topics, such as politics, religion, and race. Tay became a source of embarrassm...