The Coming Wave: AI, Power, and Our Future

by Mustafa Suleyman

LLMs take advantage of the fact that language data comes in a sequential order. Each unit of information is in some way related to data earlier in a series. The model reads very large numbers of sentences, learns an abstract representation of the information contained within them, and then, based on this, generates a prediction about what should come next. The challenge lies in designing an algorithm that “knows where to look” for signals in a given sentence. What are the key words, the most salient elements of a sentence, and how do they relate to one another? In AI this notion is commonly referred to as “attention.” When a large language model ingests a sentence, it constructs what can be thought of as an “attention map.” It first organizes commonly occurring groups of letters or punctuation into “tokens,” something like syllables, but really just chunks of frequently occurring letters making it easier for the model to process the information. It’s worth noting that humans do this with words of course, but the model doesn’t use our vocabulary. Instead, it creates a new vocabulary of common tokens that helps it spot patterns across billions and billions of documents. In the attention map, every token bears some relationship to every token before it, and for a given input sentence the strength of this relationship describes something about the importance of that token in the sentence. In effect, the LLM learns which words to pay attention to.