How AI guesses the next word

In the early 1950s, Claude Shannon, a mathematician at Bell Labs, played a simple game with his wife, Betty. He’d open a detective story by Raymond Chandler and move through a sentence one letter at a time. Betty, who had studied math herself and had worked at the same lab for a while, had to guess the next letter. If she got it wrong, Shannon told her the right one and marked whether she’d got it on the first try.

The sentence was “THE ROOM WAS NOT VERY LIGHT A SMALL OBLONG…” Out of the 129 letters in the passage, Betty got 89 right on the first guess. That’s 69 percent, nearly seven letters out of every ten! Nobody had taught Betty a table of English statistics. She just knew the language, and that was enough to know most of the sentence before she saw it.

Take her seat for a minute and see how many letters you get on the first try.

Take Betty’s seat

Betty got 69 percent of the letters on her first guess. Can you?

Guess the next letter: type it, or tap a key below. A space counts as a letter too.

Your guess

ABCDEFGHIJKLMNOPQRSTUVWXYZspace

Tell me Start over

You play this game too. If I say “Sorry I’m late, the traffic was really…”, the next word pretty much arrives in your head on its own. Your phone keyboard plays it as well: those three words it suggests above the keys. Shannon drew a serious conclusion from the game: language is full of repetition, and a big part of any sentence can be predicted before you hear it.

Now jump 70 years ahead. ChatGPT came out at the end of November 2022 and, by UBS’s estimate, reached 100 million users in two months; no app had grown that fast before. What sits behind it is, in fact, Betty’s game: a machine that knows how to do exactly one thing, guess the next word. It’s your phone keyboard, grown very, very big.

Claude, the chatbot made by the company Anthropic, is reportedly named after this same Claude Shannon. The man who played the guessing game at the table ended up lending his name to one of the machines that play it today.

Here’s the thing: Betty knew the language. A machine that has never seen a room or switched on a lamp, where does its guess come from? The answer takes five steps, and you don’t need any math to follow them. Scroll slowly.

catdogkingqueenParispizzaRain forecast +3Cloudy +1Sunny −2Bar: 2Umbrella?The cat sat on the …mat 42%sofa 27%wall 14%roof 11%moon 6%

1. A computer only understands numbers, so first every word becomes numbers: a spot in an imaginary space, like a house’s address on a map. Words that show up in similar sentences end up neighbors. “Cat” lands next to “dog”, “king” next to “queen”.
2. A neuron is a tiny calculator. Should you take an umbrella? Rain in the forecast: 3 points. A cloudy sky: 1 point. Sunshine: minus 2. These points are called weights. The neuron adds up the ones that are true, and if the total passes a bar, say 2, it says “take it”. Learning means finding those points.
3. Stack thousands of neurons in layers and you get a network. The numbers for the words come in on the left and move forward, layer by layer: the first layers pick up simple things, the later ones more complicated combinations.
4. Learning runs the other way. The network guesses a word and is shown the right one. If it was wrong, the error travels backward and every weight turns a tiny bit toward a better guess, like an archer shifting their aim after every arrow. This happens billions of times.
5. What comes out at the end is a list: a chance for every word that could come next. The model takes one, sticks it onto the end of the text, and runs the whole thing again for the word after. Every reply from ChatGPT is built this way, word by word!

Now take the model’s seat. The sentence starts with “The cat sat on the”, and each time you get a few words with their chances. Pick one, or let the model draw.

Your turn

The model picks just one word at a time; a longer bar means a better chance

The cat sat on the…

The model has suggested these words for the blank. Tap one, or let the model draw one at random.

• mat 42%
• sofa 27%
• wall 14%
• roof 11%
• moon 6%

Let the model draw Start over

Temperature: 1 (normal)

← cautiousadventurous →

Move the slider and watch the bars: a low temperature stretches the top bar, a high one evens them out.

Don’t underestimate that temperature slider. Real models have a setting with the same name, temperature, and programmers set it themselves. A temperature of zero means the model takes the most likely word every time, which means the same answer every time. A higher temperature gives the less likely words a chance too. That’s why if you ask ChatGPT the same question twice, you usually get two different answers: the model draws again each time. But draws what, exactly? Not words, it turns out.

Words or pieces of words?

A token is a piece of text, sometimes a whole word, sometimes half a word or just a punctuation mark, and tokens are what models actually read. According to OpenAI’s own guide, one token of English text is about four characters on average, so 100 tokens come to roughly 75 words.

Why not whole words? Because the list of words never ends. “Book”, “books”, “bookshop”, “bookkeeping”, people’s names, typos… If the model needed a separate slot in its vocabulary for every one of them, the vocabulary would never be finished. On the other hand, if it read every letter separately, every sentence would get very long and the work would crawl. Tokens are the middle road. So who decides where a word gets cut?

How a word gets split

The method most of today’s models use to split text is called Byte Pair Encoding, or BPE, and it wasn’t made for AI at all. Philip Gage introduced it in 1994 as a way to make files smaller. 22 years later, in 2016, Rico Sennrich and his colleagues at the University of Edinburgh used it for machine translation, and since then almost everyone has picked it up.

The idea is simple enough to try with a pencil. Say all the text we have is these four words: “play”, “plays”, “played”, “player”.

Step one: split everything into letters. “play” becomes “p · l · a · y”.

Step two: count which two neighboring pieces show up together most often. Here “p” and “l” sit side by side in all four words. (“l · a” and “a · y” come up just as often; on a tie, we pick one.) Glue them together and add “pl” to the vocabulary as a new piece.

Step three: repeat. Next round, “pl” and “a” become “pla”, and the round after that, “pla” and “y” become “play”. Now “play” is a single token, and “player” is “play · e · r”. If “er” had come up a lot in our text, it would slowly become one piece too, and “player” would be two tokens: “play · er”.

Now try it yourself. This time the text is four words, each repeated several times. Every time you press “Glue”, the most frequent pair sticks together and a new row goes into the vocabulary, until the vocabulary is full. Keep an eye on the piece counter too: every merge makes the text shorter, which is exactly what Gage was after.

Split it yourself

Each time, the two pieces that sit next to each other most often get glued together

• low 5×
• lower 2×
• newest 6×
• widest 3×

Next: “e” and “s” sit side by side 9 times in all, more than any other pair.

Glue Start over

Pieces in this text: 79 (79 at the start)

Vocabulary (10 rows; room for 6 new ones)

1. 1 l
2. 2 o
3. 3 w
4. 4 e
5. 5 r
6. 6 n
7. 7 s
8. 8 t
9. 9 i
10. 10 d

On a real mountain of text, this gluing is repeated tens of thousands of times. The result is a vocabulary where common words are one piece and rare words are built from several. GPT-2’s vocabulary, for example, had 50,257 tokens. When you type a new sentence, the model runs the same merges in the same order on it, and every piece that comes out has a row number in that vocabulary. What actually goes into the model is those numbers.

This has a funny side effect. The model only ever sees the pieces. That’s why chatbots stumbled for a long time over a question that sounds like it’s for kids: “How many r’s are in strawberry?” Quite often they said two. A word you see letter by letter is two or three chunks to the model; it’s like asking someone how many red pixels are in a photo.

Some languages get a worse deal here. The token vocabularies were mostly built from English text, so they hold far more English merges than, say, Persian ones. That’s why a sentence in many other languages usually breaks into more tokens than its English translation. In 2023, Aleksandar Petrov and his colleagues took the same 2,000 Wikipedia sentences, translated into some 200 languages, and ran them through the tokenizers. The one ChatGPT and GPT-4 used needed about 3.3 times as many tokens for Persian as for English, and for Shan, a language spoken in Myanmar, 15 times. The study only counts pieces; it doesn’t measure how well the model understands each language. Still, that matters, because these companies charge by the token; talking to an AI in Persian, for example, is usually both slower and more expensive!

So a long answer means hundreds of guesses in a row, each one the input for the next. But a row number says nothing about meaning; knowing which row “cat” is on doesn’t tell you it’s like “dog”. Where does meaning come from?

Numbers nobody wrote

For every row number, the model keeps a long list of numbers: the list you saw on the first slide. It’s called an embedding. Picture a map: you can give the location of any city with two numbers, latitude and longitude. New York and Newark have close numbers; New York and Tokyo don’t. An embedding does the same with words, only with hundreds or thousands of numbers per word instead of two.

Nobody writes these numbers by hand. They start out random and slowly settle into place during training, until distance on this map roughly matches distance in meaning.

In 2013, Tomas Mikolov and two colleagues at Microsoft Research found something even stranger: directions have meaning too. Take the numbers for “king”, subtract the numbers for “man” and add the numbers for “woman”, and the closest word to where you land is “queen”. It’s like saying on a map, “start at king and walk the same route that gets you from man to woman.” Nobody told the model that royalty and gender are two separate things; it figured that out from reading text. But these numbers are only the input. The model’s actual knowledge lives somewhere else.

Arithmetic with words

The step that takes you from “man” to “woman”, taken from “king”, lands next to “queen”

It lands right next to “queen”: king minus man plus woman comes out close to queen.

Where does the knowledge live?

In the weights: the same points from the umbrella example. When people say a model is “big”, they mean how many weights it has. GPT-3, introduced in 2020, had 175 billion of them. To get a feel for that number: if you wrote down one weight per second and never slept, it would take about 5,500 years. Grammar and the names of capitals, everything the model seems to know, sits in those numbers. When it answers, it isn’t looking anything up on the internet, unless it has been given a separate search tool.

How much text did it take to set all those weights? GPT-3 saw about 300 billion tokens during training, from web pages, books and Wikipedia. At 75 words per 100 tokens, that’s around 225 billion words. An ordinary 300-page book is about 100,000 words, so that’s more than two million books. Reading one book a week, getting through all of it would take over 40,000 years!

So the next question is how all those numbers get set right. No person could turn 175 billion knobs by hand.

An error that travels backward

What you saw on the fourth slide is called backpropagation. In 1986, David Rumelhart, Geoffrey Hinton and Ronald Williams showed in a short paper in Nature that this method could train networks with many layers. The idea is that after every wrong guess, you work out for each individual weight which way it should turn to make the error smaller.

Picture a shower that has 175 billion taps instead of two. After every go, you notice whether the water was too hot or too cold, and you turn every tap a little, each one according to its share in getting the temperature wrong. Backpropagation is really just the way to work out that share for each tap. Training is that step, over a mountain of text, billions of times.

In 2024, Hinton shared the Nobel Prize in Physics with John Hopfield. The prize cited other work of his, but the heart of backpropagation is what Shannon was doing at the table: Betty guessed wrong, and Shannon told her the right answer. The difference is that Betty already knew the language, and the model learns everything from those right answers alone. But Betty had one more skill the early networks lacked: she knew which words to listen to.

Which words matter more?

Take this sentence: “I returned the book I borrowed from the neighborhood library last week, because it was too …” To guess the last word, “book” matters far more than “neighborhood”. Betty knew that without thinking. Older models read a sentence word by word, and by the time they reached the end, the start had faded.

In 2017, eight researchers, most of them at Google, wrote a paper with the cheeky title “Attention Is All You Need”. Their idea was for the model to look at all the previous words at once for every word, and learn by itself which ones to pay more “attention” to. They called this design the Transformer. That’s the T in GPT: Generative Pre-trained Transformer, a Transformer that has been trained in advance and generates text. Nearly every well-known chatbot today is built on this idea.

Here’s our sentence again, with that attention drawn as lines.

Attention, drawn

To fill the blank, “book” counts for far more than “neighborhood”

IreturnedthebookIborrowedfromtheneighborhoodlibrarylastweek,becauseitwastoo…

The blank looks hardest at “book”, then at “it”.

Tap any word to see which earlier words it looks at. The thicker the line, the more that word counts.

Back to the blank

So if the model guesses this well, why does it sometimes say, with complete confidence, something that doesn’t exist at all?

Six cases that never existed

In the spring of 2023, a lawyer in New York used ChatGPT for help with his client’s case against the airline Avianca. ChatGPT gave him several earlier court decisions, with case numbers and even quotes, and the lawyer put them all into his brief. The problem was that six of those cases didn’t exist. In the end, the judge fined the lawyers and their firm $5,000.

If you’ve come this far with us, you already know why. The model doesn’t ask “is this true?”, it asks “what word usually comes here?”. A court decision usually has the names of two parties and a case number, so the model makes those up, because it has the right shape. These mistakes are called hallucinations. The model isn’t really lying either, because truth and lies aren’t part of this game at all; it’s just building the next plausible sentence. The same game Betty played, and it’s older than she was.

A game older than computers

On January 23, 1913, the Russian mathematician Andrey Markov reported to the Academy of Sciences in St. Petersburg on a count he had done with paper and pencil. He had taken the first 20,000 letters of Pushkin’s Eugene Onegin, stripped out the spaces and punctuation, and counted. Of those, 8,638 were vowels, or 43 percent. Then he counted consecutive pairs and found that whether the next letter was a vowel depended on the letter before it.

Markov’s count, 1913

After a vowel, the next letter is rarely a vowel; after a consonant, it usually is

Show

Pushkin’s textIf letters were independent

Chance the next letter is a vowel

After a vowel 13%After a consonant 66% All letters: 43%

In Pushkin’s text, a vowel follows a vowel only about 13 percent of the time, but follows a consonant about 66 percent of the time. Vowels and consonants like to take turns.

Markov’s counts

Letters	20,000
Vowels	8,638
Consonants	11,362
Vowel–vowel pairs	1,104
Consonant–consonant pairs	3,827
Pairs of one vowel and one consonant	15,069
Vowel–vowel pairs expected if letters were independent	3,731

Markov looked back one letter; Betty, at the whole sentence and everything she’d ever read. In over a hundred years the game has only grown: guess the next letter from the last one, then the next word from tens of thousands of tokens, sometimes a whole book. Every time ChatGPT writes a sentence, it’s playing Betty’s game; only this time there’s nobody at the table to give it the right answer after a wrong guess.

Sources and notes

• Shannon and Betty’s game, 89 of 129 letters: C. E. Shannon, “Prediction and Entropy of Printed English”, Bell System Technical Journal 30 (1951), 50–64. Full text. The sentence is from Raymond Chandler’s story “Pickup on Noon Street”; on Betty Shannon’s role: Jimmy Soni and Rob Goodman, A Mind at Play (2017). The guessing game uses only the first line of the passage; Betty’s row in it is copied from Shannon’s record of that line (his “reduced text”), where she got 29 of its 42 characters, spaces included, on the first guess.
• 100 million users in two months: UBS estimate based on Similarweb data, reported by Reuters, “ChatGPT sets record for fastest-growing user base”, February 2023.
• Claude’s name: reportedly after Claude Shannon, per Kevin Roose, “Inside the White-Hot Center of A.I. Doomerism”, The New York Times, July 11, 2023.
• The next-word game: the probabilities in the game and on the last slide are made up and don’t come from any real model.
• About four English characters per token: OpenAI Help Center, “What are tokens and how to count them?”.
• BPE: P. Gage, “A New Algorithm for Data Compression”, C Users Journal 12(2), 1994; R. Sennrich, B. Haddow, A. Birch, “Neural Machine Translation of Rare Words with Subword Units”, ACL 2016. arXiv:1508.07909. The “play” example is made up to show the method; the word counts in the playground come from Sennrich and colleagues’ own example.
• GPT-2’s 50,257 tokens: A. Radford et al., “Language Models are Unsupervised Multitask Learners”, OpenAI, 2019.
• More tokens for languages other than English, 3.3 times for Persian: A. Petrov, E. La Malfa, P. Torr, A. Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages”, NeurIPS 2023. arXiv:2305.15425. The sentences are the FLORES-200 set; 3.28 for Western Persian and 15 for Shan are the premiums over English for the cl100k_base tokenizer used by ChatGPT and GPT-4.
• King minus man plus woman: T. Mikolov, W. Yih, G. Zweig, “Linguistic Regularities in Continuous Space Word Representations”, NAACL 2013 (all three at Microsoft Research). The word map in the graphic is schematic: the positions are placed by hand.
• GPT-3’s 175 billion weights and 300 billion tokens: T. Brown et al., “Language Models are Few-Shot Learners”, 2020. arXiv:2005.14165. The 5,500 years: 175 billion seconds divided by about 31.5 million seconds in a year. The books: 225 billion words divided by 100,000 words per book is 2.25 million books, and divided by 52 books a year is about 43,000 years.
• Backpropagation, 1986: D. Rumelhart, G. Hinton, R. Williams, “Learning representations by back-propagating errors”, Nature 323 (1986), 533–536.
• Nobel Prize in Physics 2024: NobelPrize.org, John Hopfield and Geoffrey Hinton, “for foundational discoveries and inventions that enable machine learning with artificial neural networks”; for Hinton, the prize announcement highlights the Boltzmann machine (1985), not backpropagation.
• The Transformer: A. Vaswani et al., “Attention Is All You Need”, NeurIPS 2017. arXiv:1706.03762. The attention weights in the graphic are made up to show the idea.
• The Avianca case: Mata v. Avianca, Inc., US District Court for the Southern District of New York, order by Judge P. Kevin Castel, June 22, 2023.
• Markov’s count, 8,638 vowels in 20,000 letters: Brian Hayes, “First Links in the Markov Chain”, American Scientist (2013). Link. The same article gives the pair counts in the graphic: 1,104 vowel–vowel pairs and 3,827 consonant–consonant pairs, against about 3,731 double vowels if letters were independent. The 13 and 66 percent are our arithmetic on those counts.