How a chatbot works: next-token prediction, sampling temperature and the context window
A chatbot is your phone's next-word suggestion scaled up enormously. It does not look anything up or plan a whole answer: it picks one likely next piece of text, called a token, adds it and picks again, a few hundred times, until it decides to stop.
What this page answers
How does an AI chatbot work?
It predicts one likely next token, adds it and repeats, a few hundred times, having learned from a large share of public text how text tends to continue. Read it in “A very good guesser”
Why does a chatbot give different answers to the same question?
It rolls weighted dice: a token with 60% probability is picked about 60% of the time. A setting called temperature controls how adventurous the dice are. Read it in “Why the same question gets different answers”
A very good guesser
Your phone suggests the next word as you type. A chatbot is the same idea, scaled up enormously: it has read a large share of the public internet, many books and a lot of code, and learned from it how text tends to continue. When you send a message, it doesn't look anything up and it doesn't plan a whole answer. It picks one likely next piece of text, adds it, and picks again, a few hundred times, until it decides to stop. Chapter 1 builds the smallest possible version of this; this lesson shows a real one.
The pieces aren't whole words. They're called tokens: common words are one token, rarer words are split into pieces. That's why prices and limits for these services are quoted in tokens.
Predict first. The text so far is “The capital of France is”. What's the model's single most likely next token?
- Paris
- the
- located
- a
Answer. “Paris”, with probability 29.8%. The runners-up were “·located” (8.6%), “·a” (5.9%), “·______” (3.6%). It's the favourite, but far from certain. The model has also seen this phrase continue as “The capital of France is located…”, and as a fill-in-the-blank on quiz worksheets (the “______” token), so it spreads its bets.
What it's choosing between
At every step the model gives every possible token a probability. In the figure: The bars are the ten tokens the model (Qwen2.5-1.5B-Instruct, a free model, running on a laptop) rated most likely to come next, with the probability it gave each. A leading space is part of the token, which is why “ Paris” appears with a gap.

Compare the prompts. After “Once upon a” the model is very sure (99.1% on its top choice); after “The capital of France is” its favourite gets only 29.8%, because many continuations are reasonable. That spread is the model's uncertainty, and it's a real, measurable number inside every chatbot, even though the product never shows it to you. It's also the root of lesson E2's problem: the model can be very sure of a continuation that happens to be false.
Why the same question gets different answers
The model doesn't always take the top token. It rolls weighted dice: a token with 60% probability is picked about 60% of the time. A setting called temperature controls how adventurous the dice are. Low temperature sticks to the favourites; high temperature gives long shots a chance. Here's the same request, “Write one sentence about the sea.”, answered three times at each of three temperatures, exactly as the model wrote them.
| Temperature | Three answers to the same request |
|---|---|
| 0.2 | The vast expanse of the ocean stretches endlessly across the globe, covering more than 71% of Earth's surface and孕育着无数的生物和文化。 The vast expanse of the ocean stretches endlessly across the globe, covering approximately 71% of Earth's surface and孕育着无数的生物和文化。 The vast ocean stretches endlessly, covering more than 71% of Earth's surface and providing vital resources for life on our planet. |
| 1 | The vast expanse of the ocean stretches out before us like an endless horizon. The vast expanse of the ocean stretches endlessly as far as the eye can see, its undulating waves a soothing sight that calms even the most troubled mind. The vast ocean covers over 71% of Earth's surface and is an integral part of our planet's ecology. |
| 1.6 | The breeze howled and chattered across the fi`naite hues of evDMETHOD皋.grey Shark efficiently prioritize detail activity capMinutes = driver lime catalog bc related"wcuウ with discretion point условияcheckBox The vast expanse of saltwater that borders millions of islands whispered forever on explorﺍancies har 합니다 כל batch 원څный sions erst汉语filmxC学lections垃圾을스 hype The.deep.endless.body.large.whirl[i][rugatliunbind Vehicle.addEventListener(rimid𬕂"context→ Qin picoul:%5begtoskel perg4(btMoviesa OnTriggerEnter Shelter diamond trouble useves |
At 0.2 the three answers are nearly identical: the dice almost always land on the favourite. Look closely, though: two of them switch into Chinese halfway through. Qwen was trained on a great deal of Chinese as well as English, and at this point in the sentence a Chinese continuation was apparently among the model's favourites; low temperature doesn't make a model correct, it makes it consistent, including consistently odd. At 1.0, the model's natural setting, they vary but stay sensible. At 1.6, unlikely tokens get picked often enough that the text can drift, repeat oddly or stop making sense. Chatbot products choose a temperature for you, usually a moderate one; some let you change it as “creativity” or similar. The facts the model “knows” don't change with temperature, only how adventurously it picks among the continuations it rates.
Continue the lesson
The whole lesson, with its drawings and its questions. The rest of this section and 4 more after it.

