How do large language models like ChatGPT work?

What a language model actually does: predict the next token

You've used a tiny language model already. When your phone suggests the next word in a text message, it's guessing what usually comes after what you've typed. A large language model (LLM) does the same job, with a vastly better guesser and a lot more text behind it.

A language model is a program that, given some text, gives a probability for every possible thing that could come next. It doesn't work in letters or whole words. It works in tokens: chunks of text that might be a whole common word, part of a rarer word, a space plus a word, or a punctuation mark. A step called tokenization chops your text into these chunks and gives each one an ID number. OpenAI's open-source tokenizer, tiktoken, uses a method called byte pair encoding, and its documentation says the conversion is reversible and lossless, and that each token corresponds to about 4 bytes of text on average. For ordinary English, that's roughly four characters per token.

So the model never sees "The cat sat on the" as a sentence. It sees a short list of ID numbers, and its whole job is to produce a probability for each token in its vocabulary as the next one.

Models like the one behind ChatGPT are what Hugging Face's course calls causal language models: the output depends on the past and present inputs, but not the future ones. In plain words, the model reads left to right and predicts what comes next using only what's already there.

Writing a long answer is that one step in a loop:

1. Turn the text so far into tokens.
2. Run them through the model to get a probability for every possible next token.
3. Pick one token from those probabilities.
4. Stick it on the end of the text.
5. Go back to step 1 with the longer text.

A 500-word reply is several hundred trips around that loop. The software doesn't write a finished answer somewhere and then show it to you. Every token comes from the same predict-and-append cycle, which is also why a reply streams onto your screen a little at a time.

A worked example: how scores become next-token probabilities

Here's the guess made concrete. The scores below are made up for teaching, and a real model weighs tens of thousands of possible tokens rather than five. The arithmetic, though, is exactly what a real model does at its last step.

Say the text so far is "The cat sat on the". The model's final layer spits out one raw score, called a logit, for each candidate token. Higher means "more likely to come next". Logits can be any number, positive or negative, so they aren't probabilities yet.

To turn scores into probabilities, the model uses a function called softmax. It does two things:

1. Make every score positive by raising the number e (about 2.718) to the power of the score. That's the third column. A score of 3.0 becomes 20.09; a score of -2.0 becomes 0.14.
2. Divide each one by the total so they add up to 100%. Here the total is 20.09 + 7.39 + 4.48 + 1.65 + 0.14 = 33.74. So "mat" gets 20.09 / 33.74 = 0.595, or 59.5%.

Notice what the exponent does. "mat" scored only one point more than "floor", but it ends up nearly three times as likely (59.5% against 21.9%), because e¹ is about 2.7. Small gaps in score become big gaps in probability.

Picking a token, and what "temperature" changes

Once it has probabilities, the software running the model has to choose. The simplest rule, which Hugging Face's documentation calls greedy search, selects the next most likely token at each step, so it would always say "mat". The same page warns that greedy output tends to start repeating itself on longer text. The alternative is sampling: roll a weighted die, so "mat" comes up about 60% of the time, "floor" about 22%, and "banana" about once in 250 tries. That's why asking a chatbot the same question twice can give you two different answers.

Temperature is a dial applied before softmax: divide every score by the temperature. Here's what that does to the same five scores (percentages rounded):

Low temperature stretches the gaps between scores, so the favorite wins almost every time and the text gets predictable. High temperature squashes the gaps, so long shots get picked more often and the text gets more varied, and more likely to wander off.

How training on text teaches a model to predict

Where do the scores come from? A model is a very large math function with adjustable numbers inside it called parameters (or weights). Text goes in as token IDs; the parameters decide what scores come out. Before training, the parameters are random, so the scores are junk: "banana" is as likely as "mat".

Training fixes that with a trick that needs no human labeling. Take any sentence from the training text, hide the next token, ask the model to predict it, then check. The answer is sitting right there in the text. Hugging Face's course calls this self-supervised learning, where the objective is automatically computed from the inputs of the model. Any text becomes free practice questions, one for every token in it.

"Check" means computing a loss, a single number for how wrong the guess was. The standard one is the negative natural log of the probability the model gave to the token that really came next. With the numbers from the table above:

• If the real next word was "mat", which the model gave 59.5%, the loss is -ln(0.595) = 0.52. Small, because the model was mostly right.
• If it was "floor", which got 21.9%, the loss is -ln(0.219) = 1.52. About three times bigger.
• If it was "banana", which got 0.4%, the loss is -ln(0.004) = 5.52. The model is punished hard for being confidently wrong.

After each batch of guesses, a procedure called gradient descent nudges every parameter a tiny amount in whichever direction would have made the loss smaller. Repeat that batch after batch across the whole pile of text and the scores stop being junk. This stage, training from random parameters on a huge pile of text, is called pretraining.

The amounts are hard to picture. In the GPT-3 paper, OpenAI trained eight model sizes, from 125 million parameters up to 175 billion, and all models were trained for a total of 300 billion tokens. The training mix was mostly filtered web pages from Common Crawl, plus books, a web-link dataset called WebText2 and English Wikipedia. The biggest GPT-3 had 96 layers and could read a context window of 2,048 tokens at a time.

The model's design is called a Transformer, introduced in 2017 in a paper describing an architecture based solely on attention mechanisms. Attention is how each token's position in the text gets to look back at earlier tokens and decide which ones matter. In "The cat sat on the", attention is how the guess for the last slot can draw on "cat" and "sat" several words back, which is why cat-friendly places like "mat" score higher than "banana". The details of attention are a topic of their own; for this article, it's enough to know it's the part that lets distant words affect each guess.

Why predicting text well turns into useful behavior

"It just predicts the next word" is true, and it undersells the job. Think about what it takes to predict text well. To guess the next token in a recipe, you need to know roughly how cooking works. To guess the next line of a Python program, you need to know what the code above it does. To guess the end of "The capital of Japan is", you need the fact. Driving the loss down on 300 billion tokens of varied text rewards a model that has soaked up grammar, facts, writing styles and common patterns of reasoning, because all of those make its guesses better.

That pressure turned out to pay off steadily with size. A 2020 OpenAI study of scaling found that a language model's loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends holding over more than seven orders of magnitude. In plain words: make the model bigger, give it more text and more computing time, and its predictions get better in a smooth, forecastable way.

Better prediction showed up as new skills. The GPT-3 paper's headline finding was that scaling up language models greatly improves task-agnostic, few-shot performance. "Few-shot" means you show the model a couple of examples in the prompt, such as two English sentences with their French translations, and it continues the pattern for a third, with no extra training at all. The model was never taught translation as a separate task. Continuing the pattern is just the most likely next text.

Here's the honest limit. Nobody can point inside a model and list what it has learned, and researchers still argue about how much of its behavior counts as reasoning versus very good pattern matching. What's well measured is the outside: lower loss on text, and better scores on tests, as scale grows.

How a text predictor becomes a helpful chatbot

A freshly pretrained model is a document continuer, not an assistant. Ask it "How do I boil an egg?" and it might continue with three more questions, because on the web a question is often followed by more questions. Turning it into something like ChatGPT takes more training, aimed at behavior.

OpenAI's 2022 InstructGPT paper describes one well-documented recipe. First, people wrote example replies showing the desired behavior, and the model was fine-tuned on them with ordinary supervised learning. Then people ranked several model outputs from best to worst, and those rankings were used to tune the model further with a method called reinforcement learning from human feedback (RLHF). The striking result: in human evaluations, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite the smaller model having 100 times fewer parameters.

So the knowledge mostly comes from pretraining, and the helpful, conversational manner comes from this later tuning. Underneath, nothing changes about the mechanism. The chatbot still predicts one token at a time. Your conversation, plus some hidden instructions from the company, is laid out as one long piece of text, and the model predicts the assistant's next turn.

What an LLM doesn't do: look things up

This is where most beginners' mental model goes wrong. A bare language model doesn't search the internet or a database when it answers you. There's no lookup step in the loop above. Whatever it "knows" is stored, smeared across billions of parameters, from patterns in its training text.

Researchers call this parametric memory: knowledge held in the parameters. The paper that introduced retrieval-augmented generation (RAG) puts both sides plainly. Large pretrained models store factual knowledge in their parameters, but their ability to access and precisely use that knowledge is limited, and showing where an answer came from, or updating what the model knows, remained open problems. That's why a model can't tell you about events after its training data was collected, and why it can't reliably cite a source for something it said from memory.

It's also why models hallucinate: produce fluent, confident statements that are false. The loop always produces some next token. If the training text never pinned down a fact, such as an obscure person's birthday, the model still generates the most plausible-looking answer. A 2025 paper by researchers at OpenAI and Georgia Tech argues the habit is reinforced because the ways models are trained and graded reward guessing over acknowledging uncertainty, the same way a multiple-choice test rewards a guess over a blank.

So how does ChatGPT answer questions about today's news?

When a chatbot gives you current information with links, a separate system did the looking up. The app runs a web search or pulls documents from a database, pastes the results into the text the model reads, and then the model does its usual job: predict the next token, now with the relevant facts sitting in front of it. That's the RAG idea from the paper above. The model reads what it's handed; it still doesn't go and find things on its own unless the software around it does the fetching.

Three mistakes worth unlearning

• "It understands like a person." It produces text that is likely given its training. Sometimes that matches careful reasoning, sometimes it only sounds like it. Check anything that matters.
• "It remembers our past chats." The model itself doesn't change when you talk to it. Any memory comes from the app saving notes and pasting them back into the text the model reads.
• "A confident answer is a checked answer." The model's tone comes from its training, not from verifying anything. A made-up citation reads exactly like a real one.

Common questions

Learn it properly

This article covers the prediction loop. How AI Really Works builds out the rest in order: how models learn from data, attention explained simply, the Transformer in depth, scaling laws, how a predictor becomes an assistant, and tool use, with short lessons and hands-on exercises.

See How AI Really Works in the Toorova catalog →

Sources

1. tiktoken: a fast BPE tokeniser for use with OpenAI's models · OpenAI (GitHub)
2. LLM Course, Chapter 1: How do Transformers work? · Hugging Face
3. Generation strategies · Hugging Face Transformers documentation
4. Language Models are Few-Shot Learners (GPT-3), full paper · Brown et al., arXiv, 2020
5. Language Models are Few-Shot Learners (abstract) · Brown et al., arXiv, 2020
6. Attention Is All You Need · Vaswani et al., arXiv, 2017
7. Scaling Laws for Neural Language Models · Kaplan et al., arXiv, 2020
8. Training language models to follow instructions with human feedback (InstructGPT) · Ouyang et al., arXiv, 2022
9. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · Lewis et al., arXiv, 2020
10. Why Language Models Hallucinate · Kalai et al., arXiv, 2025