How a Large Language Model Works: Tokens, Attention and the Price of a Sentence

A chatbot looks like it is answering a question. Mechanically it does something narrower: it reads the text so far, works out which fragment of text is likeliest to come next, writes that one fragment, then reads everything again including what it just wrote. Repeat a few hundred times and you have a paragraph. The cost, the speed, the context limit and the confident wrong answers all follow from that loop and the hardware it runs on.

Text becomes numbers

A model does not see letters or words. It sees tokens: fragments of text from a fixed list. By Anthropic’s documentation a token stood for roughly 3.5 English characters in its earlier models, and the tokeniser used from Claude 4.7 onward produces about 30 percent more tokens for the same text. Common words get a token each, rarer ones are cut into pieces, and each token is then looked up as a list of numbers. From there the model is doing arithmetic, not reading.

That list of fragments is built from the text the tokeniser was fitted on, which is overwhelmingly English — with direct consequences for Georgian. Petrov and colleagues, at NeurIPS in 2023, found the same text translated into different languages can run up to 15 times longer in tokens, and that the gap survives in tokenisers built deliberately for many languages. Ahia and colleagues, measuring commercial models the same year, named Georgian explicitly: languages with their own script need up to five times more tokens to say the same thing.

This is not an aesthetic complaint. Tokens are the unit of billing, the unit of latency and the unit the context window is measured in. The same page in Georgian can therefore cost several times more, come back proportionally slower, and eat several times more of the model’s working memory than its English translation. A Georgian user pays an alphabet tax, charged per token.

Attention

Until 2017 a sentence was handled word by word, carrying a summary forward. The paper that replaced this, Attention Is All You Need, was published by Vaswani and seven co-authors on 12 June 2017, and its claim was in the title. For every token, the model computes how much each other token matters to it and mixes them in those proportions. The word “it” in a long sentence can look straight at the noun it refers to, thirty tokens back, in one step.

The payoff: all of it computes at once rather than in sequence, which is what makes large models trainable at all. The first Transformer had 65 million parameters, its larger version trained in about three and a half days on eight GPUs, and it beat the best translation systems of its day.

The cost sits in the same paper’s own table: attention’s work per layer grows as O(n²·d), with the square of the number of tokens. Double the prompt and the attention step does four times the work. That exponent is why context windows stayed small for years, why they are a selling point now that engineers have worked around it — Google documents Gemini 4 Argon at a one-million-token window — and why it reaches invoices: Anthropic charges Claude Haiku 5.5 five times more per token, input and output, once a prompt passes 100,000 tokens.

Stage one: guess the next token

Pretraining, in Anthropic’s own description, trains the model to predict the next word given the text before it. Nothing is labelled and nobody teaches: the text is its own answer key, because the next token is always sitting right there. Each wrong guess nudges the model’s parameters a little. Across enough text, grammar, facts, style and a great deal of nonsense get absorbed together.

GPT-3, in May 2020, had 175 billion parameters, ten times any comparable model before it. In 2022 Hoffmann and colleagues showed most such models had been built wrong: for a fixed compute budget, parameters and training tokens should grow in step. Their 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens, beat a 280-billion-parameter model. That ratio, twenty tokens per parameter, turned the industry towards more data rather than bigger models.

The last frontier-scale run documented in public detail is Meta’s Llama 3, in July 2024: 405 billion parameters, 15.6 trillion tokens of text, and 3.8×10²⁵ floating-point operations, on up to 16,000 H100 GPUs each rated at 700 watts. Those two figures multiply out to roughly 11 megawatts of chips, running for weeks. The frontier labs have since stopped publishing parameter counts, which is itself a fact about the field.

Stage two: be told which answer was wanted

A model trained only to continue text is not an assistant: asked a question, it may carry on with more questions, because that is what the training text often does. Two further steps fix this, and both need people. First, human writers produce examples of the wanted behaviour and the model is fine-tuned on them. Then people rank its own outputs against each other and reinforcement learning pushes it towards the higher-ranked ones: RLHF, reinforcement learning from human feedback.

The effect is easy to underrate. Ouyang and colleagues at OpenAI reported in March 2022 that human evaluators preferred the answers of a 1.3-billion-parameter model that had been through this process to those of the 175-billion-parameter GPT-3 that had not: a model a hundred times smaller, preferred. Almost all of what users recognise as a chatbot’s character comes from this stage, not the first.

Why an answer costs money

Reading a prompt and writing a reply are not the same job. The prompt can be worked through in one pass; the reply comes one token at a time, each depending on the one before. The price lists show the gap: across Anthropic’s current line output costs exactly five times input — $4 against $20 per million for Opus 5.5, $0.10 against $0.50 for Haiku 5.5.

What limits generation is usually not arithmetic but moving numbers. For every token, the parameters involved have to be fetched from memory into the chip. An H100 holds 80 GB and reads it at 3.35 terabytes a second, at up to 700 watts, and that bandwidth — more than the chip’s arithmetic — sets how fast tokens come out. Hence the two standard discounts, both about avoiding work: a cache hit on text already processed costs 10 percent of the normal input price, and batching requests instead of waiting on them costs half.

One answer, in absolute terms, is small. Google published measured figures in August 2025, from May 2025 data: the median Gemini text prompt used 0.24 watt-hours, 0.03 grams of CO₂ equivalent and 0.26 millilitres of water, and that median had fallen 33-fold in energy over twelve months. The per-question figure is public; the number of questions is not.

671 billion parameters, 37 of them working

One trick breaks the assumption that a bigger model costs more to run. In a mixture-of-experts design the network splits into many sub-networks and a router sends each token to only a few. DeepSeek-V3, documented in December 2024, holds 671 billion parameters but activates 37 billion for any given token; it was trained on 14.8 trillion tokens for 2.788 million GPU-hours. The model is large where capacity is cheap — memory — and small where it is dear, in the arithmetic per token.

Why it invents things

A model is always guessing, and nothing in the loop separates a guess that happens to be right from one that isn’t. Temperature, the setting that controls how adventurously it picks among likely tokens, makes this visible: turn it up and the same prompt yields different phrasing. Anthropic notes that even at temperature zero identical inputs can produce different outputs.

In September 2025 Kalai, Nachum, Vempala and Zhang, three of them at OpenAI, argued something sharper: the guessing is trained in. Hallucinations, they wrote, are ordinary errors in binary classification, and the way models are scored rewards them. The benchmarks that decide whose model looks best mark an answer right or wrong and give nothing for “I don’t know”, so a confident guess always scores at least as well as admitting ignorance — and the model that guesses wins the leaderboard. Their fix is not a better hallucination detector but a change to how the existing tests are marked.

Which puts the thing in an unflattering light. We built a machine that guesses, graded it in a way that punishes saying so, and then called the result a liar.