Ask a local model something on message forty of a long conversation and it may have quietly forgotten what you said on message three. Not because it is small or badly made, but because of a hard architectural limit called the context window.
It is the single most useful concept for understanding why AI assistants behave the way they do — local or cloud — and it explains several behaviours that otherwise look like bugs.
This article sits under How LLMs Actually Run on Your iPhone.
The model has no memory between tokens
Start here, because everything else follows from it: a language model does not remember anything.
It is a function. Text goes in, a prediction for the next token comes out. It has no internal state carried between calls, no notion of “earlier in the conversation,” no storage.
What creates the illusion of memory is that the entire conversation is resent every single time. When you type your fifth message, the app doesn’t send just that message — it sends the system prompt, all four previous exchanges, and your new message, as one continuous block of text. The model reads all of it fresh and predicts what comes next.
Continuity is a property of the input, not the model.
The context window is simply the maximum size of that input block. Everything competes for the same budget: system prompt, full conversation history, any attached document or image description, and the space for the reply itself.
Tokens, not words
Context windows are measured in tokens — sub-word chunks produced by the tokenizer. Common words are usually one token; longer or rarer words split into several.
- “the” → 1 token
- “unbelievable” → 3 tokens (
un,belie,vable) - ”🙂” → 1–2 tokens
getUserById→ 4–5 tokens
Useful conversions for English prose:
| Tokens | Approx. words | Rough equivalent |
|---|---|---|
| 1,000 | 750 | A long email |
| 4,000 | 3,000 | A short article |
| 8,000 | 6,000 | A long article |
| 32,000 | 24,000 | ~50 paperback pages |
Two caveats worth knowing. Code burns tokens fast — punctuation, indentation, and identifiers fragment heavily, so a code-heavy conversation fills a window perhaps twice as fast as prose. Non-Latin scripts cost more for most tokenizers; the same meaning in Chinese, Japanese, or Arabic typically consumes more tokens than in English.
The KV cache: where the window costs you memory
The context window isn’t only a policy limit. It has a physical cost, and on a phone that cost is the binding constraint.
Each token generated must attend to every token before it. Recomputing that history for every new word would be intolerably slow, so intermediate results — the keys and values — are cached.
That cache grows linearly with conversation length. Every token added means more cached state, permanently, for the life of the conversation.
The consequences are direct:
Memory. On a larger model, a full 32,000-token context can cost more than a gigabyte of KV cache — on top of the model weights themselves. This is why a model that loads comfortably on your device can still run into trouble deep into a very long conversation.
Speed. More cached history means more memory read per token, and inference is bound by memory bandwidth. Generation slows gradually as the conversation grows. Starting a fresh chat restores it immediately.
If a long conversation feels sluggish, that is the mechanism — not the model degrading, and not the device failing.
What actually happens at the limit
When a conversation exceeds the window, the oldest content is dropped from the input. This is worth being precise about, because the failure mode is subtle.
The model does not know something was removed. It receives a conversation that appears to begin at whatever point the truncation left, and it responds with complete confidence. So you get:
- Instructions given early in a long chat quietly stop being followed
- A name or constraint established at the start is forgotten
- The model contradicts something it “said” an hour ago
- Referring back to “the list from earlier” produces an invented list
None of this reads as an error. It reads as the model being unreliable — which is why knowing the mechanism matters. When a long conversation starts going sideways, start a fresh one and restate the essentials. It is almost always the right move, and it also restores full generation speed.
Where local and cloud genuinely differ
This is one of the areas where cloud models hold a real advantage, and it is worth stating plainly rather than glossing over.
Frontier cloud models offer context windows in the hundreds of thousands of tokens. They can do this because they run on hardware with enormous memory and can afford the KV cache. A phone cannot. The cache for a several-hundred-thousand-token context would exceed the total RAM in the device.
So if your work genuinely requires reasoning across a whole book or a large codebase in a single pass, a local model on a phone is the wrong tool, and no amount of enthusiasm changes that. On-Device AI vs Cloud AI covers the full set of trade-offs honestly.
What local models do offer is a different kind of memory. Because everything is stored on your device, an app can maintain persistent notes and per-project instructions across conversations without any of it leaving the phone. Cloaked’s projects and memory features work this way: a project’s custom system prompt is re-supplied to every conversation in that project, so continuity survives without needing a single enormous window. It is a different shape of solution to the same problem — and one where nothing gets uploaded to make it work.
Practical habits that make the limit a non-issue
Start fresh for new topics. New conversations are faster and less prone to stale-context confusion. There is no benefit to one endless thread.
Front-load what matters. Put durable instructions in a project’s system prompt rather than in a message that can scroll out of the window.
Restate before long tasks. Before asking for something demanding, briefly restate the key constraints. Cheap insurance.
Paste the relevant excerpt, not the whole document. Two thousand words of the section that matters beats twenty thousand words of everything.
Watch for the tell. When a model starts ignoring an instruction it was following happily, suspect the window before suspecting the model.
For how context length interacts with generation speed, see Tokens Per Second Explained. For how reasoning models spend extra context on their own thinking, see Thinking Mode Explained.
Download Cloaked on the App Store — every conversation, project, and memory stays on your device, with no server to hold it.
Frequently asked questions
How many words is a 32,000-token context window?
About 24,000 words of English, or roughly 50 pages of a paperback. Tokens are sub-word chunks, and English averages about 0.75 words per token. Code and non-Latin scripts consume tokens faster, so the effective word count is lower for both.
What happens when a conversation exceeds the context window?
The oldest messages drop out of the model's view. It does not summarize them or remember them vaguely — they are simply gone from the input. The model will confidently continue as though that earlier part of the conversation never happened.
Why does a long conversation get slower?
Each new token must attend to every previous token, and the cached history it attends to grows with the conversation. More cached history means more memory read per token, and generation speed drops accordingly. Starting a fresh conversation restores full speed.
Do cloud models have bigger context windows?
Yes, substantially — frontier cloud models offer hundreds of thousands of tokens. The constraint is memory: on a phone, the key-value cache for a very long context can exceed the model itself. This is a genuine advantage of cloud inference.