Have you noticed an AI forgetting instructions from the start of a long conversation? That's usually down to tokens and context windows.
What's a token?
Models don't read whole words. They read tokens, which are chunks of text. In English, a token is roughly three-quarters of a word, so 1,000 tokens is about 750 words.
What's a context window?
The context window is how much text the model can consider at once, including your messages, its replies and any files. Modern models have large windows, but they're still limited, and very long contexts can reduce attention to details.
Signs you've hit the limit
- The AI ignores earlier instructions.
- It forgets details from documents you shared.
- Answers become repetitive or inconsistent.
How to work around it
- Start a new chat for new tasks.
- Summarise a long conversation and paste the summary into a fresh chat.
- Put the most important instructions at the start and repeat them when needed.
- Share only the relevant parts of long documents.
- Use projects or custom instructions for permanent background.
Why it matters for cost
For developers using APIs, tokens are also how usage is billed, so concise prompts save money.