Managing Small Context Windows in Language Models
Three practical strategies for managing small LLM context windows, with Python examples covering sliding windows and token budgeting with RAG.
In this article, you will learn three practical strategies for managing small context windows in large language models, along with working Python examples that demonstrate how two of those strategies are implemented.
Topics we will cover include:
- How context truncation via the sliding window approach keeps token usage flat and predictable.
- How token budgeting combined with retrieval-augmented generation ensures only the most relevant context fits within a prompt.
- A concise overview of additional strategies for more specialized use cases — rolling summaries, prompt compression, and observation masking.
Introduction
Top-tier AI industries have become somewhat obsessed with language models capable of ingesting massive context windows — for example, an entire book in a single prompt. However, what they won’t admit easily is that in real-world LLM applications, these massive context windows come with various limitations and challenges, including soaring API costs, unacceptable response times, and even worse, the so-called “lost in the middle” problem whereby a model ignores data deeply buried in the middle of the giant prompt. No surprise, then, that working with small yet smartly managed context windows could yield superior outcomes, reducing latency, minimizing costs, and forcing the model to concentrate on what truly matters to generate its response.
This article unveils three of the most widely adopted practical strategies for managing and mastering small context windows in language models, along with examples that mimic the implementation of some of them for better understanding.
Context Truncation: Sliding Window
There is a consensus that sliding windows are arguably the most common and simplest strategy for managing shortened context windows in language models. Instead of providing an entire user conversation history to the model, the context is treated as a FIFO (First-In-First-Out) queue: as new interactions (exchanged messages) come in, the oldest ones are simply dropped. All it takes is defining the size of the context window and striking a balance between sufficient past context retention and latency-cost control.
The main advantage of truncating the context via sliding windows is absolute control and predictability over token usage and computing overhead. The maximum number of interactions dealt with by the model at a given time remains fixed, keeping latency flat and surprise-free.
To better understand how this approach works, look at the following Python code in which you can freely adjust the value of max_turns (context window size) and see how it affects the “memory” injected into the current prompt:
class SlidingWindowMemory:
def __init__(self, max_turns=3):
"""Keep only the last `max_turns` of a conversation."""
self.max_turns = max_turns
self.history = []
def add_interaction(self, user_text, ai_text):
self.history.append({"user": user_text, "ai": ai_text})
# The logic behind a sliding window: drop the oldest turns if limits are surpassed
if len(self.history) > self.max_turns:
self.history = self.history[-self.max_turns:]
def build_prompt(self, new_query):
prompt = "System: Answer concisely based on recent context.\n\n"
for turn in self.history:
prompt += f"User: {turn['user']}\nAI: {turn['ai']}\n"
prompt += f"User: {new_query}\nAI:"
return prompt
# --- Testing the Sliding Window mechanism: feel free to adjust the value of max_turns ---
memory = SlidingWindowMemory(max_turns=2)
# Simulating a long conversation
memory.add_interaction("Hi, I'm learning Python.", "Great choice!")
memory.add_interaction("What are lists?", "Lists are mutable arrays.")
memory.add_interaction("Can they hold mixed types?", "Yes, they can.")
# The prompt will only contain the last 'max_turns' interactions, saving tokens
print(memory.build_prompt("How do I append to one?"))Output:
System: Answer concisely based on recent context.
User: What are lists?
AI: Lists are mutable arrays.
User: Can they hold mixed types?
AI: Yes, they can.
User: How do I append to one?
AI:You can also try extending the conversation history by appending new memory.add_interaction() calls with extra query-response pairs of your own, to test the mechanism for larger context windows.
Token Budgeting and RAG (Retrieval-Augmented Generation)
RAG systems supplement LLMs with engines that reference and retrieve external documents to enrich the original user prompt with founded, relevant context. Small context windows may intuitively force a ruthless attitude toward the data to include in the context. To address this, token budgeting splits the context window into zones with strict limits per zone. For instance, a token budgeting criterion could allow up to 20% of the context for system instructions, 20% for the chat history (including the latest user query), and the remaining 60% for retrieved data. This incorporates a more dynamic retrieval and data chunking behavior, halting insertion as soon as budget limits are hit.
The main advantage of token budgeting is preventing unduly large retrieved documents from quickly exhausting the prompt and ensuring only highly relevant, concentrated information is included, thus avoiding side issues like the aforementioned “lost in the middle” problem.
The following code exemplifies the use of this mechanism in Python, using a simple word count as a free, lightweight proxy for token budgeting. To make it more realistic, you could apply the commonly accepted heuristic of 1 word = 1.3 tokens on average. The loop inside the function shows how to reliably pack a prompt without surpassing enforced limits:
def build_budgeted_prompt(system_prompt, retrieved_chunks, user_query,
chat_history="", total_word_budget=300):
# Zone allocations as fractions of total budget
system_budget = int(total_word_budget * 0.20)
history_budget = int(total_word_budget * 0.20)
retrieval_budget = int(total_word_budget * 0.60)
def word_count(text):
return len(text.split())
# Truncate system prompt if needed
system_words = system_prompt.split()
if len(system_words) > system_budget:
system_prompt = " ".join(system_words[:system_budget])
# Truncate chat history if needed
history_words = chat_history.split()
if len(history_words) > history_budget:
chat_history = " ".join(history_words[-history_budget:])
# Pack retrieved chunks until retrieval budget is exhausted
packed_chunks = []
used = 0
for chunk in retrieved_chunks:
chunk_wc = word_count(chunk)
if used + chunk_wc > retrieval_budget:
break
packed_chunks.append(chunk)
used += chunk_wc
retrieval_block = "\n---\n".join(packed_chunks)
prompt = (
f"[System]\n{system_prompt}\n\n"
f"[Retrieved Context]\n{retrieval_block}\n\n"
f"[Chat History]\n{chat_history}\n\n"
f"[User Query]\n{user_query}\n\n"
f"[Answer]"
)
return prompt
# --- Example usage ---
system_prompt = (
"You are a helpful assistant. Answer only from the provided context. "
"Be concise and factual."
)
retrieved_chunks = [
"Python lists are ordered, mutable collections that can store elements of any type.",
"To append an element to a list, use the list.append(element) method.",
"Lists support slicing, indexing, and a variety of built-in methods such as sort() and reverse().",
"Dictionaries in Python store key-value pairs and are unordered as of Python 3.6 and earlier.",
"Tuples are immutable sequences, often used for fixed collections of items.",
]
chat_history = "User: What is Python?\nAI: Python is a high-level programming language."
user_query = "How do I add an item to a Python list?"
prompt = build_budgeted_prompt(
system_prompt, retrieved_chunks, user_query,
chat_history=chat_history, total_word_budget=300
)
print(prompt)Output:
[System]
You are a helpful assistant. Answer only from the provided context. Be concise and factual.
[Retrieved Context]
Python lists are ordered, mutable collections that can store elements of any type.
---
To append an element to a list, use the list.append(element) method.
---
Lists support slicing, indexing, and a variety of built-in methods such as sort() and reverse().
---
Dictionaries in Python store key-value pairs and are unordered as of Python 3.6 and earlier.
---
Tuples are immutable sequences, often used for fixed collections of items.
[Chat History]
User: What is Python?
AI: Python is a high-level programming language.
[User Query]
How do I add an item to a Python list?
[Answer]Notice how the five retrieved chunks all fit within the 60% retrieval budget, while the system prompt and chat history each stay within their respective 20% allocations. Adjusting total_word_budget or the zone fractions lets you tune the balance between context richness and token efficiency for your specific use case.
Additional Strategies
Beyond sliding windows and token budgeting with RAG, several other strategies are worth knowing for more specialized scenarios.
Rolling summaries replace older turns in conversation history with a compressed summary rather than discarding them entirely. This preserves semantic continuity across long conversations while keeping the token footprint small — the model effectively carries a “memory digest” that grows in meaning without growing in length.
Prompt compression techniques use a secondary model or heuristics to rewrite retrieved documents and chat history into shorter forms before they enter the prompt. Methods range from simple extractive summarization to learned compression models that distill paragraphs into dense, information-rich sentences.
Observation masking is particularly relevant in agentic and tool-use settings, where intermediate tool outputs — such as raw API responses or lengthy search results — can consume tokens quickly. By filtering or truncating these observations before they are appended to the context, the system ensures that only the actionable portions of each tool call are retained.
Each of these strategies can be combined with the sliding window and token budgeting approaches described above, and the right mix depends on the latency tolerance, cost constraints, and conversational depth requirements of a given application.