Coding agents context management

how do coding agents never run out of context and know how to condense things into smaller subsets to be sent in additional llm calls, what kind of logic is used on a programming level

Coding agents **do** run out of context — they just avoid *acting like they did* by continually compressing, chunking, and regenerating their working memory. Under the hood, they use a combination of **hierarchical memory**, **summarization pipelines**, **embedding retrieval**, and **state machines** that decide what to load and when. Below is the actual logic agents use *at the programming level*, without the marketing fluff. --- # ✅ **How coding agents prevent “context overflow” (real mechanisms)** There are **4 layers** of memory management used in modern agents: --- # **1. Sliding Window Context (short-term memory)** Agents treat the LLM’s context window like a *moving tape*. When the prompt grows too large: ### **Algorithm:** ```python if token_length(prompt) > MAX_CONTEXT: drop_oldest_messages() ``` This is the simplest mechanism. But it’s not enough, so more intelligent approaches came next. --- # **2. Hierarchical Summarization (compressed short-term memory)** When conversation history gets too large, agents **summarize old states** into compact structured notes. This is hierarchical summarization: ``` Full text → Dense summary → Ultra-dense summary → Key facts ``` ### Example mechanism: ```python if token_length(buffer) > THRESHOLD: summary = llm("Summarize the following preserving all action items...", buffer) memory.add(summary) buffer = [] ``` This summary becomes a small ~200-token memory that fits forever. This is used in: - OpenAI’s AutoGPT arena agents - Anthropic’s “ReAct + memory” agents - Microsoft Autogen - SmolAgents from HuggingFace - Devin clones --- # **3. Vector Memory (long-term memory via embeddings)** Instead of storing text, agents store **semantic embeddings**. ### How it works: 1. Split all text into chunks 2. Generate embeddings 3. Store embeddings in vector DB (Chroma, Faiss, LanceDB, etc.) 4. Retrieve only the relevant chunks for each new step. ### Pseudocode: ```python embedding = embed(query) results = vector_db.search(embedding, top_k=5) context = concatenate(results) response = llm(context + query) ``` This means the agent **never needs to load the entire project**—only the relevant pieces. Used in: - ReAct agents - GraphRAG - LangChain - MemoryGPT - All serious code agents --- # **4. Plan → Execute → Reflect (state machine)** Coding agents don’t keep the whole problem in memory. They generate a **plan**, then execute small pieces. ### Example: ``` PLAN: 1. Create file structure 2. Build Django backend 3. Add API routes 4. Add auth 5. Build React frontend 6. Connect APIs 7. Write tests ``` At each step, the agent loads only: - the relevant source files - the goal for that step - the reflection from previous steps ### Pseudocode: ```python for step in plan: context = summarize(previous_steps) + get_relevant_files(step) action = llm(context + step_instruction) execute(action) reflect(action) ``` This is the core idea behind “infinite context”. --- # ⚙️ **The real programming patterns agents use** ## **A. ReAct Loop (Reasoning + Acting)** ``` Thought → Action → Observation → Thought → Action... ``` Each loop only brings in **what is needed**. --- ## **B. Toolformer pattern (Call tools, not recall text)** Agents offload work to tools: - file system queries - search functions - AST parsers - code analyzers - Git diffs - documentation lookups This means the *LLM never holds everything simultaneously*. --- ## **C. Graph-Based Memory (GraphRAG or agent graphs)** Your own project uses this idea: Each entity becomes a node: - Concepts - Functions - APIs - Modules - Past errors - Fix patterns The agent only loads the subgraph that matters to the current task. --- # 🤯 **The Core Idea: Agents don’t compress everything — they avoid loading everything at once** Agents survive tiny context windows by: - **Chunking problems** - **Summarizing older text** - **Extracting key facts** - **Storing long-term memory in a vector DB** - **Planning steps** - **Running a state machine** - **Pulling relevant code only when needed** They behave like programmers: > You don’t keep the whole codebase in your head; you jump to files as needed. --- # 🔧 Example: Real Logic From a Code Agent Here’s what a coding agent's memory manager actually looks like: ```python class AgentMemory: def __init__(self): self.short_buffer = [] self.summary_memory = [] self.vector_store = Chroma() def save(self, msg): self.short_buffer.append(msg) if self.size(self.short_buffer) > 2000: summary = llm(f"Summarize this into actionable memory: {self.short_buffer}") self.summary_memory.append(summary) self.vector_store.add(summary) self.short_buffer = [] def retrieve(self, goal): embedding = embed(goal) return self.vector_store.query(embedding, top_k=5) ``` And a working prompt build: ```python def build_prompt(goal): relevant = memory.retrieve(goal) return combine([ system_instructions, memory.summary_memory, relevant, recent_interactions, goal ]) ``` That is how agents maintain “unlimited” context. --- # If you want, I can also show: ✅ The architecture diagram of a full memory-managed coding agent ✅ A minimal open-source implementation (Python) ✅ How to build this using SmolAgents, Autogen, or a custom framework ✅ How to integrate this into your GraphRAG setup ✅ Or build your own from scratch (200 lines, extremely powerful) Would you like a **diagram**, **code**, or **in-depth explanation of any layer**?