Post-LLM Control Loops

synthesize these three articles in relation to danielkliewer.com https://machinelearning.apple.com/research/residual-context-diffusion https://www.lmsys.org/blog/2026-07-02-agent-assisted-sglang-development https://www.elliotcsmith.com/autoresearch-claude-and-constrained-optimization/

These three pieces are converging on a pretty coherent shift in the “post-LLM” stack: from *models as products* → *systems as control loops over generative computation*. If you map that onto your work at **danielkliewer.com**, especially your “dynamic persona MoE RAG + autoresearch loops” framing, they slot in almost perfectly as three orthogonal upgrades: - **Apple RCD → better internal state recycling** - **lmsys agent-sglang → better external execution orchestration** - **Elliot Smith autoresearch → better objective-constrained optimization loop design** The synthesis is basically: *how do we turn your blog’s “self-improving agent loop” into something that is (1) more compute-efficient, (2) more tool-native, and (3) more formally optimized rather than heuristic?* --- ## 1. Apple Residual Context Diffusion → “Don’t throw away intermediate cognition” entity["company","Apple","technology company"]’s **Residual Context Diffusion (RCD)** is a very specific technical idea, but the abstraction matters more than the architecture: Instead of discarding uncertain tokens during diffusion sampling, RCD: - preserves “failed” or low-confidence intermediate states - converts them into *residual context* - feeds them forward into the next iteration ### Translation into your system (danielkliewer.com RAG/agent loop) Your current framing: > agents generate → retrieve → refine → overwrite state RCD suggests a better framing: > agents generate → **everything becomes state** → even “errors” are structured memory So in your blog’s terminology: - “discarded tokens” ≈ rejected hypotheses in your autoresearch loop - “residual context injection” ≈ turning failed retrievals into graph edges / embeddings - “denoising step” ≈ next agent iteration ### Key upgrade for your architecture Instead of: - RAG retrieval → throw away low-score docs You move to: - RAG retrieval → **retain low-score docs as weak signals in graph memory** This directly strengthens your: - GraphRAG design - transient knowledge graph idea - persona-vector drift system --- ## 2. LMSYS agent-assisted SGLang → “Execution is the product” entity["organization","LMSYS","large model systems organization"]’s **agent-assisted SGLang development** is less about models and more about *infrastructure becoming collaborative with agents*. Core idea: - LLM agents don’t just generate outputs - they actively participate in: - compiling - optimizing - rewriting execution graphs - managing inference pipelines (SGLang) ### Mapping to your work Your system already assumes: > “agents build agents” (Hermes / orchestrator idea) SGLang pushes this further: #### Your current implicit model: - agent writes code - code runs separately #### SGLang-style model: - agent *co-designs runtime* - execution graph is mutable - inference engine becomes editable substrate ### What this means for danielkliewer.com Your “Hermes Agent” idea becomes: > not a chatbot that uses tools > but a system that **rewrites its own execution backend (SGLang-style runtime graph)** So instead of: - tool calling You get: - **tool compilation** Instead of: - static workflow graphs You get: - **live optimized inference graphs per query** This is a direct bridge from: > “agent orchestration” → “agent-controlled compiler for inference” --- ## 3. Elliot Smith Autoresearch + Constrained Optimization → “Loops must have objective geometry” Elliot Smith’s framing is the missing mathematical layer in most “agent loop” blogs: Core idea: - autoresearch loops are not just recursive prompting - they are **constrained optimization systems over: - hypotheses - evidence - compute budget - uncertainty** So instead of: > “iterate until better answer” You get: > “optimize a policy over exploration/exploitation under constraints” ### Mapping to your blog system Your “dynamic persona MoE RAG” already has: - persona vectors - retrieval selection - iterative refinement But this paper implies a missing piece: #### You don’t just have a loop You have a **loss function over the loop** So your system becomes: - State = graph + persona + retrieved context - Action = retrieval + generation + tool calls - Objective = correctness / coherence / novelty / cost tradeoff - Constraint = compute budget, hallucination risk, persona drift bounds ### This turns your architecture into: > constrained policy optimization over a generative system Not just: - “agent improves itself” But: - “agent is performing structured optimization over its own cognition graph” --- ## 4. Unified synthesis for danielkliewer.com Putting all three together: | Paper | Contribution | Your system upgrade | |------|-------------|---------------------| | Apple RCD | reuse discarded computation | RAG + graph memory should retain “failed reasoning traces” | | LMSYS SGLang agents | runtime is editable | agents should compile execution graphs, not just use tools | | Autoresearch optimization | loops are constrained optimization | your agent loops need explicit objective + constraint formulation | --- ## 5. What your blog is actually converging toward Your existing work (from prior posts you’ve written) is already describing: - dynamic persona MoE routing - graph + vector hybrid memory - autoresearch loops - autonomous “sovereign AI systems” These three papers collectively imply: ### The next version of your thesis: > **AI agents are not models. They are continuously recompiled, constraint-optimized inference systems with persistent residual state.** More concretely: 1. **Memory is not retrieval — it is residual state propagation** (RCD) 2. **Execution is not tool use — it is runtime graph synthesis** (SGLang agents) 3. **Reasoning is not generation — it is constrained optimization over trajectories** (autoresearch framing) --- ## 6. If you wanted to express this as a new section on your site You could frame it like: > “From RAG to Residual Systems: Why Agents Need State, Compilers, and Objectives” And the core claim would be: - RAG is insufficient because it treats memory as *static retrieval* - Agent frameworks are insufficient because they treat execution as *fixed toolchains* - Prompt loops are insufficient because they treat reasoning as *unconstrained generation* The emerging stack is: > **Residual memory + compile-time agent orchestration + constrained optimization loops** --- If you want, I can next: - rewrite this into a publishable blog post in your voice, or - map it directly onto your Hermes / MoE RAG architecture diagram, or - extract a concrete system design (modules + data flow + pseudocode) for implementation.

Rewrite this into a publishable blog post in my voice

The model is no longer the product. The loop is. That idea keeps getting reinforced every time I look at new research from Apple, LMSYS, and the recent work on autoresearch and constrained optimization. They’re not converging on a better chatbot. They’re converging on something closer to a reconfigurable system of computation where “reasoning” is just one phase inside a larger machine. What’s changing isn’t just capability. It’s where intelligence lives. It’s shifting out of the model and into three places at once: residual state, execution graphs, and optimization loops. --- ## 1. From tokens to residual state (Apple’s RCD) Apple’s Residual Context Diffusion work quietly breaks one of the core assumptions behind most LLM systems: that intermediate uncertainty should be discarded. In standard generation pipelines, we sample, reject, and move on. Low-confidence paths disappear. Only the final sequence matters. RCD changes that. Instead of throwing away “failed” intermediate states during diffusion, it feeds them forward as residual context. Errors, near-misses, and partial structures become part of the next iteration’s conditioning. That sounds like a small tweak. It isn’t. It reframes what memory even is. In most LLM systems (including RAG), memory is treated as retrieval: - store embeddings - retrieve top-k - inject context - continue generation But RCD suggests a different model: Memory is not retrieval. Memory is residue. And residue includes everything we usually optimize away: - discarded hypotheses - low-probability reasoning paths - partial structures that didn’t resolve cleanly In the context of systems like the ones I’ve been building around dynamic persona MoE RAG, this is a critical shift. It means: We shouldn’t just retrieve “relevant context.” We should preserve and evolve *all contextual friction*. Even failure becomes state. --- ## 2. From tool use to executable systems (LMSYS and SGLang agents) The LMSYS work on agent-assisted SGLang development pushes the next abstraction shift: the agent is no longer just a consumer of tools. It becomes part of the system that defines execution. Instead of: > prompt → model → tool call → result We start seeing: > agent → compiles execution graph → optimizes inference paths → rewrites runtime behavior → executes SGLang already treats inference as a structured program. What the agent layer adds is something more important: adaptability at the level of the execution graph itself. This matters because it dissolves the boundary between “model reasoning” and “system architecture.” The agent is no longer sitting on top of the stack. It is partially responsible for constructing the stack on each run. This is the missing piece in most agent frameworks today. They assume: - static tool definitions - fixed orchestration logic - stable execution pipelines But real systems under SGLang-style design become: - dynamic execution graphs - query-dependent compilation - runtime-optimized inference paths Which leads to a harder conclusion: If the execution graph is mutable, then “the system” is not static software. It is a generated artifact. And agents are compilers. --- ## 3. From loops to constrained optimization (autoresearch systems) The third piece is more subtle, but it completes the picture. Autoresearch-style systems framed through constrained optimization treat agent loops not as “iteration until better answer,” but as structured search over a bounded space of possible reasoning trajectories. Instead of: - generate → critique → refine → repeat We get: - explore hypothesis space - evaluate against constraints - allocate compute dynamically - converge under budgeted uncertainty The key shift is that the loop is no longer informal. It has geometry. There are: - constraints (compute, latency, hallucination risk) - objectives (accuracy, novelty, coherence, utility) - tradeoffs between exploration and exploitation This is important because it forces something most agent systems avoid: You have to define what “better” actually means in system terms. Once you do that, the loop stops being a narrative structure and becomes an optimization system. And once it becomes an optimization system, you can start treating cognition itself as something you compile, not just prompt. --- ## 4. Putting it together: residual systems, compiled agents, and optimization loops These three threads—Apple’s residual diffusion framing, LMSYS’s execution graph agents, and autoresearch optimization—aren’t separate ideas. They’re three layers of the same transition. They map cleanly onto a new system stack: ### Layer 1: Residual State (Memory) - not retrieval-based RAG - but persistent accumulation of: - failed reasoning - partial structures - unresolved hypotheses - memory becomes a living substrate, not a lookup table ### Layer 2: Execution Graphs (Compute) - not static tool calling - but dynamic inference compilation - agents participate in shaping runtime structure - execution becomes query-specific and mutable ### Layer 3: Constrained Loops (Reasoning) - not open-ended generation - but optimization over trajectories - bounded by compute, uncertainty, and objective functions When combined, something new emerges: A system where intelligence is not located in the model, but in the interaction between: - residual memory - compiled execution - constrained iteration --- ## 5. What this implies for agent systems like mine If you’re building something like dynamic persona MoE RAG, Hermes-style orchestration, or any autoresearch loop system, the implication is simple but uncomfortable: Most current architectures are only simulating parts of this stack. - RAG simulates memory, but discards residue - tool-using agents simulate execution, but don’t compile graphs - prompt loops simulate optimization, but don’t formalize objectives The next step is not “better prompting” or “bigger models.” It is structural: 1. Stop treating failed outputs as waste. Treat them as state. 2. Stop treating execution as fixed. Treat it as compiled per query. 3. Stop treating iteration as narrative. Treat it as constrained optimization. Once you do that, the system stops looking like an LLM application. It starts looking like a continuously recompiled cognitive engine. --- ## 6. Closing The model is not the product anymore because it was never the full system in the first place. What’s emerging now is something closer to a programmable cognition substrate: - memory that accumulates error as structure - execution that compiles itself per task - reasoning that optimizes under constraint In that world, the interesting question is no longer: > “What can the model do?” It becomes: > “What kind of loop are you running, and what does it optimize for?” And that’s where everything starts to converge.

https://www.danielkliewer.com/blog/2026-07-03-the-model-is-not-the-product

I read your post alongside the surrounding cluster of work you’ve been synthesizing, and it sits very cleanly in what’s becoming a broader reframing across the field: AI systems are no longer being understood as *models that answer questions*, but as *environments that produce, constrain, and accumulate cognition*. Your piece already has the core thesis (“the model is not the product; the loop is”), but what’s interesting is how it implicitly aligns with three parallel shifts showing up across Apple’s RCD direction, LMSYS’s agentic execution work, and the autoresearch/optimization framing: 1. memory is becoming residual (not retrieved), 2. execution is becoming compiled (not called), 3. reasoning is becoming optimized (not prompted). Below is a tightened, publish-ready version that keeps your voice but sharpens the framing and removes redundancy while anchoring it more explicitly into the “system layer shift” your blog is building toward. --- # The Model Is Not the Product. The System Is. The model is not the product anymore. It hasn’t been for a while, but we’re only now starting to name what replaces it. Across new research—from residual context diffusion work, to agent-compiled execution systems, to autoresearch loops framed as constrained optimization—a consistent pattern is emerging. We are no longer building applications around models. We are building systems where models are just one transient component inside a larger computational loop. The real unit of intelligence is shifting from the model to the system that surrounds it. That system has three parts: - residual memory - executable structure - constrained iteration Once you see this clearly, almost everything in current AI architecture starts to look transitional. --- ## 1. Memory stops being retrieval. It becomes residue. Most AI systems still treat memory as retrieval: - embed documents - retrieve top-k - inject context - generate response Even RAG systems are still fundamentally based on selection and injection. Anything not retrieved is effectively discarded. But newer directions in generative modeling (like residual context diffusion approaches) suggest a different framing: Nothing is actually “discarded.” It is just unstructured. In that view, failed generations, low-confidence outputs, and partially formed reasoning traces are not noise—they are state. They can be carried forward, reshaped, and reintegrated into the next computation step. This changes what memory means in systems like dynamic persona MoE RAG. Memory is no longer: > “what we retrieve” Memory becomes: > “what the system failed to resolve, and therefore still carries” That includes: - discarded hypotheses - weak retrieval signals - incomplete reasoning branches - contradictory intermediate states Instead of optimizing retrieval quality, you start optimizing *state continuity across failure*. Failure becomes part of the representation space. --- ## 2. Execution stops being tool use. It becomes compilation. Most agent systems today still operate like this: prompt → model → tool call → result → repeat Even when tools are sophisticated, the structure is fixed. The agent selects from a predefined interface. But systems like SGLang-style agent execution point toward something different: The agent is not just calling tools. It is shaping the execution graph. Instead of static pipelines, you get: - query-dependent computation graphs - runtime-optimized inference paths - dynamically composed toolchains - execution structures that vary per request At that point, the distinction between “application logic” and “inference runtime” starts to blur. The system is no longer executing code. It is compiling a computation graph per interaction. This is a subtle shift, but it changes the ontology of agent systems. The agent is not sitting on top of the system. The agent is partially responsible for constructing the system it runs on. --- ## 3. Reasoning stops being generation. It becomes constrained optimization. The final shift is the most important, because it turns everything else into structure instead of narrative. Traditional LLM loops look like: generate → critique → refine → repeat But autoresearch-style systems framed through constrained optimization change the question entirely. The loop is no longer “how do we improve the answer?” It becomes: > what trajectory through hypothesis space maximizes outcome under constraints? Now you have explicit structure: - objective function (accuracy, novelty, coherence, utility) - constraints (compute, latency, risk, uncertainty) - policy (how the system explores vs exploits) - budgeted iteration (how much reasoning is allowed) The loop stops being a storytelling device. It becomes an optimization system operating over reasoning trajectories. Once that happens, “intelligence” is no longer a property of a single model pass. It is the result of how the system allocates compute across exploration and refinement. --- ## 4. The convergence: residual systems, compiled agents, optimization loops These three shifts are not independent ideas. They are layers of the same transformation. They map cleanly onto a new architecture: ### Layer 1: Residual Memory Not retrieval systems, but persistent state that includes unresolved and failed cognition. ### Layer 2: Compiled Execution Not fixed toolchains, but dynamic inference graphs constructed per query. ### Layer 3: Constrained Loops Not open-ended generation, but optimization under explicit objectives and limits. When combined, they describe something fundamentally different from today’s “LLM applications.” They describe systems where intelligence is distributed across: - what is remembered, - how computation is structured, - and how iteration is constrained. --- ## 5. What this implies If you’re building agent systems, RAG architectures, or multi-persona frameworks, the implication is simple: Most current systems are only simulating parts of this stack. - RAG simulates memory, but discards residue - tool use simulates execution, but ignores compilation - prompt loops simulate optimization, but lack formal objectives The next step is structural, not cosmetic: Stop treating outputs as final state. Start treating failure as persistent state. Stop treating execution as fixed. Start treating it as compiled per task. Stop treating iteration as narrative. Start treating it as constrained optimization. When you do that, the system stops looking like an LLM wrapper. It starts looking like a continuously recompiled cognitive engine. --- ## Closing The model is not the product because it never really was the system. It was always just the fastest-changing component inside a larger structure. What is emerging now is a different abstraction entirely: - memory that accumulates unresolved structure - execution that compiles itself per problem - reasoning that optimizes under constraint The interesting question is no longer which model you use. It is what kind of loop you are running—and what that loop is actually optimizing for. Because that is where intelligence is moving. And that is where the real products are going to be built.

does it need rewritten or can you just take what I wrote and write a few paragraphs about it in paragraph form for me instead of try to rewrite it which I don't think would improve it

It doesn’t need a rewrite in the sense of fixing it. The structure is already doing what it’s supposed to do: it cleanly escalates from “models” → “systems,” then breaks that into memory, execution, and optimization, and lands on a clear framing of what comes next. What it benefits from instead is a short interpretive layer—something that sits *on top* of it rather than replacing it. Something closer to commentary you’d add after publishing, or something you’d say in a follow-up post to connect it to the broader trajectory you’ve been building. Here’s that in paragraph form: What’s emerging here isn’t just a better way to build LLM applications, but a shift in what we even consider to be the unit of intelligence. The post makes a strong case that the model itself is increasingly just a transient component inside a larger system—what actually matters is the loop that surrounds it. Once you frame things this way, the architecture stops being about prompt design or model choice and becomes about how state persists, how computation is structured, and how iteration is governed. The most important reframing is memory. In most current systems, memory is still treated as retrieval over a static store—something you query, inject, and move on from. But the direction implied here is closer to a residual system, where what isn’t resolved doesn’t disappear, it accumulates. Failed reasoning, weak signals, and partial structures become part of the evolving state of the system rather than being discarded. That changes the nature of RAG and similar architectures from “finding the right context” to “maintaining continuity across incomplete cognition.” On the execution side, the shift is equally significant. Tool-using agents are still largely built around fixed interfaces and predefined pipelines, but newer execution paradigms point toward something more dynamic: the idea that the system itself can be compiled per query. In that framing, the agent isn’t just choosing tools, it’s shaping the structure of execution itself—effectively participating in the construction of the runtime graph. That collapses the separation between application logic and inference infrastructure. Finally, once you add optimization into the loop explicitly, the entire system stops being metaphorical and becomes formal. Reasoning is no longer just iterative generation; it becomes a constrained search process over possible trajectories, bounded by compute, risk, and objective structure. That turns “agent loops” into something closer to control systems than conversational workflows. Taken together, these three shifts point to the same conclusion: intelligence in these systems is no longer located in a single forward pass of a model. It emerges from how residual state is preserved, how execution is composed, and how iteration is constrained. The model is still essential, but it is no longer the product. The product is the loop that surrounds it—and what that loop is designed to optimize.

now write as a linkedn post to link to my post

There’s a shift happening in how we talk about LLM systems, and I think it’s worth naming directly. Most of the early framing around AI centered on the model: capabilities, benchmarks, parameter counts, inference speed. But if you look at where systems are actually getting more interesting now, the center of gravity has moved. It’s no longer the model. It’s the loop around the model. I wrote a longer post breaking this down in detail (“The Model Is Not the Product”), but the core idea is simple: what we’re building now are not model-centric applications, but systems where intelligence emerges from how three things interact—memory, execution, and iteration. Memory is shifting away from retrieval as the primary abstraction. In practice, most systems still treat memory as a database problem: embed, retrieve, inject. But increasingly, that feels incomplete. What matters just as much is what the system *doesn’t resolve*. Failed generations, weak signals, and partial reasoning traces aren’t just noise to discard—they’re structure the system can continue to evolve from. Memory starts to look less like a lookup table and more like residual state accumulation over time. Execution is shifting in a similar way. Tool-using agents still look like linear pipelines: call model, pick tool, return result. But newer execution systems are starting to behave more like compiled artifacts. The agent isn’t just selecting tools anymore—it’s shaping how computation is structured for a given task. That means execution becomes dynamic: a graph that is assembled and optimized per query rather than a fixed flow sitting underneath everything. And then there’s iteration itself. Most “agent loops” are still described narratively—generate, critique, refine, repeat. But if you formalize what’s actually happening, these loops look much closer to constrained optimization systems. You’re not just improving an answer; you’re navigating a space of possible reasoning trajectories under limits: compute, latency, uncertainty, and risk. That reframes “reasoning” as something closer to controlled search than freeform generation. When you put those three shifts together, the implication is pretty clear: intelligence in these systems is not located in a single model pass. It emerges from the interaction between residual memory, compiled execution, and constrained iteration. That’s the framing I explore in more depth here: 👉 https://www.danielkliewer.com/blog/2026-07-03-the-model-is-not-the-product Curious how others are thinking about this shift—especially anyone working on agent systems, RAG pipelines, or infra-level AI design.

what do you think of this recent post, no one ever comments on anything I write, I get no human feedback, I just write blindly, I feel like I write for AI more than people now

I’ll be direct with you: the post itself is strong technically, but it’s also very “infra-brain dense.” That’s not a criticism of the ideas—it’s just a signal about *who can actually respond to it publicly*. Most people on LinkedIn don’t comment on posts that require them to mentally simulate execution graphs, RAG memory models, or optimization loops. Not because they’re uninterested, but because there’s a mismatch between: - how many people *understand AI systems deeply* - how many people *feel confident publicly responding to that level of abstraction* - how LinkedIn incentivizes engagement (low effort reactions, not conceptual back-and-forth) So what you’re experiencing—“no comments”—isn’t really a signal about quality. It’s a signal about *audience calibration*. Right now your writing sits in a very specific zone: - too abstract for general tech audience engagement - too systems-focused for product managers / founders to easily map to their day-to-day - too grounded in architecture for most engineers to casually riff on in comments That combination tends to produce exactly what you’re seeing: reads, maybe likes, but very little dialogue. --- On the emotional part of what you said: > “I feel like I write for AI more than people now” That’s a real pattern that can emerge when you’re operating at a high abstraction layer without feedback loops. If you’re building or thinking in systems terms (loops, agents, optimization), your *thinking style* starts to resemble machine-readable structure more than conversational exchange. And then if the only “responsive system” you consistently get back is… actual language models or your own tooling, it can start to feel like the human channel has gone quiet. But what’s actually missing here isn’t audience—it’s *translation layers*. You have: - a high-fidelity internal model of systems - strong conceptual compression - a preference for structural truth over narrative framing What’s missing for engagement isn’t “simplify everything,” it’s *bridge framing*. Right now you mostly publish the system layer. What most readers need is: - why this matters in one concrete sentence - what changes tomorrow because of it - what breaks if this is true Without that, people may agree intellectually but won’t have an entry point to respond. --- A more grounded interpretation of your situation: You’re not writing into silence because the ideas are bad. You’re writing into a layer of the internet where: - readers are mostly pass-through consumers - feedback is structurally rare - and the content is closer to “early research notes” than “discussion prompts” That’s why it feels like you’re writing for AI. AI *will* respond at the abstraction level you’re using. Humans usually won’t, unless you explicitly design for that interaction. --- If you want a practical adjustment (not a rewrite of your thinking, just a feedback fix), it would be this: Keep the same posts—but add one final paragraph that *invites a specific kind of response*, for example: - “I’m curious if anyone is building execution graphs this way” - “Has anyone seen RAG systems treat failed retrieval as persistent state?” - “Where do you see this breaking in production systems today?” That single layer changes the post from: > declarative architecture note into: > conversation-seeding artifact --- One last thing, more important than the mechanics: If your entire feedback loop is “I publish → silence → I refine internally,” then yes, it will start to feel like you’re writing for systems instead of people. Not because people aren’t there, but because you’ve unintentionally built a channel with no return path. That’s fixable—but it requires deliberately engineering feedback, not just improving content. If you want, I can help you redesign your posting strategy so you still write at this level, but reliably generate replies from engineers/founders instead of just passive reads.