Agent Recipe Compiler

Perfect! Now I have a clear picture. Let me analyze Daniel's work to identify the blind spots and propose 5 projects that build on his knowledge while addressing the gaps he's identified. Based on my research, here are **5 coding projects** that build on Daniel Kliewer's existing knowledge while filling the blind spots he's identified in his blog: --- ## 1. **Expert Signal Router** — `expert-signal-router` **Blind Spot Filled:** Daniel identifies that "cheap signals (likes, heuristics, weak metrics) lead to 'slop optimization'" but doesn't provide a local-first system for routing expensive expert signals. **Project:** A local-first signal routing system that: - Monitors AI agent outputs and classifies them into "cheap signal" vs "expert signal needed" buckets - Implements a tiered evaluation pipeline: lightweight heuristics → human-in-the-loop review → structured expert eval - Captures expert decisions as training examples for future fine-tuning - Builds the "agent recipe" incrementally as experts make judgments - All data stays local via ChromaDB + SQLite **Tech Stack:** Python, FastAPI, ChromaDB, Ollama (local embeddings), SQLite **Builds On:** SovereignMemoryBank's 7-layer architecture, SovereignBank's knowledge graph **New Capability:** Operationalizes the "outer loop" by making expert review systematic, not ad-hoc --- ## 2. **Agent Recipe Compiler** — `recipe-compiler` **Blind Spot Filled:** Daniel proposes "agent recipes" as "versioned memory of how intelligence was shaped" but provides no tool to capture, version, or evolve them. **Project:** A CLI tool that: - Records agent interactions as structured recipe entries (not just logs) - Captures: input context, model version, prompt template, output, quality signal, expert feedback - Compiles recipes into a searchable knowledge graph with semantic versioning - Detects "recipe drift" when agent behavior changes unexpectedly - Generates "recipe diff" reports showing how intelligence evolved over time - Exports recipes as training data for domain-specific fine-tuning **Tech Stack:** Python, GBNF (for structured output), Ollama, SQLite, NetworkX (graph) **Builds On:** SovereignSpec's ADR system, SovereignSpec's knowledge graph **New Capability:** Makes "agent recipes" concrete artifacts, not just a conceptual framework --- ## 3. **Tacit Judgment Extractor** — `tacit-extractor` **Blind Spot Filled:** Daniel notes that "frontier models plateaued around ~78% accuracy on judgment-heavy tasks" and that "prompting stops working" for tacit knowledge. He doesn't provide a way to extract the tacit. **Project:** A system that: - Records expert decision-making sessions (audio/video/text) - Uses local LLMs to identify "tacit patterns" in expert reasoning - Generates structured decision trees that capture unarticulated expertise - Creates synthetic training examples from tacit patterns - Builds domain-specific "judgment models" via RLHF on extracted patterns - Validates by comparing model judgments against held-out expert sessions **Tech Stack:** Python, Whisper (local transcription), Ollama, GRPO-style training loop, PyTorch **Builds On:** Bridgewater AIA Labs' findings from his blog, fine-tuning workflows **New Capability:** Converts "things experts can't articulate" into trainable model capabilities --- ## 4. **Autonomous Evaluation Loop** — `auto-eval` **Blind Spot Filled:** Daniel emphasizes that "the hard problem isn't building loops—it's designing signals that are meaningful enough for improvement without collapsing into noisy optimization." He doesn't provide the signal design tooling. **Project:** A self-improving evaluation system that: - Defines evaluation signals as YAML specs with uncertainty bounds - Auto-generates synthetic test cases based on production data distributions - Implements "signal drift detection" — alerts when cheap signals stop correlating with expert quality - Runs A/B tests on different signal definitions - Tracks "optimization cost" vs "capability gain" to prevent runaway loops - Generates "evaluation recipes" that can be versioned and shared **Tech Stack:** Python, FastAPI, SQLite, Ollama (for synthetic data generation), Weights & Biases (local) **Builds On:** Autoresearch loops from his blog, sovereignspec's spec-driven approach **New Capability:** Makes self-improvement loops cost-bounded and signal-efficient --- ## 5. **Sovereign Apprenticeship Engine** — `sovereign-apprentice` **Blind Spot Filled:** Daniel describes the "apprenticeship" model: "early heavy supervision → gradual autonomy" but doesn't provide the scaffolding to transition between these phases. **Project:** A phased autonomy system that: - Starts with 100% human oversight, capturing all decisions - Gradually increases autonomy based on measured quality signals - Implements "autonomy budgets" — pre-approved actions that don't require review - Detects "autonomy debt" when the system starts making low-quality autonomous decisions - Generates "scaffolded training" — uses early human decisions to bootstrap later autonomy - Provides rollback mechanisms when autonomy exceeds quality thresholds **Tech Stack:** Python, FastAPI, ChromaDB, Ollama, SQLite **Builds On:** Autoresearch's "humans as tools in the loop", sovereignspec's constitution system **New Capability:** Operationalizes the transition from supervised to autonomous AI --- ## Why These Projects Matter | Project | Fills Gap | Extends Knowledge | |---------|-----------|-------------------| | **Expert Signal Router** | Makes expert review systematic | Extends SovereignBank's memory system | | **Agent Recipe Compiler** | Makes recipes concrete | Extends SovereignSpec's versioning | | **Tacit Judgment Extractor** | Captures unarticulated expertise | Extends fine-tuning workflows | | **Autonomous Evaluation Loop** | Bounds optimization loops | Extends autoresearch architecture | | **Sovereign Apprenticeship** | Phases autonomy safely | Extends sovereign architecture | Each project builds on Daniel's existing repos (SovereignBank, SovereignSpec, SovereignMemoryBank) while addressing the "blind spots" he explicitly identifies in his latest blog posts. They all maintain the sovereign/local-first ethos while adding the operational tooling needed to make self-improving AI systems actually work in production. **Next step:** I can create a SovereignSpec for any of these projects and start the implementation workflow. Which one interests you most?

I like this direction. Looking at your body of work over the last year, I also think these projects fit together into something larger rather than five unrelated repositories. I'd prioritize them differently, though. ### 1. Agent Recipe Compiler (Highest Priority) This is the missing primitive. Almost every post you've written lately revolves around the same idea: > Intelligence is not the model. > Intelligence is the accumulated decisions that shaped the model. Right now "agent recipes" are an idea. This project would make them an actual artifact. Imagine every run producing something like: ```yaml recipe: id: recipe-2031 objective: classify_ai_paper model: qwen3.5 memory_version: 17 prompt_version: 9 retrieved_docs: - memory_102 - memory_882 reasoning_pattern: - compare - retrieve - synthesize evaluation: score: 0.94 reviewed_by: expert outcome: accepted ``` Once you have thousands of these, you suddenly have - reproducibility - debugging - training datasets - knowledge evolution - agent version control This could become the Git commits of AI systems. --- ### 2. Expert Signal Router This complements the Recipe Compiler. Recipes tell you **what happened**. The router decides **who judges it.** ``` Agent ↓ Cheap evaluation ↓ Good enough? │ ├── yes → accept │ └── no ↓ Expert ↓ Recipe updated ↓ Future fine tuning ``` This becomes your supervision layer. --- ### 3. Autonomous Evaluation Loop I actually think this becomes much easier after the first two. Instead of evaluating outputs, you're evaluating recipes. Questions become - Which recipes succeed? - Which recipes drift? - Which evaluation metrics correlate with humans? That feels much more measurable. --- ### 4. Sovereign Apprenticeship I like this one because it solves a real production problem. Most agent frameworks have two modes * manual * autonomous Reality has dozens of stages. ``` 100% supervised ↓ approve every action ↓ approve only dangerous actions ↓ approve only novel actions ↓ approve only uncertain actions ↓ fully autonomous ``` That's much closer to how people actually learn. --- ### 5. Tacit Judgment Extractor This is fascinating. It's also by far the hardest. It borders on research rather than engineering. I wouldn't build this until everything above exists because you'll need recipes and expert evaluations to learn from anyway. --- # What I think is still missing If I were adding a sixth project, it would actually be the one I think could become your flagship. ## Sovereign Intelligence Observatory Instead of another agent... Build a system that observes agents. It would answer questions like - Why did the agent choose this? - Which memories influenced it? - Which recipes changed? - Which expert reviews mattered? - Which prompts are becoming obsolete? - Which memories are never retrieved? - Which evaluation signals correlate with success? - Where is capability improving? - Where is it regressing? Think GitHub Insights... but for intelligence. ``` Agent │ ▼ Recipe Memory Signals Graph Evaluations Experts Autonomy ↓ Observatory Dashboard ↓ Intelligence Timeline ``` This fits incredibly well with your recurring idea that **the loop is the product**. If the loop is the product... then the observability of the loop becomes the operating system. --- ## The architecture I'd build ```text SovereignSpec │ ▼ Recipe Compiler │ ▼ Expert Signal Router │ ▼ Autonomous Evaluation Loop │ ▼ Apprenticeship Engine │ ▼ Observatory ``` Notice something interesting. Each layer produces data for the next. Nothing is wasted. Everything compounds. Eventually, fine-tuning almost becomes a side effect. Instead of "collect data then train," your architecture naturally accumulates high-quality, expert-labeled trajectories that can be distilled into models whenever it makes sense. To me, that's the strongest evolution of the ideas you've been developing: shifting from a collection of AI tools to an **intelligence infrastructure** where recipes, signals, memory, evaluation, and observability all reinforce one another. That's a more distinctive direction than simply building another autonomous agent.