Unified Agent Framework

I want to engineer this better it is based on https://github.com/kliewerdaniel/chris04.git I Would like you to research what I am trying to do and to engineer an entirely new unified solution which uses a single framework instead of multiple and write a coding agent prompt which will generate the specs I need to guide someone to build this. Thank you.

Alright, I see exactly what you built—and more importantly, where it’s fighting itself. You’ve essentially got: - Angular (UI) - .NET (orchestration + API) - Python (TTS sidecar) - llama.cpp (LLM server) - SQLite + JSON (state) It *works*, but it’s fragmented. You’re paying a complexity tax every time something crosses a boundary. Let’s fix that properly. --- # 🔧 What You’re Actually Trying To Build Underneath all the implementation details, your system is: > A **fully local, persistent, agentic companion system** with: - memory (long-term + extracted facts) - persona (configurable identity) - autonomous behavior (auto mode) - multimodal output (text + voice) - local inference (LLM + TTS) That’s not a web app. That’s an **agent runtime**. --- # ⚠️ Core Problem With Current Architecture From your docs: - Multiple runtimes (.NET + Python) - IPC via stdin/stdout - HTTP for LLM - Angular separate build system - Split persistence (SQLite + JSON) This creates: - ❌ duplicated logic boundaries - ❌ hard debugging (cross-process) - ❌ brittle orchestration - ❌ slow iteration - ❌ no unified agent abstraction --- # ✅ New Direction: Single-Framework Unified System ## Recommendation: **Python-First Unified Agent Runtime** Not because Python is trendy—because: - llama.cpp bindings exist - TTS already Python-native - best ecosystem for local AI - async orchestration is trivial - can still ship a UI cleanly --- ## 🧠 New Architecture (Unified) ``` ┌──────────────────────────────┐ │ FastAPI App │ │ (Single Runtime System) │ ├──────────────────────────────┤ │ Agent Core (State Machine) │ │ Memory Engine (SQLite + vec) │ │ Persona Engine │ │ LLM Interface (llama.cpp) │ │ TTS Engine (mlx-audio) │ │ Auto Mode Scheduler │ ├──────────────────────────────┤ │ WebSocket / REST API │ └──────────────┬───────────────┘ │ ▼ Frontend (Svelte/React minimal) ``` --- ## 🔁 Key Shift Instead of: > Controllers calling services calling subprocesses You move to: > **Agent loop controlling everything** --- ## 🧠 Core Concept: Agent State Loop ```python while True: input = get_user_input_or_idle_event() state = load_state() context = build_context(state) response = llm.generate(context) memory = extract_memory(response) if should_speak(response): audio = tts.generate(response) save_state() emit_to_ui(response, audio) ``` --- # 🧱 System Components (Re-Engineered) ## 1. Agent Core (THE CENTER) Single class controlling everything: ```python class CompanionAgent: def __init__(self): self.memory = MemoryEngine() self.persona = PersonaEngine() self.llm = LLMEngine() self.tts = TTSEngine() async def handle_event(self, event): context = self.build_context(event) response = await self.llm.generate(context) self.memory.update(response) if self.should_speak(response): audio = await self.tts.generate(response) return response, audio ``` --- ## 2. Memory Engine (Unified) Replace: - SQLite + JSON split With: - SQLite (structured) - optional vector store (Chroma) ```sql memories ( id, content, embedding, timestamp, importance ) ``` --- ## 3. Persona Engine Single JSON → prompt builder No controller layer needed. --- ## 4. LLM Engine Direct llama.cpp binding: - no HTTP - no external server ```python from llama_cpp import Llama ``` --- ## 5. TTS Engine Direct integration: - remove sidecar entirely --- ## 6. API Layer (Thin) FastAPI: - `/chat` - `/memory` - `/persona` - `/stream` (WebSocket) --- ## 7. Frontend (Simplified) Drop Angular. Use: - SvelteKit OR minimal React Why: - You don’t need enterprise frontend complexity - Your system is agent-first, not UI-first --- # 🚀 What You Gain - ✅ Zero IPC - ✅ One language - ✅ Easier debugging - ✅ True agent abstraction - ✅ Faster iteration - ✅ Easier local deployment - ✅ Cleaner mental model --- # 🧠 Coding Agent Prompt (What You Asked For) Here’s the important part. You can feed this into your local LLM / coding agent: --- :::writing{variant="standard" id="48291"} You are a senior systems architect and AI engineer. Your task is to design a fully local, unified AI companion system using a SINGLE runtime and framework. The system must replace a fragmented architecture consisting of Angular frontend, ASP.NET backend, Python TTS sidecar, and llama.cpp HTTP server. ## Objective Design a clean, production-ready architecture for a local-first AI companion agent with: - persistent memory - persona system - autonomous behavior (auto mode) - text + voice output - local LLM inference - no cloud dependencies ## Hard Constraints - Use Python as the ONLY backend/runtime language - Use FastAPI for API layer - No subprocess-based sidecars - No HTTP calls to local LLM (must use direct bindings) - All components must run in a single process - System must be modular but unified ## Required Components 1. Agent Core - Central event loop - Handles user input + idle events - Orchestrates all subsystems 2. Memory System - SQLite for structured storage - Optional vector store (Chroma or FAISS) - Memory extraction logic from conversations - Context window management 3. Persona System - JSON-based persona definition - Dynamic system prompt construction 4. LLM Engine - Direct llama.cpp bindings - Streaming support - Configurable context size 5. TTS Engine - Integrated Python TTS (mlx-audio or equivalent) - No external processes - Returns file paths or byte streams 6. Auto Mode - Background task scheduler - Generates proactive messages 7. API Layer - FastAPI endpoints - WebSocket streaming for chat + audio events 8. Frontend (optional but defined) - Minimal UI (React or Svelte) - Audio playback queue - Live streaming responses ## Deliverables Generate: 1. Full system architecture diagram (text) 2. Folder structure 3. Detailed module responsibilities 4. Data models (SQLite schema) 5. Agent loop pseudocode 6. API endpoint definitions 7. Event flow (chat + auto mode) 8. Concurrency model (async design) 9. Example config files (persona, memory) 10. Deployment instructions (local machine) ## Design Philosophy - Prioritize simplicity over abstraction - Eliminate unnecessary boundaries - Treat the system as an "agent runtime", not a web app - Optimize for local execution and iteration speed - Avoid enterprise patterns unless absolutely necessary ## Output Format Structured technical specification with clear sections. No fluff. No marketing language. Only engineering detail. ::: --- # 🧭 Final Thought (Important) What you built already proves something: You’re not building apps anymore. You’re building **systems that behave**. The next step is aligning the architecture with that reality: - less “services talking” - more “agent thinking” --- If you want, I can: - rewrite your repo into this structure - or generate the actual code scaffold for it Just say the word.