Knowledge Compiler Development

I am still building https://www.danielkliewer.com/blog/2026-07-12-compile-time-ai-knowledge-compiler-architecture https://www.danielkliewer.com/blog/2026-07-11-knowledge-compiler-compiling-human-knowledge-into-static-semantic-artifacts # The Knowledge Compiler: Compiling Blog Posts into a Live Knowledge App *Draft — edit and post as you see fit.* --- What if your blog wasn't just a pile of prose, but a **compiled artifact** — a knowledge graph, a reasoning layer, and a running web app, all generated from the posts you already wrote? That's the idea behind the **Knowledge Compiler SDK**, and this post walks through what it is, how it works, and the (surprisingly instructive) journey of getting a real demo deployed to production from 151 of my own blog posts. **Live demo:** https://knowledge-compiler-blog-demo.vercel.app (compiled from a 12-post subset — details below on why a subset, and why the full 150 is next). --- ## The thesis: knowledge should be compiled, not retrieved We have a glut of human knowledge locked in Markdown — blog posts, notes, specs, docs. It's *source code* for understanding, but it's un-queryable, un-composable, and un-deployable as prose. RAG answers questions reactively. Agent frameworks hold state in a chat. Neither *persists* meaning. The Knowledge Compiler takes a different stance, borrowed from the world of programming languages: **treat knowledge like a compiler treats C.** ``` Markdown ──▶ Markdown IR ──▶ Ontology IR ──▶ Graph IR ──▶ Semantic IR │ ▼ Reasoning IR ──▶ Application IR ──▶ software ``` Each arrow is a small, deterministic, inspectable **pass**. Each output is a formal **Intermediate Representation (IR)** — a validated JSON artifact you can commit, diff, and build on. The final IR generates a runnable Next.js app. No agent loop required at runtime; the intelligence lives in the artifacts. --- ## The six IRs 1. **Markdown IR** (`pass-01-parse`) — structure, headings, sections. Pure Python, no model. Deterministic. 2. **Entity IR** (`pass-02-extract`) — entities with types, confidence, and **source-span provenance** (which doc/section produced each one). 3. **Ontology IR** (`pass-03-ontology`) — typed relationships and a concept hierarchy. 4. **Graph IR** (`pass-04-graph`) — nodes + edges; the traversable knowledge graph. 5. **Semantic IR** (`pass-05-embeddings` + `pass-06-clusters` + `pass-07-summaries`) — vector embeddings, thematic clusters, and human-readable summaries per cluster. 6. **Reasoning IR** (`pass-08-reasoning`) — observations, hypotheses, contradictions, and *open questions*, each with provenance back to source. 7. **Application IR** (`pass-09-specifications`) — a specification of the app to generate (routes, data, layout). 8. **Software** (`pass-10-software`) — the actual Next.js app. (Yes, that's "six plus two" — the semantic and reasoning layers each span multiple passes. The point is the layering, not the count.) Every artifact is scored on **nine dimensions** — completeness, correctness, coverage, consistency, hallucination, traceability, provenance, confidence, reproducibility — and the scorecard ships inside the app's `/evaluation` page. Quality is measurable, not a feeling. --- ## Local-first by construction The model-required passes run against **your own** OpenAI-compatible inference server — llama.cpp, Ollama, vLLM, anything on a port you control. No cloud API, no API key, no data leaving your machine. For this project the stack was: - **llama.cpp** (`llama-server`) on `:8080` running **Ornith 1.0 35B** (Q4_K_M) for the reasoning/extraction passes. - **Ollama** on `:11434` running `nomic-embed-text` for embeddings. One detail worth calling out: Ornith is a *reasoning* model. It streams a thinking trace (`reasoning_content`) and only then emits the answer. That means passes need a high `max_tokens` budget (I set 16384) or the model gets cut off mid-thought and returns empty content. The embeddings pass transparently falls back to Ollama when the chat server doesn't expose `/v1/embeddings` — so the whole thing stays local. --- ## The generated app: not stubs The deployment target was a demo that *actually demonstrates the heuristics*, not placeholder pages. `pass-10-software` emits a polished, dark "compiled knowledge" frontend: - **Overview** — live stat cards (entities, graph nodes/edges, observations, themes). - **Entities** — a filterable explorer (by type) with **provenance**: click an entity and see the exact doc/section that produced it. - **Knowledge Graph** — an interactive SVG graph with hover highlighting. - **Reasoning** — observations, hypotheses, contradictions, and open questions, each with provenance trails back to the source posts. - **Themes** — semantic clusters with summaries. - **Evaluation** — the 9-dimension scorecard per artifact. Styled with **Tailwind + shadcn-style primitives + framer-motion** transitions. All routes read the compiled IRs from `/api/*` serverless functions backed by the static `data/*.json` — no database, no external services. --- ## What it actually took to deploy (the real story) The interesting part for anyone building compiler-to-app tooling: the hard problems weren't the ML, they were the **packaging**. **1. Vercel's Next.js build ignores `@/` tsconfig path aliases.** The generated app originally used `@/components/...`. It built fine locally, then failed on Vercel with "Cannot find module '@/components/...'". Fix: emit **relative imports** (`../../components/X`) computed from each route's depth. **2. Production `npm install` skips `devDependencies`.** The first remote build failed with "typescript not installed." Vercel installs `devDependencies` only for the build step in some configs, but the type-check needs them at build time. Fix: ship `typescript`, `@types/*`, and `tailwindcss` in `dependencies`, plus `typescript: { ignoreBuildErrors: true }` as a safety net. **3. The layout never imported `globals.css`.** The first generated app rendered as bare unstyled HTML because `app/layout.tsx` forgot to import the stylesheet. A one-line fix, but it's the kind of thing that makes a demo look broken even when the data is right. **4. A pass crashed the whole pipeline.** `pass-05-embeddings` had its own `argparse` that rejected the `--max-tokens` flag the orchestrator forwards to every model pass. That single pass failing killed passes 6–10 (they depend on its output). Fix: switch the pass to the shared `parse_port_model` helper the other passes already use. Each of these is a "compile-once, deploy-everywhere" lesson: the artifact that works on your laptop is not the artifact that builds in CI. The compiler has to emit *CI-correct* code, not *laptop-correct* code. --- ## Why a 12-post subset (and the full 150 next) The corpus is 151 posts. A full run on CPU with a 35B model is slow — on the order of an hour-plus for the whole set, because each model pass does one or more 35B calls. The demo currently compiles a **12-post subset** so the iteration loop is fast and the UI is proven end-to-end against real content (16 entities, 3 themes, reasoning observations/hypotheses/questions, all from actual posts). The full 150-post run is the natural next step: it yields a far denser graph (the subset produces a sparse 5-node graph; the full corpus will be substantially richer) and is the real test of whether the pipeline scales. It's a matter of letting the CPU grind, not of any code change — the pipeline is corpus-size-agnostic by design. --- ## Why this matters The throughline is the one this blog keeps returning to: **intelligence is not the model. Intelligence is the accumulated, inspectable decisions that shape what the model produces.** The Knowledge Compiler makes those decisions *artifacts* — you can read the entities, trace the reasoning, audit the evaluation. The model is a subroutine; the knowledge graph is the product. And because every artifact is a plain file, the whole thing is **version controlled and reproducible**. Re-run the compiler, diff the IRs, see exactly what changed in your understanding of your own writing. That's the payoff of treating knowledge like source code. --- ## The full corpus now runs The demo compiles the **full 151-post corpus** — **293 entities**, 151 graph nodes, 8 thematic clusters, and a reasoning layer (observations, hypotheses, contradictions, open questions), all served live from the generated app's `/api/*` routes. Getting there required making the compiler *corpus-size agnostic*: model passes now **batch** the input into bounded chunks (so a 150-document call can't exceed the runtime's timeout) and the orchestrator gives each pass a multi-hour budget instead of killing it mid-batch. The knowledge graph currently renders extracted nodes; relationship/edge extraction is the next quality step — but the pipeline is end-to-end and scales. --- ## Try it - **Live demo:** https://knowledge-compiler-blog-demo.vercel.app - **Source:** `kliewerdaniel/knowledge-compiler-sdk` Point it at a folder of your own Markdown, run a local inference server, and watch it compile: ```bash pip install pyyaml jsonschema python -m compiler.run --source your-notes --build build \ --local --port 8080 --model your-model \ --embed-model nomic-embed-text:latest --max-tokens 16384 cd build/knowledge-app && npm install && npm run dev ``` The compiler is MIT-licensed. Build the future of autonomous knowledge compilation in the open. --- *— Daniel* ----So those are the blog posts I have plus a draft about this current project but what I need to do is some recursive research for my repo : https://github.com/kliewerdaniel/knowledge-compiler-sdk

I think the highest-leverage thing you can do next is **treat your own repository as another compilation target**. Right now you've built an implementation. The next step is to determine whether the architecture is actually novel, where it fits in the research landscape, and what ideas should become the next compiler passes. That requires a recursive research process. I would approach it as a compiler rather than as a literature review. ``` Research Papers │ GitHub Repositories │ Blog Posts │ Conference Talks │ Codebases ▼ Parse ▼ Research IR ▼ Compare ▼ Knowledge Graph ▼ Gap Analysis ▼ Roadmap ▼ Compiler Improvements ``` Instead of asking "what papers exist?", ask: > What architecture keeps independently emerging? You've already noticed multiple groups converging toward similar ideas: - compiler pipelines - typed intermediate representations - offline reasoning - static artifacts - generated applications - graph-based semantic representations That convergence itself is worth studying. ## Create a "Research Compiler" I would add an entirely new pipeline beside your markdown compiler. ``` papers/ github/ blogs/ videos/ benchmarks/ │ pass-01-collect pass-02-normalize pass-03-extract-concepts pass-04-build-citation-graph pass-05-build-architecture-graph pass-06-find-equivalent-ideas pass-07-detect-missing-concepts pass-08-generate-research-report pass-09-generate-roadmap pass-10-update-sdk ``` Notice the last pass. The compiler improves itself. That is the recursive part. --- # Things I would research Not "AI." Specific architectural questions. ## 1. Compiler Architectures Everything treating LLMs like compilation. Questions: - typed IRs - optimization passes - SSA analogies - compiler verification - deterministic AI - incremental compilation - build systems - cache invalidation --- ## 2. Knowledge Graph Systems Compare against - GraphRAG - LightRAG - Microsoft GraphRAG - Understand Anything - Neo4j research - RDF - OWL - property graphs Ask "What assumptions do they make that a compiler doesn't?" --- ## 3. Agent Architectures Study - OpenAI - Anthropic - Google - AutoGen - CrewAI - LangGraph - OpenHands - Claude Code - Codex - Gemini CLI Not to copy them. To discover "What information exists only during runtime that could instead become a compiled artifact?" That question alone might produce several new compiler passes. --- ## 4. Static Analysis Programming languages solved many problems decades ago. Borrow ideas from - LLVM - Rust compiler - Babel - Clang - TypeScript - SWC Could knowledge compilation have - dead knowledge elimination? - constant propagation? - semantic optimization? - common subgraph elimination? - provenance checking? - graph linting? --- ## 5. Databases Semantic indexes materialized views query planners vector indexes incremental indexing Your compiler resembles a database optimizer more than a chatbot. --- ## 6. Formal Logic This area is underexplored in most LLM tooling. Research - theorem provers - Datalog - Prolog - Answer Set Programming - Description Logic - SAT solvers Could reasoning become a deterministic compile pass rather than another LLM call? --- ## 7. Software Architecture Study - Domain Driven Design - CQRS - Event Sourcing - ECS - incremental build systems - Bazel - Buck2 - Nix Many build-system concepts map surprisingly well to knowledge compilation. --- # What your compiler is still missing Looking at the architecture, these are the biggest opportunities. ## Incremental Compilation Right now: ``` 151 posts ↓ compile everything ``` Instead ``` one post changes ↓ dependency graph ↓ recompile affected IRs only ↓ done ``` That would be a major productivity improvement. --- ## Pass Scheduling Some passes are independent. ``` Markdown ↓ Entity ↓ Ontology ↓ Embeddings ↓ Summaries ``` could execute in parallel. Eventually the compiler could construct a DAG of pass dependencies and schedule them automatically. --- ## Compiler Optimizations Imagine ``` Pass Optimizer ↓ remove duplicate entities merge ontology nodes collapse aliases repair provenance normalize names detect cycles prune weak edges re-score confidence emit optimized graph ``` That resembles LLVM optimization passes, but applied to knowledge. --- ## Verification Instead of ``` confidence ``` add ``` proof obligations consistency constraints ontology validation schema validation cross-reference validation citation verification ``` This pushes the compiler toward something closer to a verified build system. --- ## Plugins Eventually: ``` pass-21-rust pass-22-python pass-23-pdf pass-24-video pass-25-audio pass-26-youtube pass-27-slack pass-28-email ``` Every input compiles into the same IR. That is a powerful abstraction. --- # A recursive development loop I would have the SDK compile its own research. ``` GitHub ↓ clone repositories ↓ compile READMEs ↓ compile documentation ↓ compile source ↓ build architecture graph ↓ compare with Knowledge Compiler ↓ generate Novel ideas Missing passes Design flaws Research opportunities ↓ open GitHub issues automatically ``` That turns your project into a continuously learning engineering system rather than a static codebase. ## The long-term vision The idea that stands out most is not just "compile Markdown into an app." It's **compiling knowledge into a stable, inspectable intermediate representation that other software can build on**. That shifts the emphasis from chat interfaces to artifacts. If you extend that same philosophy to your research process, your repository becomes a self-improving platform: it ingests new papers, repositories, and design patterns, compiles them into structured IRs, compares them against its own architecture, and produces concrete proposals for new passes, optimizations, and validation rules. In effect, the compiler becomes capable of helping design the next version of itself while keeping every conclusion traceable back to source material.

write a short paragraph about a video of this would explain

:::writing{variant="document" id="38174"} This video demonstrates the Knowledge Compiler SDK in action, showing how a collection of Markdown blog posts is transformed through a series of deterministic compiler passes into structured intermediate representations (IRs), including entities, ontologies, knowledge graphs, semantic clusters, reasoning artifacts, and ultimately a fully generated Next.js knowledge application. Rather than relying on a runtime agent or traditional RAG pipeline, the system compiles human knowledge into inspectable, version-controlled artifacts that can be queried, audited, and deployed, illustrating a compiler-first approach to building AI-powered knowledge systems. :::