Local Models vs APIs

summarize, include all details and rewrite in my voice: There are massive differences. One is that unlike getting something through someone else who is eating a cost and has to think of ways to cut costs as much as possible versus hosting something from your own resources which gives you the freedom to invest whatever amount of time and effort into soemthing. That is you can alter the token counts, have it reason indefinitely, have it run as quickly as you want, not worry about latency, not worrying about internet connection, not worrying about transmitting information off of a local set up. Etc. I initially did it in order to test my apps because APIs get expensive really quick when you make a lot of mistakes and want to test over and over very quickly to try to debug something. It makes the debugging process so much easier. Personally I use Ollama. I know there are other options, but that is what I started with and I have had no reason to change as I have been very happy with their contributions and ability to keep up to date with everything. And it is easy to setup. Not only that but you do not have to worry about changing the code structure or how it operates constantly like you do if you use different APIs, like Anthropic uses a different structure than OpenaAI and OpenAI just introduced a new Responses API with the OpenAI Agents SDK they just released. That is what I am working on, kind of. I am also working on MCP with Ollama and testing that out for the hackathon tomorrow. Just go to r/LocalLLaMA and they are a great and very nice community that are very willing to be helpful as well. There are downsides though. Namely quality. It will not be the same as SOTA models which use black magic from their alien lizard overlords. By that I mean money. So local models for one, are going to be limited by your hardware. Unless you have really good hardware you are just not going to get the same level of performance as you get with API. For example. I just bought the newest Macbook Pro with the M4 Pro and 48GB unified memory. If you want the same speed of output as what you get from an API you will only be running much smaller models than what they use. For example. So the rule of thumb is that you want to be able to fit the entire model on your VRAM. So with unified memory they say it is like having that much VRAM but you have to think there is also everything else you have running, like Docker eats so much memory for example. So here are the sizes of the models I can run: gemma3:27b 30ddded7fba6 17 GB 5 days ago mistral:latest f974a74358d6 4.1 GB 8 days ago qwq:latest cc1091b0e276 19 GB 8 days ago llama3:latest 365c0bd3c000 4.7 GB 12 days ago llama2:latest 78e26419b446 3.8 GB 3 weeks ago nomic-embed-text:latest 0a109f422b47 274 MB 3 weeks ago llava:latest 8dd30f6b0cb1 4.7 GB 3 weeks ago mistral-small:24b-instruct-2501-q8_0 20ffe5db0161 25 GB 6 weeks ago deepseek-r1-qwen-2.5-32B-ablated-Q8_0.gguf:latest 62d46152c169 34 GB 6 weeks ago bakllava:latest 3dd68bd4447c 4.7 GB 7 weeks ago llama3.2:latest a80c4f17acd5 2.0 GB 7 weeks ago deepseek-r1:70b 0c1615a8ca32 42 GB 7 weeks ago gemma2:27b 53261bc9c192 15 GB 2 months ago vanilj/Phi-4:latest f00b090d5270 9.1 GB 2 months ago llama3.3:latest a6eb4748fd29 42 GB 2 months ago qwen2.5:32b 9f13ba1299af 19 GB 2 months ago dolphin-mistral:7b 5dc8c5a2be65 4.1 GB 2 months ago As you can see they are not very large and the parameter count is much lower than what SOTA models use. So you are going to have performance issues. Like for instance. I could test my first PersonaGen program using OpenAI and get a solid response and everything works. But if I run it locally it breaks. Why? Because the parameter count for the model is lower, as well as the lack of black magic, that is a reasoning graph. I built one, basically a for glorified for loop, but it turns any Ollama model into a reasoning model. I built it quick to show how you can put any logic you want. It even had a nice frontend I made for it and this one dude made fun of me because it just had a for loop for the reasoning logic because I was just making something to teach myself, not something to impress others, geez. So in this instance the reason was because I wanted the LLM to respond back only with JSON and the SOTA models could do it fine because of the reasoning graph with RAG and now MCP they use. But I am building all of that for local models as well. They are trying to tie you into their ecosystem by offering more and different things quickly, very quickly. It is hard to keep up. But today I am building an MCP browser client and server using Ollama, which I need to get back to.

There’s a massive difference between relying on APIs and running models locally. When you use an API, you're at the mercy of someone else’s infrastructure—they have to cut costs, optimize for efficiency, and introduce limitations like rate limits and token caps. Running things locally gives you total control: you can push token counts as high as your hardware allows, let models reason indefinitely, eliminate latency issues, and never worry about an internet connection or data privacy. Originally, I set up a local model to test my apps. API costs add up fast, especially when you're debugging and need to iterate quickly. Having a local setup made troubleshooting way easier. I use Ollama—it was the first one I tried, and I’ve stuck with it because it stays updated, is easy to set up, and just works. Plus, you don’t have to deal with constantly shifting API structures. OpenAI, for example, just introduced their Responses API with the OpenAI Agents SDK, which would’ve required adjusting my entire workflow. Local models avoid that headache. Right now, I’m also experimenting with MCP using Ollama, testing it for the hackathon tomorrow. If you're into local LLMs, r/LocalLLaMA is a great community—super helpful and knowledgeable. But there are trade-offs. The biggest one? Quality. Local models aren’t on the same level as state-of-the-art (SOTA) models, which are trained on absurdly expensive hardware and optimized with proprietary techniques (aka "black magic from their alien lizard overlords"—read: money). Your performance is limited by your hardware. If you don’t have a high-end setup, you won’t get the same speed or accuracy as an API. For reference, I just got a MacBook Pro with the M4 Pro and 48GB of unified memory, and even then, I can only run smaller models compared to what APIs offer. The rule of thumb is that the model needs to fit in VRAM. Unified memory helps, but other processes (like Docker) eat up a lot of it. Here’s what I can run: - **gemma3:27b** – 17GB - **mistral** – 4.1GB - **qwq** – 19GB - **llama3** – 4.7GB - **mistral-small:24b-instruct-2501-q8_0** – 25GB - **deepseek-r1-qwen-2.5-32B-ablated-Q8_0.gguf** – 34GB - **deepseek-r1:70b** – 42GB These are nowhere near the size of SOTA models, and that affects performance. For example, I tested my first **PersonaGen** program using OpenAI’s API, and it worked perfectly. Running it locally? It broke. Why? Lower parameter count and lack of reasoning graphs. To address this, I built a reasoning graph—a glorified `for` loop—but it lets any Ollama model perform reasoning like a bigger model. I even made a frontend for it, but some guy mocked it for being too simple. Whatever. It was a proof of concept, not a flex. One issue I ran into: I needed the model to output only JSON. SOTA models handle this fine with retrieval-augmented generation (RAG) and MCP, but local models struggle. That’s why I’m now working on adding those features myself. The big companies are racing to lock people into their ecosystems by rolling out new features constantly, making it tough to keep up. Anyway, today I’m building an MCP browser client and server with Ollama. Time to get back to it.

make this short enough to be a Reddit comment reply but as long as possible: There’s a massive difference between relying on APIs and running models locally. When you use an API, you’re at the mercy of someone else’s infrastructure—they have to cut costs, optimize for efficiency, and introduce limitations like rate limits and token caps. Running things locally gives you total control: you can push token counts as high as your hardware allows, let models reason indefinitely, eliminate latency issues, and never worry about an internet connection or data privacy. Originally, I set up a local model to test my apps. API costs add up fast, especially when you’re debugging and need to iterate quickly. Having a local setup made troubleshooting way easier. I use Ollama—it was the first one I tried, and I’ve stuck with it because it stays updated, is easy to set up, and just works. Plus, you don’t have to deal with constantly shifting API structures. OpenAI, for example, just introduced their Responses API with the OpenAI Agents SDK, which would’ve required adjusting my entire workflow. Local models avoid that headache. Right now, I’m also experimenting with MCP using Ollama, testing it for the hackathon tomorrow. If you’re into local LLMs, r/LocalLLaMA is a great community—super helpful and knowledgeable. But there are trade-offs. The biggest one? Quality. Local models aren’t on the same level as state-of-the-art (SOTA) models, which are trained on absurdly expensive hardware and optimized with proprietary techniques (aka “black magic from their alien lizard overlords”—read: money). Your performance is limited by your hardware. If you don’t have a high-end setup, you won’t get the same speed or accuracy as an API. For reference, I just got a MacBook Pro with the M4 Pro and 48GB of unified memory, and even then, I can only run smaller models compared to what APIs offer. The rule of thumb is that the model needs to fit in VRAM. Unified memory helps, but other processes (like Docker) eat up a lot of it. Here’s what I can run: • gemma3:27b – 17GB • mistral – 4.1GB • qwq – 19GB • llama3 – 4.7GB • mistral-small:24b-instruct-2501-q8_0 – 25GB • deepseek-r1-qwen-2.5-32B-ablated-Q8_0.gguf – 34GB • deepseek-r1:70b – 42GB These are nowhere near the size of SOTA models, and that affects performance. For example, I tested my first PersonaGen program using OpenAI’s API, and it worked perfectly. Running it locally? It broke. Why? Lower parameter count and lack of reasoning graphs. To address this, I built a reasoning graph—a glorified for loop—but it lets any Ollama model perform reasoning like a bigger model. I even made a frontend for it, but some guy mocked it for being too simple. Whatever. It was a proof of concept, not a flex. One issue I ran into: I needed the model to output only JSON. SOTA models handle this fine with retrieval-augmented generation (RAG) and MCP, but local models struggle. That’s why I’m now working on adding those features myself. The big companies are racing to lock people into their ecosystems by rolling out new features constantly, making it tough to keep up. Anyway, today I’m building an MCP browser client and server with Ollama. Time to get back to it.

Running local models vs. APIs is night and day. APIs lock you into someone else’s infrastructure—rate limits, token caps, changing SDKs—while local models give you full control: higher token counts, lower latency, offline access, and total privacy. I started with Ollama for testing since API costs stack up fast, and it just works. Biggest trade-off? Quality. SOTA models are trained with insane resources, and local models lag behind. Even with my M4 Pro MacBook (48GB RAM), I can’t run anything close to API models. For example, deepseek-r1:70b eats 42GB, and mistral-small:24b takes 25GB. More parameters = better reasoning, which is why my PersonaGen tool worked fine with OpenAI but broke locally. To fix that, I built a reasoning graph (a glorified for loop) to mimic bigger models. Someone mocked my frontend for being too simple—whatever, it was proof of concept. Now, I’m working on MCP and JSON-constrained output since local models struggle with that. If you're into this, check out r/LocalLLaMA—it’s a solid community.

again with this: Running local models vs. APIs is night and day. APIs lock you into someone else’s infrastructure—rate limits, token caps, changing SDKs—while local models give you full control: higher token counts, lower latency, offline access, and total privacy. I started with Ollama for testing since API costs stack up fast, and it just works. Biggest trade-off? Quality. SOTA models are trained with insane resources, and local models lag behind. Even with my M4 Pro MacBook (48GB RAM), I can’t run anything close to API models. For example, deepseek-r1:70b eats 42GB, and mistral-small:24b takes 25GB. More parameters = better reasoning, which is why my PersonaGen tool worked fine with OpenAI but broke locally. To fix that, I built a reasoning graph (a glorified for loop) to mimic bigger models. Now, I’m working on MCP and JSON-constrained output since local models struggle with that. If you’re into this, check out r/LocalLLaMA—it’s a solid community. Again I had to summarize this one, twice, in order for it to fit with the LLM, sorry. Haha, this one looks AI generated, but it was really just bad photoshopping on my phone, I made these on my phone everyday while I was studying ocw.mit.edu like it was my job as part of my routine. I did the exact same thing every day for 4 years without missing one day 7 days a week. That is my record I think. That is when I painted the majority of my painting that is in the background of these cat pictures. But that was a lifetime ago. It was before everything went horribly horribly wrong. Or right, it is really how you look at things. Because now I am a super human when before I was barely scraping by. I am in the best shape now, well that is not true, I used to lift weights everyday, well I do now too. I work my manual labor job and have to walk 2 miles to work and 2 miles home each day. So maybe I am in the best shape, except I am older, and wiser and braver. When you get old you know what to fear and what not to fear and make fewer mistakes on the streets. At least that is what living on the streets teaches you. I lived in and out of transitional housing 8 years before I started this manual labor job. I want to work as an AI developer though. I think that I have the background for it and have invested as much time as many other people have. I am not the best developer. I am not really a great developer at all. I am horrible at coding. But I know how to and I know how it all works from the beginning to the end of the data pipeline. I have over a decade's experience in data annotation and have personally taken tens of thousands of surveys and intelligence tasks with high accuracy. I did it to see what people were researching all over the world. That was back in the hayday of Mechanical Turk when you could Turk for 30 minutes and have $20 real quick. Those were the days.

Local models vs. APIs is night and day—APIs come with rate limits, token caps, and ever-changing SDKs, while local models give you full control over token counts, latency, privacy, and offline access. I started using Ollama to cut API costs, and it just works. The trade-off? Quality. SOTA models run on absurdly expensive hardware, while local models lag behind. Even with my M4 Pro MacBook (48GB RAM), I can’t match API performance—deepseek-r1:70b eats 42GB, mistral-small:24b takes 25GB. My PersonaGen tool worked fine with OpenAI but broke locally due to parameter limits. To fix that, I built a reasoning graph (basically a glorified for loop). Now, I’m working on MCP and JSON-constrained output since local models struggle there. If you’re into this, r/LocalLLaMA is a great community. Outside AI, I’ve lived a life—8 years in and out of transitional housing, now working manual labor while walking 4 miles a day. Studied OCW MIT daily for 4 years straight. Spent a decade in data annotation, back when you could Turk for $20 in 30 minutes. Not the best coder, but I get how AI works end to end. Just trying to break into AI dev.

now this: Here is the repo it is based on, https://github.com/kliewerdaniel/Orchestrator-Ollama it is not a finished product, it was a way for me to teach the concepts to myself. That is what I do. I don't finish repos or programs. I just learn as much as I can and then move on to the next thing. So this is one of those repos, so you would have to just analyze the structure and code. I used this more to teach me an idea I had about graphs. I use networkx to define the flow between agents. You build a graph composed on nodes(agents) and edges(prompts, context, metadata, vectorstore, now MCP) Again it is very basic, so not trying to impress anyone just trying to teach myself concepts. The last part about ai_guidelines.md is still new. I was inspired by the way that guidelines are built for the data annotation jobs I have had, because they are all very similar and they were created by the big tech companies as a standardized software development process. So I am trying to put together a bunch of coding principles. A UI for adjusting variables for these principles. That is what I should do. Thank you for the inspiration. I am going to use this repo for the hackathon with the MCP server I just built. Sorry I just went off the rails there but had to follow a train of thought which is going to be great I just know it. Anyway. I am trying to describe the entire pipeline, awwwww shit I just thought of something. I still have all the PDF forms of the guidelines given to me by the different companies over the years. I could use a RAG and reference them to few shot my own ai_guidelines.md That is a great idea. Now summarize this.

The repo [Orchestrator-Ollama](https://github.com/kliewerdaniel/Orchestrator-Ollama) isn’t a finished product—it was a way for me to teach myself concepts. That’s my process: I don’t finish repos, I learn as much as I can and move on. This one helped me explore graph-based AI orchestration using `networkx` to define agent flow. Nodes represent agents, and edges represent prompts, context, metadata, vector stores, and now MCP. It’s basic—not meant to impress, just for learning. Lately, I’ve been working on `ai_guidelines.md`, inspired by the standardized guidelines used in big tech data annotation jobs. I want to build a UI to adjust coding principles dynamically. For the hackathon, I’m integrating this repo with my MCP server. Also, I realized I still have PDFs of annotation guidelines from various companies—I could use RAG to reference them and few-shot my own `ai_guidelines.md`. That’s a great idea.

now this: The repo Orchestrator-Ollama isn’t a finished product—it was a way for me to teach myself concepts. That’s my process: I don’t finish repos, I learn as much as I can and move on. This one helped me explore graph-based AI orchestration using networkx to define agent flow. Nodes represent agents, and edges represent prompts, context, metadata, vector stores, and now MCP. It’s basic—not meant to impress, just for learning. Lately, I’ve been working on ai_guidelines.md, inspired by the standardized guidelines used in big tech data annotation jobs. I want to build a UI to adjust coding principles dynamically, I have an idea, I should make that a tool for the MCP server I am building with Ollama. For the hackathon, I’m integrating this repo with a MCP server I am coding to work with Ollama. Also, I realized I still have PDFs of annotation guidelines from various companies—I could use RAG to reference them and few-shot my own ai_guidelines.md. That’s a great idea. I could do that with my Simulacra App That is the app that I have to hide from Zuck's Robot Jesus. But the end product will be a Persona based reasoning agent which you can call functions using MCP to execute things like browser-use and such. I imagine that is all that Manus is, joking.

The **Orchestrator-Ollama** repo isn’t a finished product—it was a way for me to teach myself concepts. That’s my process: I don’t finish repos, I learn as much as I can and move on. This one helped me explore **graph-based AI orchestration** using `networkx` to define agent flow. Nodes represent agents, and edges represent prompts, context, metadata, vector stores, and now **MCP**. It’s basic—not meant to impress, just for learning. Lately, I’ve been working on **ai_guidelines.md**, inspired by the standardized guidelines from big tech data annotation jobs. I want to build a **UI for adjusting coding principles dynamically**—I should turn that into a **tool for the MCP server I’m building with Ollama**. For the **hackathon**, I’m integrating this repo with an **MCP server** I’m coding to work with Ollama. I also realized I still have **PDFs of annotation guidelines** from various companies—I could use **RAG** to reference them and **few-shot my own ai_guidelines.md**. That’s a great idea. I could do that with my **Simulacra App**—the one I have to hide from *Zuck’s Robot Jesus*. But in the end, it’ll be a **persona-based reasoning agent** that can **call functions using MCP** to execute tasks like browser automation. Honestly, that’s probably all **Manus** is—just kidding. 😆

now this: The repo Orchestrator-Ollama isn’t a finished product—it was a way for me to teach myself concepts. That’s my process: I don’t finish repos, I learn as much as I can and move on. This one helped me explore graph-based AI orchestration using networkx to define agent flow. Nodes represent agents, and edges represent prompts, context, metadata, vector stores, and now MCP. It’s basic—not meant to impress, just for learning. Lately, I’ve been working on ai_guidelines.md, inspired by the standardized guidelines from big tech data annotation jobs. I want to build a UI for adjusting coding principles dynamically—I should turn that into a tool for the MCP server I’m building with Ollama. But I want to build a graph composed of each and every aspect of a tech company including all of the different jobs that compose each part of it and individually build the prompts for each. Then you could use fstrings in the prompts. I don't know why people don't do this more. Basically you use keys and values in the prompt which are stored in the db in something as simple as a JSON file or even SQLite. That way you can dynamically alter them on and control all of it through a frontend. Then you could run A/B testing or use RLHF and optimize. Or you could create few shot prompts composed of the structured JSON you want to pass along with the context along the edges of the graph between the glorified system prompt nodes. For the hackathon, I’m integrating this repo with an MCP server I’m coding to work with Ollama. I also realized I still have PDFs of annotation guidelines from various companies—I could use RAG to reference them and few-shot my own ai_guidelines.md. That’s a great idea. I could do that with my Simulacra App—the one I have to hide from Zuck’s Robot Jesus. But in the end, it’ll be a persona-based reasoning agent that can call functions using MCP to execute tasks like browser automation. Honestly, that’s probably all Manus is—just kidding. No need to pay me. I will teach you for free. I publish everything I make on my website: danielkliewer.com and you can follow me on github at github.com/kliewerdaniel/ Plus I just like to write a lot so just keep asking questions. But to answer your question again, this is just my idea of fun so you don't have to pay me, just build something that helps people in some way rather than for zuck's killer robots. I work for Meta right now. I did not work at all for them last month though, I work as a third party annotator so I technically do not work for them and set my own schedule and they apparently don't mind if you don't work for a month. I still have accounts for other tech companies as well. Like I am 100% sure I worked for Anthropic at one point and they were the contract which got me out of poverty.

The **Orchestrator-Ollama** repo isn’t a finished product—it was a way for me to teach myself concepts. That’s my process: I don’t finish repos, I learn as much as I can and move on. This one helped me explore **graph-based AI orchestration** using `networkx` to define agent flow. Nodes represent agents, and edges represent prompts, context, metadata, vector stores, and now **MCP**. It’s basic—not meant to impress, just for learning. Lately, I’ve been working on **ai_guidelines.md**, inspired by the standardized guidelines from big tech data annotation jobs. I want to build a **UI for adjusting coding principles dynamically**, but I also want to take it further: - Build a **graph mapping every aspect of a tech company**, including all job roles. - Individually define **prompts for each role** using **f-strings**. - Store prompts in a **database (JSON, SQLite, etc.)** for dynamic updates via a frontend. - Enable **A/B testing, RLHF**, and optimization. - Structure **few-shot prompts** with contextual edges linking system prompt nodes. For the **hackathon**, I’m integrating this repo with an **MCP server** I’m coding to work with Ollama. I also realized I still have **PDFs of annotation guidelines** from various companies—I could use **RAG** to reference them and **few-shot my own ai_guidelines.md**. That’s a great idea. I could do that with my **Simulacra App**—the one I have to hide from *Zuck’s Robot Jesus*. But in the end, it’ll be a **persona-based reasoning agent** that can **call functions using MCP** to execute tasks like browser automation. Honestly, that’s probably all **Manus** is—just kidding. 😆 No need to pay me—I’ll teach you for free. I publish everything I make on my website: **[danielkliewer.com](https://danielkliewer.com)**, and you can follow me on GitHub: **[github.com/kliewerdaniel](https://github.com/kliewerdaniel/)**. I just like to write, so keep asking questions. My only request? **Build something that helps people**, not *Zuck’s killer robots*. Also, fun fact: I technically work for Meta right now as a **third-party annotator**, but I set my own schedule and didn’t work for them at all last month. I’ve worked for other tech companies too—pretty sure Anthropic was the contract that got me out of poverty.