AI Cost Curve Evolution

Weighty Thoughts Weighty Thoughts AI's Plummeting Prices Are a Software Story, Not a Hardware One This has made local, open-weight models a real competitor to the frontier James Wang May 19, 2026 Why is model inference getting cheaper? How did I drop a soon-to-be $2,000+/month bill for AI agents to next to nothing? And why are local models on commodity hardware potentially “good enough” for most people? There are two macro trends here that feed directly into each other. First, AI inference costs, as I’ve mentioned before, have been dropping 70-90% per year. Guido Appenzeller coined the term “LLMflation” through his original observation that costs have “dropped by a factor of 1,000 in three years.” No matter how many times I say it—and however many smart observers point it out—it still shocks most people because AI feels like it’s getting more expensive. That’s because costs are dropping for the same capacity (same model, same query), and we’re constantly ramping up what we use (bigger model, more expensive query). It’s the same reason why, despite Moore’s Law (which is slower than LLMflation), computers don’t cost $0.00001—we made computers bigger as we went, even as their cost exponentially plummeted. This, however, is an old story. The interesting part of the piece is what drives this plummeting cost. It’s not (mainly) hardware. It’s software. A chart of the 10x drop per year from a16z. The second is a consequence of that: local, open-weight models on kind-of-old commodity hardware are becoming more and more competitive with models on the frontier. That obviously has big implications—and potential consequences—on what the frontier labs will ultimately be able to charge for the biggest models. Local AI + Old GPU Beats Frontier Model I ended up researching and writing this piece accidentally. I’ve always experimented with open-weight models myself, even when it made no sense to run them. My (public) history playing around with language models goes back to 2014, when I published an implementation of a Google model paper for a data science certification. It’s always largely been a hobby/learning experience with no practical output—until recently. A few weeks ago, I started playing with Qwen 3.6 27B—released about a month ago in April 2026. I didn’t run it on an Nvidia H100/B200 or some other data center GPU. I ran it on a consumer gaming card, an Nvidia RTX 3090 Ti, released in 2022. While I heard good reviews of its capabilities, I didn’t expect much of it. To my surprise, it felt like running Claude Sonnet, Anthropic’s model one tier below the biggest model (the one they encourage people to run as their “everyday” model). Sure, it isn’t Opus (Anthropic’s biggest, priciest model) or GPT-5.5 Pro (OpenAI’s biggest model), but this thing is open-weight/free and runs on a 4-year-old consumer-grade GPU! Of course, “feelings” don’t mean much for model quality. Fortunately, Anthropic forced my hand on finding alternatives to how I run my rather extensive agent workflow. I detailed that workflow here (it’s still one of my most popular pieces ever). As a note, that was a (relative) eternity ago. I have way more agents running at this point—enough so that I need to actually have agents that manage my agents (I reinvented the org chart…). How many agents do I have? Who can say anyway? I’m sure it’s like having too many children. At some point you just start rounding. The Coming Anthropic Agent Crackdown Why did they force my hand? Well, starting June 15, 2026, claude -p—which, as per my agents article, is how most of my autonomous AI agents run—is no longer included in the subscription. Instead, you get $200 worth of credits, which at full API rates is 25x more expensive. So, how much does my agent stack cost? Am I somewhere close to $200? Uh, no. My automated systems alone would run north of $2,000-$3,000/month at those rates, and the all-in personal number is, well, higher. I do not begrudge Anthropic cracking down on this. I may have been following guidelines from Boris Cherny, head of Claude Code, on acceptable use... but still, that’s a lot of money multiplied across a ton of users. So an experiment that would otherwise have been “interesting but academic” suddenly had a budget attached to it. Dropping down from Opus to Sonnet for almost everything only got me down to around $1,000/month (and, for some of it, I didn’t get acceptable results from Sonnet). For part of it, I could use codex exec (OpenAI's equivalent of claude -p) on my $20 ChatGPT plan. Replace one frontier model with another. But $20 isn't enough—especially since I need GPT 5.5 for the harder tasks Sonnet fails at. The question is, can Qwen 3.6 27B (meaning 27B parameters) really step in for Sonnet (a model that likely has hundreds of billions of parameters)? Well, the public benchmarks seem to suggest it can. Qwen 3.6 27B holding its own against frontier offerings on Terminal-Bench. Qwen’s model card has plenty of other benchmarks too. Benchmarks don’t always reflect real performance, however. So I ran a few side-by-side benchmarks on the workloads I cared most about: daily briefing synthesis, chart annotation (especially for various medical applications and AI papers), and arXiv (research paper preprint archive) triage. Same prompts, same context, four models. For the paper-triage task, “quality” is partly a matter of taste, so I used Opus and Codex as a consensus jury—where they agree on the threshold, that’s the closest proxy I have to ground truth without hand-labeling everything myself. As a whole, the briefings were largely the same across all of them (so Opus was always overkill; Qwen is fine). Neither Qwen nor Sonnet was good enough for annotation (so I moved it to Codex). Finally, Sonnet was actually worse than Qwen for paper scoring. This means a 27B-parameter open-weight model, quantized to Q4, on a four-year-old consumer GPU, is doing comparable work to a paid mid-tier cloud API. Yes, it’s not Anthropic’s Opus, but it’s roughly matching Sonnet—a model still very much in Anthropic’s frontier lineup. Qwen 3.6 27B is a particular standout, and it’s unlikely to beat Sonnet on every workflow. However, it and many recent, “small” open-weight models are at everyday-use quality (not “small language models” but small “large language models”). Heck, Qwen 3.6 27B even has vision capabilities! While, as per Nathan Lambert, open-weight models have been in “perpetual catch-up” and underappreciated generally, what strikes me is how far down the hardware requirements curve we’ve come for “acceptable” results. After all of these moves—the scoring workload alone was ~$120/month if done on Sonnet but became $0—I got my projected claude -p budget to under $200/month with a comfortable margin. (As a note for astute readers, my own power cost is likely not truly $0—especially not being in the Bay Area and having PG&E grace me with some of the most expensive power rates in the country. Still, even with those rates, it’s an order of magnitude less. Definitely less than $0.004 per run.) Thanks for reading Weighty Thoughts! Subscribe for free to receive new posts and support my work. Type your email... Subscribe The Moving Frontier is “Model” At one point, an old consumer card like the 3090 Ti couldn’t run anything reasonably competitive with a frontier model. Now, it’s capable of running a model in the same league as a core offering from the top AI lab in the world. My hardware stayed constant—no Santa Claus secretly upgraded my GPU—so obviously something else must have changed. In August 2023, I wrote a piece called “Compute is Overrated as AI’s Bottleneck,” and the basic argument was that under the Model-Data-Compute framework, the model was doing more of the work than the breathless GPU extrapolations of the day suggested. For me, as per my book, I define “model” as all of the algorithms and techniques that go into making AI work—including post-training, RLHF... but also base improvements in the underlying model architectures themselves. At the time, the prevailing view was that AI training costs would exceed US GDP by 2035 if you just drew the curve forward. My argument—which I'm pleased to say held up—was that “MOAR compute!” mattered less than the architectural and algorithmic gains. Deep learning (and CNNs, transformers, and more) enabled the current boom, not just “infinite compute”—and new techniques are helping make it cheap enough that anything that can be AI, will be AI. (Though, as per my book, not everything will be AI… because not everything can be.) Source: Center for Emerging Technology and Security (AI and Compute Brief 2022). Obviously, this didn’t quite happen… Hardware vs. Software, with Actual Numbers So, I have a nice anecdote. What does this look like in the broader landscape? For the recent 2024-2025 window, the best available decompositions suggest that a majority of inference efficiency gains came from non-hardware technical progress—especially model-side or algorithmic improvements—rather than silicon alone. Hardware accounts for roughly one-quarter to one-third, depending on methodology. MIT FutureTech (Gundlach et al., Nov 2025) and Stanford Hazy (Saad-Falcon et al., Nov 2025). In MIT’s paper, the authors include non-hardware technical improvements, such as data, distillation, MoE, and related efficiency improvements. Stanford mainly focuses on model-side improvements (in local models across consumer/edge hardware). Regardless of methodology, both reports agree that most of the decline is not silicon. Perhaps not all of it is literal software, but it is “model” writ large, in the way I use it in Model-Data-Compute (“model” would have been confusing in the title without context, though). There’s also a useful natural experiment beyond my own hardware. NVIDIA’s own benchmarks show that H100 throughput on Llama 2 70B improved by roughly 1.5× over a year on identical silicon, from software updates alone. That’s a hardware-generation-sized gain delivered without buying new hardware. H200 added another ~28% on top, and Blackwell another ~3× on top of that—but the same-hardware software work is meaningfully larger than people give it credit for. On a much smaller scale, the same thing keeps happening to my setup! Software keeps making it better. While I was working on this piece (literally), a llama.cpp pull request adding multi-token speculative decoding merged and roughly doubled the throughput of my Qwen 3.6 27B on the same 3090 Ti: My throughput (speed) basically doubled overnight for free (for nitpickers, yes, I did have to slightly decrease my context window—but it basically makes no difference in practice given the rather marginal change). Model Improvements, Spotlighted Software improvements, tricks, and techniques… that’s great and all, but just to be clear, even though we’ve stuck with transformers, there has been a substantial shift in “models.” Among the notable advances in architecture or formulation have been MoE (Mixture of Experts), which many of the big models have been based on (most famously, DeepSeek v3/R1, which caused the “DeepSeek moment”), distillation (which helps larger models train smaller ones—enabling bigger model breakthroughs to “trickle down” to smaller ones), and quantization. Quantization is part of what helped Qwen 3.6 27B run on my 3090 Ti—which would barely fit the full size and likely have no context window (read here for a reminder on what that is and why it’s important). However, it has also allowed many hyperscalers and labs, similar to distillation, to bring down the cost of running these models while preserving most of their capacity/performance. Hardware’s Still Important… Just Not as Much as Most People Think Of course, hardware is still important. Chinese labs have a thing or two they could tell you about that… HBM (high-bandwidth memory) on Nvidia’s H200 versus H100—which have the same compute capacity, just more memory bandwidth—gave roughly a 40% “free” inference speedup on memory-bound workloads. That’s pure hardware improvement. Going from Ampere to Hopper to Blackwell for Nvidia has created significant improvements (that’s why people buy their chips!). Cerebras, an AI chip company, went public last week at a peak of just under $100B (and has since dropped materially...). Groq was purchased by Nvidia late last year for $20B. These hardware improvements obviously matter. Still, the unspoken consensus is that the main binding constraint of AI is GPUs and chips. As we’ve seen in multiple ways, that’s wrong. Why This All Matters In my book, I wrote that “everything that can be AI, will be.” The reason is this cost curve for AI inference. It’s faster than Moore’s Law, which is what got us from giant mainframes in 1980 to smartphones in our pockets, which are a million times more powerful than those room-sized machines. Epoch AI estimates frontier capability is now runnable on a single top-end consumer GPU within 6-12 months of being released at the frontier. That’s more or less what I saw (and was surprised by) when I ran Qwen 3.6 27B on my 3090 Ti. Of course, this also means that if you try to raise your prices (or, as Anthropic did, squeeze out certain use cases I have), people have a lot of other options. That puts a cap on the pricing power of the frontier labs. A long time ago (i.e., April last year), I discussed two possible paths for frontier labs—fixed costs for training could keep increasing and they could become natural monopolies... or overall costs could fall through the floor and their capabilities could become commodities. It’s not necessarily going to truly be that binary—and it’s still early on—but cloud prices for open-weight models are converging at the local hardware cost of electricity—roughly $0.20-$0.50 per million tokens. Anthropic commands a huge premium per token—which I’ve been happy to pay—but would most people stay if they 10x’d their price? 100x? I suspect not. Despite Anthropic's crackdown, I can keep running a lot of AI agents—because, for better or worse, I don't actually need Claude. I subbed it out with ChatGPT and Qwen without any issue. That says something about the future, even as Anthropic enjoys its time in the sun as the “leading lab.” After all, if things keep going this way, we’ll be running frontier-level models on our phones in 5-10 years. That’d be because of both hardware and software (though probably still mostly software). Thanks for reading! I hope you enjoyed this article. If you’d like to learn more about AI’s past, present, and future in an easy-to-understand way, I’ve published a book titled What You Need to Know About AI. You can order the book on Amazon, Barnes & Noble, Bookshop, or pick up a copy in-person at a local bookstore. Subscribe to Weighty Thoughts By James Wang · Launched 5 years ago VC on AI, deep tech, startups. Former Bridgewater, Google[x], startup founder. Read by top engineers, fund managers, and policymakers. Type your email... Subscribe By subscribing, you agree Substack's Terms of Use, and acknowledge its Information Collection Notice and Privacy Policy. 34 Likes ∙ 6 Restacks Discussion about this post Write a comment... Amit Gaur 16h But if software gets better isn't the demand for hardware go up? Jevons paradox or am I misinterpreting it? Like (1) Reply Share 1 reply by James Wang Discentix 3h Edited Nice post! As someone who tried to run local models on a 16GB M4 Mac Mini (and got horrible performance), running Qwen 3.6 27B on 2022 video hardware is impressive. Wondering what the surrounding machine is and how much the hardware costs (i.e., did you custom build a Windows machine?). Like Reply Share 1 more comment... Most AI startups are doomed Just because it matters doesn’t mean it’s defensible or profitable Sep 27, 2023 • James Wang 48 9 3 CUDA is Still a Giant Moat for NVIDIA Despite everyone’s focus on hardware, the software of AI is what protects NVIDIA Mar 23, 2024 • James Wang 52 9 6 How I Utilize AI Agents An Illustrated Tour of My Setup + Please Don't Use OpenClaw Feb 28 • James Wang 132 16 20 Ready for more? Type your email... Subscribe © 2026 James Wang · Privacy ∙ Terms ∙ Collection notice Start your Substack Get the app Substack is the home for great culture

That’s a thoughtful piece, and I think James Wang is pointing at something real: **the AI cost curve is increasingly a software/algorithm story, not just a “buy more GPUs” story.** But there are a few layers worth unpacking. ## 1. The core claim is mostly right: software is doing a surprising amount of the work The article’s strongest argument is this: > *Inference got dramatically cheaper even when hardware stayed constant.* That’s hard to ignore. Examples he gives are actually important: - **Quantization** → running 27B models in 4-bit instead of full precision. - **Distillation** → large models teaching smaller models. - **MoE (Mixture of Experts)** → sparse activation instead of firing the whole network. - **Inference optimizations** → speculative decoding, KV cache tricks, batching improvements. - **Better architectures/post-training** → RL, synthetic data, alignment refinements. Those are not small tweaks. They’re massive multipliers. A 2022 RTX 3090 running something competitive with a mainstream frontier cloud model would have sounded absurd a few years ago. He’s also correct that **software-only improvements routinely unlock hardware-generation-sized gains**. --- ## 2. But hardware is probably being understated The title says: > “AI’s plummeting prices are a software story, not a hardware one.” I’d soften that. It’s closer to: > **AI cost decline is a software–hardware coevolution story where software currently gets under-credited.** Because hardware still matters enormously. Training frontier models still depends on: - HBM bandwidth - interconnect topology - networking fabric - accelerator memory capacity - tensor compute efficiency - datacenter power/cooling A 3090 can run an open model. A 3090 cannot realistically train GPT-class systems. Different problem. --- ## 3. Jevons paradox absolutely applies here (the commenter is onto something) The commenter asking about **Jevons paradox** is asking a good question. The paradox: > **Efficiency improvements often increase total consumption rather than decrease it.** Example: If tokens become 100× cheaper… people don't spend 1/100th as much. They do: - longer contexts - more agents - persistent memory - multimodal inference - video generation - background automation - swarms of autonomous processes Exactly what Wang himself describes. His agent stack didn’t shrink. It exploded until costs became painful. That is classic Jevons behavior. So both things can be true: - **unit inference cost falls** - **total hardware demand rises** That’s basically what we're watching happen right now. --- ## 4. The local-model argument is becoming hard to dismiss This may be the most important long-term part of the article. A few years ago: Local = hobbyist toy. Today: Local often means: - coding - summarization - research triage - retrieval workflows - private document work - automation pipelines - offline inference And increasingly… *good enough*. That changes economics. Because cloud labs are no longer competing against “nothing.” They’re competing against: - local Ollama stacks - consumer GPUs - Apple silicon - open weights - cheap inference providers That creates pricing pressure. --- ## 5. The frontier moat question is the big unresolved issue This is where the article becomes strategically interesting. If: - open models compress capability gaps quickly - inference costs collapse - local hardware improves then frontier labs risk becoming partially commoditized. But the counterargument is: Frontier labs may keep moats via: - proprietary reasoning improvements - closed post-training pipelines - tool integration - memory/orchestration layers - enterprise ecosystems - safety/compliance - massive training budgets Think of it like: Open source Linux exists. Cloud hyperscalers still make enormous amounts of money. Capability ≠ business defensibility. --- ## 6. The user perspective: this article matches what many practitioners are quietly experiencing This part rang true: > “I don’t actually need Claude.” That’s increasingly common. A lot of people are drifting toward a stack like: - hardest tasks → premium frontier model - routine coding/research → strong open model - automation → local inference - batch workloads → cheap cloud provider Hybridization. Not “one model to rule them all.” --- My short takeaway: **The article is directionally correct.** AI cost collapse is being driven heavily by **algorithms, compression, inference engineering, and model design improvements**, not merely newer GPUs. But I’d revise the thesis slightly: > Software is not replacing hardware as the story. > > Software is making existing hardware dramatically more productive — which then changes the economics of the entire AI ecosystem. And if that trend continues, the practical question becomes less: > *“Who owns the biggest GPU cluster?”* and more: > *“What level of intelligence becomes commodity compute?”* That’s a very different world.

This is a strong argument, and it lines up with a lot of what practitioners have been quietly observing: **AI cost declines are increasingly a software story, not just a hardware story**. But there are a few places where I’d tighten or challenge the thesis. ## The core claim is mostly right The article’s main idea is: > inference got dramatically cheaper because models, algorithms, quantization, distillation, runtimes, and serving techniques improved—not simply because GPUs got faster. That is broadly true. If you freeze hardware and compare 2023 → 2026: - better architectures (Mixture-of-Experts, attention improvements, speculative decoding) - quantization (Q4/Q5, AWQ, GPTQ, GGUF, KV cache tricks) - distillation and synthetic data - inference runtimes (e.g., llama.cpp, vLLM, TensorRT optimizations) - routing and agent orchestration improvements have delivered huge gains. The important observation is: **same GPU → much better capability** A 3090 running a competent 27B model in 2026 versus what a 3090 could realistically run in 2023 is night-and-day. ## But “software vs hardware” is a false dichotomy The article slightly overstates the split. A better framing is: > software multiplies hardware efficiency. The reason quantization matters is because memory bandwidth exists. The reason speculative decoding matters is because GPUs parallelize well. The reason local models became usable is because VRAM capacity and CUDA ecosystems matured. For example: - a 3090 (24GB VRAM) was unusually future-proof - CUDA optimization mattered enormously - consumer GPUs became capable enough to host meaningful context windows So hardware didn’t stop mattering—it created the substrate on which software compounded. This is closer to: > hardware gives ceilings, software compresses costs toward those ceilings. ## His strongest point: “good enough” beats frontier for most workflows This is the part I think people underestimate. Most users don’t need the best model. They need: - drafting - coding assistance - summarization - research triage - workflow automation - classification - structured extraction For that, a competent open-weight model is often enough. If your task quality curve looks like this: | Model | Quality | |---|---:| | frontier model | 100 | | good open model | 88 | | weak model | 45 | and the price curve is: | Model | Cost | |---|---:| | frontier | $$$$ | | open/local | near-zero | then economics become brutal. A company paying 20× more for a 12% quality improvement only works when that delta materially changes outcomes. For many workflows, it doesn’t. That’s why you’re seeing pressure on premium APIs. ## The real economic force is commoditization The article implicitly argues frontier labs face a ceiling on pricing. I think that’s directionally correct. If: - open models keep catching up - inference keeps collapsing in price - local hardware improves - orchestration becomes model-agnostic then premium labs become constrained. The economic question becomes: > What remains scarce? Possible answers: 1. **best reasoning** (hard problems, scientific research, deep planning) 2. **tool ecosystems and integrations** 3. **trust, safety, reliability** 4. **latency + infrastructure** 5. **workflow products rather than raw models** In other words: The moat may shift from “the smartest model” to “the best system.” This is similar to how databases commoditized but enterprise software still made money. ## The article undersells Jevons paradox The commenter asking about Jevons paradox is asking the right question. entity["people","William Stanley Jevons","economist associated with Jevons paradox"]’s idea: > when efficiency rises, usage often rises faster. Historically: - cheaper compute → more software - cheaper bandwidth → streaming everywhere - cheaper storage → infinite photos/video AI likely behaves similarly. As inference gets cheaper: - more agents - longer context - more multimodal processing - always-on assistants - embedded AI in every workflow So yes: **cost per inference falls, while total compute demand may explode**. This is actually bullish for GPU demand even if software keeps improving. That’s why companies like entity["company","NVIDIA","GPU company"] can still win even if efficiency improves. Efficiency lowers cost and expands market size. ## The “frontier on your phone” prediction I’d temper this. “Frontier-level” depends on definition. A phone in 2031 probably runs: - an excellent local personal model - personalized memory - lightweight reasoning - multimodal capabilities But the absolute frontier will likely also move. You don’t catch the frontier—you catch **yesterday’s frontier**. That’s been true in computing forever. Your laptop today exceeds old supercomputers, but not current ones. So the likely future is: > 2031 phones run something equivalent to a 2026 frontier assistant. Still transformative. ## What this means for people building things For builders (especially people doing local workflows like you often experiment with), the practical implication is huge: Stop asking: > “Can I match GPT-5.5?” Ask: > “What is good enough for this workflow?” Examples: - SEO drafting → local - structured extraction → local - ranking/scoring → local - summarization → local - coding autocomplete → local - difficult reasoning/debugging → frontier That hybrid architecture is increasingly rational. You pay for intelligence only where intelligence matters. That is probably the deepest truth in the whole essay.