We Put a 120-Billion-Parameter Model on a Desk: The Economics and Discipline of a Local AI Fleet

Every AI-native product has two kinds of model calls. The first kind has a human waiting: a chat reply, a suggestion appearing under a field. The second kind has nobody waiting: overnight sweeps that grade a content library, batch passes that decide whether two uploaded documents are duplicates, parsers that turn invoices into rows. In T2D3 OS, that second kind is most of the volume — and this week we moved its largest-model work onto a computer that sits on a desk.

The numbers first, because they are the argument. The new node — a Framework Desktop with an AMD Strix Halo processor and 128GB of unified memory, roughly a $2,500 machine — serves a 120-billion-parameter model at a measured $0.05 per million input tokens and $0.70 per million output tokens, power and hardware amortization included. The cloud tier this work previously ran on lists at $2 to $25 per million output tokens depending on the model. For background judgment work, that is not a discount; it is a different budget category. Volume stops being a cost decision at all — which is exactly what our frontier-quality principle demands: the price of tokens must never be the reason the work gets less thorough.

How a desktop runs a 120B model (and why memory bandwidth is the number that matters)

Three things had to become true before a desk-sized computer could serve models this large, and all three happened in the last two years.

Unified memory made the capacity problem cheap. A 120B-parameter model quantized to ~4 bits needs about 61GB just for its weights. On a traditional PC that means multiple high-end GPUs. On a unified-memory machine — where CPU and GPU share one large pool — a single 128GB desktop holds it with room for context. This is the same architecture trick that lets a MacBook run large models; the Strix Halo brings it to a mini-PC price.

Quantization made the quality cost acceptable. Compressing weights from 16 bits to ~4 bits loses little measurable quality on judgment tasks — classify, grade, compare, extract — which is precisely the work we route locally. We do not have to take that on faith: every routed task family gets evaluated against the model it replaces before traffic moves (more on that below).

Mixture-of-experts made big models fast on slow memory. Here is the counterintuitive part. On this machine, our dense 70B model generates about 5 tokens per second, while the 120B model generates 41 — the bigger model is eight times faster. The reason is memory bandwidth. Generating one token requires reading every active parameter from memory, and this class of machine moves about 256GB/s — a quarter of a high-end GPU. A dense model reads all 70 billion parameters per token; a mixture-of-experts model activates only ~5 billion of its 120 billion per token, so each token costs an eighth of the memory traffic. When you shop for local-AI hardware, the spec sheet number that predicts your experience is not TFLOPs and not even memory size — it is memory bandwidth divided by active parameters.

That asymmetry decides our routing: the fast 120B MoE is the workhorse; the slow-but-dense 70B stays available for the rare task where dense-model quality wins and nobody is waiting.

The discipline: a fleet is only cheap if it is boring

This is our second node, not our first — a 4090/5090 GPU rig has served our fast local tiers for months. The lesson from running the first one is that the hardware is the easy part. What makes a mixed local/cloud fleet safe is a set of unglamorous rules:

Routing keys off model identity, never load balancing. Each node has a distinct catalog — the new box carries the large models the GPU rig cannot fit; nothing is mirrored. A prompt pinned to a model routes to the one node that serves it, and if that node is down, the request falls back to a cloud model of equivalent quality — never sideways to the other box, which doesn't have the model anyway. Fallbacks go up in quality, never down.

Cost is measured, never asserted. "Local is basically free" is how self-hosting decisions rot. Every local model carries a real price — watts drawn, hardware amortized over three years, divided by measured tokens per second — and every call is logged with its cost in the same ledger as our cloud API calls, so the finance view compares fleets honestly. When the new node came online, its catalog carried placeholder prices flagged as placeholders; they were replaced with benchmarked numbers the same day, and the ledger records which is which.

Latency posture is written down, and enforced by review. The new node's memory bandwidth makes it a batch machine, full stop. Our routing rules name the surfaces that must never run on it — anything a user is actively waiting on stays on the fast rig or in the cloud. Before we moved any traffic, we timed the worst realistic case end to end through the tunnel that fronts the box — a forced model swap plus a full grading call: 55 seconds, comfortably inside every timeout budget in the path. Measure the ugly case before it measures you.

Silent degradation gets hunted. The most instructive thing this project fixed was not a failure — it was a success that had quietly stopped being one. Weeks ago, the GPU rig's capable 35B model was retired to make room for image generation, and its gateway began transparently serving those prompts with the small 9B model instead. Everything kept working; the quality tier had silently dropped, invisible unless you read the serving logs. Because our call log records which model actually served each request — not just which was asked for — the drift was visible and attributable, and the new node's first routed traffic was exactly those two prompts: content-library grading and duplicate-signal adjudication, back on a large model, now at 1/35th of the cloud price. If your AI stack cannot tell you which model really answered, you do not know what quality you are shipping.

What we refuse to run locally

The honest list matters as much as the wins. Nothing user-waiting goes to the new node — a 5-token-per-second model behind a chat box is a resignation letter. Nothing tool-using goes to any local node; tool-calling prompts stay on the frontier models where our security posture keeps them. And no frontier-judgment work moves local to save money: strategy synthesis, customer-facing writing, and anything that grades our own AI's output stays on the strongest models we can buy. Local inference is task fit — high-volume, structured, background judgment — never a down-tier.

The pattern generalizes beyond marketing software. If your product makes background AI calls in volume, the combination of unified-memory hardware, aggressively quantized open-weight models, and mixture-of-experts economics has quietly moved the break-even. A desk-sized machine now serves 120B-parameter judgment at cents per million tokens. The engineering that makes it responsible — identity-keyed routing, measured cost, written latency posture, and a log that names the model that actually answered — is the part most teams skip, and it is the part that lets you sleep.