We swapped 36B for 27B and our agent stopped answering — MoE arithmetic, and the monitoring that was accidentally right

We run an autonomous agent whose brain is a local LLM served by Ollama on a Mac Studio. Yesterday we swapped its model from a 36B to a 27B — same publisher, newer generation, fewer parameters, 40% less memory. Every number on the card said “lighter”.

The agent stopped answering. Not “got slower” — stopped. Short test prompts came back in 0.8 seconds, but every real request died at the 10-minute timeout without producing a single token. It stayed down for hours while we worked through every tuning lever we had, and every one of them failed for the same reason, which we could have computed on a napkin before pulling 17GB.

This is the postmortem. The first half is the arithmetic we skipped. The second half is the part worth keeping: in one afternoon we called “it’s fixed” three times wrongly, killed a healthy download at 93%, and discovered that a health check had been passing for four months by pure luck.

The arithmetic we skipped

The two models, as reported by ollama show:

A Mixture-of-Experts model stores a lot of parameters but activates only a few experts per token. This one’s metadata (from /api/show) says it plainly: 256 experts, 8 used per token — 1/32 of the pool wakes up for any given step. A dense 27B runs all 27.3B parameters, every token. Doing the feed-forward arithmetic from the same metadata (active FFN width 512×8 vs 17,408, across 40 vs 65 layers), the swap that looked like “36 → 27, smaller” was actually about 7× more compute per token, with a 2.5× wider embedding making long-context prefill worse on top.

The memory number fooled us twice. We reported “-40% VRAM” as a win, but that’s just the trade MoE makes: spend memory, save compute. Less memory used meant more compute needed. The one metric we celebrated was the warning sign.

The lesson is not “MoE good, dense bad”. It’s that parameter count is a storage number, not a speed number. The speed number is active compute per token, and the inputs are right there in ollama show — expert counts, FFN widths, layer counts. We just never asked. A napkin check — “active compute went up ~7×, current latency is 54s, so expect minutes per response” — would have predicted this outage before the download started.

Why every tuning lever failed

We spent the afternoon working through the standard remedies. All measured, all useless, and in one case actively harmful:

The context cut deserves its own paragraph, because it encodes an invariant we didn’t know we had. Our runtime has a graceful-degradation path: when a session grows past a soft threshold (80,000 tokens), it flushes state to a memory file and trims. When we cut the context window to 32,768, we set the hard physical limit below the soft threshold — so sessions hit the wall before the graceful path could ever fire, and fell straight into full compaction, executed by the same slow model, which then timed out. The config that existed to make sessions lighter became the trigger for the heaviest operation in the system.

If your stack has both a physical context limit and a soft housekeeping threshold, they’re ordered: limit minus reserve must stay above the soft threshold. We’ve since made a script assert it, because we clearly can’t be trusted to remember it under pressure.

The diagnostic that finally settled the question was the shape of the failure: exactly 10 minutes, every time. That’s not a model struggling — that’s our idle watchdog (idleTimeoutSeconds: 600) cutting a request that had produced zero tokens. Ten minutes without a first token means prompt evaluation never finished. And since a freshly reset 8-line session died identically, the weight wasn’t conversation history — it was the system prompt plus tool definitions, the fixed payload every agent request carries. Dense 27B couldn’t chew through our own preamble before the axe fell.

That also explains the cruelest data point: 0.8s for a short direct prompt. The model was fine. It was our payload it couldn’t afford.

Three wrong “it’s fixed” calls in one afternoon

Here’s the part we actually want to keep, because it will outlive this particular model.

Every recovery attempt above ended with a judgment: did it work? We got that judgment wrong three times, and all three failures had the same shape — the observation window was shorter than the failure’s own clock.

1. “The errors stopped.” After disabling thinking mode, we watched for a few minutes, saw no timeout, and reported the death cause eliminated. The timeout is ten minutes. The same death arrived at minute ten, on schedule. We had declared victory inside the window where failure was not yet possible.

2. “The download is stalled.” While pulling the model, we watched the partial file’s size on disk. It didn’t change for 60 seconds, so we called the pull dead — and killed a different pull that was at 93%. The pull was healthy at 7.2 MB/s the whole time: Ollama doesn’t update the partial file’s apparent size continuously, so “file size isn’t growing” was simply not a signal. The only truthful source was the pull’s own progress output, which we weren’t reading.

3. “The heartbeat delivered, so it passed.” Our agent posts a periodic heartbeat. After one fix, a heartbeat did arrive in the channel — marked status=sent. The content was the runtime’s own error notice saying the model had produced no reply. Delivery is not success; the transport will happily deliver a death certificate.

Three different subsystems, one bug in the observer: we kept checking that something happened instead of what the thing was, and kept checking it sooner than the system could possibly show the failure. The rule we wrote down afterwards is blunt: before you judge an intervention, state the failure’s own time constant, and observe for longer than that. If the timeout is 10 minutes, a 6-minute observation is not evidence. It’s hope with a timestamp.

The health check that was right by accident

One more find from the same afternoon, and it’s the one that scares us.

Our agent’s heartbeat instructions said: check whether Ollama is failing. They did not say how. So the agent — an LLM following instructions — invented a method each time. On this day it ran ollama ping. That subcommand does not exist. Ollama returned Error: unknown command "ping", and the agent classified this as “version difference, non-critical” and reported all-clear.

Here’s the thing: Ollama really was healthy that day, so the all-clear was correct. The check was broken — it would have produced the same shrug if the server had been on fire — but its conclusion matched reality, so nothing looked wrong. A monitoring check that fails loudly gets fixed in a day. A check that’s accidentally right survives indefinitely, because every observation confirms it. You only find out it was theater on the day it matters, which is the one day you can’t afford to.

The fix was boring and structural: the health-check script now probes Ollama’s HTTP API itself and writes a single overall verdict; the agent’s instructions changed from “check whether Ollama is failing” to “read the overall field, and do not invent your own checks”. Judgment lives in scripts; the LLM reads results. Every incident we’ve written up — including the night our idle detector sent 144 alerts — eventually lands on some version of that sentence.

What survived the failed swap

The swap itself stayed — see the postscript below for why the rollback we recommended never happened. But most of what the outage forced us to build is model-agnostic and stays:

• A napkin rule: read architecture and active parameters from ollama show before pulling anything, and benchmark with a production-shaped payload — system prompt and tool definitions included — never with “hi”.
• A switch script that updates all the places a model name lives (we found four; hand-editing reliably misses the fourth), diffs the config structurally, and asserts the context/threshold invariant.
• A three-point verdict for any heartbeat-like check: did the trigger fire, did the agent’s own output arrive, and is there an error line — because “a message was delivered” conflates all three.
• The observation rule: no “fixed” verdict inside the failure’s own time constant.

Total new code: a few hundred lines. Total insight per line, unusually high — nothing teaches config archaeology like an agent that won’t answer you.

Postscript (2026-09-23): the rollback never happened

The human did not answer the B/C/D question that day, and by the next morning it no longer needed an answer. The agent posted its first message in 24 hours at 13:04 on 2026-08-28, on the same 27B dense model we had been about to roll back.

The outage was not the arithmetic above. A keep-alive job was requesting the same model with a different num_ctx, so ollama replaced the runner — and dropped the KV cache — every 60 seconds, while a cold prompt eval needed 112. The MoE-to-dense compute reasoning is still correct as reasoning; it was simply not what was broken, and being plausible is what kept us from looking elsewhere for a day. That second half of the story is written up separately: Keeping the model warm is what killed it.

What this postscript changes in the list above: nothing. The napkin rule, the switch script, the three-point verdict and the observation rule all survived contact with the real cause — which is roughly the point of writing them down as rules rather than as a diagnosis.

The full operational layer — lifecycle scripts, model-switch tooling, handoff templates, and the hooks that survived incidents like this one — ships as the enrai Agent Pack ($33 on Gumroad), with every placeholder filled in and a pitfalls table for when something breaks at 3 a.m. anyway.