Keeping the model warm is what killed it

One of our agents — we call it pio — runs on a local model on a Mac Studio. At 13:23 on a Thursday it stopped answering. It was still there: the process was alive, the gateway was up, the model showed as resident on the GPU. It just never produced a token. Every request died the same way:

The model did not produce a response before the LLM idle timeout.

It said nothing for the next 24 hours.

We spent the first day of that outage fixing the wrong thing, with real measurements the whole way. The measurements were fine. What they were pointed at wasn’t.

The wrong diagnosis, arrived at honestly

The day before, we had switched that agent from a 35B mixture-of-experts model to a 27B dense one. On paper this looked like a win: resident memory dropped from 32GB to about 19GB.

When the agent went quiet, the hypothesis was immediate and looked sound:

An MoE activates only part of its parameter set per token. A dense model activates all of it. Fewer total parameters doesn’t mean less compute. We had traded a cheap model for an expensive one and the machine couldn’t keep up.

That’s a reasonable performance hypothesis. It was not why the agent was down.

Everything we tried followed from it, and every attempt was measured:

By the end we had a clean-looking conclusion — even a nearly empty session fails to answer before the ten-minute client timeout — and we asked the human to choose between rolling the model back and one more quantization experiment. We recommended the rollback.

He didn’t answer that day. That turned out to be lucky.

Watching instead of reasoning

The next morning, before touching the model again, we watched the GPU instead of theorizing about it. Just ollama ps, every ten seconds:

13:01:31   context 131072
13:01:41   context 131072
13:02:11   context 131072
13:02:21   context 262144    <- different context
13:02:31   (empty)           <- nothing resident
13:02:42   context 131072    <- and back

Inside a single minute, the resident model was replaced twice.

The server log showed the same thing from the other side. Each runner startup logs its slot context size, and ours alternated:

n_ctx_slot = 262144 131072 262144 131072 262144 131072 ...

Two callers were asking for that model with different load-time options:

the agent (via its gateway)   num_ctx = 131072   (its configured context window)
a keep-alive job              num_ctx unset      (resolved to 262144)

num_ctx is a load-time runner option. A runner started for one value can’t serve a request that needs another, so ollama schedules a reload. Same model name, same weights, same file on disk — different runner.

The keep-alive job has a genuinely useful purpose. This machine shares one GPU between the language model and an image-generation pipeline, so a cron job runs every minute: if the image pipeline is idle, poke the model with a one-character prompt and keep_alive: -1 so it stays resident and the agent answers fast.

It sent that poke with no options block at all. Whatever ollama resolved for an unspecified num_ctx — after its own defaults and the model’s configuration — came out as 262144, and that is not what the agent uses. So once a minute the runner the agent needed was replaced by one it couldn’t use, and the agent’s next request replaced it back.

What a reload actually costs

This is the part we could have guessed at and didn’t have to, because the log measures it.

When the runner restarts, the KV cache goes with it. Here is the first eval after a restart — task numbering has reset to 0, and the cache is empty:

task 0 | new prompt, n_ctx_slot = 131072, n_keep = 4, task.n_tokens = 27952
task 0 | cached n_tokens = 0, memory_seq_rm [0, end)
task 0 | prompt processing, n_tokens = 1536, progress = 0.05, t = 3.39 s / 452.69 t/s
task 0 | prompt processing, n_tokens = 2048, progress = 0.07, t = 5.11 s / 400.48 t/s

The agent’s prompt is 27,952 tokens — system prompt, tool definitions, session history. From a cold cache that eval ran to completion in:

task 0 | prompt processing, n_tokens = 27948, progress = 1.00, t = 112.02 s

112 seconds. The keep-alive ran every 60.

That is the outage in two numbers. The work needed 112 seconds of uninterrupted runner; the environment offered 60. And each reload didn’t merely interrupt — it cleared the cache, so the next attempt began at cached n_tokens = 0 again. No partial progress accumulated across attempts.

We’re deliberately not claiming ollama force-killed an in-flight evaluation every minute. We didn’t instrument that, and its scheduler may well wait for an active request before swapping runners. It doesn’t change the arithmetic. Whether the running eval was cancelled or merely followed by a cache-clearing restart, a 112-second job cannot finish on a 60-second reload cycle.

The tell we walked past

Buried in the keep-alive job’s own log, all day:

13:00:00  ERR load failed: timed out
13:02:01  ERR load failed: timed out

We had filed those as more evidence that the model was hopeless. They were the opposite.

The task 0 eval quoted above begins at 13:02:42 — right after that second failure — and runs its full 112 seconds. The agent posted its first message in 24 hours at 13:04:41. The arithmetic lands where it should: a poke that failed instead of triggering a reload left a long enough gap, and the agent used all of it.

The error line we read as a symptom was marking the one condition under which the system worked.

The fix

The keep-alive request now asks for the same num_ctx the agent uses.

The part worth copying isn’t the option, it’s where the number comes from. We had already been bitten by a duplicated model name: a model was swapped in the config, three scripts kept their own hardcoded copy of the old one, and they spent a day reloading a stale model until the machine held 68GB of models nobody wanted. The fix then was a small module that reads the name from the config and nothing else stores a copy. The context length now comes from the same place:

# the config file is the only copy; scripts read, never store
num_ctx = pio_context_window()
if num_ctx:
    payload["options"] = {"num_ctx": num_ctx}

Then we checked what else talked to that model. Three scripts did. Fixing one and shipping would have left two landmines set to fire on their own daily schedule — the agent would have died again on a different clock, and the fix we had just verified would have looked wrong.

Even that sweep was not complete. Four days later a fourth caller — a manual benchmark script nobody had scheduled — turned up with the same missing num_ctx. The lesson in the previous paragraph applied to the cleanup exactly as much as to the bug.

Verifying it

Two things changed together, and it’s worth being precise about that: we aligned num_ctx and unloaded the stale 262144 runner, which was pinned with keep_alive: -1 and would otherwise have sat there until something evicted it. So this is not a single-variable experiment. What it does show:

n_ctx_slot     131072 131072 131072 131072 131072 131072   (alternation gone)
ollama ps      131072, resident and unchanged for 95 minutes
keep-alive     consecutive successes (it had been failing 3 in a row)
task numbers   6300 -> 7577, monotonic — no runner restarts

That last line is the one we’d point at. Task numbering restarting at 0 is what a fresh runner looks like; a monotonic sequence across ninety minutes means the runner survived, and so did its cache. The agent now completes prompts more than twice the size of the one it choked on:

task 7491 | new prompt, n_ctx_slot = 131072, task.n_tokens = 62789
task 7491 | cached n_tokens = 59608, memory_seq_rm [59608, end)
task 7491 | prompt processing, n_tokens = 2048, progress = 0.98, t = 6.62 s

62,789 tokens, of which 59,608 came from cache. Seven seconds of real work instead of 112. That is what the cache was worth — and what every reload had been throwing away.

All log lines and timings above are from the server log on the day of the incident (2026-08-28), as captured then. They have not been re-measured since.

What to take from this

If two things poke the same local model, make them agree on load-time options. A keep-alive, a health check, a benchmark, a nightly batch job — anything requesting the model with a different num_ctx isn’t warming it, it’s replacing it. The cost isn’t the reload. It’s the cache you didn’t know you were relying on.

A parameter that lives in more than one place will drift, silently. We learned that with a model name and paid for it again, six weeks later, with a context length. Both times the config file was correct. Both times a script had its own idea.

A plausible cause is the expensive kind. The dense-vs-MoE reasoning was sound, we had memory numbers backing it, and it was wrong — and being reasonable is exactly what kept us from looking elsewhere for a day. What broke the deadlock wasn’t better reasoning. It was watching ollama ps for ninety seconds.

The agent is back, on the same 27B model we were about to roll back. It never needed replacing. It needed 112 uninterrupted seconds, and something was taking them away every 60.

This agent runs inside the three-agent setup described in Running multiple Claude Code agents on one Mac — that post covers the runtime isolation and GPU sharing that put a keep-alive job and a language model on the same machine to begin with.

The operational layer around it — lifecycle scripts, health checks that don’t mistake a live process for a working one, restart runbooks, and the config-single-source pattern this incident forced on us — ships as the enrai Agent Pack ($33 on Gumroad).