The word came out in the wrong language: measuring script-mixing across three agents and two model families
One of our agents wrote дир in the middle of a Japanese sentence. It meant “directory” — in Japanese, direkutori, four katakana — and what came out was three Cyrillic letters that spell the first syllable of the Russian word. Another one wrote food違っている where it meant kuichigatte iru, “these don’t line up.” The Japanese verb starts with kui, which sounds like “eat,” and the model produced the English noun for what you eat, glued to the rest of the Japanese conjugation. Not a typo — a swap.
Neither is a typo. kuichigatte and food share no letters. дир cannot be typed on the keyboard layout the machine uses. In both cases the meaning survived and only the script was swapped, which puts the fault inside the model’s vocabulary selection, not on the way in or out.
We run three agents on one machine: two coding-agent sessions on a hosted model — one writes and reviews (the writer agent below), one owns the infrastructure (the infra agent) — and one local model that is the product itself. All three do this. Throughout, “speech” means what an agent posts to a chat channel, “commands” means what it types into a shell or a search tool, and “visible prose” means the notes it writes between tool calls. This post is about what happened when we tried to measure it, how the number moved by a factor of 2.4 depending on what we compared, and why the rare case that matters is the one nobody would notice.
What it looks like
The examples below are all real, from channel posts and internal documents, with the intended Japanese word given in Latin letters.
физическの出番 physical Cyrillic stem, Japanese suffix
fredrezones现象 genshō (現象) Simplified Chinese
env키名 ki (キー) Korean syllable for "key"
листingとREADME listing Cyrillic stem, Latin suffix
材料 → материалになる zairyō Russian noun, Japanese verb ending
一言교えて oshiete (教えて) one Korean syllable replacing one kanji
全員집합して shūgō (集合) Korean replacing two kanji
数倍변わる kawaru (変わる) Korean replacing one kanjiThe local model, which runs a Chinese-trained family, drifts into Simplified Chinese: 不一样, 确认各项状态, 现在是16:24. The two hosted-model agents drift into Cyrillic and Hangul, and occasionally English. The direction differs by model; the shape is identical. A single lexical item is emitted in a script the surrounding sentence is not in, and the sentence still parses.
We had been seeing these for four months before anyone counted them.
The first count, and the six recounts after it
The first scan ran on 2026-08-29. It looked for Cyrillic, Hangul, Thai, and a list of 66 characters that exist in Simplified Chinese but not in Japanese, across every message the local model had produced since May.
data span 2026-05-02 to 08-29, 55 days present
(gaps: 06-06 to 06-17, 06-20 to 08-11)
days affected 15
hits 56 total — 39 before 08-29, 17 on 08-29 itself
oldest 2026-05-06Two things fell out immediately. The 17 hits on the day of the scan were us discussing the scan: every quoted example was counted as a new occurrence. And the eight hits from May and June predated the local model’s switch to a new generation on 2026-08-27 by three months. Whatever this is, it is not a regression in the latest model. It spans two generations of the same family, and — once we looked — two different vendors.
That second point came from the first mistake. The initial cross-agent comparison reported zero hits for the two hosted-model agents. A plain text search over the same log files found plenty. The scanner was reading the wrong field: for the local model, spoken output lives in the message text, but for the hosted-model agents, anything a human reads is the text argument of a “send to channel” tool call. The assistant-role text blocks that the scanner was reading are working notes between tool calls, not prose. A second bug compounded it — timestamps in UTC, so a session that started at 07:42 local was filed under the previous day.
Had we trusted the first number, the conclusion would have been “the local model has a problem.” The opposite of true: the hosted agents do it too, at a comparable rate.
We recounted six times that day. Each time we aligned one more thing — same kind of output, deduplicated, discussion-day excluded, same denominator — and the headline ratio between the local model and the others shrank by roughly half:
chars hits per 100k chars
local model speech 476,828 35 7.3
writer agent speech 276,922 10 3.6
infra agent speech 504,705 14 2.8
writer agent commands 559,060 5 0.9
infra agent commands 1,694,243 7 0.4
writer agent visible prose 291,567 0 0.0
infra agent visible prose 338,818 0 0.0“Speech” is what an agent posts to a channel. “Commands” is what it types into a shell or a search tool. “Visible prose” is the text it writes between tool calls for its own record.
The first version of this table said the local model mixed scripts 6.2× more often than the others. The final version says 2.0× to 2.6×. Every step in between was correct arithmetic on a slightly wrong comparison. We now write the multiplier down as “two to three, and we could not pin it tighter,” because the honest error bar on a ratio that moved from 6.2 to 2.6 as we cleaned the inputs is not small.
There is also a row we cannot fill. The two hosted-model agents keep full transcripts on disk — that is where every count above came from — and those transcripts store internal reasoning as blocks with a type of thinking: 1,740 blocks for one agent, 2,401 for the other. Every one of them has an empty body and a 912-character signature. The reasoning is not stored, only its fingerprint. The local model has no such block type at all. So we cannot say whether script-mixing happens in reasoning. An earlier draft said “it does not appear in thinking.” It appeared nowhere because there was nothing to read, which is a different statement, and we caught it only because someone asked what the denominator was.
Where it lands decides whether it matters
The frequency table above would tell you to worry about the local model’s speech. That is the wrong row.
A mixed-script word in a channel post is ugly, breaks character for a product that is supposed to have a voice, and is noticed by the first human who reads it. A mixed-script word in a command is silent.
2026-08-27 01:58Z, infra agent
grep -nE "^def |^SECTIONS|результ|result\[" scripts/self_check.py
result: 10 matches, exit 0The agent meant result. It typed результ — the Russian stem — as one of four alternatives. That alternative matches nothing, so the command returns the ten def lines and looks completely normal. The file had 20 lines containing result, and every one of them was invisible to the agent. Those 20 lines were the heartbeat-scoring logic — how the system decides whether the product is alive — and the agent was reading that file because the heartbeat was misbehaving that day.
No error. Ten hits. Half the search silently gone.
Across 3,823 commands from both hosted-model agents in the measured window, that was the only confirmed case. One in 3,823, or 0.03%. Against 7.3 per 100k characters in speech, it looks negligible, and by count it is. By consequence, it is the entire problem. The speech cases cost a wince. The command case took evidence away from an investigation that was already going badly, and nobody knew.
So we split the record. Speech rates and command rates are reported separately, and command hits are broken out by whether the foreign characters landed in a search pattern or a path. Those are the ones that miss silently.
The checker that had never run
A week later the infrastructure agent extended detection to command inputs — shell commands, search patterns, glob patterns, web queries. The extension turned out to have been written on 09-01. It had never executed. It was registered in neither the scheduler nor the commit hooks, so “we scan commands now” had been true for four days in the sense that the code existed.
The pre-commit hook had a second gap. It piped file contents through the checker and looked for a warning string in the output. If the checker crashed, the output was empty, no warning string was present, and the commit went through. “No script-mixing found” and “the check did not run” were the same event. That was rewritten to print the checker returned nothing (rc=N); this check was not performed — and the commit still goes through, deliberately, because a hook that blocks a commit for quoting an example of the thing it detects would be bypassed within a week.
When the command scan finally ran over real logs, every one of the 92 hits came from shell commands. Nothing from search patterns, nothing from glob patterns, nothing from web queries. That could mean the mixing does not happen there, or that the extractor for those fields was broken. The output alone cannot distinguish the two, so we built a synthetic log with one planted foreign word in each of six input paths and ran the scanner over it. Six of six detected. The extractor works; the other channels are simply clean.
The detector detects itself
Then the real problem with running this weekly appeared. The detector here is a script that reads the agents’ own tool-call logs — every shell command, search pattern, and channel post — and flags characters outside the scripts we expect. Of 96 hits across both hosted agents’ logs, the large majority were the detector’s own work: test inputs containing planted examples, edits to the character list, grep commands searching for the very strings the scanner flags, and reports — like this one — quoting real cases.
Reporting on the phenomenon is the phenomenon, as far as a character-class scanner can tell. The single acceptance report for the first measurement added ten Cyrillic hits, three Hangul, and six Simplified Chinese to the count.
Filtering those out is where we got fooled the second time. The obvious rule is “exclude anything in a search-pattern position — a grep argument, a regex, a query.” That removes 34 of 96. It also removes seven hits that sit inside file paths, and it would remove the 2026-08-27 результ case, which is exactly a foreign word in a search-pattern position, arrived at by accident instead of on purpose. An investigator’s grep and a mistaken grep have the same shape. Structure cannot separate them, and we accepted that blind spot for bare search words rather than ship a report nobody would read.
What structure can separate is what the word is next to. A foreign string sitting in a file path, or fused to an identifier, is a miss waiting to happen regardless of why it got there. So path-shaped hits are ranked above every exclusion rule and always printed, even when the count is zero. The first version of “path-shaped” was “contains a slash,” which promoted 401/403, a sed substitution that was fixing a mixed word, and a character list separated by slashes. It now requires an ASCII path segment or an identifier marker like a file extension. Ten test cases, positive and negative, and the report still lists everything that fell out of the path category under “other suspects” — the tightening reorders, it does not delete.
The order that survived is short. Path-shaped hits come first and are always shown, even at zero. Then the two structural exclusions — the detector’s own character list, and search-position — with one rule on top: a path-shaped hit is never dropped by either. Then anything previously logged, which is a weak exclusion because it also hides recurrences. Whatever is left is the list of suspects, and every bucket prints its count.
And the report never prints “0 new” alone. It prints 0 new — 20 hidden as previously logged (which does not mean fixed), because a clean line that was produced by suppression should say so on the line.
What a week looks like now
The weekly job runs Monday morning and posts one message. Two recent ones:
week of 09-08 4,222 tool calls scanned
3 hits → 3 excluded (2 detector source, 1 discussion) → 0 remaining
path-shaped: 0
week of 09-15 1,551 tool calls scanned
0 hits → 0 remaining
path-shaped: 0Before trusting the exclusion rules, one of us went through 46 flagged items by hand with 40 characters of context each. Forty-four were quotations of one kind or another — including two that the review itself had produced that morning: a grep for результ, and the judgment table listing the examples. Two were real: 数倍변わる and 調査결果, both from the infra agent on the same day, both in the body of a document it was writing, both self-caught and corrected within the hour. Two genuine cases in 11,490 tool calls is 0.017%; the original baseline was one in 3,823, 0.026%. Same order of magnitude, larger sample. The rate did not change. What changed is that we can now say so with a number instead of an impression.
Both real cases were in document prose, not commands — outside the scope the command scanner was built for. They were caught because the scanner reads the whole command string, heredoc body included. That side effect is the only reason the two true positives of the month exist in the record.
One more correction, since the theme of this post is being wrong in public. Our most-cited example of the dangerous path-shaped type was a data directory named after a cryptocurrency ticker in Simplified Chinese. It appeared in commands from both hosted agents. On hand review it turned out to be a real ticker symbol — a meme coin with a Chinese name, listed on the exchange we pull from. Real data, correctly reproduced. It is now excluded against a list of 1,143 live symbols refreshed weekly, and the harm example we actually stand behind is the результ grep and a 設定дир (“config directory”) path — one confirmed miss and one near-miss.
What we can and cannot say
can say all three agents do it, across two model families and two generations
the shape is constant: meaning kept, one word's script swapped
it does not appear in the agents' visible working prose (630k chars, 0 hits)
where it lands matters more than how often: speech is cosmetic,
a search pattern or path is a silent miss
rate over four weeks: ~0.02% of tool calls, stable
cannot say the precise multiplier between agents (6.2× → 2.6× as we aligned inputs)
whether it happens in reasoning (bodies are not stored)
when it started (65 of 120 days are missing from the logs; 05-06 is the oldest we have)
why — we have not tried to fix it, only to see itThat last line is deliberate. We have not added a prompt instruction, changed a sampling parameter, or filed anything with a vendor. Three agents, four months, and a rate that sits at two in ten thousand is not yet a thing to fix. It is a thing to see, because the one case in that ten thousand that lands in a grep pattern will not announce itself, and before 2026-08-29 nothing in the system would have told us it had happened.
If you run one agent and it writes in a language other than English, the cheap version of all of this is a single check: scan the arguments of every search and file command for characters outside the scripts you expect, print the path-shaped ones every time even when there are none, and read the list weekly rather than alerting daily. The daily alert will be about the detector. The weekly list, once every few months, will be about the thing you were looking for and did not find.
For the measurement habit that produced the six recounts — run the new code against everything you already have, and count — see Every fix opened the next hole. The hooks, watchers, and weekly-report scaffolding this checker lives in ship as the enrai Agent Pack ($33 on Gumroad).