A Perfect Zero in Broken English: Cereby's Humanizer, Round Two
Why our detector's Human band turned out to measure damage, and the humanizer we shipped once we knew
Here is a paragraph of the kind students paste into Cereby's humanizer. It is synthetic, the same one our last post opened with, and our AI detector scores it 100:
Regular exercise offers numerous benefits for both physical and mental health. It strengthens the heart, improves circulation, and helps maintain a healthy weight. In addition, physical activity releases endorphins, which can reduce stress and improve overall mood. Experts recommend at least 150 minutes of moderate activity each week, such as brisk walking or cycling. Making exercise a consistent habit is one of the most effective ways to support long-term well-being.
This round, our lab searched hard for a rewrite of it that the same detector would call human. It found one that scores 0:
Moderate exercise helps people both be healthy and have good mental health. Heart strengthening, improving the circulation and keeping healthy weight are good things to do. Endorphins are the natural endocrine compound that will help people feel good and make them look and feel better. [...]
A perfect score, in English nobody would hand in. That would be an anecdote if it were rare. We ran the same search over 60 real AI paragraphs. 17 reached the Human band with every one of our guards passing, and not one kept both correct grammar and the original meaning.
This post is round two of the humanizer lab, the setup that runs our production detector, grammar and meaning checks side by side on a laptop. Two in Three Hundred built it and ended with a model that kept text intact but did not fool the detector. It also promised that users would keep the old humanizer until a model beat it on the same 300 test texts. None did. This round explains why, and what we shipped instead: a rewrite engine that changes a paragraph only when it can do so without changing what it says, and tells you plainly when it could not.
Search instead of imitation
Round one's model learned from pairs we built backwards: a human paragraph, restyled into polished AI prose by a local model, and the original as the answer. It learned to undo that one restyler. On the AI text students actually paste, it mostly copied.
So this round started from real AI paragraphs, 6,678 of them from the training split of our dataset, kept apart from the texts we grade on. They have no human original, so the targets had to be found: run slow generators offline, keep only the rewrites that pass every check, and train a small model on those. We call a rewrite usable when it passes the engine's guards, scores 20 or below, adds no LanguageTool error, has no sentence that reads worse than the source's worst (a grammaticality model trained on CoLA, a corpus of sentences labelled acceptable or not, judges that), and keeps the meaning in both directions by natural-language inference.
Before building that pipeline at scale, we measured the ceiling on 60 paragraphs with every generator we had. Here is the search traced for the paragraph above:
Bigger models still write like models
The obvious generator is a bigger model with a better prompt. We asked Qwen3.5-4B, running locally, to rewrite each paragraph in six registers, including the ones our detector calls human most often: a student essay, a forum answer, a news brief.
Scores alone could not show whether that helped, because the detector's 0 to 100 score is a rounded probability, and 100 only says the model is more than 99.5% sure. So the lab now ranks by log-odds, the detector's raw confidence before rounding: 0 is a coin flip, 5.3 is where the score hits 100, and the 60 paragraphs sat at a median of 9.2. The 4B model's best rewrite of each, in any register, sat at a median of 9.18. Its text read exactly as AI as the source. The same was true of the 9-billion-parameter model in round one. Polished prose from an instruction-tuned model is exactly what this detector was trained to catch, whatever it is asked to sound like.
Human, as this detector means it
The candidates that did reach the Human band came from the round 1 model and from the sentence beam, which keeps, sentence by sentence, whichever version lowers the paragraph's log-odds most. Followed through every check, the 60 paragraphs thinned out like this:
Before trusting that zero, we checked the checkers. Of the 43 candidates that passed the guards and scored 20 or below, 36 failed the meaning check and 33 had a sentence that read worse than the source's worst. We read them. The graders were right. On the synthetic paragraphs we wrote for this post the pattern is the same: "A sleep deficit bruises attention and decision-making the next morning", "Printing press invention in the fifteenth century revolutionated the spread of information", and a paragraph about exercise that turned into one about kids. The detector rewards text that is a little broken. A search against it finds that first.
Voice was not the lever
The old humanizer gets 53 of 300 dev texts into the Human band, and it adds a first person to nearly all of its outputs, in the Human band or not (52 of 53, and 232 of 247). Two things set its Human-band outputs apart: filler openers such as "Honestly" or "Here's the thing" (42 of 53, against 109 of 247) and new grammar errors (a median of one, against none). So we tested voice on its own, added to the 90-text dev slice in ways that keep every claim:
| Edit | Paragraphs changed | Median change in log-odds | Reached the Human band |
|---|---|---|---|
| "Honestly," and "Look," on two sentences | 116 | -1.20 | 6 |
| An opinion frame on every sentence | 154 | -1.36 | 1 |
| Two filler sentences ("Not obvious at first glance.") | 111 | -0.98 | 3 |
| The reader addressed once | 143 | -0.15 | 0 |
| One rhetorical question | 111 | +0.03 | 0 |
From a median of 8.42, a drop of about one point changes nothing a user sees. The best of these edits reached the Human band on 6 of 116 paragraphs, and the meaning check rejected most of the filler sentences outright, since a sentence that says nothing is not in the source. On the 12 rewrites that already kept their meaning and scored 50 or below, frames moved the log-odds by less than half a point and a question pushed it up by two. The old humanizer stacks all of these edits at once, with the grammar errors, and keeps the meaning of 11 texts in 300.
What we shipped
That left a choice between a humanizer that reaches the Human band by damaging text and one that keeps text correct and rarely reaches it. We chose correct text and an honest score. The engine from round one now has two more checks of its own, one for meaning and one for whether a rewrite reads as well as the original, and the app says what happened:
Both checks run only on candidates that would otherwise win, because they are the slowest steps. When nothing clears them, the paragraph stays as the student wrote it, and when no paragraph changed at all the app says so, "No paragraph could be rewritten to read more human without changing what it says", and charges nothing. Before, it would have handed back the student's own text as a rewrite to judge.
Here is the synthetic paragraph from the top through all three: the old humanizer, the lab's search, and what we shipped. The input is ours; every output and score is what the system returned.
input (synthetic) "Regular exercise offers numerous benefits for both physical and mental health. ..."
detector 100
the old humanizer → "If you ask me, regular exercise gives numerous benefits ... The interesting part? ..."
detector 54
checks failed: length, first person added, "you" added, question added
meaning lost
lab search (best of 4B, round 1 model and sentence beam)
winner → "Moderate exercise helps people both be healthy and have good mental health. ..."
detector 0
acceptability weakest sentence 0.78, under the 0.80 floor: reads worse
meaning lost: "regular" became "moderate", and the last sentence says something else
shipped engine (4 candidates, two CPU threads)
winner none: no candidate both beat 100 by 15 points and passed every check
result → the paragraph exactly as written, in 7.5 seconds
detector 100
app "No paragraph could be rewritten to read more human without changing
what it says, so your text is unchanged. You weren't charged."We graded the shipping configuration the way users get it, through the app's own pipeline and the server code with the model on two CPU threads, on the 300 dev texts and then, once, on 300 test texts neither system was tuned on:
| On 300 texts, dev / test | The old humanizer | What we shipped |
|---|---|---|
| Meaning kept | 11 / 15 | 299 / 299 |
| No new grammar error | 167 / 165 | 298 / 299 |
| Texts changed at all | 300 / 300 | 19 / 19 |
| Changed and damaged | 296 / 299 | 6 / 6 |
| Human band (20 or below) | 53 / 40 | 0 / 0 |
| Average score after | 71.3 / 72.7 | 98.7 / 98.7 |
"Damaged" counts a text with a new grammar error, a changed meaning, or a paragraph that reads worse than the original. The shipped engine changes about one text in sixteen, and about a third of what it changes still trips one of those checks, mostly the readability one. The old humanizer changes everything and damages nearly all of it. On two threads, the slowest 5% of texts took 34 seconds on dev and 38 on test. It went live for every student on October 26.
The cost is the number students came for. Not one of the 300 dev texts reached the Human band, against 53 for the old humanizer:
Two CPUs and 1.3 GB
The engine runs on our detector server: two virtual CPUs, with 1.3 GB set aside for the model. We timed the student there in a throwaway container before choosing a file size:
The 8-bit file used 1.14 GB of its 1.27 GB allowance under load, and two 300-word rewrites at once took 46 seconds. The 4-bit file took 8.8 seconds for the same pair, and on our 90-text check it was as safe as the 8-bit one (meaning kept on 90, against 88). We shipped the 4-bit file.
The meaning check had the same squeeze. The model the lab grades with is far too slow for two CPUs, and the small ones we tried, run sentence by sentence, would have rejected half of the good rewrites to catch most of the bad. Comparing whole paragraphs in both directions instead, a 70-million-parameter model catches 87% of the meaning changes the lab's large model finds, at about a quarter of a second per check, while wrongly rejecting 32% of good rewrites. For this engine that trade is cheap: a wrong rejection only means a paragraph stays as the student wrote it.
Boring failures
The safe version that was not. Our first full run of the shipping configuration had only the meaning check. It kept the meaning of 297 of 300 dev texts, and we nearly called it done. Then we looked only at the 29 texts it had changed: 17 came back with a new grammar error, a changed meaning, or a paragraph that read worse than the original, such as "camera cameras" and "Another reason being a human body has receptors on the tongue". A rewrite can keep every claim and still loosen the grammar, and loosening the grammar is what lowers this detector's score. The second check, a 67-million-parameter model trained on CoLA that requires every sentence to read at least as well as the source's weakest, catches 93% of those cases in calibration, at under a tenth of a second.
The score that stopped at 100. Round one ranked candidates by the rounded score, so a search could not tell a paragraph the detector was 99.6% sure of from one it was certain of. The log-odds fixed that, and changed which candidates every search kept.
The decoder that ignored the detector. Our token-by-token "detector-guided" decoder from round one scored each choice as the AI probability minus a small weight on how likely the model found the token. On AI text that probability is 0.9999 for every choice, so the second term decided everything, and the decoder was plain greedy decoding. It now uses log-odds and scores the whole paragraph at each step, not the fragment written so far.
A backslash that meant AI. In the public dataset we use, 1,614 of 12,933 AI training texts, and 31 dev and 34 test texts, contained a literal backslash-n where a line break should be. No human text did. A model trained on them could learn that a backslash means AI. We repaired them and re-ran the old humanizer on the repaired sets; its numbers barely moved.
Every candidate empty. The first run with a repetition penalty that discourages copying the source returned nothing at all. The server rejected one setting's value of -1, and our client, built to treat one failed sample as one fewer candidate, quietly turned every rejection into an empty one.
A curly apostrophe. Replacing straight apostrophes with curly ones, and nothing else, moved 4 of 70 paragraphs into the Human band. The guards now reject a rewrite with more curly quotes than its source.
What this taught us
A detector's Human band is only worth chasing if it measures something a reader would recognize. Ours, on this kind of text, mostly measures damage: of 17 paragraphs it scored human, none kept correct grammar and its meaning together.
A cheap search before an expensive training run is how to find a ceiling. The bake-off cost an afternoon and showed that no amount of distillation could reach a target the search could not.
Graders have to be read, not just run. The zero was believable only after we read the 43 rewrites behind it.
What's next
The detector is the next thing to examine. It calls 61 of 150 Wikipedia paragraphs AI and calls broken English human, and a humanizer can only be as meaningful as the detector it answers to. The meaning check still misses about one change in eight, and the engine's guards are the backstop for the rest.