Two in Three Hundred: A Lab for Cereby's Humanizer
A local copy of our detector, an honest grade for today's humanizer, and the pipeline we built to train the next one
Here is a paragraph of the kind students paste into Cereby's humanizer. It is synthetic, written for this post, and our AI detector scores it 100:
Regular exercise offers numerous benefits for both physical and mental health. It strengthens the heart, improves circulation, and helps maintain a healthy weight. In addition, physical activity releases endorphins, which can reduce stress and improve overall mood. Experts recommend at least 150 minutes of moderate activity each week, such as brisk walking or cycling. Making exercise a consistent habit is one of the most effective ways to support long-term well-being.
This is what today's humanizer returned for it:
If you ask me, regular exercise gives numerous benefits for physical and mental health, which is to say, it strengthens the heart, improves circulation, and helps keep a healthy weight. Here's what I noticed: and yet here we're. From my perspective, also, physical activity releases endorphins, which can reduce stress and improve on the whole mood. [...] The interesting part? Making exercise a consistent habit is among the most solid ways to support long-term well-being. Not obvious at first glance.
The detector now scores it 54, "Leans AI". The text gained an opinion nobody held, a sentence that means nothing and two questions to itself. That would be an anecdote if it were rare. We ran today's humanizer over 300 AI-written texts it had never seen. 52 came out in the Human band, 10 kept their meaning, and 2 did both with no new grammar errors.
This post is about the humanizer lab, the setup we built to measure that: our production detector and humanizer running on a laptop, with grammar and meaning checks next to the score. It is also about the rewrite engine and training pipeline we built on top of the lab. The model that pipeline trained is not live yet, and the last sections say why.
The grade nobody had run
Until now the humanizer had one benchmark: 55 texts, scored by calling the production detector, with nothing measuring grammar or meaning. It had also only ever graded earlier versions of the pipeline. The version that ships, with its rule passes and a gated model step, had been simulated but never run end to end.
So the first job was a lab that could run the real thing without touching production. It runs the same detector model at the same revision, on CPU in 32-bit floats as the server does, plus the humanizer service itself, a grammar checker (LanguageTool, an open-source rule-based checker) and a meaning check (a natural-language-inference model that asks, in both directions, whether each sentence of one text follows from the other). The lab scripts refuse any address that is not the laptop's own.
A copy of a detector is only worth something if it agrees with the original. We replayed every text we had a production score for. All 455 texts that fit in one detector window scored exactly what production had recorded, with no label flips. Longer texts spanning several windows matched in 9 of 17, so the lab grades the way the detector tab does, paragraph by paragraph, where parity is exact:
The headline metric is all three at once: the paragraph-level score in the Human band (20 or below), no new LanguageTool errors, and the meaning check passing. We call it Success@20. Today's humanizer scores 2 of 300 on it.
What the score was made of
Today's humanizer runs about fifteen rule-based steps, then, for a deep rewrite still scoring above 90, a small Qwen model on our detector server, then the rules again. Nothing re-scores the result the way the user will see it. Here is the synthetic paragraph going through it, with the steps that did the damage in red:
The rules win points with exactly what you see there. Across the 300 dev texts, the rewrite added a first person the original never had in 289, a "you" in 199, a question in 153 and a filler opener in 150. Run one at a time on 150 paragraphs, the individual steps barely moved the score. The full pass lowered it by 26 points on average, but 149 of those 150 paragraphs broke at least one of the checks described below, and 81 gained grammar errors.
The detector has a strong opinion about what human sounds like, and it is not "correct". We scored real human writing: 150 Wikipedia paragraphs, 150 news paragraphs and 150 paragraphs from student essays. It called 61 of the Wikipedia paragraphs AI (80 or above), 47 of the news paragraphs and 24 of the essays. To this detector, sounding like an encyclopedia is suspicious and sounding like a student is safe. That explains why a rule set that makes text chattier and sloppier works on it.
It also corrects our own post from April. Twelve Rules and a Paraphraser reported the score falling from 68% to 51% "with the text still grammatical and faithful". Nothing in that benchmark measured either. Measured now, the shipped pipeline keeps the meaning of 10 texts in 300.
The shape we built
The new rewrite path changes who decides. Instead of applying rules and hoping, it asks the detector. Our detector runs on the same server as the humanizer, so it can sit inside the loop:
Two properties come from the shape rather than from the model. It is never worse: a paragraph changes only when a candidate beats it by a margin, so the worst case is the user's own text back. And the number the user sees is the number the detector gives the final text, computed the way the detector tab computes it.
The guards are deterministic checks, and they are where "correct English" becomes enforceable. A candidate is dropped if it changes a number, a date, a quotation or a name's capitalization, if it adds a first or second person or a question the source did not have, if it adds dashes, lookalike Unicode or invisible characters, if it repeats itself, or if its length drifts too far. Those are the tricks a search against a detector finds first, and today's rules use several of them.
The model runs in llama.cpp's server (an open-source engine for small language models on ordinary CPUs) in its own container beside the detector, on the same two virtual CPUs. The deploy script refuses a model file whose checksum does not match the one pinned in the repository. All of this is built and tested. None of it is switched on.
Teaching a small model what human looks like
The model needed training data that showed it what human text looks like. We could not take AI text and ask a big model to make it human: of 32 rewrites from a 9-billion-parameter model running locally, the 20 that passed the guards all scored 99 or 100. So we ran the problem backwards. We collected real human paragraphs that the detector scores as human, had a local model restyle each one into polished AI prose, and trained a small model to turn the AI version back into the original:
Out of 10,898 paragraphs the detector called human, 7,781 passed LanguageTool and 3,022 passed the acceptability check described below. After an audit of our own autocorrection, 2,374 remained. The pair filters are strict on purpose. If the AI version dropped a name or the writer's "I", the original would look like a rewrite that adds one, and the model would learn to invent.
Boring failures
The grammar checker with blind spots. LanguageTool only knows the mistakes it has rules for, and it passed "they would be able to successful complete the project". We added a second check, a RoBERTa model trained on CoLA (a corpus of sentences labelled grammatical or not) that scores each sentence. On known examples it rated good sentences 0.99 and broken ones between 0.02 and 0.31. Requiring every sentence to score 0.8 kept 500 of our first 1,096 targets.
The autocorrect that renamed things. We let LanguageTool fix up to four small slips per paragraph. An audit of 1,483 corrected targets found "louvers" turned into "lovers", "Atletico" into "Atlético", "colour" into "color" and "potentionaly" into "potential". Autocorrect now applies grammar, punctuation and casing fixes and restores missing apostrophes, but never applies a dictionary spelling suggestion and never changes the letters of a capitalized word. A paragraph that still has a real misspelling is dropped, which cost 645 of the 1,483.
The prompt that made AI text human. Our first restyling prompts dropped the writer's voice, so we told the model to keep the point of view, every name and every number exactly. It obeyed so well that its "AI" versions stopped reading as AI: the share scoring 80 or above fell from 96 to 98% depending on style to between 1 and 49%. Four prompt variants on the same 64 targets all landed between 7 and 10 usable pairs, so we kept the plain prompts and generated more versions.
The model that trained at seven tokens a second. Our first choice of model, Qwen3.5 at 0.8 billion parameters, uses a newer layer type whose fast kernel in the training library has no backward pass, so training falls back to a loop over every token. It trained at 7 tokens a second. Qwen3 at 0.6 billion parameters trains at about 250 to 330 on the same laptop and makes a smaller file for the server, so we switched.
The grader that failed quietly. We stopped the lab service before one grading run without noticing it also hosted LanguageTool. Every grammar check failed, and the grader recorded 0 of 60 texts as grammar-clean instead of stopping. It now refuses to grade when any check fails.
Smaller ones: a learning rate twice as high diverged within 30 steps; prompting the restyler with real AI paragraphs as style examples gave 1 usable pair in 64; and running two rewrites in a row, to make training inputs look more like what users paste, gave 0 usable pairs from 240 candidates.
Clean text that still reads as AI
We graded the fine-tuned model on the first 60 dev texts, the same 60 that today's humanizer had already been graded on:
| On the same 60 dev texts | Today's humanizer | New engine and model |
|---|---|---|
| Human band (20 or below) | 19 | 1 |
| Grammar clean | 44 | 60 |
| Meaning kept | 2 | 57 |
| All three (Success@20) | 0 | 0 |
| Average score after | 62 | 95 |
The new path does what it promises on quality, and not what users come for:
It only rewrote 10 of the 64 paragraphs, because most of its candidates kept most of the source's wording and still scored 100. That turned out to be specific to what it was reading. On inputs like its training data, a checkpoint of the model rewrote freely, keeping about half the wording, and its median candidate scored 34. On the dev texts, which read like plain chatbot answers rather than our restyler's formal prose, it copied: 82% of the wording kept, a median score of 100.
We tried the obvious fixes. Eight candidates and two rounds per paragraph got 2 texts into the Human band, and 1 text passed all three checks, the only Success@20 either system scored on these 60. Training on 962 pairs that let the model add "I" and "you" got none. Scoring the candidates the guards had rejected found only 5 of 172 at 20 or below. The guards are not what holds the score up. The model's own writing still reads as AI to this detector.
Here is the synthetic paragraph from the top, through both systems:
input (synthetic) "Regular exercise offers numerous benefits for both physical and mental health. ..."
detector 100
today's humanizer → "If you ask me, regular exercise gives numerous benefits ... The interesting part? ..."
detector 54
guards failed: length, first person added, second person added, question added
meaning lost
new engine (0.6B model, 8 candidates × 2 rounds)
candidates 16
dropped by guards 9, every one for adding "you"
scores of the rest 98, 98, 100, 83, 96, 100, 100
winner 83 "Regular exercise helps kids and overall health. It strengthens the heart, ..."
meaning lost: "kids" is not in the source, and no guard looks for itThe contrast case is the useful part. The new engine refused every candidate that addressed the reader, which today's humanizer does freely, but it still let through a candidate that changed who the paragraph is about. The guards check names and numbers, not nouns.
What this taught us
A humanizer graded only by the score it gets will learn to vandalize text, because vandalism is what the detector rewards. The tell was the meaning check: 10 texts in 300.
Our detector's idea of human is partly a persona. It calls an encyclopedia AI and a chatty student human, so a rewrite that keeps an essay's register is fighting the detector, not just the model's limits.
Data tools need audits like code does. Autocorrect was supposed to clean training targets and instead renamed people and places, and the grammar checker passed sentences a person would mark in red.
Imitating human paragraphs taught the model the form of its training inputs, not the general move from AI to human. It rewrote what looked like its data and copied what did not.
What's next
Three things before any model reaches users. The engine needs a meaning check of its own, because the guards let "kids" through. The model needs training that uses the detector directly, rewarding the lower-scoring of two guard-clean rewrites of the same real AI paragraph rather than imitating human ones. And if that is not enough, a larger model, which our server's two CPUs and 3 GB will have to fit. Until one of those beats today's humanizer on these same 300 texts, users keep today's humanizer, and the lab keeps the score honest.