A Test Became Flashcards: From First-Match-Wins to Parallel Scoring

Why a regex that answered first could overrule a model that read the whole sentence, and the scoring that settles it now.

---

A student types "create a algebra test i can study on" and gets back a flashcard set. (The message is synthetic, typo included, but it has the shape of the ones that broke.)

The classifier had seen the word "study", matched a flashcards pattern at confidence 0.90, and returned at once, without asking the model for a second opinion. The word that mattered was "test". The system got credit for being fast, and the student lost a coin.

In February, Cells Could Mean Three Subjects: Inside Cereby's Intent Classifier described a six-phase pipeline that took accuracy from about 77% to about 95% and cut corrections by 80%. Its flaw was structural: a regex match returned early, so for a message with competing signals the model never got a vote. In April we rebuilt the middle of it as the semantic scoring pipeline: every classifier runs in parallel, each proposal is scored, and the strongest case wins. This is the second of three posts on Cereby's intent classifier; the third is The Gate That Never Fired: Auditing Cereby's Intent Classifier, from October.

The model never got a vote

The old pipeline ran in a fixed order: structural checks (an attached file, pinned context, a message too unclear to route), then regex patterns, then the model. A regex match ended the search:

The confidence score could not catch it. It ran after the winner was chosen and graded how well the winner fit, never whether another candidate fit better, so a strong pattern with complete parameters scored well while being wrong.

Every classifier competes

We kept the structural early returns: a file is attached or it is not. Everything else became one parallel step. Regex reported every pattern that matched, the model and the multi-intent parser ran beside it, and each proposed a candidate in one shared shape (the action, its source, an ambiguity score, an optional follow-up), which is what made them comparable. The same message again:

From April, Cereby also asked when the best candidate scored below 0.5, or when the top two (for different actions) were within 0.1 of each other. Before, it asked only when the model flagged a question itself.

A collision you can measure

Every action has signals, words that support it, and anti-signals, words that point elsewhere. For flashcards, "memorize" is a signal and "test" or "quiz" an anti-signal; for a quiz it is roughly the reverse. A regex candidate's ambiguity is the share of its matched words that are anti-signals. A model candidate's is 0, since it reads the sentence rather than isolated words.

The penalty weight, 0.35, put a regex match with ambiguity above 0.5 below a model match of moderate confidence. We tried 0.25, 0.30, 0.35 and 0.40 against our corpus of misclassified messages. At 0.25, some regex candidates with moderate ambiguity still beat the model. At 0.35, every regex candidate that had wrongly won flipped, and no correct result changed. We would not have found that number by reasoning about it.

Worked examples

Each candidate starts from its source's confidence and is adjusted:

final score = base confidence
            - 0.35 x ambiguity     penalty for competing signals
            + context bonus        up to +0.10 for matching active subjects
            + parameter bonus      up to +0.10 for complete parameters
            + agreement bonus      +0.15 if another source agrees
            capped at 0 and 1

The misclassification case:

"create a algebra test i can study on"                     synthetic

regex candidate     create_flashcards
  base              0.90
  ambiguity        -0.35 x 0.67 = -0.23
                    signals "study", anti-signals "test" and "create a"
  parameters       +0.00    no subject extracted
  agreement        +0.00    the model disagrees
  final             0.67

model candidate     generate_quiz
  base              0.88
  ambiguity         0.00
  parameters       +0.10    subject "algebra"
  agreement        +0.00    regex disagrees
  final             0.98

winner              generate_quiz    correct

And an unambiguous one, where agreement lifts both candidates and scoring costs nothing:

"create a podcast about biology"                           synthetic

regex candidate     create_podcast    no anti-signals     agreement +0.15
model candidate     create_podcast                        agreement +0.15
final               both capped at 1.00

winner              create_podcast    no regression

The flashcards that vanished

The old order had a second casualty. The multi-intent parser saw both actions in "create a quiz then flashcard on algebra", stored the second as a follow-up, and stopped. Nothing downstream read it:

The fix reused the pending action Cereby already ran on a "yes" to confirm schedules. The response already carried a follow-up and a pending action, so connecting them took no new UI and no new API fields.

Every message paid 200 ms

The old pipeline skipped the model for regex-matched messages, roughly 60% of traffic. The new one always called it, in parallel, so those messages went from about 1 ms to about 201 ms, and messages that needed the model stayed at about 201 ms. A misclassification costs a student a coin, a rephrase, and 3 to 5 seconds of wasted generation, so we took the trade. If the model call failed, the pipeline scored whatever candidates remained, which beat the old fallback and its separate keyword logic.

47 messages that used to go wrong

We tested the formula against 47 previously misclassified messages:

Scenario	Count	Old winner	New winner	Correct
Regex matched the wrong keyword	31	Regex, returned early	The model, after the ambiguity penalty	31/31
Multi-intent dropped the second action	9	First action only	First action, then the follow-up offered	9/9
Close confidence, wrong pick	7	Higher raw confidence	The candidate that fit the context	6/7

When both classifiers are wrong. In the one miss, regex and the model agreed on the wrong action, and the agreement bonus amplified the shared mistake. Scoring cannot help when every source is wrong. The existing suite, 261 tests across 30 files, passed with zero regressions.

What this taught us

Ambiguity is measurable. The share of a candidate's words that point elsewhere was a reliable proxy for misclassification risk, and the weight that made it work came from real failures, not intuition.

Existing code often has the right fields and missing wiring. The follow-up, the pending action and the check for a "yes" all existed before this work.

What's next

A module that already detected data-flow dependencies ("analyze my weak points then quiz me on them") was not wired in yet; with it, a second action could use the first one's output. And a cache could spare regex-matched messages their 200 ms, for patterns where regex and the model had agreed more than 95% of the time.

Postscript. Three weeks later, a question about a pinned quiz showed what scoring cannot see. The classifier chose performance analysis, correctly, but the model took the subject from the quiz's title, and the analysis found no history for it. We fixed it in the prompt, in an override that cleared that subject, and in the analyzer, which from then on read the pinned quiz itself.