Law Review, Law Firm: Auditing Cereby's Memory

We expected a decision model to make Cereby's memory smarter. What the numbers said instead, and the seven plain fixes that shipped.

---

A law student has ten things saved in Cereby's memory. One says they want to make law review. Another says they work part-time at a law firm. They type: "forget that I want to make law review."

The model we were testing chose the law firm, 73% sure. The student and their memories are made up for this post, but that choice is what the model really returned. In 25 held-out forget requests it was the only wrong delete, and the simple word match it was meant to replace, which looks for the request's words inside each memory, made none.

Cereby's memory is the short list of facts Cereby keeps about each student so that a new chat does not start from zero: their course, their exam date, that they like bullet points. In October we read all of it end to end, measured it against production, and tested whether a decision model, the kind that had just improved Cereby Mini's edit check, should make memory's choices too. It should not, yet. This post is about how we found that out, and about what we fixed instead.

Seventy-eight of eighty

First, scale, because it decides what matters. Since May, Cereby has saved 80 memories for 49 students. Twenty-eight of those students have exactly one. The classifier has routed a request to remember three times, from two students, and no memory has ever been forgotten from a chat. Nearly everything came from one place:

"Noticed in passing" is the detector: a small, fast language model that reads every chat message beside the answer and pulls out the one fact worth keeping, if there is one. Our April post described how it took over from a save tool the chat model rarely used. In September a second step joined it, the reconciler, which decides whether a new fact is new, already known, or a better version of a memory it resembles ("its at 3pm" refining "Quiz on November 7").

So in practice, memory is one decision made on every message: keep this or not. The explicit paths, "remember that" and "forget that", were barely exercised. Several of the bugs below lived there. In fairness to the old code, none of them had hurt a real student yet. They were still broken promises.

A promise the save path never heard

Cereby has temporary chats, the quiz tutor's chat runs as one, and a temporary chat promises that nothing from it is kept. That promise was enforced in the chat route, which stopped memories from being read and stopped the detector's passive save. It never reached the two other ways a memory gets written: the classifier routing "remember that…" to the save tool, and the answering model calling the save tool itself. The step that actually writes a memory checked only the account-wide setting.

The fix is a rule scoped to the request rather than an argument passed down. The controller hands memory state through a dozen call sites, and any new path that forgot the argument would save again. Now the chat route runs the turn inside a "this chat may not save" scope. The one function every write passes through refuses inside it, and the answering model is not even offered the save tool. A test reads the rule from inside the controller, on both the buffered and the streamed path, so it cannot pass by accident.

A cousin of the same bug: with memory switched off in Settings, every message still went to the detector, which found facts that were then refused. Detection now waits for the setting.

“Memory removed.”

The forget path was worse, because it was confident. When the answering model asked to forget a memory, the tool loop ran the real handler, which knows whether it found one, found none, or found several and should ask. Then the loop threw that answer away and told the model "Memory removed." every time. The request below is synthetic; the strings are the code's own:

Underneath that, the two delete functions reported success whenever the database did not error, including when no row matched. The forget tool also took a memory id, and the memories in the prompt carry no ids, so any id the model sent was invented and "forgot" successfully. Deletes now say whether they deleted. The handler's own words go back to the model, and the id parameter is gone.

A question with probabilities

Those were bugs. The decision model was meant to be the upgrade.

A decision model does not write a reply. You give it text and a question, and it returns probabilities: P(yes) for a yes-or-no question, or a probability for every option of a multiple-choice one. It cannot copy a confidence from an example the way a language model can, because it does not write one. That was exactly the flaw we fixed in Mini, and memory looked like it had the same flaw. The detector writes a confidence into its answer, and Cereby keeps a fact only at 0.7 or above. Its instructions show one example, at 0.9. On four save-worthy synthetic messages it wrote 0.9, 0.9, 1.0 and 0.95.

Memory makes five decisions. Only two of them are pure decisions, with no text to write:

Decision	Fit for a decision model
Is this message worth remembering?	Yes or no, on every message: tested
Which saved memory does a "forget" mean?	One option among the student's memories, or none: tested
Is this the same fact as memory N?	Mostly text: an update needs the merged sentence, which a decision model cannot write
Is "remember this chat" a request?	Yes or no, but nobody has used it
Which memories belong in this prompt?	Every student's whole memory already fits

For saving, the shape we tested put the decision model in front of the detector. Below the bar, the language model is never called. Above it, the detector writes the fact as it does today, so this design can only remove saves, never invent one:

For forgetting, the decision model saw the student's memories as options plus "none of these". Above a bar it forgets the memory it chose; below it, Cereby lists the likeliest and asks which one.

We wrote the test sets first: 150 synthetic messages labelled save or skip (plus 8 ambiguous ones, reported and never scored), and 36 forget requests over four synthetic students. Each set was split into a smaller part for tuning and a larger held-out part, and the rule for a win was written into each file before the first run. Wording and bars were chosen on the tuning part only. Both engines ran against the real production functions. All of it, every run, cost about five cents.

Ninety-six of ninety-seven

The tuning part looked like a win. The detector saved three messages it should have skipped, each on all three of its tries. One was a fact about the student's sister. One was a pasted passage written in the first person. The third was "Make that last answer shorter", read as a lasting preference. The decision model stopped the first two outright (probabilities of 0 and 0.01), and after one change to its wording, a line about one-off requests, it stopped the third.

Then the 97 held-out messages, run once with everything frozen:

97 held-out messages	The detector alone	Decision model, then detector
Saved when it should not have	1 (“Explain it again but simpler”)	0
Missed a fact worth saving	0	1 (a paralegal learning contract law, P of 0.10)
Cost per message	baseline	1.46 times
Time per message, median (beside the answer)	595 ms	754 ms

That is a tie, and our pre-registered rule said a tie keeps what is there. The interesting part is why. The detector's confidence really is copied, and it barely matters: the yes-or-no answer it comes with was right on 96 of 97 messages. A calibrated probability is worth a lot when the thing it replaces is often wrong. Here it was rarely wrong. The mistakes it fixed while tuning barely appeared in the held-out set.

The exact words were the hard part

Forgetting went the other way. The decision model understood paraphrases the old matcher could not: "I'm not in nursing school anymore, forget that" means the memory saying the student is a second-year nursing student, and it chose that one at 93%. But it gave up exactness to do it. Here are the 25 held-out requests by kind, one row per engine:

It answered 19 of 25 correctly against 16, and it deleted the wrong memory once against never. A forget that removes the wrong fact is the one mistake this feature must not make. A student who said "forget X" does not go and check that Y survived. The rule said no wrong deletes beyond the old matcher's, so it lost.

The shape we landed on

What shipped is the language-model path, with its honesty restored. Every write passes through one gate that knows what the chat allows. The model calls have deadlines. A "forget" reports what happened. And each chat turn now records what it did with memory:

Smaller things we fixed on the way

The “already known” that dropped a time. The reconciler re-checked a model's "this is new" against a similarity floor, but trusted its "already known" at 60% confidence with no check at all. "Quiz at 3:00 PM on November 7" against a saved "Quiz on November 7" could be called known, and the time was lost. That was the exact example from the bug report that created the reconciler, and two of our own tests asserted the losing behaviour. Now "already known" must hold every date and time in the new fact, or the safer path decides.

No deadline at all. The detector, the reconciler and the reply after a save ran with the AI library's defaults: ten minutes and two retries each. The intent classifier got a six-second limit in October; memory had been left out. It has the same limit now.

The rest, briefly. "Do you remember our conversation about mitosis?" used to save the chat and skip the answer; questions and refusals no longer count as "remember this chat", and its summary keeps the newest turns instead of the oldest. "Don’t forget", typed with the curly apostrophe phones use, is now read as a request to remember. Telling Cereby your nickname no longer stores it twice. Each chat turn's analytics record now carries how many memories went into the prompt and, for a typed message, what memory did (added, updated, refused and why). Before, a passive save left no trace beyond a log line that lived 48 hours.

Two corrections to our April post. It said a memory Cereby forgot on its own could be restored from Settings; Settings only lists active memories, so it cannot. And the 0.7 confidence floor it described is a floor on a number the model writes itself, which in our checks sat at or near the one example in its instructions. That mattered less than it sounds, for the reason above.

What this taught us

A calibrated probability is only an upgrade over a step that is often wrong. The detector's confidence was fake, and its decisions were right 96 times in 97, so there was almost nothing for the honest number to fix.

Understanding meaning and matching words are different skills, and forget needs the second one first. The decision model read paraphrases that the substring match never could, then deleted "law firm" for "law review".

A promise enforced in one place is a promise kept in one place. The temporary-chat flag was checked carefully where it was set and nowhere a memory was actually written. Moving the check to the one gate every write passes through fixed paths we had not listed.

Write the rule for a win before the first run. While tuning, the decision model looked like the obvious choice. Without a rule written in advance, the held-out tie would have been easy to call a win.

What's next

The forget result suggests a hybrid we have not tested: trust the word match when it finds exactly one memory, where it made no mistakes, and ask the decision model only when it finds none or several. We designed that after seeing the held-out results, so it needs a fresh test set before we can claim it. Memories added by hand in Settings still skip the reconciler, so a fact typed there twice is saved twice. And the new per-turn record will tell us, for the first time, how often memory actually does anything for a student.