Engineering · AI · Feature · Product
The Model Chose Not to Save It: Rebuilding Cereby's Memory
A save tool the chat model rarely called, and the parallel classifier that replaced it.
Picture a student who tells Cereby they are studying for the MCAT. The next day they open a new chat, and Cereby has no idea. The fact was not lost in transit, and no database dropped it. The chat model simply decided not to save it.
That was Cereby's memory, the facts about a student that every chat can draw on, as it stood in early April. The Grab-Bag Prompt: Cereby's Four-Layer Context Model describes how each request is assembled, with user memory as one layer. The architecture was right; the memory layer just had almost nothing in it. In April we rebuilt it, with a separate classifier to decide what to save, duplicates caught by meaning, and retrieval moved into the database.
The save was optional
We had given the chat model a save-memory tool among a dozen others (quizzes, notes, flashcards) and trusted it to notice what was worth keeping while it answered. It did not. On messages with a durable, save-worthy fact, it called the tool roughly 5 to 10% of the time. Explicit "remember this" commands worked reliably; implicit facts, preferences and goals slipped through:
When it did save, it never checked existing memories, so a major mentioned three times could become three near-identical rows.
A second model, in parallel
The fix was to stop asking one model to do two jobs. A cheap, fast classifier ran alongside the answer and decided what to save:
We chose a small, fast model. It answered in 1 to 2 seconds, and at roughly 300 tokens in and 50 out it cost about $0.00003 a call: $0.03 a month for a student sending 1,000 messages.
Most messages had nothing to save
The classifier looked for durable facts in six categories: a preference ("I prefer short explanations"), a fact ("I'm a sophomore at UCLA"), a goal ("I want to ace my physics final"), a learning style ("Visual explanations help me"), a schedule ("My quiz is next Thursday"), and a general catch-all. It ignored task requests, greetings, questions about Cereby and passing context, gave each candidate a confidence from 0 to 1, and dropped anything below 0.7. Most chat is questions and task requests, not self-disclosure, so it found nothing in about 85% of messages:
Those skipped the embedding, the similarity check and the insert, for a median overhead under 50 ms.
“I study biology” is “I'm a biology major”
Before saving, we embedded each candidate (turned it into a 1536-dimensional vector, a list of numbers that places text by meaning) and asked the database, through pgvector, for the nearest existing memory. Above 0.92 cosine similarity, the save was skipped:
We arrived at 0.92 after testing a corpus of 200 memory pairs. Below 0.90, unrelated memories in one domain ("studying biology", "biology quiz next week") collided; above 0.95, legitimate duplicates slipped through.
Every memory, on every request
Retrieval read everything. It fetched all of a student's memories, embedded them in the application and ranked them: fine at 50 memories, and 200 to 400 ms added to every chat request at 500. We moved it into one database call backed by an HNSW index (a structure for fast approximate nearest-neighbor search):
One round trip, and under 2 ms however many memories a student had. Existing memories were backfilled once.
The badge raced the reply
We wanted a "Memory updated" badge inline, without blocking the reply or polling for it. Detection started as soon as the request confirmed memory was on, and just before the reply went out, we raced it against a 3-second timer. If detection had finished, the badge rode along with the reply. If not, the reply went without it, and the memory was still saved in the background. About 90% of the time, detection finished before the reply was ready.
Plans capped memories at 50 (Free), 100 (Lite) and 500 (Pro), with no cap on Premium; past the cap, automatic saves were dropped silently and explicit ones offered an upgrade. A memory Cereby forgot on its own could be restored from settings; one the student deleted was gone.
Before and after
Saves, as a range for each system:
| Before | After | |
|---|---|---|
| Who decides to save | The chat model, by a tool call, inconsistently | A parallel classifier, automatically |
| Save-worthy messages saved | About 5 to 10% | About 85 to 90% |
| Duplicates | Not caught | Skipped above 0.92 similarity |
| Retrieval at 500 memories | 200 to 400 ms, every memory ranked in the app | Under 2 ms, one indexed query |
| Feedback | Only on an explicit save | An inline "Memory updated" badge |
| Memory settings | A flat list and delete | Categories, search, inline edit, manual add |
What this taught us
Making the chat model both answer and save was a design error, not an oversight. A cheap classifier in parallel does one job well for almost nothing.
Embeddings belong in the database. Similarity computed in the application did not scale; an index made retrieval time-constant and removed a class of latency bugs.
Deduplication thresholds need empirical tuning. Our first guess, 0.90, blocked distinct memories in the same domain. 0.92 suited our corpus, and it needs review whenever the embedding model changes.
What's next
Two gaps led the backlog. Related memories ("studying for MCAT", "MCAT is in June", "nervous about MCAT verbal") should merge into one richer memory automatically. And a student should be able to see which memories shaped an answer: "Cereby used: your preference for bullet points, your biology major."