Engineering · AI · Product
Pin the Assessment Again: Performance Analysis Without the Pin
How we rebuilt the chat surface so a student could ask about a subject and get an answer.
A student types "analyze my performance in math." We had a real answer ready: attempt normalization, topic aggregation, weak-point detection and study-consistency scoring, all one prop (a value handed to a UI component) away from the screen. Instead the chat showed:
Per-source attempt detail and question-by-question review will appear here once the pinned assessment finishes loading. If this persists, pin the assessment again from the @ menu.
The message was technically accurate. The system was waiting for context that would never come, because nobody pins a quiz before asking about it. What every student actually does is type the subject and expect an answer.
This post is about performance analysis in Cereby's chat, the per-source breakdown of a student's attempts, and how in May we made it answer for a named subject without a pin. What a pinned item carries is covered in The Card Cereby Couldn't See: Enriching Pinned Study Materials.
One path always lost
The detailed UI was built for pinned content, whose attempt history the chat could resolve directly. The renderer asked one question, "did the user pin something?": the rich accordion if so, the placeholder if not. But the chat agent also had a tool for analyzing performance by subject, and when a student typed a subject, it ran:
The analyzer produced real data and a narrative, and then the UI counted zero pinned summaries and showed the placeholder anyway. Two paths converged on one component, and one always lost.
A second producer, the same shape
The fix was additive. The pinned flow stayed byte-for-byte unchanged, and a second producer of the same per-source summary shape ran on a different condition:
The handler chose the producer, and the component saw the same data type wherever it came from. Students who pinned noticed nothing; everyone else stopped seeing the placeholder.
Geometry is mathematics
The first subject classifier was deliberately dumb: a substring search of every quiz, flashcard set and exam's title, tags, metadata and inner topic, for the word and its alias ("math" and "Mathematics", "bio" and "Biology"). It worked when the link was lexical: "algebra" matched "Algebra Diagnostic".
We shipped it, opened the debug page on a real account, and typed "math." Zero matches. The account had a quiz titled "Quiz: geometry." Geometry is mathematics. Every literate human knows that. A substring search does not:
That was the moment we knew keyword matching was the wrong primitive.
One model call, and no fallback
We ripped the keyword path out and made one call to a small, fast language model, cheap and quick enough for every request. It got the subjects asked about and the titles, topics and tags of every quiz and flashcard set, and said which materials belonged to each subject. The model was fixed, not the student's chat model, because classification should not burn the student's chosen budget, and the temperature was 0.1, so the same query surfaced the same materials. There was no keyword fallback:
A failed or malformed call reported every named subject as unmatched, with chips (one-tap suggestions) that make material for each. A keyword backfill behind the model would have made the experience harder to reason about: strict failure surfaces real problems, and soft failure hides them. Malformed replies were a known risk, since JSON mode is not portable across providers and the gateway does not paper over the differences.
After the swap, "math" found both the algebra quiz and the geometry quiz. "Calculus" would find a "Derivatives Practice" quiz, and "literature" would find "Romeo and Juliet Test."
A walkthrough
A student getting ready for finals has three materials in their account (the account is synthetic):
- quiz "Quiz: the Algebra" (id=q_alg, 3 attempts, avgScore 60%, topics=[Linear Equations, Quadratic Functions])
- quiz "Quiz: geometry" (id=q_geo, 1 attempt, avgScore 80%, topics=[Triangles, Circles])
- flashcards "Spanish Vocab Set 1" (id=fc_es, 2 sessions, tags=[spanish])Nothing pinned, nothing tagged manually. They open the chat and type:
analyze my performance in math and science
The handler splits the request into two subjects, sees no pinned content, and hands the classifier the full material catalog. The response below is illustrative: the shape of what the classifier returned for this catalog, not a logged call.
classifier input:
subjects: ["Mathematics", "Science"]
materials: [q_alg, q_geo, fc_es] // titles, topics, tags as above
classifier response, illustrative (temperature 0.1):
{
"Mathematics": ["q_alg", "q_geo"],
"Science": []
}The component builds per-source summaries from the matched IDs:
Mathematics (2)
q_alg → 3 attempts, avgScore 60%, sparkline + topic breakdown, openByDefault: true
q_geo → 1 attempt, avgScore 80%, sparkline + topic breakdown, openByDefault: false
Science (0)
empty-state: "We couldn't find any quizzes or flashcards related to Science yet. Want me to make some?"
chips: [create quiz on Science, generate flashcards on Science, build learning path]Both math cards render under an "In Mathematics (2)" header, the quiz with more attempts expanded. Science renders as a named empty state, not the comprehensive readout: if somebody asks about Science, we owe them an honest answer about Science. A chip submits a fresh chat turn, and two minutes later the student has practice material they did not have before.
The contrast case is the one that killed keyword matching. The same student types "how am I doing in math":
substring scan:
"math" in "Quiz: the Algebra" → false
"math" in "Quiz: geometry" → false
"math" in "Spanish Vocab Set 1" → false
matched: [] → renders placeholder
classifier:
{ "Mathematics": ["q_alg", "q_geo"] } → renders both quizzesSame materials, same metadata, two different answers. No title contains "math", so the substring scan returns nothing; the classifier returns both quizzes because geometry is mathematics.
Before and after
| Before | After |
|---|---|
| "Analyze my performance in math": a placeholder asking for a pin | The full per-source breakdown |
| Substring matching on titles and tags | Semantic, via a fast classifier model |
| "math" missed quizzes titled "Geometry" or "Algebra" | Both, plus calculus, statistics and anything else the model sees as math |
| One subject only | "math, science" renders a group per subject |
| No matches: a placeholder | The subject by name, and three chips that make material |
What this taught us
The hardest fix was finding the one prop where rendering gave up. The pipeline had been right for months; the bug was a boolean question asked at the wrong altitude.
Language models are absurdly good at the boring classification we keep solving with regex. One small-model call cost less and worked better than our heuristics; a fuzzy "is X about Y" decision is a service call, not a function.
Pinned-only flows are easy to ship and easy to forget. Students pick the most natural form of a query, and that is the path that has to work.
And an empty state with next actions beats one with an apology. "We couldn't find Algebra materials, here are three buttons that make some" is product. "No data" is a wall.
What's next
With no semantic search on materials yet, the classifier received every relevant material's metadata. Embedding each material once at write time and pre-filtering by similarity would shrink the prompt and the latency for power users with hundreds of quizzes: one migration and a backfill job. And auto-tagging at creation would remove the long tail of cases where the model has to guess, though "Untitled Quiz 3" still classified on its content.