Seven Queries Nobody Read: Trimming Cereby's Orchestration Layer

File chat fetched context it then dropped and redid compression it had already done. How we made it skip the work.

---

A student with a textbook PDF open in chat asks a third question about Chapter 7. (The file and the questions are synthetic.) First Cereby runs seven database queries, about 200 ms. Then the classifier sees a question about the file, and the file handler drops everything those queries fetched. Then it compresses the chapter, about 300 ms with calls to an embedding API, just as it did for the last two questions:

The answer was right; the student paid in latency and we paid in API costs. The waste lived in the orchestration layer, the code around Cereby's file pipeline that decides what to fetch, what to compress and how much room each file gets. (The Proof on Page 12: From File Upload to Grounded Answer describes the pipeline itself, and Five to Eight Seconds for a Follow-Up: Making Cereby AI Faster how this layer's queries were made parallel and cached.) In April we profiled file chat's request path and made six changes. Three are worth telling in full.

The classifier knew first, and ran second

The intent classifier (Cells Could Mean Three Subjects: Inside Cereby's Intent Classifier) had a fast first pass, pattern matching with no model call, that could recognize a question about a file in under a millisecond. It ran after the aggregator, so the seven queries always fired first. We moved it to the front. When it resolved the intent, as it did for questions about a file or about the student's performance, Cereby built a minimal context (the student and today's date) and went straight to the handler, with no queries. Otherwise the old path ran unchanged. A feature flag and per-request path logging let us measure the hit rate before removing the flag:

Three questions, three identical compressions

Every follow-up re-ran the whole two-phase pipeline from Everything but the Answer: Query-Aware Compression for Oversized Pages: chunking, keyword scoring, picking candidates, then embedding calls to rerank them. So we cached its output, keyed on a fingerprint of the file, the question's significant keywords and the token budget. Two questions with the same keywords shared an entry for 10 minutes, long enough for a follow-up session and short enough that a re-upload expires naturally. The in-memory cache grew from 100 to 200 entries to make room:

The first question waited for 200 embeddings

The first question about a new upload paid the full embedding latency: a 50-page PDF with 200 paragraph chunks meant 200 embedding calls, batched, around 500 ms. So, once the pages were extracted, a background task split them into chunks and embedded them in batches of 50, and the embeddings were cached before the student typed.

Three smaller cuts

The file budget had been a fixed 8,000 tokens, a number chosen for an older 32K-context model. We made it scale with the model's window up to a 16,000-token cap, which every chat model we offered reached; with no model ID it stayed at 8,000. Several files no longer split the budget equally: each got a share in proportion to its relevance to the question (0.3 to 1.0, from keyword overlap), with at least 400 tokens. With 16,000 tokens, a textbook scoring 0.9 got 12,000 and a syllabus scoring 0.3 got 4,000. And the request handler's 20 positional parameters (one call site passed 14 placeholders) became one options object, which changed no behavior and made moving the classifier much cleaner.

The boring failures

The zero that meant forever. Prefetching exposed why embeddings were not staying cached. The embedding service stored them with a TTL (time to live) of 0, meaning permanent. The cache manager read it with ||, which treats 0 as missing, and fell back to its 5-minute default:

Embeddings had been silently expiring for months. With ??, which falls back only when a value is missing, and a permanent path for a TTL of 0, they stayed, and cache hit rates rose across every file-chat flow, not only the prefetch.

The floor that overshot the budget. When many files hit the 400-token floor, their shares can add up to slightly more than the budget. At two to five files that is negligible; at twenty or more, we decided the floor mattered more than the total.

Before and after

	Before	After
Database queries on file chat	7 per request, in parallel	None when the fast pass finds a file question
Compression on follow-ups	The full two-phase pipeline every turn	A cache hit when the keywords match (10 minutes)
File budget on the default chat model	8,000 tokens	16,000 tokens
Two files	An equal split	By relevance, roughly 75/25 in typical use
First-question embedding latency	About 500 ms	About 0 ms, prefetched

What this taught us

The cheapest query is the one you skip. The aggregator was already parallel and cached, but on file chat the right move was no aggregation at all. Classifying is cheaper than fetching, and it tells you whether to fetch.

Cache the output, not only the input. We already cached embeddings; the expensive thing was the pipeline that used them.

A budget is a product decision with a model-shaped input, and it needs a cap: students do not benefit from 150,000 tokens of file context when attention degrades past 16,000.

What's next

Prefetched embeddings live in memory and vanish on every server restart; a database or Redis would make them durable. Each pinned file is still compressed on its own, where a shared retrieval index would send the budget to the best chunks in any file. And casual messages ("hi", "thanks") still run the full aggregator, because the casual handler reads calendar events.