Why Our Compressor Kept Reading the Preface (part 3)
What eight months of real textbooks did to the file compression we wrote about in February, and how a public-domain book helped us fix it.
"Explain more"
A student attaches a textbook, hundreds of pages long, and asks how natural selection could produce an organ as complex as the eye. The answer is good: it quotes the right chapter and explains the argument. The next message is the most natural follow-up there is: "explain more."
When we replayed that exchange against Charles Darwin's On the Origin of Species, the follow-up was answered from the wrong end of the book. The model had been handed 26 of the first 48 pages and one page from the middle: the title page, a note about which ebook edition this was, two epigraphs, the table of contents, the introduction and parts of the first two chapters. The chapter about the eye was not in it. The model did what a model does with that material. It wrote something fluent and polite and beside the point.
This kind of bug rarely gets reported. A vague answer to a vague follow-up does not look broken. It looks like the AI being a little dim.
The system we described was not the system running
In February we wrote two posts about how Cereby keeps a large file out of every request. Hierarchical Context Compression described summaries made for every page at upload, and a ranking that sent the best pages in full and the rest as summaries. Query-Aware Smart Compression described splitting an oversized page into chunks and keeping the ones that match the question.
When we sat down to audit it, the first finding was that the first post no longer described anything that ran. The page ranking had no callers. Upload-time summaries were still being written for some files, and nothing read them. What actually ran on every question was the chunk approach from the second post, stretched over the whole file: split the text into chunks, score each one against the question's keywords, then let embeddings (vectors that capture meaning, so "body parts they no longer use" can match "rudimentary organs") re-rank the 50 best keyword matches.
That last step is the whole story of the opening. When the question shares no words with the passage that answers it, there are no keyword matches, and "the 50 best" is just the first 50 chunks in file order. The embeddings were added to fix exactly the case where vocabulary does not line up, and in exactly that case they were shown only the preface. Here is the follow-up from the opening, step by step:
We had never measured any of this on a real book, so we built an eval that does. It runs the real compressor over Darwin (public domain, 412 pages) with ten questions: direct ones, a paraphrase, a question in Spanish, one that hinges on a three-letter word, two bare follow-ups and two requests about the whole book. Eight have a known answer passage. Before any change, that passage reached the model in 3 of the 8, with or without embeddings.
The shape we landed on
Three decisions carry most of the change.
We ask what the question names before ranking anything. "Make notes on this", "summarize this book" and the message our Create Note button sends name nothing to search for, so they now get coverage: the largest set of chunks spaced evenly across the whole file that fits the budget. Ranking such a request by its words was how "make notes" ended up searching the book for the word "notes". A bare follow-up like "explain more" borrows the last question in the conversation that did name something. Only then does anything get ranked.
Each row below is the same book as the eval saw it, front matter on the left and the index on the right. The first two rows are the follow-up from the opening, before and after the change. The third is "summarize this book", which before the change got much the same pages as the first. Red is text the model got only because of where it sits in the file, green is text chosen for what was asked, and grey was not sent.
The embeddings see the whole book. The pool they rank is still the best keyword matches, plus 50 chunks spaced across the file. When the keywords found little, it also takes the neighbors of the few chunks closest in meaning, since the answer often sits next to the nearest page the spread happened to land on.
We fuse ranks, not scores. The old code blended 40% keyword score with 60% semantic similarity. That sounds principled until you notice that the keyword score included bonuses that could push it past 1, while similarity scores from our embedding model sit in a narrow band. Reciprocal rank fusion sidesteps the question: each ranking contributes one over a constant plus the chunk's position in it, and positions are on the same scale by construction.
Around those, the excerpt now says what it is. Every run of text starts with its page label, each gap names the pages it skips, and a line under the header lists every page shown.
Boring failures and how we fixed them
The cache that answered the wrong question
Compressing a book takes long enough that we cache the result for ten minutes. The cache key was the file plus the question's keywords, and the question analyzer only counted words longer than three letters as keywords. So "What is DNA?" and "What is RNA?" both had none, and a student who asked the second question within ten minutes, on the same server, was answered from the pages chosen for the first.
The same length rule was quietly doing damage everywhere else: "dna", "atp" and "bee" were never searched for, because they were too short. The key is now the whole question, and short words are judged by a stoplist in four languages instead of by their length.
We told the model it had the whole book
The prompt around an attached file said, in capital letters, that the content below was the complete file and should be searched thoroughly before saying anything was missing. It was a few percent of the file. So when the answer sat on a page the model had not been shown, it said the document did not cover the topic.
Behind that sat a second filter. If a reply contained "not mentioned", "no information" or any of a list of similar phrases anywhere, the whole reply was replaced with a canned "I couldn't find a clear answer", and the turn was not charged. A full, correct answer that happened to say "the text doesn't specify the year" vanished.
Excerpts are now called excerpts, with the pages listed, and the model is asked to say what is not in the pages it was given. Only a short reply that opens with a refusal is replaced.
The index matched everything
Once short words and the scoring were fixed, a new winner appeared for almost every question: the back-of-book index. A line like "Galapagos Archipelago, birds of, 390, 398" is dense with exactly the words a question uses, and it answers nothing. Worse, its numbers point at the printed edition's pages, not ours. One excerpt carried seven index pages. A chunk where most lines end in page references is now treated as an index or a table of contents, and ranked excerpts leave it out unless it is all that matches.
Cutting from the top, one layer down
Part 2 of this series was about a cut that kept the first N tokens of a page and threw away the answer. The same failure had come back one layer down. Five features (the video scene planner, the podcast script writer, the podcast's content analysis, the topic picker every file action uses and the study-sheet generator) took the excerpt they were handed and kept its first 3,000 to 28,000 characters. Because excerpts are in page order, those cuts kept the earliest picks and dropped the best ones.
One helper now fits text to a length by relevance to the request, or as an even spread when there is no topic, and all five use it.
The rest were smaller, and just as silent
Voice mode loaded the student's attached file, compressed it, and then answered without it: the spoken path built its prompt from a context that never received the file. And the embedding call had no timeout at all, so when one took 31 seconds during our eval, the answer would have waited 31 seconds; there is now a four-second deadline, after which the excerpt is built from keywords alone. Beyond those two, headings and one-line formulas were dropped before scoring, selection could never take more than 50 chunks (so a vague question on a book with short paragraphs used about half its budget), a keyword deduplication was quadratic (2.3 seconds of blocked server on 4.5 MB of synthetic text), and an upload-time prefetch embedded every paragraph under keys nothing ever read.
A worked example
The same two turns as the opening, against On the Origin of Species. The eye passage is on page 155, in chapter six.
conversation
turn 1 "How could natural selection produce an organ as perfect as the eye?"
turn 2 "explain more"Before, for turn 2:
query to the compressor "explain more"
search terms none ("explain" and "more" are stop words)
keyword pool no scores, so file order: the first chunks, the middle one and the last
embedding re-rank reorders those 50
excerpt 27 of 412 sections
pages shown 1 to 48 (26 pages) and 207
front matter title page, ebook notes, epigraphs, contents, introduction
page 155 not sentAfter, for the same turn:
query to the compressor turn 1, then "explain more" (turn 2 names nothing, so it borrows)
search terms natural, selection, produce, organ, perfect, eye
pool best keyword matches + 50 chunks spaced across the book
ranking keyword rank and semantic rank, fused
excerpt 28 of 421 sections, with a "pages shown" line
pages shown 2, 70, 72-73, 93, 108, 115, 125, 128-129, 142, 144, 155-158, 160-161, ...
page 155 sent, under its own page labelThe model also gets a short note that turn 2 follows up on turn 1, because earlier user questions are otherwise kept out of a file answer's messages so the model answers the current one.
Before and after
All figures come from the eval: Darwin, 412 pages, at the budget a file question gets.
| Measure | Before | After |
|---|---|---|
| Answering passage reaches the model, keywords only | 3 of 8 | 6 of 8 |
| Answering passage reaches the model, with embeddings | 3 of 8 | 7 of 8 |
| "Explain more" after the eye question | pages 1 to 48 and 207, no answer | the eye's chapter, answer included |
| "Summarize this book" | 4 of 16 sections, 8 pages of front matter | all 16 sections, 1 page of front matter |
| Budget a vague question used, book with short paragraphs | 54% | 95% |
| Index pages in a ranked excerpt | up to 3 | 0 |
| Ranking by meaning, cold, median | 0.77 s | 1.0 to 1.2 s |
The last row is the price of showing the embeddings twice as many chunks. On a warm server the spread's vectors are cached, so later questions about the same file reuse that half of the work. The one case still missing in each layout is a sampling miss: the Spanish question and the paraphrase each land when the spread happens to fall near the answer, which it does in one of our two layouts and not the other.
What this taught us
Every failure in this post produced a fluent answer. None crashed or timed out, and none showed an error. The cost of a silent degradation is that it is only found by someone who already knows the answer, so the eval with known answer passages did more for us than any review.
The fallback is the product. "Rank anyway" with nothing to rank by quietly defaults to file order, and for a textbook, file order means the preface. Deciding what a request is before deciding how to rank it fixed more cases than any change to the ranking itself.
Our tests had a book in them, a synthetic one where every page but one said the same thing. A ranker cannot tell front matter from content when all the content looks alike, so the fixture passed while the real behavior failed. A public-domain book with an index, a table of contents and a vocabulary of its own showed the preface problem, the lost three-letter words and the 50-chunk ceiling in its first run.
Telling the model the truth about what it was given is a quality fix, not a disclaimer. "These are excerpts, here are the pages" turns "the document doesn't cover this" into "the pages I was given don't cover this, ask me about that chapter", which is both honest and more useful.
What's next
The remaining misses come from sampling: on each question, the embeddings see about a hundred chunks of a four-hundred-chunk book. Next we store chunk embeddings for attached files at upload, so every passage is searchable by meaning on every turn and only the question itself needs embedding. The page summaries we still write at upload and never read go with it.