The Toggle That Felt Broken: Streaming Read-Aloud by the Sentence

How we made Cereby speak its replies as it writes them, not after it has finished.

---

Inside Cereby's chat input there is a small toggle: "Read AI responses aloud." With it on, a student would ask a question, watch the answer stream onto the screen, finish reading it, and only then hear the audio start. The voice was always behind the eyes, and for a feature meant for eyes-off study (the commute, the kitchen, a walk), that is the wrong way to fail.

This post is about Cereby's read-aloud pipeline, the path from the model's reply to the speaker, which in April we rebuilt to speak each sentence as soon as it formed. On a four-paragraph answer, that was roughly 500ms of dead air instead of several seconds, with the same toggle, speech model, voice and per-character cost. Its companion, on dictation and podcasts, is The Question Nobody Typed: Dictation and Podcasts in Cereby.

Every second of generation was a second of silence

The old shape was the obvious thing to build. Here is one reply through it:

The model already streamed its tokens; collecting them into one payload threw away latency we had paid for. Each chunk's speech was requested only after the previous MP3 came back, so round trips stacked end to end instead of overlapping with playback. And chunks were sized for the API's 4,000-character limit, not the ear, so a student could finish a paragraph before its audio was even requested. The text streamed at modern speeds; the audio arrived like it was being faxed.

A reply is a stream of sentences

We started treating the reply as a stream of sentences, in three layers that each stream on their own: text deltas over server-sent events (one HTTP response that arrives in pieces), a sentence splitter, and a speech queue that synthesizes each sentence and the two after it:

None of the three is exotic alone. Composed, they make the wait a student hears one speech round trip, during generation rather than after it:

Sentences from a moving buffer

While text is still arriving, the trailing fragment may not be a whole sentence yet. The splitter's rule, as pseudocode:

keep a pending buffer that grows with each delta
on every delta:
    look for sentence terminators (. ? ! and language-aware variants)
    for every terminator followed by whitespace (or end of buffer, then idle):
        emit the sentence up to and including the terminator
        keep the rest as the new pending buffer
on stream end:
    flush pending if non-empty

The client kept a queue, a cache of audio keyed by sentence text, and the requests in flight, so no sentence was synthesized twice at once:

to speak a sentence:
    add it to the queue
    if nothing is playing: start the playback loop
    prefetch the next two queued sentences

playback loop:
    take the head of the queue
    if it is cached: play it now
    else if it is being synthesized: wait for that, then play
    else: synthesize it, then play
    when it ends: loop again, and prefetch the new next two

Two sentences ahead

How far ahead to synthesize took tuning:

Each clip played from an object URL (a short-lived browser address for data in memory), revoked when it left the cache or the player unmounted; otherwise long sessions leak tens of megabytes of decoded audio into the tab. And the speech endpoint now returned chat audio as raw bytes, not base64 for the client to decode on the hot path; audio we keep, such as podcasts, still came back as JSON with a stored copy.

The failures the design doc missed

Cancelling mid-reply. Cancelling dropped the queued sentences and revoked the prefetched object URLs. The speech request in flight was usually already returning, so we let it finish and threw the result away.

The one-word reply. A very short reply made prefetch run on a queue that was already empty. The playback loop and the prefetch scheduler read "currently playing" from a value React updates on its next render, so the second sentence saw a stale snapshot. We made them share one flag, read and written synchronously.

"Sure!" on repeat. Identical short sentences ("Sure!", "Got it.", "Let me think.") were synthesized over and over. The fix was the cache, keyed by a sentence's first 100 characters: cheap, hash-free, good enough.

Safari's gesture rule. Mobile Safari plays audio programmatically only after an audio element is created inside a user gesture, so we primed an empty one the first time the toggle was turned on.

Before and after

The two versions differ at every layer:

	Before (buffered)	After (streaming, pipelined)
Time to first audio	Full model latency plus 1 speech call	About 1 sentence of model latency plus 1 speech call
Gap between sentences	1 speech call (sequential)	Near zero (prefetched)
Speech calls per reply	One per chunk of up to 4,000 characters	One per sentence
Chat over the wire	Buffered JSON	Server-sent deltas (opt-in)
Speech over the wire	Base64 in JSON	Raw audio/mpeg
Client cache	None	Object URLs by sentence, in-flight requests shared

What this taught us

Buffer one layer up, never two. When two streaming systems sit back to back, a model and then a speech engine, a buffer between them throws away both streams. The fix is almost never a bigger buffer; it is removing the buffer.

Size the pieces for the listener, not the API. A 4,000-character chunk arrived a paragraph late; a sentence is what the ear waits for.

What's next

The right prefetch depth for lists and short bullets differs from prose, so the lookahead should adapt. And nothing yet measured, end to end, the time until the first sentence is audible: the metric that maps to the original complaint.