A Port and a Key: Hardening Cereby's AI Detector API

How a scoring endpoint that worked became one we could operate, and the old way in that outlived the migration

---

The first production version of Cereby's AI detector was a port and a key. A VM ran the detector with its port open to the internet, and the app checked an API key on each request. If you knew the URL and the key, you got a score; if not, a 401. Docker, meanwhile, reported the container as unhealthy while it served real traffic perfectly. Its healthcheck called a key-protected route without a key, got the same 401, and called that sick.

The scores were never in doubt. This post is about operating the detector API, the service that scores a text for how likely it is to be AI-written: what the first version left exposed, the stack we put around it in March, and what an audit three weeks later found we had missed. It is one of three posts on the detector, with 62% and Nowhere to Look: Cereby's Standalone AI Text Detector on the workspace students use and Twelve Rules and a Paraphraser: Building the Cereby Humanizer on the humanizer.

Nothing stood in front of the model

As the only thing between the internet and a model that costs CPU on every request, a port and a key stopped being fine quickly. Here is one request on that path, and the healthcheck beside it:

There was no TLS at a stable hostname, and no edge (a server in front of the app) to absorb bursts, cap request size or answer with a 429 before a request reached the app. A leaked key would have meant unbounded inference until someone noticed. After a reboot, someone had to remember which containers to start. None of it was the model's fault. It was operational debt, the kind that shows up all at once, when traffic does.

Edge work at the edge, scoring in the app

We kept the scoring app exactly as it was and wrapped it in a small Docker Compose stack (a file that declares which containers run and how they connect). Nginx, a reverse proxy, became the only thing listening on ports 80 and 443. The detector moved to an internal port that was never published, the product app called detector.cereby.ai over TLS with a shared key, and a systemd unit (the host's service manager) brought the stack back up on boot:

That split made the rest tractable. Nginx owned TLS (Let's Encrypt certificates through certbot, with plain HTTP kept only for certificate challenges and a redirect), request size, timeouts, connection caps and coarse throttling per IP and per key. The container owned what a key means and the scoring itself, with a hard ceiling on concurrent scoring jobs. Past it, a request came straight back as a 503 with a Retry-After header: refuse work the VM cannot do, fast, instead of queueing it into oblivion.

It also made bad traffic visible. Nginx wrote the scoring routes to their own access log on the host, and fail2ban (a tool that watches log files and bans IPs that misbehave) read it, beside a jail for SSH. Repeated failed keys got an IP banned at the firewall for a while, which caught credential probing that would otherwise cost a TLS handshake and a proxy hop per attempt.

What it was built to stop

The detector was a CPU-bound model behind a key, so bad traffic cost CPU on one VM and a little bandwidth, not stolen data. The stack was shaped for the steady drip that finds any exposed API: probes with key lists, scanners looking for .env and .git, scripted bursts against the scoring route, none of it sophisticated and all of it constant. The jail and the per-IP limits made an evening of probing uneconomical, not impossible.

A distributed campaign from thousands of IPs with valid-looking keys would call for Cloudflare or similar in front of the hostname, which we had not added because nothing of that shape had shown up. And no perimeter helps with a leaked key; short rotation cycles and one-line revocation do.

Version one never shut down

On April 19, three weeks in, an independent audit checked everything above. Most of it held. The biggest finding was that the listener from before the Compose stack was still running, added to rather than replaced by the new setup, with its plaintext port still open to the internet. Here is the probing traffic from the last section, on both ports:

That port skipped Nginx, TLS, the rate limits, the body cap and the jail, leaving the app's key check alone in front of it: version two shipped, and version one never shut down. We disabled the old listener and closed its port at the firewall. Here is the stack the audit left, with the dead branch kept on purpose, so nobody recreates it:

The boring failures

The healthcheck that needed a key. The probe in the opening broke a simple rule: if production needs a key, the probe needs the same key. Otherwise you spend an afternoon chasing a healthy container that Docker insists is sick.

The certificate nobody renewed. The audit's second finding was a valid Let's Encrypt certificate with nothing to renew it: no timer, no scheduled job, no certbot container. It would have expired silently months later. A systemd timer then ran a throwaway certbot container twice a day against the files Nginx already used, and reloaded Nginx on success.

The log the jails could not fully see. Not every request reached the log the jails read. Moving its declaration up a level in Nginx's configuration put every request, on 443 and on 80, in one file.

Logs without a ceiling. Container logs had no size cap, and the scoring log only grew. The Docker daemon got a cap per container, and the scoring log was rotated and compressed weekly, with Nginx reopening it and fail2ban reloading after each rotation.

The audit also added a jail for scanners asking for .git, .env, wp-admin, backups and database dumps, and found SSH settings that had drifted and a build context that no longer matched our tracked source. The rest each got a one-paragraph cause and fix in the runbook: certificate ordering, missing Compose files, hairpin 502s (a server calling its own public address and getting a bad gateway back) and broken build contexts (the folder Docker builds an image from).

Before and after

After any change, three requests from outside proved the deploy (plain HTTP redirects, the health route refuses a request with no key, a scoring request with the key returns a score), and the scoring log and its jail had to be live.

	Before	After
Public surface	The app's own port	Nginx on 80 and 443, once the old port was closed
TLS	Not standardized	HTTPS by default on detector.cereby.ai
Overload	Best effort	429 at the edge, 503 when scoring slots are full
Failed keys	The app's key check	A dedicated log, and bans for repeated failures
Boot	By hand	systemd and Compose
Proof after a deploy	Tribal knowledge	Three requests in a short runbook

What this taught us

A migration is not finished until the old path is closed. Ours sat beside the new stack, skipping every layer we had built, until an audit found it.

The boring failures are most of the incident time, and a one-paragraph cause and fix beats heroics at 2 a.m. And if three requests cannot prove a deploy worked, nobody running it will trust it.