i summon one kim

The manual pages on man.netbsd.org are rendered by man-cgi, a ~1400-line POSIX shell script forked per request by fcgiwrap behind an nginx FastCGI cache and Fastly. This weekend I overhauled how the service caches.

The overhaul was driven by three problems:

• Surviving traffic peaks. A crawler surge can overwhelm the service: when enough requests miss the caches, renders can queue faster than the fcgiwrap worker pool drains them, and requests start failing with 502.
• Cache size. The nginx cache on the origin had grown to 110 GB, most of it duplicates: a machine-independent page was cached separately under every architecture prefix (/i386/ls.1, /amd64/ls.1, … alongside /ls.1) — up to ~60 objects for one page — and bots dutifully crawled the whole alias space.
• Cacheability. A single Cache-Control header steered browsers, nginx, and Fastly alike, nothing could be invalidated remotely, and every page embedded the full architecture and collection lists in its query form, so each NetBSD release invalidated every cached page at every tier.

An accident was also hiding in the TTL logic: the pattern meant to give NetBSD-current pages a short lifetime could never match the default collection, so pages that are rebuilt daily were being cached for 90 days. An earlier round of tuning in April had coped with load by caching longer; this round replaces that approach with revalidation and purging.

What changed

Each cache tier now gets its own header:

• Cache-Control for browsers, which cache briefly (they cannot be purged),
• X-Accel-Expires for nginx, which also caches briefly (an expired entry costs only a revalidation), and
• Surrogate-Control plus Surrogate-Key for Fastly, which caches long (purgeable by key at any time).

A small purge tool resolves keys like coll:NetBSD-current or a single page into a soft purge, so a collection rebuild can be made visible immediately.

Importantly, the CGI gained a fast 304 path: when If-Modified-Since matches the Last-Modified it would send, it answers 304 without rendering, so the origin cost of a revalidation drops to a man -w plus a stat. Before trusting this in production I verified nginx’s side in a disposable lab rig against a synthetic FastCGI backend: with caching enabled, nginx never forwards client conditionals to the backend, so the only If-Modified-Since the CGI ever sees is nginx’s own revalidation echoing the CGI’s Last-Modified back — which makes the exact string match a complete test, not an approximation.

The query form no longer embeds the architecture and collection lists. The server renders fallback options and an inline script populates the selects from two plain-text endpoints (/api/v1/archlist, /api/v1/colllist); the browser remembers the chosen values in localStorage, not in cookies, so no request grows a cookie header and nothing varies per user at any cache tier. A list change now invalidates two tiny objects instead of every page. Without JavaScript the selects offer only the current values, but the command field and all page URLs keep working.

With the form remembering the architecture, the URL no longer needs to carry it for pages shared across architectures — which really was the only reason it was carried. So the CGI now redirects every page to its canonical URL: /i386/ls.1 becomes /ls.1 (301), machine-dependent pages keep their arch, and machine-class pages redirect into their class directory (/i386/est.4 → /x86/est.4). This collapses up to ~60 cached objects per machine-independent page into one page plus small cacheable redirect objects, and it is the main lever on cache size. The full effect appears only as crawlers re-follow the new 301s.

The work was aided by Claude Code sessions: characterization tests first, cold-context reviews of the changes, and an architecture decision record whenever a real decision was made — ten ADRs by the end.

The cutover, measured

I deployed twice today, and both deployments deliberately dropped every cache tier: the nginx caches were wiped on both origin hosts and Fastly got a full purge (objects cached before the cutover carried no purge keys, so purging the first batch selectively was not an option). That made the day an unplanned load test, and the origin access logs across it split into clear phases:

The first purge hurt: for over three hours roughly a tenth of the non-rate-limited traffic — some 32,500 requests — failed with 502, every one of them fcgiwrap’s socket refusing the connection. Renders queued faster than the worker pool drained them; nothing broke inside the CGI itself. The second purge, served by the version with the canonical-arch redirects, produced no 502s at all: the failover origin carried the transition at ~6 QPS and the refill ran at 8.6 QPS mean, 37% of it the new cheap, cacheable 301s. That is the object-space collapse working as designed. Note the numbers bound CGI load only from above — the log format unfortunately doesn’t record nginx cache hits — and no warm-cache measurement of the new version exists yet; it will likely sit below that 8.6 QPS except when some inconsiderate new bot hits it.

The measurements settled a question that has hung over this script for years: should it be rewritten in a faster or persistent-server language? With refill load at single-digit QPS even against deliberately cold caches, and revalidation priced at a man -w, the answer is no — the renderer cost stopped mattering before the shell did. The only observed failure mode is a worker-pool capacity limit under a full cache drop, which cache policy already avoids (wipes are rare and operator-initiated), though I still owe fcgiwrap a capacity review. Thirty years of accreted rendering behaviour carries no migration risk this way; the sed HTMLizer does keep blocking UTF-8 output, and that trade-off is recorded, not forgotten.

The bots

The origin identifies clients only by User-Agent — client addresses are Fastly POPs — and the breakdown over the 23-hour window (both origin hosts combined, 1,197,000 requests) is striking:

One client — identifying as Lightpanda/1.0 — made 72% of everything that reached the origin, 87% of the storm window, and collected 93% of the 429s and 94% of the 502s, ignoring sustained rate limiting throughout. Lightpanda is just the software, an open-source headless browser, so the User-Agent names the tool and not the operator; I have no idea who was behind it.

The self-identified crawlers, by contrast, behaved better than people tend to expect. GPTBot peaked at about three requests a second; the nine smaller ones (bingbot, YandexBot, Googlebot, Applebot, ClaudeBot, Bytespider, meta-externalagent, Barkrowler, CCBot) never peaked above 41 requests a minute, and all of them were served throughout — even during the storm. Together they still add up to about two requests per second around the clock, a constant background noise whose necessity is not always given, but the days when the identifiable bots were the ones knocking the service over appear to be behind us.

I hope the improved cacheability makes life better for the bots too: a crawler that revalidates politely now gets its 304 from Fastly or nginx without the origin doing any heavy work. Whether we should be asking bots to cache more aggressively is worth some research — the same headers serve browsers and crawlers alike, so any longer client TTL would also be dangled in front of the actual end users, the humans reading manual pages; serving them staler pages to placate crawlers would be the wrong trade.