[mekarpeles]

Closes #13672.

other had become the dominant series in both the crawler and partner API panels, so neither panel told you who was actually hitting the site. Both paths classified against a hardcoded allowlist that ages; anything off the list collapsed into other and the agent's name was discarded.

Read the caveat below before the design section — two adversarial review rounds found that the numbers justifying this issue were substantially a measurement artifact, and that changes how much of this change is load-bearing.

Caveat: the issue's numbers were inflated, and the two-line fix may be most of the win

The issue justifies the work with other at 7.89K against labelled crawlers of 338/280, and 2.27K against labelled partners of 74/25/11. Those came from matching the whole nginx log line rather than the User-Agent field. So:

• crawler other counted /robots.txt fetched by ordinary browsers — \b[a-z_-]+bot matches "robot" inside "robots" — and book titles, so /works/OL1W/I-Robot registered as a crawler named i_robot
• partner other counted anything with @ anywhere in the line, so its largest entries were "partners" literally named GET, POST and https — request lines and referrers with an email in a query string

A reviewer measured 622 req/min of ordinary human traffic landing in partner other as a partner called https.

Scoping both pipelines to the UA field (awk -F'"' '{print $6}') removes that. That two-line change is the load-bearing fix; promotion.py handles whatever is left. Whether the remainder justifies a rolling-window promoter can only be settled by watching real other for a day after this deploys. If partners.other comes back near-empty, the partner half of the promoter can be deleted — the 89-name pin list already covers the known world, and new partners introduce themselves by email before they do by volume.

The threshold calibration is unaffected: I checked that the committed snapshot is 92 rows of genuine partner UAs with zero request-lines, so >=50 selecting 47 agents covering 91.3% of volume was computed over clean data. What was inflated is the size of the bucket those thresholds drain, not where they sit on the distribution.

No existing series disappears

A PR titled "auto-promote out of other" invites the assumption that the allowlist was deleted and quiet partners like Gleeph (11) and Bookscovery (25) are about to vanish. They are not. The snapshot stops being a gate and becomes a pin list: pinned partners keep their series however quiet, bypassing both the threshold and the cap, while new partners are promoted on volume.

Verified rather than assumed — I reimplemented the old extract_agent_counts and diffed its labelling against the new path: 89 labels before, 89 after, none lost, none gained. Two reviewers independently confirmed it.

Three series will step at deploy. That is bugs being fixed, not traffic changing.

Partner undercounts. The old code did agent_counts[name] = count — assignment, not accumulation — so UA variants sharing a first token overwrote each other:

partner	was reporting	actually
Pinakes-POC	73	238
ReRoll	39	83
WonderclubBot	32	140

partners.other drops sharply, because it was largely GET/POST/referrer junk.

bot_traffic.other changes definition, from whole-line lines to UA-field first-match occurrences. Direction is mixed: browser noise disappears, self-naming crawlers no longer double-count. Expect a step; it needs a dashboard annotation.

How promotion works

Both jobs tick once a minute, but the volume justifying a label accrues over hours — so the promotion decision and the metric emission are separated. Emission is unchanged: each tick reports that tick's counts. Only which names get their own label is decided over a rolling 60-tick window.

Applying the thresholds literally per-minute would promote nobody. Simulated at the issue's own volumes:

agent	per hour	promoted at	under a literal per-minute threshold
perplexity	338	tick 14	never
googlebot	280	tick 17	never
UMDB	74	tick 41	never

max_labels (25 crawlers / 50 partners) caps concurrent labels so the legend cannot fill with drive-by bots, and promotion lapses when an agent ages out of the window, so a one-off burst holds a slot for at most an hour.

What two adversarial review rounds found and fixed

1. Labels minted from the whole log line — robot, i_robot, GET, https. Fixed: UA field only.
2. Pinned partners competed in the ranking, so the 89 highest-volume names ate all 50 slots and starved the new agents the feature exists to find — it would have shipped and done nothing. Replaying the snapshot plus 10 new partners: 5 of 10 promoted before, 10 of 10 after.
3. One agent became two series. grep -o emitted every match, and polite crawlers name themselves twice: OAI-SearchBot/1.0; +https://openai.com/searchbot produced oai_searchbot and a phantom searchbot, each at full count, burning two slots. Fixed: first match only.
4. non_bot and other stopped summing to the total once the bot path was UA-scoped and non_bot was not — a browser fetching /robots.txt was counted in neither (~11.6% of lines on one reviewer's fixture). Both UA-scoped now.
5. A UA could overwrite existing series. Only other was reserved, but bash emits <bucket>.non_bot into the same bucket and Python submits second, so non-bot overwrote the real human-traffic count. Aranet_SearchBot likewise slipped the known-bot filter (literal hyphens) and minted the real bot's label. Both closed.
6. Promoted labels emitted gaps on quiet minutes, rendering as sawtooths. Now zero-filled. Pinned names are deliberately left sparse — zero-filling those would change what every mean/average panel over the 89 existing series reads, with no traffic behind it.

Plus a socket timeout on graphite submission, and split("\n") over splitlines().

Limitations, stated plainly

• The Graphite tree grows over time. max_labels bounds concurrent series, not whisper files; a promoted name that later drops out leaves its file behind. MAX_CREATES_PER_MINUTE = inf in olsystem's carbon.conf, so there is no create throttle — the cost is disk and inodes, not starvation of other services. (That config is in olsystem; I could not confirm from here that it is live on the graphite host.)
• User-Agent is attacker-controlled. Labels are length-bounded, reserved names are protected, and max_labels caps concurrency — but someone willing to sustain volume can still churn new metric paths or evict a legitimate agent from the ranking. Deliberately not hardened further; this is observability tooling.
• The window is in-process and resets on deploy, rebuilding over the following hour.
• This improves other, not non_bot. The regex needs a letter immediately before bot/spider/crawler, so Scrapy, python-requests, facebookexternalhit, Go-http-client, Screaming Frog SEO Spider and Barkrowler remain invisible. If those are a large share of traffic, that is a separate change.
• The obfi.sh / USER_AGENT_BOTS duplication is untouched; this adds no new copy of the bot list, and the FIXME stays. ($is_sus_user_agent in web_nginx.conf is a 403 blocklist, not a third copy.)

Before merge

• Confirm nc -zv graphite.us.archive.org 2004 from ol-covers0. That host only ever used plaintext 2003; the promotion path uses pickle 2004. The server binds it on 0.0.0.0 exactly as 2003, and ol-www0/ol-home0/ol-solr0-2 already submit there, so this is expected to work — but covers0 has not done it before.
• Check whether anything alerts on .other. Moving spikes out of other is the point, which defeats any "unknown traffic spike" alert pointed at it. Re-point at an aggregate such as sumSeries(stats.ol.partners.*).

Testing

49 tests, pre-commit clean. test_monitor.py covers the jobs themselves — metric paths and submission — because the helpers being green did not stop the submission from being deletable without a failure. Beyond the assertions, the fixes above were each verified by running the real bash pipeline against the reviewers' own repro inputs.