Known Agents; Bye Bye Bad Bots?; Post-Quantum Bots

I’ve been doing some extended work in the “bot” realm, especially as it pertains to our AI overlords, so today we take a fairly deep dive across three sections into the current bot landscape.

In other news, the new Freedium we linked to in the previous Bonus Drop has added support for some of the more egregiously paywalled newspapers. I know folks gotta eat, but Democracy dies in paywalls, so I’m fine with subverting them (esp. the NYT and WaPo) until we’re out of this mess.

TL;DR

(This is an LLM/GPT-generated summary of today’s Drop. Ollama and ornith:35b.)

• Known Agents tracks 2,000+ AI crawlers, scrapers, and agents across 15+ categories with an identification API, robots.txt generation, and Global Insights data showing 31% of web visits now come from artificial agents (https://knownagents.com/)
• New York’s Stealth Crawler Prohibition Act and the EDPB’s Guidelines 03/2026 on web scraping for generative AI signal a transatlantic regulatory shift from “scrape first, ask later” toward mandatory crawler identification, with GDPR applying in full to AI training scrapes and penalties up to $15,000 per day in NY (https://www.edpb.europa.eu/our-work-tools/documents/public-consultations/2026/guidelines-032026-web-scraping-context)
• Post-quantum TLS handshakes (X25519MLKEM768 key shares) have become a cheap, retroactive bot-detection signal at CDNs like Cloudflare and Akamai, exposing scraping tools whose ClientHello no longer matches real browsers, a shift further cemented by Executive Order 14412’s federal PQ migration deadlines (https://scrapfly.io/blog/posts/post-quantum-tls-bot-detection)

Known Agents

The Site Observatory experiment I’m running on one of my other sites has had a pretty brutal observation regarding what actors hit small blog sites. Now, I’ve geared theat one to attract AI agents (as part of the experiment), but — in general — half of any given website traffic isn’t human – and hasn’t been for a while. However, a major shift has happened and a growing slice of that non-human traffic is now AI agents. Sure, the search engine crawlers still exist, but also autonomous browser agents shopping your store, AI assistants fetching your content to summarize for someone else’s users, and training-data scrapers hoovering your pages into model training sets.

Known Agents tracks this ecosystem: who the agents are, which companies operate them, and how to identify them at the network level. The directory covers 2,000+ crawlers, scrapers, and AI agents across 15+ categories. Their identification API verifies whether an incoming request is genuinely from who it claims to be — doing the cryptographic IP matching so you don’t have to – and there’s automatic robots.txt generation that keeps pace with new bots as they appear. The Global Insights page publishes anonymized aggregate data from thousands of subscribing sites, and the numbers are clarifying: 31% of all website visits come from artificial agents, ChatGPT-User alone accounts for 1.3% of global bot traffic, GPTBot another 0.5%, and the count of top-1,000 sites blocking AI scrapers in their robots.txt keeps climbing. (Keep in mind this data is biased by the sites that have opted-in to using Known Agents).

The full agent directory (HTTP user-agent strings included) lives on that Global Insights page in a table that’s trivially scrape-able (no API key, no auth, just a curl pipe and a few lines of R):

curl --silent "https://knownagents.com/insights" | Rscript -e \
  'rvest::read_html("/dev/stdin") |>
   rvest::html_table() |> _[[1]][,1:3] |>
   janitor::clean_names() |>
   dplyr::mutate_all(stringi::stri_replace_all_regex, "\\n.*+", "", multiline=TRUE) |>
   setNames(c("agent", "type", "company")) |>
   write.csv(row.names=FALSE)'

That gives you a clean CSV of every tracked agent, its category, and the operating company. The names alone are worth skimming. Alongside the expected GPTBot, ClaudeBot, and Google-Extended, you’ll find things like AmazonBuyForMe (Amazon’s shopping agent that visits product pages to gather purchasing info), Claude-Code (tracking how often Anthropic’s coding agent is reading your docs), Manus-User (a browser-enabled agent from China’s Butterfly Effect, up 34% month-over-month), and opencode itself clocking a 246% monthly traffic increase. The “Undocumented AI Agent” category is particularly worth some attention, as these are crawlers that don’t publicly disclose their purpose, and several of them (anthropic-ai, Claude-Web, cohere-ai) appear in 13-15% of top-1,000 site robots.txt files, meaning enough operators have noticed and blocked them to register at that scale.

Known Agents is free for most sites and takes under five minutes to set up.

If you’ve ever wondered who’s actually reading your pages and what they’re doing with the data afterward, this is where I’d start (either use the site or at least grab the agent data).

Bye Bye Bad Bots?

Two regulatory developments in the past month may be a signal that the free-for-all on web scraping for AI training is closing — simultaneously — from both sides of the Atlantic.

New York’s legislature passed the Stealth Crawler Prohibition Act (S9934A) on June 5, sending it to Governor Hochul for signature as the first state-level law in the US directly targeting AI data scraping. The bill defines “crawler” broadly as “any software that retrieves, scans, indexes, scrapes, or otherwise accesses a website, including bots, user agents, fetchers, spiders, and AI agents”. A “stealth crawler” is one that accesses a “covered news source” without identifying itself or its purpose, or that ignores technical controls like robots.txt and paywalls. Civil penalties run up to $15,000 per day per violation, enforceable by the state attorney general. The bill’s sponsor, Senate Deputy Majority Leader Michael Gianaris, did a pretty good job framing the problem. He asserts that stealth crawlers “have enabled tech companies to free ride off of the hard work of dedicated journalists, all while diverting readers away from the publishers’ own websites.” The New York State Broadcasters Association and the New York News Publishers Association both lobbied in support. Search engine crawlers and other bots that “announce their presence and provide a direct benefit” get a carve-out – which puts the burden squarely on every crawler operator to prove it belongs in that bucket rather than the stealth one.

Mere days later, the European Data Protection Board (EDPB) adopted Guidelines 03/2026 on Web Scraping in the Context of Generative AI at its July 8 plenary session in Brussels, resolving the most consequential open question in European AI privacy law. GDPR applies in full to web scraping for AI training, with no carve-out, no exemption for publicly visible data, and no blanket legitimate-interest defense. Four obligations now apply in sequence before a single page can be scraped when EU residents’ personal data is involved — which, given how much of the open web includes personal data, is most of the time. Developers must complete a case-by-case legitimate interest assessment (LIA) before each deployment, not once per project. Data minimization filters must run before the crawl, not downstream during dataset cleaning. Transparency obligations apply, with individual notification excused only in a narrow “impossible or disproportionate” exception that the guidelines caution cannot be routinely relied upon. And the accuracy principle requires scraping only from reliable sources, with timestamps and validation baked in.

The companion anonymization guidelines adopted the same day establish a three-criterion test that’s widely expected to function as a trapdoor rather than an escape hatch. Data qualifies as truly “anonymous” (and, therefore outside GDPR’s scope) only if it satisfies no record isolation, no linkage, and no inference simultaneously. For large language models, the “no inference” criterion creates a structural problem: generative transformers are known to regurgitate fragments of their training data verbatim when prompted (a phenomenon the EDPB had already flagged in its December 2024 Opinion on AI models, finding that few AI systems meet” the anonymization standard in practice). An academic audit of Common Crawl found that approximately 155 million of 1 billion audited URLs fell into sensitive-data categories under GDPR Article 9. This highly suggests that special-category data is structurally embedded in the raw material the industry relies on, not an edge case that careful filtering can sidestep.

Now, the two laws are structurally different: New York targets stealth specifically and applies only to news sites, while the EDPB guidelines apply to any scraping that touches EU residents’ data, full stop. But they converge on a practical implication that anyone running a website really needs to grok. The legal environment is shifting from “scrape first, ask later” toward “identify yourself before you access, or face consequences.” This causes the aforementioned Known Agents directory and identification API to sit exactly at that inflection point. The question of who’s crawling your site is transitioning from operational curiosity to regulatory compliance, and neither the New York law nor the EDPB guidelines offer a mechanism for answering it that doesn’t require knowing which agent is making each request, and whether it is who it claims to be.

It’ll be interesting to see what might happen if more states and countries/regions pile on to this trend. Sadly, these feel like Potemkin rules, since enforcing them really does have both high-cost and high complexity to implement well.

Post-Quantum Bots

Post-quantum cryptography was sold as future-proofing against quantum computers that don’t exist yet. The actual near-term consequence turned out to be something nobody in the scraping community saw coming: it completely changed TLS handshakes, and that new shape is now a [potential] bot detection signal.

Scrapfly published a thorough breakdown in April (Post-Quantum TLS: Why Scraping Tools Are Now Exposed) of how the transition to post-quantum key exchange created an entirely new fingerprinting vector. The mechanism is both elegant and easy to grok.

Chrome since April 2024, Firefox since November 2024, and Apple since October 2025 all send an X25519MLKEM768 key share by default in every TLS 1.3 handshake. That key share is 1,124 bytes. The X25519 key share it replaced was 36 bytes. The ClientHello, which used to fit comfortably in a single TCP packet at 300-500 bytes, now exceeds 1,400 bytes and fragments across two or more packets. CDNs cross-reference the key share presence against the User-Agent header at the TLS layer (before any HTTP traffic arrives). A request claiming to be Chrome 131 without a PQ key share doesn’t match any known-good Chrome 131 ClientHello, and the check requires no machine learning, no behavioral analysis, and no session history.

Also, when a browser vendor (Chrome, Firefox, Apple) ships post-quantum TLS support, the change is backward-looking in its detection effect. A scraper that was working fine yesterday (e.g., Chrome 130 with a 36-byte key share) breaks against today’s CDN rules even though the scraper didn’t change anything. The CDN doesn’t just check what the scraper is doing now; it checks whether the scraper’s handshake matches what a real browser would have sent since the PQ deadline. Rolling forward your User-Agent string to “Chrome 131” while your TLS stack still produces a pre-PQ handshake is the fingerprint equivalent of putting a 2026 license plate on a 2018 car with 60,000 miles on the odometer. The mismatch is detectable immediately (and cheaply) and the check is retroactive to whenever the browser turned PQ on by default.

This retroactive “bite” is what makes this particularly nasty for scraper maintainers. By early 2026, 57.4% of all browser connections already included a PQ key share; Cloudflare had been rolling out PQ support across all plan tiers since 2024; Akamai made post-quantum key exchange the default for every client connection on January 31, 2026. Go-based scraping libraries that pinned pre-PQ uTLS fingerprints found themselves especially exposed, as two named CVEs confirmed that their Chrome impersonation profiles were trivially distinguishable from real browsers (which is a genuinely embarrassing way to get caught).

I should note that those stats may not hold for your personal site. I’ve got a “PQ Watch” going at https://rud.is/pqwatch/ and the hacking bots + dumber bots are really not PQ ready at all.

Three weeks after the Scrapfly post went live, the White House signed Executive Order 14412 on June 22 (Cloudflare’s write-up), setting a December 2030 deadline for federal agencies to transition high-value systems to post-quantum encryption and a December 2031 deadline for post-quantum authentication. Cloudflare noted that over two-thirds of browser traffic to its network is already PQ-protected and that it had moved its own full-PQ target to 2029 following research breakthroughs from Google and Oratomic. The EO is nominally about federal compliance timelines, but the infrastructure it mandates is the same infrastructure already reshaping bot detection at the TLS layer. The upgrade cycle that federal contractors will spend the next four years navigating is the same one that turned the ClientHello into a fingerprinting surface.

The practical upshot for anyone running a website: your server’s TLS handshake now carries a signal that didn’t exist eighteen months ago, and CDNs are already using it. This is structurally separate from the Known Agents directory, from robots.txt, from any agent identification API. It’s a change in how the internet transports encrypted traffic, and it’s classifying bots at a layer that most bot operators can’t easily spoof. Even attempting to spoof requires implementing a post-quantum TLS stack that produces the same packet-size distribution as the browser it claims to be, which is a significant undertaking that’s likely not worth the effort.

FIN

Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:

• 🐘 Mastodon via @dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev
• 🦋 Bluesky via https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy

☮️