Scraper Poisoning: Defending Your Data From AI Crawlers
Blocking AI crawlers only stops the polite ones. ekmire's scraper poisoning module lets bad-faith scrapers in — then quietly degrades what they collect.
2026 is the year the open web started closing to AI crawlers. Cloudflare now blocks declared AI bots by default and added Pay-Per-Crawl — a 402 paywall response for crawlers that don't have a deal in place. More than 2.5 million sites have added Disallow rules for AI training user-agents. The signal is everywhere: sites don't want their content going into someone else's training run for free.
The problem is that robots.txt is a request, not a lock. A compliant crawler honors it. A scraper that doesn't care rotates IPs, spoofs a browser user-agent, and takes what it wants anyway. Blocking only filters out the crawlers you didn't need to worry about in the first place.
ekmire's proxy takes a different approach for the ones that don't ask nicely: let them in, and quietly make what they collect worthless.
How ekmire detects AI training crawlers
Detection runs as the last step of the proxy's request pipeline, after the response has already been generated for a legitimate user or bot — it doesn't block anything upstream of it. The signals it looks at:
- Request cadence. The same tarpit logic that catches brute-force automation (three or more requests under an 800ms gap) flags the mechanical, systematic crawl pattern that separates a scraper from a human browsing session.
- Honeypot anchors. Every HTML response gets an invisible anchor injected into an HTML comment — a link no human will ever click and no legitimate crawler following
robots.txtwould ever fetch. A request to that honeypot URL is close to unambiguous proof of an automated, non-compliant scraper. It immediately tarpits the source IP. - Content negotiation and header fingerprints. Combined with IP reputation signals already running in the proxy's pipeline, these narrow down "probably a training crawler" from "probably a person."
What "poisoning" means in practice
Once a source is confidently identified as an unauthorized scraper, ekmire doesn't serve a 403. It serves a response that looks completely normal — and is subtly wrong. Zero-width characters are injected at randomized word boundaries throughout the HTML response. Invisible to a person reading the page in a browser. Present in whatever text a scraper's parser extracts and feeds into a training or embedding pipeline.
This is the same principle behind Nightshade and Glaze, the image-poisoning tools that add invisible pixel-level noise to artwork so a model trained on the poisoned version learns something subtly wrong about the underlying concept. ekmire applies the same idea to text and HTML: a scraper doesn't get an error to work around, it gets data that looks fine and degrades whatever it's used to build.
Compare that to Cloudflare's approach — block by default, or charge per crawl via Pay-Per-Crawl. Both are visible, both are things a determined bad actor can route around with a new IP block or a spoofed identity. Poisoning doesn't announce itself. There's nothing to detect and evade, because from the scraper's point of view, nothing went wrong.
The legal and ethical angle
This only works as a defensive posture if it's disclosed. ekmire's own Acceptable Use Policy and every self-hosted deployment's robots.txt state plainly that automated training scraping outside stated terms is not permitted — the honeypot and poisoning behavior only ever applies to traffic that already ignored that notice. That distinction matters: this is a response to a scraper breaking a stated rule, not an attack on anyone browsing normally.
The regulatory backdrop is moving in the same direction. Full enforcement provisions of the EU AI Act for high-risk systems land on August 2, 2026, and the Text and Data Mining Reservation Protocol is emerging as the standard mechanism for European sites to formally opt out of AI training use. None of that stops a scraper that ignores the signal — but it does mean sites have a clearer, more defensible line to point to when they explain why a crawler wasn't welcome in the first place.
Who actually benefits from this
Scraper poisoning matters most for sites where the content itself is the product: documentation sites with detailed technical answers, pricing pages competitors scrape for positioning, and any LLM-powered product with a proprietary knowledge base sitting behind an API response. If your differentiator is the quality of the content you've written or the data you've structured, an unauthorized scraper isn't a nuisance — it's a competitor's next training set.
It runs as one config toggle in the same proxy already handling request-level WAF protection — no separate service, no additional infrastructure, no SDK in the app it's protecting.
Three things to take away
- Blocking and poisoning solve different problems. A 403 stops traffic. Poisoning degrades what unauthorized traffic collects. Use both — the deterministic rule engine still blocks known bad actors outright; poisoning is what happens to the ones that get past that first pass looking legitimate.
- Disclosure is what makes this defensible. An AUP and a
robots.txtentry stating your training-use terms is the difference between an active defense and an undisclosed trap. Publish the terms first. - The honeypot is the highest-confidence signal you have. No legitimate browser or compliant crawler will ever request an invisible, comment-embedded anchor link. A hit there is close to a certainty, not a heuristic guess — which is why it triggers an immediate tarpit rather than just a log entry.
The scraper poisoning docs cover the toggle, the honeypot behavior, and how it interacts with the rest of the detection pipeline. It ships in the same proxy image already running the WAF layer — nothing extra to deploy.