More than half of internet traffic is now non-human, according to Cloudflare, and a growing share of it is designed to avoid being noticed. Stealth crawlers, bots that mask their identity or ignore robots.txt to scrape content for AI training and agent use, have become common enough that publishers are rewriting their bot defense playbooks around them. People Inc, the publisher behind titles including People and Better Homes & Gardens, has expanded the user agents it actively blocks from roughly 2,100 to more than 30,000.
The shift matters to marketers because it is reshaping the content economics underneath every campaign. Cloudflare’s own data shows AI training now accounts for 52% of crawler requests, up from 22% in early 2025, and some heavily crawled content categories have seen human traffic decline by as much as 40% in under a year. When the audience reading a brand’s content is increasingly a bot feeding a model rather than a person who might convert, the entire logic of content marketing, and the traffic based deals built on top of it, starts to wobble.
Regulators are moving faster than expected. New York’s legislature passed the Stealth Crawler Prohibition Act in June, requiring any crawler touching a covered news site to identify itself with an accurate user agent string and disclose its purpose, backed by penalties of up to $15,000 per day. It now awaits Governor Kathy Hochul’s signature and is not yet law. If signed, New York becomes the first state to force AI crawlers to identify themselves, a template other states and a similar U.S. House bill are already watching. For marketing and content teams, that is the clearest signal yet that “who is actually reading this” is about to become a compliance question, not just an analytics one. Publishers are not waiting on Albany either: some are already building ad inventory meant specifically for AI agents, and the same disclosure pressure now hitting bots echoes what New Jersey’s new data broker law is doing to identity data one state over.
Source: Cloudflare
