Cloudflare, a leading internet infrastructure and cybersecurity company, announced on Tuesday the introduction of a default setting that allows websites to automatically block artificial intelligence companies from scraping their content, the New York Times reports.
The move comes amid growing concerns from publishers and content creators over the unauthorized use of digital material to train AI models.
Under this new policy, bots that attempt to collect data from websites hosted by Cloudflare will now be blocked by default unless the site owner explicitly grants permission. Previously, AI crawlers that were not flagged as malicious could often access and collect website data freely.
“We’re changing the rules of the internet across all of Cloudflare,” said Matthew Prince, the company’s CEO.
He described the new approach as a way to support digital publishers by ensuring AI developers seek consent before using their data.
“If you’re a robot, now you have to go on the toll road in order to get the content of all of these publishers,” Prince said.
Cloudflare, whose global infrastructure reportedly handles around 20% of internet traffic, said it has observed a notable rise in AI-related data scraping. AI developers—including companies like OpenAI, Anthropic, and Google—rely heavily on large datasets from the internet to train their models, making the issue of data access increasingly sensitive.
The company’s move reflects broader tensions in the tech industry over the use of online content to develop AI tools. Several content creators and organizations have raised concerns that their materials are being used without permission or compensation. Notable legal actions have included Reddit’s lawsuit against Anthropic and The New York Times’ suit against OpenAI and Microsoft, both involving allegations of unauthorized data use.
While some publishers have chosen to license their content to AI firms—such as The Times’ agreement with Amazon and similar deals by Axel Springer and Condé Nast—others continue to seek greater control and compensation. Mark Howard, COO of Time, welcomed Cloudflare’s move, saying it represents an initial step in addressing a complex issue.
“News publishers deserve fair compensation for what they publish,” he said.
Cloudflare had already introduced an optional AI-blocking setting last year. Tuesday’s announcement marks a shift toward making such protections standard, rather than opt-in.
Prince expressed concern about the broader implications for online publishing, warning that if content creators are not fairly compensated, the motivation to produce original content could decline.
“The incentives for content creation are dead,” he said. “And if [AI companies] don’t get access to the content, then their product will be worse.”
Major AI companies affected by the move, including OpenAI, Anthropic, and Google, did not immediately comment.
