Cloudflare, a tech company that helps websites secure and manage their internet traffic, said on Tuesday that it had rolled out a new permission-based setting that allows customers to automatically block artificial intelligence companies from collecting their digital data, a move that has implications for publishers and the race to build A.I.
With Cloudflare’s new setting, websites can block — by default — online bots that scrape their data, requiring the website owner to grant access for a bot to collect the content, the company said. In the past, those whom Cloudflare did not flag as a hacker or malicious actor could get through to a website to gather its information.
“We’re changing the rules of the internet across all of Cloudflare,” said Matthew Prince, the chief executive of the company, which provides tools that protect websites from cyberattacks and helps them load content more efficiently. “If you’re a robot, now you have to go on the toll road in order to get the content of all of these publishers.”
Cloudflare is making the change to protect original content on the internet, Mr. Prince said. If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content, he said. The company, which says its network of servers handles about 20 percent of internet traffic, has seen a sharp increase in A.I. data crawlers on the web.
Data for A.I. systems has become an increasingly contentious issue. OpenAI, Anthropic, Google and other companies building A.I. systems have amassed reams of information from across the internet to train their A.I. models. High-quality data is particularly prized because it helps A.I. models become more proficient in generating accurate answers, videos and images.
But website publishers, authors, news organizations and other content creators have accused A.I. companies of using their material without permission and payment. Last month, Reddit sued Anthropic, saying the start-up had unlawfully used the data of its more than 100 million daily users to train its A.I. systems. In 2023, The New York Times sued OpenAI and its partner, Microsoft, accusing them of copyright infringement of news content related to A.I. systems. OpenAI and Microsoft have denied those claims.
Some publishers have struck licensing deals with A.I. companies to receive compensation for their content. In May, The Times agreed to license its editorial content to Amazon for use in the tech giant’s A.I. platforms. Axel Springer, Condé Nast and News Corp have also entered into agreements with A.I. companies to receive revenue for the use of their material.
Mark Howard, the chief operating officer of Time, said he welcomed Cloudflare’s move. Data scraping by A.I. companies threatens anyone who creates content, he said, adding that news publishers like Time deserved fair compensation for what they published.
Still, what Cloudflare is enabling “is really just the very, very first step in what will be a very long process,” he said. “But you have to start somewhere, and you have to start at some time.”
OpenAI, Anthropic and Google did not respond to requests for comment.
Cloudflare began considering how to help online publishers about 18 months ago, Mr. Prince said. For the past few decades, getting people to go to their websites was how publishers and content creators made money, he said. But A.I. has changed those dynamics, with people increasingly turning to A.I. tools like ChatGPT instead of a search engine or a primary source article.
Mr. Prince said he was “deeply concerned that the incentives for content creation are dead.” Last July, Cloudflare introduced an optional setting to allow website publishers to block A.I. scrapers if they wanted. That led to the default setting on Tuesday.
A.I. companies that do not pay for content will ultimately lose out on access to it, Mr. Prince said.
“I am 100 percent confident we can block them from accessing the content,” he said. “And if they don’t get access to the content, then their product will be worse.”
The post Cloudflare Introduces Default Blocking of A.I. Data Scrapers appeared first on New York Times.
