Cloudflare has unveiled a new feature for web hosting customers to block AI bots from scraping website content without permission. Responding to customer concerns, Cloudflare introduced a one-click option to block all AI bots, aiming to protect content creators and maintain a safe internet environment.
“We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly. To help, we’ve added a brand new one-click to block all AI bots,” Cloudflare stated.
While the robots.txt file has been a traditional method to block bots, it’s often ignored by unscrupulous AI bots. Notably, OpenAI and Google provided guidance last year on blocking their crawlers using robots.txt. However, Cloudflare’s analysis revealed that about 39 percent of the top one million web properties served by Cloudflare are visited by AI bots that often bypass robots.txt directives.
Cloudflare’s machine-learning model has been effective in detecting bots, including those disguised by spoofed user agents. This detection relies on digital fingerprinting, allowing Cloudflare to identify and block bots even when they attempt to mimic real browsers.
To help preserve a safe Internet for content creators, we’ve just launched a brand new “easy button” to block all AI bots. It’s available for all customers, including those on our free tier. Read our blog post for more details: https://t.co/csWFFgqbKM
— Cloudflare (@Cloudflare) July 3, 2024
With this new feature, available even to free-tier customers, users can simply toggle the “Block AI Scrapers and Crawlers” button in the Security -> Bots menu to activate the block.
Cloudflare emphasized its commitment to continually adapting its machine learning models to combat evolving bot tactics, ensuring content creators retain control over their data.
Maintaining control over website content and preventing unauthorized AI training is crucial in today’s digital landscape.
Try Out Cloudflare Enterprise for Better Performance and Security!
Cloudflare Enterprise offers advanced tools for businesses that prefer performance and security over anything.
Start Growing with Cloudways Today.
Our Clients Love us because we never compromise on these
Abdul Rehman
Abdul is a tech-savvy, coffee-fueled, and creatively driven marketer who loves keeping up with the latest software updates and tech gadgets. He’s also a skilled technical writer who can explain complex concepts simply for a broad audience. Abdul enjoys sharing his knowledge of the Cloud industry through user manuals, documentation, and blog posts.