Cloudflare has released its sixth annual Year in Review, offering a detailed look at the state of the internet, security, and the booming world of AI. The report highlights a major disparity in web crawling activity. According to the data, Googlebot crawled significantly more web pages than its AI-native competitors. Specifically, Google’s bot visited 200 times more pages than PerplexityBot.
This comprehensive Cloudflare Review draws data from a massive network spanning over 330 cities in 125 countries. The findings reveal that Googlebot reached 11.6% of unique web pages in late 2025. In comparison, OpenAI’s GPTBot only reached 3.6%, while PerplexityBot crawled a mere 0.06%. The report suggests this dominance exists because Google uses a dual-purpose approach. Googlebot crawls for both search engine indexing and AI model training. Consequently, web publishers cannot block Google’s AI training without also removing their sites from Google Search results.
Cloudflare Review: Key Findings from the 2025 Report
Beyond AI crawling, the report provides essential data regarding global connectivity and security threats. Here are the specific highlights:
- Traffic Growth: Global Internet traffic increased by 19% year-over-year, with a notable acceleration after mid-August.
- Security Shifts: For the first time, civil society and nonprofit organizations became the most attacked sector, facing 23.2% of attacks by July.
- Blocking Trends: Publishers frequently block AI-specific bots like GPTBot via robots.txt files, while Googlebot and Bingbot mostly face partial restrictions to protect login pages.
- Crawl-to-Refer Ratios: Anthropic showed the highest ratio of crawling to referring, meaning it scans content heavily but sends very few users back to the source websites.
- User-Action Crawling: Bots acting on behalf of users (like ChatGPT answering a specific question) grew over 15 times throughout the year.
Also Read: Podcast Statistics: Market Growth, Trends, and Strategies for Success
The report also notes a shift in traffic composition. While human-generated traffic remains dominant, non-AI bot traffic is close behind. By December, humans generated 47% of requests compared to 44% from bots. As AI tools continue to evolve, this Cloudflare Review serves as a vital baseline for understanding how these automated agents interact with the open web.
More News To Read: Publisher AI Partnerships: Google Strengthens Ties
