Cloudflare Radar Year in Review 2024: big source of traffic is AI crawlers

The internet is increasingly where we live today. In fact, global internet traffic grew 17.2% this year alone, according to Cloudflare.

The network provider has released its fifth annual internet radar report, offering insights into connectivity, security, outage frequencies, device usage, and a multitude of other trends.

Not surprisingly, Google, Facebook, Apple, TikTok, and Amazon Web Services (AWS) are the most popular internet services worldwide, while Chrome led the pack (65.8%) as the most popular web browser globally.

One big source of traffic, it noted, is AI crawlers, which are increasingly under scrutiny as they scan the web and gobble up voluminous amounts of data to train large language models (LLMs). A big concern is that some take data even when they’re not supposed to, as opposed to “verified” good bots that typically come from search engines and are transparent about who they are (such as GoogleBot, GPTBot, Qualys, and BingBot).

Cloudflare tracks AI bot traffic to determine which are the most aggressive, which have the highest volume of requests, and which perform crawls on a regular basis. Researchers found that “facebookexternalhit” accounted for the most traffic throughout the year (27.16%) — the bot is notorious for creating excessive traffic — followed by Bytespider (from TikTok owner ByteDance) at 23.35%, Amazonbot (13.34%), Anthropic’s ClaudeBot (8.06%), and GPTBot (5.60%).

Interestingly, Bytespider traffic gradually declined over the year, ending roughly 80% to 85% lower than at the start of the year, while Anthropic’s ClaudeBot traffic saw a spike in the middle of the year, then flattened out. GPTBot traffic, for its part, remained pretty consistent throughout 2024.

How we connect (or don’t)

HyperText Transfer Protocol (HTTP)  is the backbone of data transmission, first standardized in 1996. HTTP/2 was released in 2015 and HTTP/3 rolled out in 2022. Cloudflare found that HTTP/2 still accounts for nearly half of web requests (49.6%), while 29.9% use older HTTP/1 and 20.5% use HTTP/3.

Cloudflare also keeps close track of another critical communications standard, transmission control protocol (TCP), which ensures reliable data transfer between network devices. The company found that 20.7% of TCP connections were unexpectedly terminated before the exchange of any useful data. TCP anomalies can occur due to denial of service (DoS) attacks, network scanning, client disconnects, connection tampering, or “quirky client behavior,” Cloudflare pointed out. 

The largest share of TCP connection terminations identified by Cloudflare took place “post SYN,” or after a server received a synchronization request, but before it received an acknowledgement.

On the security front, Cloudflare found that, of the trillions upon trillions of emails sent this year, an average of 4.3% were malicious. These most commonly contained deceptive links (42.9%) and deceptive identities (35.1%). Both methods were found in up to 70% of analyzed emails at different times throughout the year.

Cloudflare also noted that the Log4j vulnerability is still a tried-and-true attack method, being anywhere from 4x to 100x more active than other common vulnerabilities and exposures (CVEs).

In addition, nearly 100% of email messages processed by Cloudflare from the .bar (bar and pub) .rest (restaurant), and .uno (Latin America) domains were found to be either spam or outright malicious.

Beyond CrowdStrike

While many accuse CrowdStrike of breaking the internet — the July outage will undoubtedly go down as one of the largest in history — Cloudflare noted that there were actually 225 major internet outages around the world this year. The majority of these occurred in Africa, across the Middle East and India.

More than half of these outages were the result of government-directed shutdowns; others were caused by cable cutting, power outages, technical problems, weather, maintenance, and cyberattacks. Cloudflare reported that many were short-lived (lasting just a few hours) while others “stretched on for days or weeks,” such as one in Bangladesh that lasted over 10 days in July.

Who has the fastest internet (and what are they connecting on)?

Cloudflare ranked countries across the globe on internet quality, based on upload speed, download speed, idle latency, and loaded latency. Who leads the pack? Spain, which boasts download speeds of 292.6 Mbps and upload speeds of 192.6 Mbps. All top countries experienced download speeds above 200Mbps.

As for how people around the world connect, 41.3% of global internet traffic came from mobile devices, and 58.7% from laptops and PCs. However, in roughly 100 regions of the world, the majority of traffic came from mobile devices. Cuba and Syria had the largest mobile device traffic share (accounting for 77%), with other high demand areas including the Middle East/Africa, Asia Pacific and South/Central America.

Cloudflare pointed out that these traffic measurements are similar to those of 2023 and 2022, “suggesting that mobile device usage has achieved a steady state.” This should come as no surprise, as roughly 70% of the world’s population uses smartphones today.