· New F5 Labs report finds that over half of page requests for web content are now automated as use of LLM scrapers intensifies.
· More than a third of login attempts in the technology industry are account takeover attacks.
· Healthcare and hospitality are the most targeted industries on the web; entertainment is the most targeted on mobile.
By ; Bakinam Khaled
The rise of generative AI means that bots now access some types of web content more than humans, according to new research from F5.
F5 Labs’ 2025 Advanced Persistent Bots Report analyzes 207 billion web and API transactions from November 2023 to September 2024. It examines records from customers with existing bot defenses in place, demonstrating how automated traffic operators behave when confronted with countermeasures.
The report finds that 50.04% of page requests for Content[1] were from automated sources, compared to 22.3% of Search requests on the web and 21.5% of Add to Cart transactions. It suggests significant growth in the kind of web scrapers used by LLM providers such as OpenAI, Anthropic, and Perplexity, and the persistence of these bots in continuing to send requests when blocked.
In total, 21.22 billion of the transactions monitored (10.2%) came from a variety of automated sources, some of them benign, but with 10 billion (4.8%) consisting of malicious bot traffic.
“For years, bot traffic has primarily been targeted at Search flows, as well as aspects of the user journey where someone signs up or logs in to use a service, adds an item to their basket, checks out, or seeks to change their password,” said David Warburton, Director, F5 Labs. “The huge upsurge in content scraping, undoubtedly associated with the explosion of generative AI and LLMs, underlines how dynamic bot traffic is and the need for organizations to be constantly on watch for changes in attack patterns.”
Bots ease up—but not for everyone
Patterns and prevalence of bot traffic varied according to industry. The most targeted industries on the web were hospitality (44.6% of traffic from bots), healthcare (32.6%) and eCommerce (22.7%).
On mobile, entertainment (23%) was by far the most targeted sector, well ahead of eCommerce (4.5%) and QSR (4.2%).
Several industries still experience high levels of credential stuffing attacks that seek to take control of user accounts. On the web, over a third of login attempts for companies in the technology sector were attempted account takeovers (33.5%), ahead of general retail (25.7%) and gaming (19.6%). On mobile, such attacks were most prevalent against entertainment companies (24.7%) and eCommerce providers (23.8%).
The sophistication of attacks also varied by industry. The vast majority of automated traffic targeting healthcare on both web and mobile was classed as ‘basic.’ Other industries experienced relatively high levels of more sophisticated traffic considered ‘advanced’—the top three on the web being general retail, banks, and airlines.
Despite the high levels of bot traffic, the majority of industries tracked experienced a decline in automated activity compared to 2023, suggesting that bot controls in place were having the desired effect.
The outliers were hospitality on the web, which increased by 18.3% and QSR on the web, up by 11.2%. Although it experienced a much greater share of bot traffic on mobile than any other industry, the entertainment sector still recorded an 11.5% decrease from 2023.
“Certain industries are perennial targets for unwanted bot traffic,” added Warburton. “Hospitality experiences high volumes because aggregators want to scrape hotel room rate and availability data, or malicious actors are trying to steal loyalty points. Similarly, eCommerce providers are targeted by resellers and bots trained to exploit voucher and gift card details.”
From the data, it’s clear that certain industries have adapted over time: widely targeted sectors such as airlines and financial services have built up defenses to frustrate less sophisticated attackers, meaning they must now contend with a higher proportion of traffic from more advanced, highly persistent operators.
Mitigation: a double-edged sword?
The report also assessed the impact of deterrence on bot traffic, comparing the experiences of customers who were monitoring automated traffic with those who were mitigating it.
On mobile, the trend was clear and expected. Organizations mitigating traffic saw a significantly lower share of automated activity in their search traffic (0.9% compared with 24.8% for those just monitoring), a pattern matched in login (5% for mitigators compared with 21.7% for those monitoring) and sign up (2.4% compared with 21.7%).
On the web it was a different story. In most workflows, automated traffic was higher for organizations that were actively mitigating bots. These customers saw 20.9% of automated traffic in search versus 14.9% for those simply monitoring, and the equation was the same in Add to Cart (19.2% vs. 18.2%), Checkout (8.6% vs. 7.4%) and Account Recovery (6.6% vs. 4.6%).
“Typically, we’d expect mitigation to lead to a decline in bot traffic, as operators that are blocked move on in search of weaker targets,” said Warburton. “However, there are now whole business models built around the scraping of data, prices, and intellectual property: those operators are not going to give up easily when they are deterred. An increase in traffic can mean these actors are trying harder, and in more ways—not necessarily that they are succeeding. The consistent trend of this research, and all of our experience at F5, is that mitigation works, and deterrence makes a difference.”
