- AI
- A
AI bots scan even TLS certificate logs. All information is used to train LLMs.
According to Human Security's State of AI Traffic statistical report, the activity of AI agents, bots and AI tools on the internet has increased sharply over the past year.
For example, traffic from AI agents has increased almost 80-fold (by 7851%), while traffic from AI scrapers has risen by 597%. Total AI traffic has grown by 187% over the past year, and overall it is expanding 8 times faster than human traffic.
Websites that are hardest hit are those that frequently publish fresh content and useful information for AI tools. For instance, online stores with product information, or media websites. The share of bots on these sites is already approaching 90%:
As of early 2026, total AI traffic has already surpassed human traffic, so it can be concluded that AI bots have officially taken over the internet.
The number of fake accounts across all services is growing exponentially.
«The internet as a whole was built on this very simple assumption that there is a human behind a computer screen, and this assumption is changing very quickly,» — says Stu Solomon, CEO of Human Security.
The report is based on data from the Human Defense Platform, which runs for the company's clients and is theoretically capable of detecting automated traffic.
While this report is not exhaustive, it marks a significant milestone in the AI era of the internet. The industry has tracked steady growth in automated traffic since the launch of ChatGPT in 2022, and now bot activity online has officially surpassed human activity.
At the SXSW conference in March 2026, Cloudflare CEO Matthew Prince said that before the AI era, bots made up roughly 20% of internet traffic, most of which was related to Google's web crawler. He predicted that by 2027, traffic from AI bots would exceed human traffic, citing the growth of generative AI and its insatiable demand for data.
But this happened earlier — in 2026.
The End of the Open Web
The threat is so serious that some experts refer to the current state of affairs as the end of the Open Web as we know it.
In the Open Web of the past, every user could freely publish content for a wide audience. Now, all content is immediately absorbed by AI bots and used to train LLMs, which then deliver distilled summaries to users. As a result, "human" traffic to websites has dropped catastrophically. In some cases, it makes up a tiny fraction of AI bot traffic.
Website traffic in some categories, such as tech media, has dropped by more than 50%.
Publishers are trying to protect themselves from bots by moving more and more content behind closed paywalls, blocking even the Internet Archive — and this further weakens the position of the Open Web.
Open platforms such as Wikipedia are targeted by bot attacks that generate AI slop. Open-source software projects struggle to keep up with patching vulnerabilities detected by AI.
Directives in robots.txt to prohibit indexing are universally ignored by AI companies.
Open-source licenses are now completely meaningless as well, because they are trivially circumvented using LLMs.
Scanning certificate logs
AI bots collect any data they can reach on the internet. They even scan TLS certificate logs to identify new domains for crawling.
As soon as a user registers a TLS certificate for a domain (in this case autoconfig.benjojo.uk), the first visitor arrives at the server almost instantly:
Dec 12 20:43:04 xxxx xxx[719]:
l=debug
m="http request"
pkg=http
httpaccess=
handler=(nomatch)
method=get
url=/robots.txt
host=autoconfig.benjojo.uk
duration="162.176µs"
statuscode=404
proto=http/2.0
remoteaddr=74.7.175.182:38242
tlsinfo=tls1.3
useragent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; robots.txt; +https://openai.com/searchbot"
referrr=
size=19
cid=19b14416d95
This is about CT logs (Certificate Transparency), where all certificates issued by all Certificate Authorities are logged.
Tools for viewing CT logs:
Real employees are content generators
Any information that data brokers can source is currently being snapped up on the market for LLM training. For instance, when companies go bankrupt, archives containing years of internal employee correspondence, work files, and code can now be sold for a high price. Previously, this kind of data from abandoned projects was completely worthless, but now it is highly valuable training data.
Operating companies upload to LLMs their employees' information archives (chats from messengers, emails, meeting recordings, work files), creating their 'digital twins' that can partially replicate the employee in the event of their dismissal, see the collegue-skill repository on Github.
Even non-verbal behavior (hand movements) is valuable information for training AI in select professional fields, which is why all employees at some factories now work with head-mounted cameras:
To better manage their companies, top executives create their own 'digital twins' to personally monitor every current employee and maintain continuous contact with them.
Any information generated by real people (UGC) gains additional value in this day and age, when AI companies are competing extremely fiercely with one another as they try to outpace their peers in the training speed of their language models. That is why human employees have become more valuable than ever before.
Under these conditions, experts recommend exercising extreme caution when using LLM services and protecting confidential information from leaks.
Write comment