Ahrefs Study Reveals 97% of llms.txt Files Remain Unread Across 137,000 Domains
Ahrefs has conducted an extensive analysis involving 137,000 domains, utilizing its Web Analytics and Bot Analytics tools. The study focused on examining server logs, live traffic, and user agents to understand the interaction with `llms.txt` files. A key finding indicates that a significant majority, 97%, of these files are not accessed or read by bots and crawlers.
A recent study performed by Ahrefs scrutinized 137,000 domains to gain insight into the readership of `llms.txt` files. The research employed Ahrefs' proprietary Web Analytics and Bot Analytics platforms.
The methodology involved a detailed examination of server logs, live traffic data, and the user agents that interacted with the vast number of domains under review. This comprehensive approach aimed to uncover patterns regarding how `llms.txt` files are handled by various automated agents.
The analysis concluded that a substantial portion of these files, specifically 97%, are never read by the bots or crawlers that access the domains. This suggests a widespread lack of engagement with `llms.txt` files within the observed web ecosystem.
According to Ahrefs Blog, the study utilized Ahrefs Web Analytics and Bot Analytics to analyze server logs, live traffic, and user agents across 137,000 domains.
