From Raw Logs to Analysis: How We Track AI Bot Traffic at Certible
The Challenge: With increasing bot traffic hitting our llms.txt endpoint, we needed a scalable way to analyze CloudFront access patterns at low log analysis cost.
Our Solution: AWS Athena with partitioned CloudFront logs
How we set it up:
- Partitioned CloudFront logs by year/month/day so that queries scan only the relevant data
- Athena for serverless SQL queries - pay only for what you scan
- Time-series analysis to track monthly growth patterns
The result:
Our data shows a steady increase in llms.txt requests throughout 2025.
Sample query that shows the trend:
WITH monthly_accesses AS (
SELECT month, COUNT(*) AS access_count
FROM default.cloudfront_logs
WHERE uri LIKE '/llms.txt%'
AND domain = 'www.certible.com'
AND year = '2025'
GROUP BY month
)
SELECT month, access_count,
ROUND((access_count - LAG(access_count) OVER (ORDER BY month))
* 100.0 / LAG(access_count) OVER (ORDER BY month), 2) AS percent_change
FROM monthly_accesses;
Key Takeaways:
- Partitioning saves money: queries scan only the relevant data
- Athena is billed per query, by the amount of data scanned
- Access patterns in the logs show how often AI agents request llms.txt
This setup helps us understand how AI agents interact with our website.