Analyse AI Bot Traffic with AWS Athena and Cloudfront logs

AWSAthenaCloudFrontData AnalyticsAIllmstxtTech InfrastructureData-DrivenServerless AnalyticsLog Analysis

From Raw Logs to Analysis: How We Track AI Bot Traffic at Certible

The Challenge: With increasing bot traffic hitting our llms.txt endpoint, we needed a scalable way to analyze CloudFront access patterns at low log analysis cost.

Our Solution: AWS Athena with partitioned CloudFront logs

How we set it up:

The result:

Our data shows a steady increase in llms.txt requests throughout 2025.

Sample query that shows the trend:

WITH monthly_accesses AS (
  SELECT month, COUNT(*) AS access_count
  FROM default.cloudfront_logs
  WHERE uri LIKE '/llms.txt%' 
    AND domain = 'www.certible.com'
    AND year = '2025'
  GROUP BY month
)
SELECT month, access_count,
  ROUND((access_count - LAG(access_count) OVER (ORDER BY month)) 
    * 100.0 / LAG(access_count) OVER (ORDER BY month), 2) AS percent_change
FROM monthly_accesses;

Key Takeaways:

This setup helps us understand how AI agents interact with our website.