What to upload
- A raw access log (not an error log, and not a report or analytics export).
- Formats: Apache or Nginx combined, or Amazon CloudFront standard logs.
- Plain text (
.log,.txt) or gzipped (.gz). No need to unzip. - Size limit per file: 50 MB on Standard, 200 MB on Pro, 500 MB on Enterprise.
- A week or more of traffic gives a much better picture of crawler behaviour than a single day.
Where to find the log on your host
cPanel
Open cPanel, go to Metrics → Raw Access, and download the archive for your domain. The file is gzipped (.gz); upload it as is.
If the list is empty, tick "Archive logs in your home directory" on the same page. Older archives then collect in the logs folder of your home directory.
Plesk
In Websites & Domains, open your domain and choose Logs. Download the access log (on many servers it lives at /var/www/vhosts/system/your-domain/logs/access_ssl_log for HTTPS traffic).
Your own server (Apache)
Usually /var/log/apache2/access.log (Debian/Ubuntu) or /var/log/httpd/access_log (RHEL/CentOS). Rotated files such as access.log.2.gz work too. The log must use the standard combined format, which includes the user agent we need to identify bots.
Your own server (Nginx)
Usually /var/log/nginx/access.log. Nginx's default format is compatible. If you changed log_format, make sure it still records the request, status code and user agent.
WP Engine
In the WP Engine User Portal, open the environment, go to Logs, and download the access log.
Kinsta
Connect with SFTP (credentials are in MyKinsta under your site's Info page) and download access.log from the logs folder. The Logs viewer in MyKinsta only shows recent lines, so use SFTP for a full file.
SiteGround
Look in Site Tools under Statistics for the access log, or download it over SFTP. Menu names change between plans; if you can't find raw access logs, SiteGround support can tell you where they are for your plan.
Amazon CloudFront
Turn on standard logging for the distribution (the logs land in an S3 bucket), then download one of the log files. CloudFront's standard log format is supported directly.
Cloudflare
Cloudflare's own request logs (Logpush) are an Enterprise feature and come as JSON, which CrawlTide doesn't read yet. Upload your origin server's access log instead (cPanel, Apache, Nginx or your host, above). Bots are still identified by their user agent even when requests pass through Cloudflare.
Hosted site builders (Shopify, Wix, Squarespace, Webflow)
These platforms don't give customers raw server access logs, so Log Files can't be used for sites hosted on them. Everything else in CrawlTide (audits, keywords, AI visibility) works normally.
If the upload fails
- "No parseable request lines": the file isn't a raw access log in a supported format. Check you downloaded the access log, not the error log or a stats report.
- "File exceeds the limit": upload a shorter date range, or a single rotated file instead of the whole archive.
- Uncrawled pages look wrong: run a fresh audit first. Log Files compares bot visits with the pages found in your latest completed crawl.
Ready? Open your project and go to Log Files to upload. See also AI Visibility explained.