Free Technical SEO Tool
Sitemap Checker & Robots.txt Tester
Check your XML sitemap and robots.txt in one pass: crawlability and indexing risk, free
How it works
No black box. Here's exactly what Sitemap Checker & Robots.txt Tester checks.
- 1
You enter your domain
We use the root: protocol, host. Anything after is ignored.
- 2
We fetch /robots.txt
Verify it exists, returns 200, and check for accidental "Disallow: /" rules and missing Sitemap directives.
- 3
We fetch /sitemap.xml
Or whatever URL is declared in robots.txt. We confirm it parses, count URLs, and detect index sitemaps.
- 4
You get a verdict + fixes
Severity-coded issues. Each fix is concrete enough to do in 5 minutes.
Why this matters
A misconfigured robots.txt or missing sitemap is the most expensive small bug in SEO. It silently kills your visibility for weeks before anyone notices. The classic horror story: someone copies a staging robots.txt with "Disallow: /" into production. Site disappears from Google. Three weeks pass. By the time anyone catches it, the rebuild costs months.
- "Disallow: /" or "noindex" left over from a staging environment is the #1 silent SEO killer.
- A missing or empty sitemap forces crawlers to discover URLs through internal links only. Orphan pages never get indexed.
- Sitemap declarations in robots.txt are still the canonical way to point crawlers at your sitemap, even with Search Console submitted.
- Sitemap index files matter for sites over ~50,000 URLs. Google rejects single sitemaps over that limit.
Want the full story across every page?
The Sitemap Checker & Robots.txt Tester checks one URL. CrawlTide audits your whole site, tracks issues over time, watches your AI Visibility weekly, and publishes articles to WordPress, Webflow, and Shopify.
No credit card. Free tier covers a small site end-to-end.
Frequently asked questions
What does a sitemap checker check?
A sitemap checker confirms that your XML sitemap loads (a 200 response, not a 404 or an HTML error page), parses as valid XML, and lists the URLs you expect. This one also reads your robots.txt, confirms the sitemap is declared there, counts the URLs, recognizes sitemap index files, and flags rules such as "Disallow: /" that would stop crawlers from reaching your pages.
Is this a crawlability checker?
Yes, for the site-wide part of crawlability. Robots.txt decides which paths crawlers may fetch, and the sitemap tells them which URLs exist, so a bad rule or a missing sitemap blocks crawling for the whole site. For a single page, the indexability checker adds the page-level checks: status code, redirects, noindex tags, X-Robots-Tag headers and canonical.
How do I check my website's sitemap?
Open yourdomain.com/sitemap.xml in a browser. You want a 200 response and valid XML, not a 404 or an HTML error page dressed up as one. Check that the URLs inside use your canonical domain (https, and www or non-www consistently), that they are live pages rather than redirects or dead ones, and that the file is listed in robots.txt as a Sitemap directive. Paste your domain above and this tool does all of that in one pass.
How do I verify my sitemap is working?
Working means three separate things, and most checks only cover the first. One, it resolves: 200 status and parseable XML. Two, it is discoverable: declared in robots.txt and submitted in Search Console. Three, Google has actually read it. That last one is the one people miss, and it is the one that bites.
How do I verify Google has actually read my sitemap?
Search Console, Indexing, Sitemaps. The "Last read" column is the date Google last fetched it, and "Discovered pages" is how many URLs it found. Compare that count against the number of URLs actually in your file. If Last read is weeks or months old, or the discovered count is lower than your real count, Google is working from a stale copy and your newer pages have not reached it through the sitemap at all. Resubmitting the same URL queues a fresh fetch, though the date can take a day to update.
Do I really need a robots.txt?
You don't. Without one, crawlers default to "allow everything." But you almost always want one for two reasons: declaring your sitemap location, and blocking specific paths (admin areas, search-result pages, faceted-navigation URLs that produce infinite duplicates).
Where should robots.txt and sitemap.xml live?
Both must be at the domain root: yourdomain.com/robots.txt and yourdomain.com/sitemap.xml. Google looks there first. You can override the sitemap location by declaring it in robots.txt, but the convention is the root.
How big can my sitemap be?
Single sitemap: max 50,000 URLs and 50 MB uncompressed. Larger sites use a sitemap index file that points to multiple sitemap files (each under the limits). Most CMSes do this automatically.
Will this catch noindex tags too?
Not in this tool. Those are per-page meta tags. The AI Visibility and SEO Score tools both flag noindex on the page they audit. CrawlTide's full audit flags noindex across your whole site after a crawl.
My sitemap is dynamically generated. Will this still work?
Yes, as long as it returns 200 with valid XML. We don't care whether it's a static file or generated on demand by your framework (Next.js sitemap.ts, Rails routes, etc).
Should I block AI crawlers like GPTBot?
GPTBot is OpenAI's training crawler, so blocking it is a training opt-out. It does not remove you from ChatGPT search, which uses OAI-SearchBot. The same split exists elsewhere: ClaudeBot trains while Claude-SearchBot powers search, and Google-Extended is a training control that does not affect Google Search. If you want to appear in AI search results, keep the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) allowed. The full CrawlTide product checks each of these separately and never penalizes a training opt-out.