Robots & Sitemap

The Robots & Sitemap tool scans your site's robots.txt and XML sitemap files in seconds and shows exactly which pages search engine bots can reach, which are blocked, and whether your sitemap is valid. You enter your URL; the tool flags crawl blocks, conflicting rules, and missing sitemap declarations, then hands you actionable fixes.

These two files are the gatekeepers of technical SEO: robots.txt defines where bots may go on your site, while the XML sitemap lists the URLs you want indexed. Per Google's official documentation, a single sitemap file may contain at most 50,000 URLs or 50 MB uncompressed (Google Search Central, 2025), and any robots.txt content beyond 500 KiB is ignored (Google robots.txt Spec, 2025).

The tool fetches your /robots.txt and XML sitemap files and answers three core questions: Can bots reach your important pages? Is your sitemap valid and declared in robots.txt? Do the two files contradict each other? You get the result instantly, without leaving your browser.

Concretely, the tool checks:

• Accessibility: whether robots.txt returns a 200 status or is accidentally blocking the entire site with Disallow: /.
• Critical file blocks: whether CSS, JavaScript, or image directories are closed off — Google needs these to render pages correctly.
• Sitemap declaration: whether a Sitemap: line exists in robots.txt and the address it points to actually works.
• Sitemap validity: whether the XML is malformed, exceeds the 50,000-URL / 50 MB limit, or lists blocked URLs.
• Bot-specific rules: whether rules for Googlebot, Bingbot, and AI crawlers like GPTBot are written correctly.

The output is not a raw file dump; each finding comes with a "why it matters" and a "how to fix it."

Correctly configured robots.txt and sitemap files stop search engines from wasting crawl budget and speed up discovery of the right pages. Misconfiguration, on the other hand, causes silent disasters that can go as far as dropping your entire site from the index.

The most common mistake is treating robots.txt as an indexing control. Google states plainly: robots.txt is not a mechanism for keeping a page out of Google; a blocked URL that receives links from other sites can still appear in search results even though its content is never crawled (Google Search Central, 2025). To truly hide a page you need a noindex tag or password protection.

The second big trap is combining noindex with Disallow. When a page is blocked in robots.txt, the bot never opens it and therefore never sees the noindex tag — so the page stays in search results. The tool catches this conflict automatically. You can further verify indexing status page by page with our Indexability Check tool.

In short: robots.txt tells bots what not to crawl, while the sitemap tells them what you want crawled and indexed. They complement each other but must never conflict — if a URL is blocked in robots.txt, it should not appear in the sitemap.

Sources: Google robots.txt Spec, 2025 and Google Sitemap Docs, 2025. Both files are signals only; without content quality and technical soundness, neither guarantees indexing on its own.

Enter your domain and run the scan; the tool produces a robots.txt and sitemap report in seconds. You then apply each finding in priority order.

1. Enter your root domain (e.g. https://yoursite.com). The tool automatically fetches /robots.txt and the declared sitemaps.
2. Fix red-flagged blocks first: an accidental Disallow: / or a CSS/JS block is the highest priority — in production it causes severe traffic loss.
3. Add the sitemap line: append Sitemap: https://yoursite.com/sitemap.xml to the bottom of robots.txt. Google allows multiple sitemap lines with no limit.
4. Clean up conflicting URLs: for any URL listed in the sitemap but blocked in robots.txt, resolve it in one of the two files.
5. Submit to Search Console and monitor the discovered-vs-indexed URL counts.

Once you've applied the output, run the other technical checks for your whole site: validate duplicate-page signals with Canonical Check, multilingual tags with Hreflang Check, and structured data with Schema Generator. Together they build a solid technical SEO foundation.

The most destructive mistake is a Disallow: / line carried over from a staging environment into production — this single line closes your entire site to crawling. The tool flags the following errors:

• Blocking CSS/JS: closing off directories like /wp-content/ or /assets/ entirely prevents Google from rendering the page and breaks mobile-friendliness signals.
• Blocking the sitemap: if a rule in robots.txt also covers the sitemap URL, Googlebot can never reach it.
• Case and path errors: robots.txt paths are case-sensitive; /Folder/ and /folder/ are interpreted differently.
• Using robots.txt for indexing control: for a page you want hidden, use a noindex tag verified via Indexability Check, not a block.
• Ignoring AI bots: without separate rules for crawlers like GPTBot and Google-Extended, you have no control over how your content is used in AI training.

If you also want to review meta-level signals, the Meta Description Generator tool complements your SERP visibility.

No. A sitemap only speeds up discovery; it does not guarantee indexing. Google states clearly that a URL being in your sitemap does not mean it will be indexed — crawl budget, content quality, and technical soundness are the deciding factors (Google Sitemaps Report, 2025).

That is why adding thin, duplicate, or low-value pages to your sitemap backfires: it wastes crawl budget and lowers the perceived quality of your site. An ideal sitemap contains only the canonical, valuable URLs you actually want indexed. If the "discovered" count in Search Console is far higher than the "indexed" count, you need to investigate the errors in the index coverage report.

Sitemap health is just one piece of a holistic technical SEO strategy. If you want an expert audit of your robots.txt and sitemap configuration, or an enterprise-grade technical SEO foundation, get in touch with Sora Yazılım. Our team delivers end-to-end solutions for crawling, indexing, and GEO/AEO visibility.

In 2026, robots.txt decides more than your Google rankings: it determines whether ChatGPT, Perplexity, and Google AI Overviews can cite your site as a source. Blocking an AI crawler removes you from that platform's answers, so every rule is now a visibility decision, not just a crawl-budget decision.

The key distinction is between crawlers that collect training data and crawlers that fetch pages to cite them in live answers:

For a business that wants to be quoted in AI answers, the sensible default is to keep answer and search bots open, and treat training bots as a separate policy decision. A reflexive block-everything rule can erase your brand from the AI answer ecosystem entirely. The checker shows which rule applies to each of these bots at a glance, so you know exactly which door is open to whom.

Two steps prevent most migration disasters: confirm that the staging Disallow: / never reaches the production robots.txt, and rebuild the sitemap so it lists only final URLs that return 200. Skipping either one is the classic cause of post-launch traffic drops.

A practical launch-day checklist:

1. Before go-live: verify that staging and production robots.txt files differ — test-environment blocking rules must not ship.
2. On launch day: regenerate the sitemap for the new URL structure. No 301-redirecting or 404 URLs should remain in it.
3. Update robots.txt: point the Sitemap: line at the new sitemap. If the domain changes, keep the old domain's robots.txt open so Google can crawl your 301 redirects.
4. Notify Search Console: submit the new sitemap and, for domain moves, use the Change of Address tool.
5. Monitor the first weeks: check crawl errors and indexed-page counts daily until they stabilize.

The sitemap is only one discovery path; a coherent internal link network is the other. After restructuring URLs, review page-to-page connections with the Internal Linking tool so bots can rediscover the new structure through links as well as the sitemap.

Three signals are enough: the submitted-vs-indexed ratio in Search Console's Sitemaps report, the exclusion reasons in the Page Indexing report, and the status-code breakdown in the Crawl Stats report. Together they tell you whether the two files are doing real work.

What success looks like in practice:

• Sitemaps report: the indexed count converges toward the submitted count; a widening gap means low-value or conflicting URLs are polluting the sitemap.
• Page Indexing report: "Blocked by robots.txt" and "Discovered – currently not indexed" both trend downward over time.
• Crawl Stats: the vast majority of Googlebot requests return 200, with 4xx/5xx responses and redirect chains staying low.
• Tool scan: re-run the checker on this page after every significant release and confirm the red-finding count stays at zero.

The step most teams skip is making this a monthly routine — a single deploy can silently overwrite robots.txt, so "fixed it once" is not a durable state. And once pages are reliably indexed, the job shifts to earning clicks: surface queries with weak click-through using the CTR Opportunities tool and strengthen the titles and descriptions behind them.