robots.txt and Sitemap.xml Guide: Correct Setup and Common Mistakes

Q: Should I block AI bots?

That's your call. User agents like `GPTBot`, `ClaudeBot`, and `PerplexityBot` can be blocked via robots.txt. But if you want your content cited in AI-generated answers, blocking them costs you that visibility — we discuss this trade-off in our GEO guide.

Sometimes all it takes to erase a site from search entirely is a single line:

User-agent: *
Disallow: /

These two lines are correct in a development environment; pushed to production by mistake, they wipe the site from Google. It's one of the most common — and most expensive — technical mistakes made right after launch.

robots.txt and sitemap.xml are the two fundamental communication channels you have with search engines. This guide covers what each one does, how to write them correctly, and where things commonly go wrong.

What Is robots.txt and What Does It Do?

It's a plain text file that sits at your site's root (yoursite.com/robots.txt). It tells search engine bots which addresses not to crawl.

The critical distinction: robots.txt blocks crawling, not indexing.

These two get mixed up constantly. If you block a page with robots.txt, Google can't read it — but if other sites link to it, Google can still index it without ever seeing its content. The result is a strange, description-less listing in search results.

Goal	Right tool	Why
Keep a page out of search results	noindex meta tag	Google reads the page and learns not to index it
Stop bots from straining your server	robots.txt disallow	Crawling never happens at all
Neither crawled nor shown	noindex first, disallow once it's dropped from the index	A noindex tag on a blocked page can't be read

The last row matters most: a noindex tag on a page blocked by robots.txt can't be seen by Google. If you want a page out of the index, apply noindex first, confirm it's dropped, then block it.

How Do You Write a Correct robots.txt?

For most sites, the file can be this simple:

User-agent: *
Disallow: /wp-admin/
Disallow: /cart
Disallow: /checkout
Disallow: /search
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap.xml

Rules:

User-agent specifies which bot you're addressing; * means all of them.
Disallow specifies the path not to crawl. Leaving it empty (Disallow:) means everything is allowed.
Allow opens an exception within a Disallow.
Sitemap gives the full address of your sitemap; you can list more than one.
The file must be UTF-8 and sit at the root. A robots.txt in a subfolder is invalid.

Worth blocking: admin panels, cart and checkout steps, on-site search result pages, endless filter combinations, and areas containing personal data.

Never block: CSS and JavaScript files. Google needs them to render the page the way a user sees it; blocking them makes the page look broken and hurts your mobile-usability evaluation.

What Is Sitemap.xml?

A sitemap is a list of the important addresses on your site. It's the most direct way to tell Google "here's what I want you to crawl."

Basic structure:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yoursite.com/</loc>
    <lastmod>2026-08-13</lastmod>
  </url>
  <url>
    <loc>https://yoursite.com/services</loc>
    <lastmod>2026-07-02</lastmod>
  </url>
</urlset>

Limits: a single file can hold up to 50,000 URLs and 50 MB (uncompressed). Larger sites use a sitemap index:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://yoursite.com/sitemap-products.xml</loc></sitemap>
  <sitemap><loc>https://yoursite.com/sitemap-blog.xml</loc></sitemap>
</sitemapindex>

The hidden benefit of splitting it up: Search Console reports on each sitemap separately, so you can see directly which section isn't getting indexed. If 90% of your products are indexed but only 30% of your blog is, you spot the problem at a glance.

Six Sitemap Mistakes

1. Listing URLs that shouldn't be indexed. Addresses that are noindex, canonical elsewhere, or return a 301 shouldn't be in the sitemap. A sitemap says "this matters"; a canonical says "this isn't the original." When the two conflict, Google may end up ignoring your canonical tag.

2. Updating lastmod every single day. Refreshing the date when content hasn't actually changed erodes trust; eventually Google stops paying attention to the field. Only update it when content genuinely changes.

3. Trying to make priority and changefreq mean something. Google largely ignores both fields. Not worth the time.

4. Mixing protocols or subdomains. URLs inside a sitemap should match the domain the sitemap itself is served from, and match the canonical version.

5. Leaving 404ing URLs in place. Deleted pages need to come out of the sitemap too; otherwise you're wasting crawl budget.

6. Never submitting the sitemap at all. It's common to build the file and forget to add it in Search Console. Adding a Sitemap: line to robots.txt is also worth doing.

The Relationship With Crawl Budget

For small sites, crawl budget isn't an issue. But on sites generating tens of thousands of URLs, a cycle sets in: Google crawls a limited number of pages; if those pages are clogged with filter combinations, your actual product pages never get their turn.

In that case, robots.txt becomes real leverage:

Block endless filter combinations (Disallow: /*?color=)
Block on-site search results
Block calendar/date-based infinite pages

But be careful: internal links pointing at pages you block now lead into a blind spot. Review the internal link structure pointing to those pages before you block them.

Verifying After Launch

Open yoursite.com/robots.txt in a browser. It should return a 200 and be readable.
Confirm in Search Console → Settings → robots.txt report that the file has been fetched with no errors.
Test a few key pages with the URL Inspection tool — the "Crawl allowed?" line should say Yes.
Submit the sitemap to Search Console and, a few days later, compare "Discovered URLs" against "Indexed."
If the gap is large, check the exclusion reasons in the Pages report.

All of these steps are covered screen by screen in our Search Console guide.

Pre-Launch Checklist

The two things most often forgotten when a new site goes live:

Has the Disallow: / line from the development environment been removed?
Has the theme/template's "block search engines" setting been turned off?

Both are one-click fixes but can go unnoticed for weeks. Opening robots.txt should be the first thing you check on launch day. For a broader check, see our technical SEO checklist.

Frequently Asked Questions

Does a site without robots.txt get penalized?

No. If the file is missing, Google treats everything as crawlable. It's still worth creating one, since it's a convenient place to point to your sitemap.

Should I block AI bots?

That's your call. User agents like GPTBot, ClaudeBot, and PerplexityBot can be blocked via robots.txt. But if you want your content cited in AI-generated answers, blocking them costs you that visibility — we discuss this trade-off in our GEO guide.

How often should I update my sitemap?

Generate it automatically. If your CMS doesn't update the sitemap when a new page is added, manual upkeep won't hold up.

A page I blocked is still showing up in Google — why?

robots.txt blocks crawling, not indexing. Getting a page out of the index requires noindex — and for noindex to be read, the page needs to be crawlable in the first place.

Related Articles: