Sitemaps, robots.txt and Search Console for a new site

A new site's relationship with Google runs through a sitemap, a robots.txt file, an indexing instruction and one free tool. They are often confused because they all concern "what Google sees", but each does a different job, and robots.txt in particular does not do the job most people assume.

What each piece does

Sitemaps

A sitemap is an XML file listing the canonical URLs you want crawled. A minimal one:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-10-17</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/journal/sample-post/</loc>
    <lastmod>2026-10-17</lastmod>
  </url>
</urlset>

Rules from Google's documentation worth remembering:

• A single sitemap is limited to 50 MB uncompressed or 50,000 URLs, in every format Google supports. Larger sets are split into several sitemaps, optionally tied together by a sitemap index.
• The file must be UTF-8 encoded, and URLs must be fully qualified and absolute, because Google tries to crawl them exactly as listed.
• Google ignores <priority> and <changefreq>. It uses <lastmod> if it is consistently and verifiably accurate. Google's 2023 sitemap announcement says "last modification" means a significant change: the main text, structured data or links, not a footer tweak. Do not stamp every URL with today's date on each build, and leaving <lastmod> out for a page you cannot date is acceptable.
• List only URLs you want in results: the preferred canonical form, returning 200. Redirected, noindexed or duplicate URLs create contradictions with your canonical signals.
• Host it at the site root. Unless you submit it through Search Console, a sitemap affects only the directory it sits in and the paths below it.

A sitemap is a hint, not a promise. Google's page on submitting says that submitting one is merely a hint and does not guarantee Google will download it or use it for crawling.

How big is too big (fictional arithmetic)

A fictional shop has 120,000 product URLs. The 50,000-URL limit means at least 120,000 / 50,000 = 2.4, rounded up to 3 sitemap files, plus an index file that lists them. The size limit is rarely the first to bite: the <url> entry in the sample above is 111 bytes with its whitespace, so 50,000 of them come to about 5.5 million bytes, far below 50 MB. For a plain list of URLs the count binds first.

robots.txt

robots.txt lives at the root (https://www.example.com/robots.txt) and follows the Robots Exclusion Protocol, standardised as RFC 9309. A reasonable file for a small site is nearly empty:

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

The empty Disallow allows everything. Add rules only for URLs that waste crawler effort, such as internal search results or a cart.

The single most important fact, which Google states at the top of its introduction to robots.txt: the file is used mainly to avoid overloading your site with requests, and it is not a mechanism for keeping a web page out of Google. A disallowed URL can still be indexed without its content, and Google says it may show the URL with no snippet. And if you block a page, Google cannot read any noindex rule on it.

Check for a leftover Disallow: / from staging, which blocks the whole site.

How the file is read

• Rules match by path prefix, and the value is case-sensitive: Disallow: /Private does not cover /private. Under RFC 9309 the most specific match, meaning the one with the most octets, is used, and if an allow and a disallow rule are equivalent the allow rule should be used.
• One file per host, protocol and port. https://www.example.com/robots.txt governs that origin only. A file in a subdirectory is not valid.
• The Sitemap: line is understood by Google, Bing and other major engines, may appear anywhere in the file, must be a full URL, and is not tied to any user agent.
• Fields Google does not support include crawl-delay, and noindex written inside robots.txt is not supported either.
• Size. RFC 9309 says a crawler's parsing limit must be at least 500 kibibytes (512,000 bytes). Google separately states that it enforces a limit of 500 KiB and ignores content after it. The RFC sets a floor for any crawler; Google describes its own behaviour.

When the file cannot be fetched

What a crawler does when robots.txt is missing or broken is a place where the standard and Google's behaviour differ, so keep them apart.

RFC 9309. If the server answers with a 4xx status, the file is "unavailable" (section 2.3.1.3) and the crawler may access any resources. If the file is unreachable because of server or network errors, such as 5xx statuses (section 2.3.1.4), the crawler must assume complete disallow. If the file stays undefined for a reasonably long time, for example 30 days, the crawler may treat it as unavailable or keep using a cached copy. A crawler should follow at least five consecutive redirects (section 2.3.1.2).

Google. Its own documentation describes its behaviour, with different details:

Under both, a robots.txt that starts returning 500 can halt crawling and a 404 means "everything allowed", so check its status code after every release.

noindex versus disallow

A noindex instruction is written as <meta name="robots" content="noindex"> in the page head, or as an X-Robots-Tag: noindex HTTP header for non-HTML files. For Google to obey it the page must be reachable: Google says that if the page is blocked by robots.txt or the crawler cannot access it, the crawler never sees the rule and the page can still appear in results. Google also says it must crawl the page to see the tag, so a page already in results can stay there until it is recrawled, which may take months for a page of little importance; URL Inspection lets you request a recrawl.

Search Console: add and verify the site

Search Console is Google's free tool for seeing how it crawls and indexes your site. Start with ownership.

1. Choose a property type. A Domain property covers the domain, every subdomain and both HTTP and HTTPS in one view. A URL-prefix property covers only addresses that begin with the exact prefix you enter.
2. Verify. Google's verification page lists these methods: HTML file upload, HTML tag, a Google Analytics tracking code, Google Tag Manager, a Google Sites or Blogger account, and a domain name provider record. The last is the only way to verify a Domain property, using a TXT record or, in some configurations, a CNAME record. The file and tag methods work for URL-prefix properties only.
3. Keep the proof in place. Verification lasts as long as Search Console can confirm the token, and it re-checks periodically. If the token disappears you are notified and, if it is not restored, your permissions expire after a grace period. The file method does not follow redirects to another domain, and the tag must be in the head of the home page.

The DNS method fits well with the roles described in how DNS works for someone who just bought a domain: the record goes in the zone at the DNS host, not at whichever service serves the pages.

Day-one sequence

1. Confirm the site responds on exactly one preferred address over HTTPS, and the others redirect to it.
2. Check that robots.txt exists, returns 200, allows the pages you want indexed, and has no stray Disallow: /.
3. Confirm no page you want indexed carries a noindex, which is a common leftover from staging.
4. Publish sitemap.xml with preferred URLs only, and add its address to robots.txt.
5. Add and verify the property in Search Console.
6. Submit the sitemap in the Sitemaps report. That report lists only sitemaps submitted through it or the API, not ones Google found through robots.txt.
7. Use the URL Inspection tool on the home page and one inner page. It shows what Google last saw, the chosen canonical, and whether the page can be indexed. If it is not indexed yet, you can request indexing; this is a request, not a guarantee.
8. Record a baseline and then wait. Do not resubmit every day.

What about ping and IndexNow?

Google deprecated its sitemap "ping" endpoint in 2023, and its blog post says requests to it return a 404. Google's supported routes are the Search Console Sitemaps report, the API and the Sitemap: line in robots.txt. IndexNow is a separate protocol for pushing changed URLs to participating search engines. The endpoint list in the IndexNow FAQ at the time of writing names Amazon, Bing, Naver, Seznam.cz, Yandex and Yep, and does not include Google, so treat it as useful for those engines only and check the list again before relying on it.

Reading the reports

• Sitemaps report. Each submitted sitemap shows the status of its last request: "Success", "Couldn't fetch" when Google could not retrieve the file, or "Sitemap had X errors" when it could read the file only in part.
• Page indexing report. Some Google pages still call this the Index coverage report. It splits URLs into indexed and not indexed, with a reason for each. Google's help says a page can be not indexed for good reasons, and that you should not expect every URL to be indexed, only the canonical pages. The reasons it describes include "Page with redirect" (a non-canonical URL that redirects, which is expected for your non-preferred variants), "URL blocked by robots.txt" and "URL marked 'noindex'" (investigate if you wanted the page indexed), "Crawled - currently not indexed" (crawled but not indexed, and Google says there is no need to resubmit it) and "Discovered - currently not indexed" (found but not yet crawled, typically because Google rescheduled the crawl to avoid overloading the site).
• URL Inspection. The per-page answer. It shows the indexed version, canonical selection and a live test of the current page.
• Performance report. Impressions, clicks and queries once the site appears. Expect it nearly empty at first.

Google's help says a site with fewer than 500 pages probably does not need the Page indexing report, and that a site:example.com search is a quicker first check. A page indexed or not after a few days says little; compare over weeks.

Fifteen pages, nine waiting (fictional)

The fictional example.org launches with fifteen pages. The owner publishes a sitemap with fifteen URLs and a robots.txt that points to it. In Search Console the Sitemaps report says "Success" with fifteen discovered. The Page indexing report shows four indexed, nine "Discovered - currently not indexed" and two "Page with redirect" (the HTTP and non-www variants). The redirect entries are expected. For the nine, the owner checks that each is linked from the home page, has a distinct title and real text, and returns 200, then leaves it alone for two weeks rather than resubmitting. The point of the example is the response: verify the causes you control, and then wait.

Because the content of a new site matters for indexing, pages that exist as complete HTML on the first request are the easiest for crawlers to process, a point covered in static HTML first.

Checklist

• Make a sitemap of preferred, indexable, 200-status URLs only.
• Treat robots.txt as crawl control, never as a way to hide a page.
• Use noindex on crawlable pages you want out of results.
• Verify a Domain property by DNS, then submit the sitemap and inspect two pages.
• Give indexing weeks, not hours, and do not resubmit repeatedly.