The Broken-Sitemap Tax: XML Sitemaps for AI Discovery

Nobody puts a line item for this in the budget. It surfaces as something else. Publishing volume climbs and organic traffic flatlines. A competitor gets named in ChatGPT answers about your category and you do not. Your log export shows tens of thousands of AI-crawler requests a month, which looks like proof the machines are paying attention, right up until you check how many of those requests ended in a document that could be cited.

That gap is the Broken-Sitemap Tax: the compounding cost of declaring a set of URLs a generative engine cannot cheaply resolve into text. The URLs redirect. They carry timestamps nobody believes. They disagree with their own canonical tags. Each one is a fetch that costs the crawler compute and returns nothing usable, and every system with a budget responds the same way, it comes back less often, on fewer URLs, with less patience.

This is not a rerun of the classical crawl-budget argument, which barely matters for most sites under a few thousand URLs. Two things changed. First, hop tolerance collapsed: real-time AI indexers enforce a ceiling of one to three redirect hops, against the ten Googlebot has tolerated and the thirty the HTTP spec permits. Second, the sitemap became a change feed: real-time indexers read its lastmod values to decide what to recrawl today, which makes timestamp integrity a ranking input rather than an administrative detail.

01How did the sitemap become a schedule instead of a manifest?

Training crawlers like GPTBot, ClaudeBot and CCBot ingest bulk corpora asynchronously. They throttle themselves between 0.5 and 5.0 requests per second and tolerate multi-hop redirects, because nothing about their mission is time-bound. Real-time search indexers like OAI-SearchBot, Claude-SearchBot and PerplexityBot ground live answers under latency constraints closer to an API call than a crawl, and they ignore Crawl-delay entirely, because a delayed fetch is useless when a user is waiting. User-triggered fetchers like ChatGPT-User fire when a person pastes your URL into a chat and behave like a headless browser with a stopwatch running.

One practical consequence: when people say their AI visibility is fine because GPTBot hits their site constantly, they are usually reading training-crawler logs. Training ingestion moves at the pace of model releases; citation in a live answer moves at the pace of the real-time index. Only one of them is affected by what you shipped this week. The full crawler roster and robots.txt directives are in how AI crawlers index your site.

02What is the three-hop ceiling?

Three hops sounds generous until you count what a legacy URL actually does. Scheme normalisation is one hop, host normalisation another, trailing-slash enforcement a third. A user arriving from an old PDF, an email footer, or a partner directory can burn the whole budget before a byte of content is served.

The failure mode is what makes it expensive. A 404 is loud; a 500 pages someone; a crawler quietly hitting its hop ceiling produces a 301 in your access log, then silence. Your monitoring sees a successful response. The engine sees a resource it could not resolve inside its budget and moves to the next candidate, which belongs to someone else.

It helps to have a number. Define discovery yield as the share of declared URLs a real-time indexer can actually resolve into a citable document in one pass: three binary conditions, all cheap to measure.

discovery_yield =  (1 / N) · Σ  [ s_i = 200 ]·[ h_i <= H ]·[ c_i = u_i ]

  N   URLs declared in the sitemap        H   agent hop ceiling (1-3)
  s   terminal status code                c   the page's declared canonical
  h   redirect hop count                  u   the URL as declared
  A URL counts only if it returns 200, resolves inside the hop budget,
  and points at itself as canonical. Most sites score 0.60 to 0.85.

#!/usr/bin/env bash
# Reports redirect hops and terminal status for every URL in a sitemap.
# Anything with hops >= 3 is at or past the real-time indexer ceiling.
SITEMAP="https://example.com/sitemap.xml"
UA="Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)"

curl -s "$SITEMAP" \
  | grep -oP '(?<=<loc>)[^<]+' \
  | while read -r url; do
      chain=$(curl -sIL -A "$UA" -o /dev/null \
        -w '%{num_redirects} %{http_code} %{url_effective}' "$url")
      hops=$(echo "$chain"  | awk '{print $1}')
      code=$(echo "$chain"  | awk '{print $2}')
      final=$(echo "$chain" | awk '{print $3}')
      if [ "$hops" -ge 1 ] || [ "$code" != "200" ]; then
        printf 'HOPS=%s STATUS=%s  %s  ->  %s\n' "$hops" "$code" "$url" "$final"
      fi
    done

Run that against your own sitemap before reading further. The output is usually the most persuasive artefact in the entire audit, because it is your URLs and your numbers rather than an industry average.

03Why are status codes cache instructions, not cosmetic detail?

A 301 caches the new target, GPTBot immediately, ClaudeBot for roughly thirty days, PerplexityBot on the next cycle. A 308 does the same while guaranteeing the request method and body survive, which makes it correct for API endpoints, JSON feeds and IndexNow payloads where a silent rewrite to GET would destroy the request. A 302 or 307 keeps the original URL in the index and re-requests it every cycle. Multiply that by a few thousand URLs and a daily cadence and you have an egress line item nobody can explain.

The pattern worth hunting is the 301 followed by a 302, which happens when a permanent migration rule and a temporary campaign or maintenance rule live in different layers and neither team knows about the other. Two more anti-patterns deserve a grep: user-agent-dependent redirects that serve bots a different path are cloaking and are detected, and geolocation redirects that bounce /en/ to /us/ create loops for crawlers that egress from a single region, which describes most AI fleets.

04Why must root discovery files get zero hops, not three?

This breaks in a specific, common way. A team reorganises static assets, moves the files into a build-output directory, and adds a catch-all rewrite that happens to emit a 301. Everything still works in a browser. Every human test passes. The discovery layer goes dark and nothing reports it.

server {
  listen 443 ssl http2;
  server_name example.com;

  # Root discovery assets. 200 OK, zero client-visible hops.
  location = /robots.txt {
    default_type text/plain;
    try_files /static_root/robots.txt =404;
    expires 1h;
    add_header Cache-Control "public, no-transform";
  }
  location = /sitemap.xml {
    default_type application/xml;
    try_files /static_root/sitemap.xml =404;
    expires 15m;
  }
  location = /llms.txt {
    default_type text/plain;
    try_files /static_root/llms.txt =404;
    expires 1h;
    add_header Access-Control-Allow-Origin "*";
  }
  location = /llms-full.txt {
    default_type text/plain;
    try_files /static_root/llms-full.txt =404;
    expires 1h;
  }

  # Everything else: collapse scheme, host, and slash in a single hop.
  location / {
    if ($http_x_forwarded_proto = "http") {
      return 301 https://example.com$request_uri;
    }
    proxy_pass http://origin_upstream;
  }
}

On llms.txt itself, keep expectations calibrated. It is a clean specification with real adoption among developer tools, but large-scale log studies found 97% of published llms.txt files received zero automated crawler requests. Publish a valid one because it costs an hour and future-proofs the surface, but do not treat it as a discovery strategy, the XML sitemap is where the discovery work happens. The dedicated llms.txt piece goes deeper on the spec.

05Why is lastmod the signal you are probably lying with?

Here is the mechanism. Your build regenerates the sitemap on every deploy and stamps today's date on all 12,000 URLs, the default behaviour of most static-site generators and half the CMS plugins in circulation. The indexer takes you at your word and recrawls aggressively, compares the semantic hash of what it fetched against what it had, and finds nothing changed, on thousands of URLs. It will not do that twice: the recrawl priority of the whole file gets demoted, and the next time you ship something important you are in a slower queue.

timestamp_trust =  (URLs whose CONTENT changed) / (URLs whose LASTMOD changed)

  1.00  every timestamp change was a real content change
  0.04  you told the indexer 12,000 pages changed when 500 did
  < 0.7 = engineering ticket.  < 0.2 = better to publish no lastmod at all.

The fix is to stop letting the build decide what changed and let a content hash decide instead: hash the rendered body with the volatile parts stripped, compare against the stored hash, and write a timestamp only when the comparison fails.

import hashlib, json, re, datetime, pathlib

STATE = pathlib.Path(".sitemap-hashes.json")
VOLATILE = [
    re.compile(r'<nav\b.*?</nav>', re.S | re.I),
    re.compile(r'<footer\b.*?</footer>', re.S | re.I),
    re.compile(r'data-build-id="[^"]*"'),
    re.compile(r'\d{4}-\d{2}-\d{2}T[\d:+.-]+'),   # rendered timestamps
]

def semantic_hash(html: str) -> str:
    """Hash only the content a retrieval system would actually keep."""
    for pattern in VOLATILE:
        html = pattern.sub('', html)
    text = re.sub(r'<[^>]+>', ' ', html)
    text = ' '.join(text.split()).lower()
    return hashlib.sha256(text.encode('utf-8')).hexdigest()

def resolve_lastmod(pages: dict[str, str]):
    """pages maps url -> rendered html. Returns url -> ISO 8601 lastmod."""
    prior = json.loads(STATE.read_text()) if STATE.exists() else {}
    now = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0).isoformat()
    out, state, changed = {}, {}, []
    for url, html in pages.items():
        digest = semantic_hash(html)
        record = prior.get(url)
        if record and record['hash'] == digest:
            out[url] = record['lastmod']          # unchanged, keep the old date
            state[url] = record
        else:
            out[url] = now                        # genuinely new or edited
            state[url] = {'hash': digest, 'lastmod': now}
            changed.append(url)
    STATE.write_text(json.dumps(state, indent=2))
    print(f'{len(changed)} of {len(pages)} URLs changed this build')
    return out, changed

The changed list that falls out of this function is the same list you push to IndexNow later, which is not a coincidence. Two formatting rules while you are in there: use full W3C datetime with a UTC offset (2026-03-31T08:30:00+00:00, not a bare date), and never write a future lastmod, which happens more than you would expect when scheduled publishing and timezone handling collide.

06Why should you partition the sitemap file?

<!-- /sitemap.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-docs.xml</loc>
    <lastmod>2026-08-24T09:12:00+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-blog.xml</loc>
    <lastmod>2026-08-26T06:40:00+00:00</lastmod>
  </sitemap>
</sitemapindex>

<!-- /sitemap-docs.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/docs/architecture</loc>
    <lastmod>2026-08-24T09:12:00+00:00</lastmod>
  </url>
  <url>
    <loc>https://example.com/docs/rate-limits</loc>
    <lastmod>2026-07-02T11:48:00+00:00</lastmod>
  </url>
</urlset>
<!-- No changefreq, no priority: ignored for years, pure parse-time cost. -->

Note what is absent: no changefreq and no priority. Both have been ignored by Google for years and appear in no documented AI-indexer behaviour, so they are bytes that cost parse time and buy nothing. The lastmod on the index entries is genuinely useful, because it lets an agent skip an entire child file it has already seen. Escape your ampersands, keep the file UTF-8, serve it over HTTPS, and declare the namespace, unescaped entities and missing namespaces are the two failures that break strict XML parsers outright.

07What does sitemap hygiene actually require?

• Zero non-200 URLs. Anything returning 301, 302, 308, 404 or 5xx should be purged automatically, not quarterly by hand.
• Canonical alignment. The URL in loc must match the page's own canonical tag exactly, including trailing slash and protocol. Mismatches are a documented cause of eviction from citation candidate pools.
• Directive alignment. A URL carrying noindex or blocked in robots.txt must never appear, declaring a page you told crawlers not to read is a contradiction resolved against you.
• Parameter discipline. Session IDs, tracking parameters and faceted-navigation combinations produce near-duplicate URLs that consume budget and dilute the canonical signal.

import asyncio, re, sys
import httpx
from xml.etree import ElementTree

NS = {'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
UA = 'Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)'
HOP_CEILING = 3
CANONICAL = re.compile(r'<link[^>]+rel=["\']canonical["\'][^>]+href=["\']([^"\']+)', re.I)

async def check(client, url):
    try:
        r = await client.get(url, follow_redirects=True, timeout=10.0)
    except httpx.RequestError as exc:
        return {'url': url, 'fail': f'network: {exc.__class__.__name__}'}
    hops = len(r.history)
    if r.status_code != 200:
        return {'url': url, 'fail': f'status {r.status_code}'}
    if hops >= HOP_CEILING:
        return {'url': url, 'fail': f'{hops} hops, at or past the ceiling'}
    if 'noindex' in r.headers.get('x-robots-tag', '').lower():
        return {'url': url, 'fail': 'x-robots-tag noindex'}
    match = CANONICAL.search(r.text[:200_000])
    if match and match.group(1).rstrip('/') != url.rstrip('/'):
        return {'url': url, 'fail': f'canonical drift -> {match.group(1)}'}
    return None

async def main(sitemap_url):
    async with httpx.AsyncClient(headers={'User-Agent': UA}) as client:
        xml = (await client.get(sitemap_url)).text
        urls = [e.text for e in ElementTree.fromstring(xml).iterfind('.//sm:loc', NS)]
        sem = asyncio.Semaphore(12)
        async def guarded(u):
            async with sem:
                return await check(client, u)
        results = await asyncio.gather(*(guarded(u) for u in urls))
    failures = [r for r in results if r]
    yield_rate = 1 - len(failures) / max(len(urls), 1)
    for f in failures[:40]:
        print(f"FAIL  {f['fail']:<38} {f['url']}")
    print(f'\ndiscovery yield: {yield_rate:.3f} over {len(urls)} URLs')
    sys.exit(1 if yield_rate < 0.98 else 0)

asyncio.run(main('https://example.com/sitemap-docs.xml'))

Wire that into CI as a blocking step on the sitemap build. A threshold of 0.98 is aggressive on purpose, the point of a gate is that it fails before the file ships, not that it produces a dashboard nobody opens. The canonical-and-structure side of this is covered in the anatomy of a high-citation page.

08Why is freshness the highest-leverage variable you control?

Google AI Overviews is more forgiving, 65% of citations from content updated within a year, 89% within three years. That difference is a strategy input: if ChatGPT Search is where your category gets decided, your update cadence is not a hygiene task, it is the campaign. Fitted to a simple exponential, the observed decay implies a citation half-life near two months for competitive commercial queries, the same shape as the 30-day content half-life.

C(t) =  e^(-λ·t)          # relative citation likelihood

  t  days since the last GENUINE content change
  λ  fitted to the fresh/stale ratio; implies a citation half-life
     near ~60 days for competitive commercial queries.

Two things follow. A page you update every ninety days spends most of its life in the shallow part of the curve, so a quarterly refresh is closer to maintenance than advantage. And the update has to be real, the decay is driven by the indexer detecting genuine change, the same hash comparison from the lastmod section. Changing a date in a byline is not a content update, and the system that decides whether you get cited is the one that already caught you doing it.

09How does IndexNow push beat pull?

The participant list includes Bing, Yandex, Naver, Seznam and Yep, which matters more than it used to because a meaningful share of live ChatGPT Search grounding resolves through the Bing index rather than a direct fetch, the same downstream dependency described in do AI crawlers render JavaScript.

Ownership verification and payloads

Before you submit, you prove you control the domain: a key of 8 to 128 characters from a-z, A-Z, 0-9 and hyphen (a 32-character UUID with dashes stripped is the default), written into a UTF-8 file at the root named exactly {key}.txt, containing the key and nothing else. If you cannot write to the root, keyLocation lets you host it elsewhere, but verification scope is bounded by the directory the key sits in, a key at /catalog/key.txt authorises submissions under /catalog/ and nothing else. Single URLs go over GET percent-encoded; bulk submissions go over POST as JSON, up to 10,000 URLs, and the host field takes a bare FQDN with no protocol, path, slash or port.

# Single URL, HTTP GET
GET /indexnow?url=https%3A%2F%2Fexample.com%2Fdocs&key=fa8c0a469da44e9b8f6a769f291829f5 HTTP/1.1
Host: api.indexnow.org

# Bulk submission, HTTP POST (up to 10,000 URLs)
POST /indexnow HTTP/1.1
Host: api.indexnow.org
Content-Type: application/json; charset=utf-8

{
  "host": "example.com",
  "key": "fa8c0a469da44e9b8f6a769f291829f5",
  "keyLocation": "https://example.com/fa8c0a469da44e9b8f6a769f291829f5.txt",
  "urlList": [
    "https://example.com/docs/rate-limits",
    "https://example.com/docs/architecture",
    "https://example.com/blog/sitemap-audit"
  ]
}

Submitting to one endpoint reaches all of them. The receiving engine verifies your key, then broadcasts the payload to every other participant within about ten seconds, and that fanout is cryptographically signed rather than trusted on faith, each forwarded payload carries an RSA-signed SHA-256 digest of the body that the receiver verifies against the sender's published key set. That is what stops a third party from spoofing submissions on your behalf.

import json, time
import requests
from typing import Iterable

class IndexNowPipeline:
    ENDPOINT = 'https://api.indexnow.org/indexnow'
    BATCH = 10_000

    def __init__(self, host: str, api_key: str, key_location: str | None = None):
        # host must be a bare FQDN. Strip anything the caller got wrong.
        self.host = host.replace('https://', '').replace('http://', '').strip('/')
        self.api_key = api_key
        self.key_location = key_location or f'https://{self.host}/{api_key}.txt'

    def preflight(self) -> bool:
        """Verify the key file is reachable and exact before submitting."""
        r = requests.get(self.key_location, timeout=10)
        if r.status_code != 200:
            print(f'key file unreachable: {r.status_code} at {self.key_location}')
            return False
        if r.text.strip() != self.api_key:
            print('key file contents do not match the key')
            return False
        return True

    def submit(self, urls: Iterable[str]) -> bool:
        urls = list(dict.fromkeys(urls))          # dedupe, preserve order
        if not urls:
            return True
        if not self.preflight():
            return False
        for i in range(0, len(urls), self.BATCH):
            batch = urls[i:i + self.BATCH]
            payload = {'host': self.host, 'key': self.api_key,
                       'keyLocation': self.key_location, 'urlList': batch}
            r = requests.post(self.ENDPOINT, data=json.dumps(payload),
                              headers={'Content-Type': 'application/json; charset=utf-8'},
                              timeout=15)
            if r.status_code == 200:
                print(f'pushed {len(batch)} URLs')
            elif r.status_code == 429:
                time.sleep(30)                    # back off, then retry
                return self.submit(urls[i:])
            else:
                print(f'submission failed: {r.status_code} {r.text[:200]}')
                return False
        return True

if __name__ == '__main__':
    # changed_urls comes from the hash comparison in the lastmod code above.
    pipeline = IndexNowPipeline('example.com', 'fa8c0a469da44e9b8f6a769f291829f5')
    pipeline.submit(changed_urls)

Call it from the deploy hook, after the sitemap is written and the validator has passed. Push only what changed, pushing your whole inventory on every deploy is the IndexNow equivalent of build-stamping your timestamps, and it earns you a 429 followed by a reputation you will not enjoy.

10What can a clean sitemap not do for you?

The Princeton, Georgia Tech, Allen Institute and IIT Delhi GEO research measured how specific content modifications change citation performance, with lifts up to 40% over unoptimised baselines. The individual results are more useful than the headline.

The rank-5 finding is the interesting one for anyone without an established domain: applying source citation to a page at position five in classical results lifted its generative visibility by 115%, which suggests evidence density can substitute for authority in a way that was never true in link-based ranking. One more distinction worth carrying into reporting, a 2026 benchmark of 21,181 search-layer interactions separates citation selection (whether an engine includes your link) from citation absorption (how much your text shapes the answer).

The practical read: optimising for ChatGPT Search rewards evidence density on a small number of pages, while optimising for Perplexity rewards breadth and concise semantic matching across many. Your sitemap strategy is the same in both cases; your content strategy is not. Scoring which of the two you are winning is the subject of Share of Model, measured properly, and the taxonomy underneath it is citation vs mention vs recommendation.

11How do you put a number on it and gate it?

The first number quantifies the waste: count the redirect hops you serve crawlers per cycle and add a penalty for every URL whose terminal response is temporary, because those get re-requested indefinitely. Multiply by your CDN egress cost per request and you have a finance-legible version, usually small, and that is fine, the real cost was never the bandwidth, it is the fetches you did not get spent on pages you wanted read.

tax =  f · Σ  h_i · (1 + τ_i·k)

  f   crawl cycles per period       τ   1 if terminal response is temporary
  h   redirect hops on URL i        k   re-request multiplier for temporaries
  x CDN egress cost/request = a finance-legible number. The real cost was
  never bandwidth; it is the fetches not spent on pages you wanted read.

SHI =  0.40·R + 0.30·H + 0.20·γ + 0.10·F

  R  share of declared URLs returning 200 directly
  H  share resolving inside the three-hop ceiling
  γ  timestamp trust (from timestamp_trust above)
  F  share of commercial URLs updated within 30 days
  Score monthly. Treat anything under 80 as an open engineering item.

The weights are a starting point, not a law. Response health carries the most because it is a precondition for everything else; freshness carries the least because it is hardest to sustain and easiest to fake. Score it monthly, plot it, and treat anything under 80 as an open engineering item.

# .github/workflows/discovery-gate.yml
name: discovery-gate
on:
  push:
    branches: [main]
  schedule:
    - cron: '0 6 * * 1'          # weekly drift check against production
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install httpx
      - name: Root files must return 200 with zero hops
        run: |
          for f in robots.txt sitemap.xml llms.txt; do
            read -r hops status < <(curl -sIL -o /dev/null \
              -w '%{num_redirects} %{http_code}' "https://example.com/$f")
            echo "$f: hops=$hops status=$status"
            [ "$hops" = "0" ] && [ "$status" = "200" ] || exit 1
          done
      - name: Sitemap discovery yield must exceed 0.98
        run: python scripts/validate_sitemap.py
      - name: Push changed URLs to IndexNow
        if: github.event_name == 'push'
        run: python scripts/push_indexnow.py
        env:
          INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}

Pair the gate with log telemetry so you catch regressions that ship from outside your repository, which is where most of them come from. Segment requests by verified bot identity rather than user-agent string alone, using reverse and forward DNS or vendor-published IP ranges, and alert when AI-crawler traffic pushes origin p95 latency past 800ms or 5xx past 1%, when spoofed AI user-agents exceed 5% of bot traffic, and immediately if any root discovery file starts returning something other than 200. The verification mechanics tie back to prompt-to-citation tracking.

12What does the ninety-day sequence look like?