The Freshness Moat: How to Tell Whether Crawlers Fetched Your Updates
Stop changing publication dates to trick crawlers. Build a Freshness Moat instead: verifiable state, honest signals, and edge-log measurement that shows whether the crawlers you care about actually re-fetched your update.
Are you still performing “date theater”? You tweak a few words in an old post, change the “Last updated” date, and hope the search engine notices.
Google, for one, says it uses a sitemap’s lastmod only when the value is “consistently and verifiably accurate,” and that a copyright-date change isn’t a significant update (Google Search Central: build and submit a sitemap, accessed 2026-10-18)1. Date theater doesn’t just fail; it teaches crawlers to ignore your dates.
The web has a deeper problem for the age of AI search: it can publish text, but it struggles to publish state.
So treat freshness as an engineering property you can measure: did the crawlers you care about re-fetch your updated pages, and how fast?
Be precise about what that measures. A fetch is not indexing, and indexing is not being used in an AI answer. This article is about the part you can observe on your own servers: the fetch. The other two need their own checks, covered at the end.
Stop Faking Freshness. Start Publishing State.
If your website can’t say what changed and when, in a form a machine can read, you don’t have a knowledge base. You have an archive.
Static pages are fine for human readers, but they are bad at communicating change. A crawler deciding what to fetch again would like to know:
- What exactly changed?
- When did it change?
- Was this a material update or a typo fix?
- Which data files should it check next?
Without answers to those questions, “freshness” is a constant churn of timestamps.
Four Concepts You Need to Understand
Principle pages vs. volatile pages
Principle pages are timeless. They outline your core architecture, long-lived frameworks, and positioning. They should rarely change.
Volatile pages are stateful. They contain your pricing, operational policies, supported regions, or software versions. They need continuous validation.
For example, a page explaining your methodology can stay stable, but volatile claims like your supported markets (DACH, United States) must reflect your current operating reality.
The material update
A material update happens only when you change:
- the core meaning of the content (not just swapping adjectives),
- structured data (schema and JSON‑LD),
- key evidentiary links (citations, audit packets, registries),
- the validity of a business claim (pricing model, service eligibility, availability).
This matches how Google describes a significant change for lastmod: main content, structured data, or on-page links count; a copyright date does not (sitemap guidance, accessed 2026-10-18).
Propagation
Propagation is the delay between publishing a material update and a crawler successfully re-fetching the affected URLs. It is a fetch measurement. Whether the new content then gets indexed, and whether any AI system uses it, are separate questions.
Propagation Half-Life (PHL)
The central metric for your Freshness Moat:
How long after a material update does a given crawler re-fetch half of the affected URLs?
It is only meaningful with its definition attached. The measurement section below spells out the cohort, the crawler, the window, and what happens when fewer than half are re-fetched.
The Freshness Moat Triad
Stop guessing whether your updates were noticed. Publish state, send signals, and measure.
In words: you publish the current state, signal that it changed, measure who re-fetched it, and feed what you learn back into how you publish.
Step One: Ship Verification, Not Rewrites
Instead of telling your marketing team to “refresh the blog post,” publish accountable, structured state that says:
- what is true right now,
- what just changed,
- when it was last verified.
The verification and updates module
Add a public section or a dedicated route that is fully machine-readable. It doesn’t need a slick design; it needs to be accurate. This block renders from this site’s golden record:
{
"entity": "Argbe.tech",
"as_of": "2026-10-20",
"entity_version": "0.130.1",
"volatile_claims": {
"availability_regions": [
"DACH",
"United States"
],
"pricing_model": "Fixed weekly rate"
},
"materiality_rule": "Material update = core meaning / structured-data / claim validity change. Cosmetic edits are explicitly excluded.",
"surfaces": {
"canonical_pages": [
"/",
"/contact",
"/geo-seo"
],
"machine_surfaces": [
"/entity.json",
"/llms.txt",
"/changes.json",
"/sitemap-index.xml"
]
}
}Because these values come from a single source of truth (the golden record), they stay aligned across the site:
- Pricing model: Fixed weekly rate Fixed weekly rate
- Regions: DACH, United States ["DACH","United States"]
Step Two: Build Your Signal Stack
Your signal stack tells crawlers, honestly, that something changed. None of these signals forces a crawl. Each one makes a re-fetch cheaper or more likely for the systems that support it.
HTTP caching validators
Make revalidation cheap with ETag and Last-Modified. Google’s crawlers support both; when both are present they use ETag, and Google recommends ETag because it avoids date-format problems, while still advising you to set both (Google: overview of crawlers and fetchers, accessed 2026-10-18)2. Google also says individual crawlers may or may not use caching, depending on the product.
Check your own surfaces before you trust them. Run on this site on 2026-10-18:
$ curl -sI https://argbe.tech/ | grep -iE '^(etag|last-modified|cache-control)'
cache-control: public, max-age=0, must-revalidate
$ curl -sI https://argbe.tech/entity.json | grep -iE '^(etag|last-modified|cache-control)'
cache-control: public, max-age=86400, stale-while-revalidate=86400
etag: "d4c0caa5f6986751c393dd6a3fc247fd"
$ curl -s -o /dev/null -w '%{http_code}\n' -H 'If-None-Match: "d4c0caa5f6986751c393dd6a3fc247fd"' https://argbe.tech/entity.json
304The JSON surfaces revalidate correctly: same ETag, 304 Not Modified. The HTML pages send neither validator, so a crawler has to download them in full to learn nothing changed. That’s our gap, not a theory.
Strict sitemap discipline
If you bump lastmod every time you fix a comma, you’re crying wolf. Bump it only for material updates. Google ignores changefreq and priority entirely (sitemap guidance, accessed 2026-10-18), so don’t spend effort there.
Push notifications with IndexNow
IndexNow lets you notify participating search engines the moment a URL is added, updated, or deleted (IndexNow documentation, accessed 2026-10-18)3. Know its limits before you rely on it:
- Participating engines include Bing, Yandex, Naver, Seznam.cz, Yep, and Amazon; a submission to one is shared with the others. Google is not listed (IndexNow FAQ, accessed 2026-10-18)4.
- Submitting a URL “does not guarantee immediate indexing”; each engine still decides whether to crawl it (same FAQ; see also Bing: IndexNow getting started, accessed 2026-10-18)5.
Honest structured data
When a material update happens, reflect it in your JSON‑LD. Update dateModified only when the change matters (schema.org/dateModified, accessed 2026-10-18)6. Keep “last verified” (you checked and it’s still accurate) separate from “last updated” (you changed the underlying facts).
Avoid Freshness Theater
| The Danger | What It Looks Like | Why It Hurts You | The Real Solution |
|---|---|---|---|
| Date churn | Automating today’s date into “Last updated” | Crawlers that check accuracy learn to ignore it | Only update lastmod for true material changes |
| Cosmetic tweaks | Swapping adjectives to force a refresh | Doesn’t change the substance of your claims | Publish structured state changes, not just prose |
| Silent updates | Shipping new data without sending signals | Crawlers have no reason to check sooner | Send validators, keep sitemaps accurate, publish a change feed |
Step Three: Route Agents with /llms.txt (an Unverified Proposal)
/llms.txt is a proposal, published by Jeremy Howard and still open for community input: a Markdown file that gives language models a short, curated map of your site, meant mainly for use at inference time (llmstxt.org, accessed 2026-10-18)7. Its format is an H1 with the site name, a short summary, and H2 sections of links.
Treat it as cheap and optional. Google says you don’t need AI text files or new machine-readable files to appear in AI Overviews or AI Mode (Google: AI features and your website, accessed 2026-10-18)8, and we know of no major AI search product that documents reading it. Publish one if it helps agents or integrators you work with; don’t expect it to change crawling.
A file in the proposed format, pointing at your machine surfaces (example domain):
# Example Co.
> Shopify development and conversion work. Facts below are generated from our golden record.
## Machine surfaces
- [Golden record](https://example.com/entity.json): identity, offers, regions, pricing model
- [Change feed](https://example.com/changes.json): material updates with dates and affected URLs
- [Sitemap](https://example.com/sitemap-index.xml)
## Key pages
- [Contact](https://example.com/contact)
- [Pricing](https://example.com/pricing)Step Four: Publish a Change Feed
Stop hiding your updates in prose. Publish a /changes.json file as your machine-readable patch notes: each event says what changed, whether it was material, and which URLs it affects. That list of affected URLs is also the cohort you’ll measure in the next step.
A hypothetical material event, rendered with this site’s current version:
{
"version": "0.130.1",
"as_of": "2026-10-20",
"events": [
{
"id": "update-example-001",
"observed_at": "(deploy timestamp)",
"material": true,
"changed_paths": [
"pricing.model",
"markets.regions"
],
"affected_urls": [
"/",
"/contact",
"/entity.json",
"/changes.json",
"/sitemap-index.xml"
],
"evidence": [
"https://github.com/argbe-tech/releases/releases"
]
}
]
}Hold your own feed to the rule. This site’s feed doesn’t meet it yet: as of 2026-10-18 it marks every release as material, including releases that changed only the version number and date. That is the date churn this article warns against.
Step Five: The Signal Observatory
If you aren’t measuring re-fetches, your freshness is performative.
You need to know which crawlers fetched your updated URLs, and how soon after you deployed. Edge logs are the most direct source. On Cloudflare, the Logpush http_requests dataset carries the fields you need: EdgeStartTimestamp, ClientRequestPath, ClientRequestUserAgent, EdgeResponseStatus, and VerifiedBotCategory (Cloudflare: http_requests dataset, accessed 2026-10-18)9.
Separate the crawlers before you count anything
Different crawlers do different jobs, and only some of them say anything about search:
- Search crawlers: Googlebot; OpenAI’s OAI-SearchBot, which surfaces sites in ChatGPT search; Anthropic’s Claude-SearchBot.
- Training crawlers: OpenAI’s GPTBot and Anthropic’s ClaudeBot collect content for model training. A GPTBot fetch tells you nothing about whether you appear in ChatGPT search.
- User-initiated fetchers: ChatGPT-User and Claude-User visit a page because a person asked; OpenAI notes robots.txt rules may not apply to them.
Sources: OpenAI: overview of OpenAI crawlers10 and Anthropic: does Anthropic crawl data from the web11, both accessed 2026-10-18.
User-agent strings are easy to fake, so count only verified crawlers. Google documents reverse-and-forward DNS checks and publishes its crawler IP ranges (Google: verifying Googlebot, accessed 2026-10-18)12; OpenAI publishes IP lists for its crawlers. Cloudflare’s VerifiedBotCategory does this classification for you; its category names are listed in Cloudflare’s verified bot categories and can change, so check them before you filter.
Unverified automation is a separate problem. Cloudflare has documented a crawler that used undeclared user agents to get around no-crawl directives (Cloudflare blog, accessed 2026-10-18)13. Watch for it, but keep it out of your propagation numbers.
Define PHL before you compute it
- Event: one material update, with its deploy timestamp (T0) from your change feed.
- Cohort and denominator: the affected URLs listed for that event. Every one of them counts, including those never re-fetched.
- Crawler: one verified crawler per measurement. Never mix search and training crawlers.
- Re-fetch: the first request by that crawler after T0 that returned
200or304. - Window: a fixed observation period, chosen before you look.
- PHL: the time until half the cohort has been re-fetched. If fewer than half were re-fetched within the window, report “longer than the window” and the unrefetched URLs. A censored result is a finding, not a zero.
A reproducible calculation
This script computes PHL from Logpush NDJSON with the Python standard library:
"""phl.py: Propagation Half-Life from Cloudflare Logpush http_requests NDJSON (Python 3, stdlib only).
Usage: python3 phl.py LOG.ndjson DEPLOYED_AT_ISO WINDOW_DAYS BOT_CATEGORY UA_TOKEN URL_PATH [URL_PATH ...]
Example: python3 phl.py sample.ndjson 2026-10-01T09:00:00Z 14 "Search Engine Crawler" Googlebot /pricing /contact
"""
import json
import sys
from datetime import datetime, timedelta, timezone
log_file, deployed_at, window_days, bot_category, ua_token, *cohort = sys.argv[1:]
t0 = datetime.fromisoformat(deployed_at.replace("Z", "+00:00"))
t_end = t0 + timedelta(days=int(window_days))
first_refetch = {}
with open(log_file) as f:
for line in f:
r = json.loads(line)
ts = datetime.fromtimestamp(r["EdgeStartTimestamp"] / 1e9, tz=timezone.utc) # nanoseconds
if not (t0 < ts <= t_end):
continue
if r.get("VerifiedBotCategory") != bot_category or ua_token not in r["ClientRequestUserAgent"]:
continue # one verified crawler at a time; never mix search and training bots
path = r["ClientRequestPath"]
if path in cohort and r["EdgeResponseStatus"] in (200, 304):
first_refetch[path] = min(ts, first_refetch.get(path, ts))
delays = sorted((ts - t0).total_seconds() / 3600 for ts in first_refetch.values())
n, k = len(cohort), len(delays)
print(f"cohort={n} refetched={k} window={window_days}d bot={ua_token}")
for path in cohort:
ts = first_refetch.get(path)
print(f" {path}: " + (f"{(ts - t0).total_seconds() / 3600:.1f}h" if ts else "not refetched in window"))
half = (n + 1) // 2 # URLs that must be refetched before half the cohort has been seen
if k >= half:
print(f"PHL = {delays[half - 1]:.1f}h (time until {half} of {n} URLs were refetched)")
else:
print(f"PHL > {window_days}d (only {k} of {n} refetched; result is censored, not zero)")It expects EdgeStartTimestamp in nanoseconds; configure your Logpush job’s timestamp format that way.
Here is a synthetic sample log, sample.ndjson: nine requests built to exercise the rules, including a pre-deploy fetch, two GPTBot fetches, a 304, a duplicate, and a request that claims to be Googlebot without verification.
{"EdgeStartTimestamp": 1790773200000000000, "ClientRequestPath": "/pricing", "ClientRequestUserAgent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "VerifiedBotCategory": "Search Engine Crawler", "EdgeResponseStatus": 200}
{"EdgeStartTimestamp": 1790857800000000000, "ClientRequestPath": "/pricing", "ClientRequestUserAgent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "VerifiedBotCategory": "Search Engine Crawler", "EdgeResponseStatus": 200}
{"EdgeStartTimestamp": 1790866800000000000, "ClientRequestPath": "/pricing", "ClientRequestUserAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)", "VerifiedBotCategory": "AI Crawler", "EdgeResponseStatus": 200}
{"EdgeStartTimestamp": 1790910720000000000, "ClientRequestPath": "/contact", "ClientRequestUserAgent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "VerifiedBotCategory": "Search Engine Crawler", "EdgeResponseStatus": 304}
{"EdgeStartTimestamp": 1790989200000000000, "ClientRequestPath": "/services/shopify", "ClientRequestUserAgent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "VerifiedBotCategory": "Search Engine Crawler", "EdgeResponseStatus": 200}
{"EdgeStartTimestamp": 1790992800000000000, "ClientRequestPath": "/services/shopify", "ClientRequestUserAgent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "VerifiedBotCategory": "Search Engine Crawler", "EdgeResponseStatus": 200}
{"EdgeStartTimestamp": 1790852400000000000, "ClientRequestPath": "/entity.json", "ClientRequestUserAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)", "VerifiedBotCategory": "AI Crawler", "EdgeResponseStatus": 200}
{"EdgeStartTimestamp": 1791313200000000000, "ClientRequestPath": "/changes.json", "ClientRequestUserAgent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", "VerifiedBotCategory": "Search Engine Crawler", "EdgeResponseStatus": 200}
{"EdgeStartTimestamp": 1790888400000000000, "ClientRequestPath": "/services/cro", "ClientRequestUserAgent": "Mozilla/5.0 (compatible; Googlebot/2.1)", "VerifiedBotCategory": "", "EdgeResponseStatus": 200}Running the script on it:
$ python3 phl.py sample.ndjson 2026-10-01T09:00:00Z 14 "Search Engine Crawler" Googlebot \
/pricing /contact /services/shopify /services/cro /entity.json /changes.json
cohort=6 refetched=4 window=14d bot=Googlebot
/pricing: 3.5h
/contact: 18.2h
/services/shopify: 40.0h
/services/cro: not refetched in window
/entity.json: not refetched in window
/changes.json: 130.0h
PHL = 40.0h (time until 3 of 6 URLs were refetched)Two of the exclusions are the point. GPTBot fetched /entity.json two hours after the deploy, but it’s a training crawler, so /entity.json still counts as not re-fetched for search. The unverified “Googlebot” request to /services/cro is ignored. Counting either would make your propagation look faster than it is.
What a fast PHL does not tell you
- Indexing: for Google, check the URL Inspection tool in Search Console for the page’s indexed version.
- AI answers: ask a fixed set of questions about the changed facts in the products you care about, record the date and answers, and repeat. That’s a sampled observation, not proof of cause.
- Cause: if PHL improves after you add a signal, other things may have changed too. Compare against events without the signal before you credit it.
The feedback loop
- Make a material update and record it in
/changes.json. - Compute PHL per crawler for that event’s cohort.
- If URLs stay unrefetched, check the basics for those URLs: status codes, validators, sitemap
lastmod, internal links, and (for engines that support it) IndexNow. - Track PHL across events; a single event is an anecdote.
The Bottom Line
Competitors can copy your blog posts. It is much harder to copy a disciplined system that ships verifiable data, sends honest signals, and measures its own propagation.
Stop rewriting content to please legacy algorithms. Publish state, signal it honestly, and measure who actually came back for it.
How This Was Checked
The header, 304, sitemap, and change-feed observations come from requests to argbe.tech on 2026-10-18. The PHL script was run on the synthetic sample described above; its numbers describe that sample, not any real crawler. Crawler, sitemap, IndexNow, and llms.txt behavior was checked against the linked documentation on 2026-10-18.
Evidence Locker
On this page