Social crawler User-Agents: the truth table for content forks

August 3, 2026 · 7 min read

open-graph
crawlers
user-agent
cloaking

You open the production page in Chrome: new hero, correct og:title, fresh og:image. Marketing pastes the same URL into Slack. The card still shows last quarter's campaign art and a title from the A/B "control" template.

Nobody rewrote the tags overnight. Two different User-Agents hit the same path and received two different HTML documents. The bot lane still had the old CMS layout. Humans got the redesign.

That gap is the entire topic of this post: what social crawlers claim to be, what they do (and do not) execute, how to prove it with curl, and when UA forking is a fix versus an accident.

One URL, two worlds

Typical ways the bot lane diverges from the browser lane:

Mechanism	What the crawler gets	What you see
SSR / SPA shell	Empty or placeholder meta in initial HTML	Tags after hydration
A/B or personalization	Control / default variant	Winning variant or logged-in experience
Bot fight / challenge	403, interstitial, or minimal challenge page	Full page after JS or cookie
CDN / WAF rules	Different cache key or block for known bot UAs	Warm edge cache for browsers
Intentional "bot HTML"	Dedicated OG-only markup	Full marketing page

The first four often look like "Open Graph is broken." The fifth can be legitimate (serve share meta without shipping a 2 MB SPA shell) — until someone forgets to update the bot template when the campaign ends.

Security note, up front: do not treat "hide secrets from non-bot UAs" as access control. Crawlers and attackers both choose their User-Agent string. Anything you put only in the browser branch is still public to anyone who requests it.

User-Agent truth table

Values below come from OGKit's platform profiles (@og-kit/core). Every row carries a source and a last verified date. Where we do not have a reliable number, we write unknown / unverified — we do not invent TTLs.

Platform	User-Agent (representative)	Executes JS?	Cache TTL (share)	Refresh path	Source · verified
Facebook / Meta	`facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)`	No	unknown	Sharing Debugger scrape	official · 2026-07-01
X (Twitter)	`Twitterbot/1.0`	No	~7 days (often cited; not an official fixed number)	Card Validator / re-share after URL change	UA official; TTL community · 2026-07-01
LinkedIn	`LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)`	No	~7 days (community)	Post Inspector	community · 2026-07-01
Slack	`Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)`	No	unknown	Re-share link (no public debugger)	UA official · 2026-07-01
Discord	`Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)`	No	unknown	Query-string change or embed tools	community · 2026-07-01
WhatsApp	`WhatsApp/2.0` (variants exist; often reuses Meta crawler infra)	No	unknown	No public debugger; wait or change URL	community · 2026-07-01
Telegram	`TelegramBot (like TwitterBot)`	No	unknown	`@WebpageBot` or re-share	community · 2026-07-01
Googlebot (contrast)	`Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)`	Yes (rendering service)	n/a for OG cards	Search Console URL Inspection	official · 2026-07-01

Full field-level detail (image budgets, truncation, field priority) lives on the live platforms pages — same data, rendered with provenance badges.

How to read the table

Social OG crawlers do not run your JavaScript. If og:* only appears after hydration, the card is empty. That is why "works in DevTools, fails on LinkedIn" is so common. Related rule: og-tags-require-javascript.
Google is not a social preview crawler. Optimizing only for Googlebot (or only testing with a browser) does not prove Slack or Meta will see the same HTML.
TTL columns are not SLAs. Where source is community or unverified, treat the number as a planning hint. After you change tags or images, use each platform's official rescrape path (rescrape helper) instead of waiting for a rumored expiry.

Prove it yourself (`curl -A`)

Do not trust View Source in a browser alone if middleware keys off UA. Fetch as the bots:

URL="https://example.com/launch"

# Baseline (generic browser-ish UA)
curl -sL -A "Mozilla/5.0" "$URL" | tee /tmp/human.html | grep -iE 'og:title|og:image' | head

# Meta
curl -sL -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" \
  "$URL" | tee /tmp/fb.html | grep -iE 'og:title|og:image' | head

# X
curl -sL -A "Twitterbot/1.0" "$URL" | tee /tmp/x.html | grep -iE 'og:title|og:image|twitter:' | head

# LinkedIn
curl -sL -A "LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)" \
  "$URL" | tee /tmp/li.html | grep -iE 'og:title|og:image' | head

Then compare documents, not vibes:

# Body hash (ignores date headers; still sensitive to any HTML drift)
sha256sum /tmp/human.html /tmp/fb.html /tmp/x.html /tmp/li.html

# Structured meta-only diff (install htmlq or use grep)
diff -u \
  <(grep -iE 'property="og:|name="twitter:' /tmp/human.html | sort) \
  <(grep -iE 'property="og:|name="twitter:' /tmp/fb.html | sort)

Also check status codes and headers for the bot UA:

curl -sI -A "facebookexternalhit/1.1" "$URL" | head -20

A 200 for you and a 403 / challenge HTML for the bot is the classic WAF false positive. Pair this with the blank-card checklist: Why your Open Graph card is blank.

What "same" means for OG

Identical full HTML is ideal but not always required. For share correctness, at least these should match across social UAs:

og:title, og:description, og:image, og:url, og:type
Absolute HTTPS image URL that returns 200 without auth
No bot-only interstitial instead of the real document

If marketing HTML and bot HTML intentionally differ, treat the bot template as a first-class surface in your release checklist — same urgency as the hero image.

When to unify HTML (and when a fork is OK)

Prefer one HTML document when:

You are not deliberately maintaining a separate share template
A/B tools or edge personalization rewrite title or hero by cookie
Bot management challenges public marketing URLs
The "fix" was only for Googlebot or Core Web Vitals, not for unfurls

A deliberate fork can be fine when:

You SSR a thin share shell for crawlers and a heavy client app for users, and both get the same OG values from one source of truth
You strip non-share chrome for bots purely for fetch size, without changing meta meaning

Never "OK":

Putting private data only in the non-bot branch as "security"
Serving a permanent 200 with wrong meta to bots while humans see the real campaign
Forgetting to deploy the bot template when the human template changes

OGKit encodes the accidental-divergence case as ua-content-diverged: different User-Agents received different HTML body hashes. That is a warn, not an automatic "you are cloaking" conviction — but it is the signal that forces a human look.

Checklist

curl without JS shows the same og:* tags you expect in the browser
curl -A facebookexternalhit/1.1 (and at least one other social UA) is not blocked
Body or meta hashes across UAs are identical, or the delta is intentional and documented
Bot-fight rules allow public share URLs for known social crawlers
After tag/image changes, you rescrape platforms that matter to you — do not rely on an invented TTL
Secrets are gated by auth, not by User-Agent

Try a multi-UA scan

Paste a public URL into the homepage scanner. OGKit fetches with platform User-Agents from the same profile table, surfaces rule findings including ua-content-diverged, and points at official rescrape links when the HTML is right but the card is still stale.

For private or localhost hosts the public crawler cannot reach, use the browser extension path — covered later in the series. The point of the product is not a prettier mock card; it is showing the document each bot actually received.