OGKit

Rules / robots-blocks-social-crawlers · ruleset 2026-07

robots / X-Robots-Tag blocks social previews

errorrobots-blocks-social-crawlers
Rule idrobots-blocks-social-crawlers
Severityerror
CategoryPlatform and crawl
Platformsall platforms
Needs humanyes — structural fix

What this is

robots / X-Robots-Tag blocks social previews. OGKit rule robots-blocks-social-crawlers (error, needs human). The id is a public API for the docs URL and CI --disable key; do not rename it.

Search phrases that should land on this page:

  • robots blocks social previews
  • X-Robots-Tag blocks
  • noindex social card
  • nosnippet preview

Why it matters

noindex / none / nosnippet (and similar) on the page or X-Robots-Tag can prevent scrapers from building a rich card.

Which platforms are affected

All OGKit platform profiles, including major social, chat, and search crawlers that consume these tags.

Platform-specific crawler budgets and field priority live in the platform behaviour reference.

How to reproduce

  1. Check meta robots and the X-Robots-Tag response header for noindex / none / nosnippet.
  2. curl -sI URL | grep -i robots

How to fix

Allow social crawlers to index the public marketing/share URLs. Keep noindex only on true private pages that should never be shared.

Example fix:

<meta name="robots" content="index,follow" />