For AI agents: a documentation index is available at /llms.txt. Markdown versions of these pages are available by appending `.md` to the URL or by sending `Accept: text/markdown`.

Rules / robots-blocks-social-crawlers · ruleset 2026-07

robots / X-Robots-Tag blocks social previews

errorrobots-blocks-social-crawlers

Name

robots / X-Robots-Tag blocks social previews robots-blocks-social-crawlers

noindex / none / nosnippet (and similar) on the page or X-Robots-Tag can prevent scrapers from building a rich card.

How to fix

Allow social crawlers to index the public marketing/share URLs. Keep noindex only on true private pages that should never be shared.

Example fix:

<meta name="robots" content="index,follow" />

Copy or download

Copyable Markdown for a teammate or an agent chat.

# robots-blocks-social-crawlers — robots / X-Robots-Tag blocks social previews

- **Name:** robots / X-Robots-Tag blocks social previews

## Description

noindex / none / nosnippet (and similar) on the page or X-Robots-Tag can prevent scrapers from building a rich card.

## How to fix

Allow social crawlers to index the public marketing/share URLs. Keep noindex only on true private pages that should never be shared.

### Example fix:

```html
<meta name="robots" content="index,follow" />
```

## Rule docs

https://ogkit.akr.moe/rules/robots-blocks-social-crawlers

Download an Agent Skill package for this rule.

Download robots-blocks-social-crawlers.skill

The file is a ZIP of skill-name/SKILL.md (Agent Skills spec). The .skill extension is that ZIP; rename to .zip if a client asks for one.

Rule idrobots-blocks-social-crawlers
Severityerror
CategoryPlatform and crawl
Platformsall platforms
Needs humanyes — structural fix

What this is

robots / X-Robots-Tag blocks social previews. OGKit rule robots-blocks-social-crawlers (error, needs human). The id is a public API for the docs URL and CI --disable key; do not rename it.

Search phrases that should land on this page:

  • robots blocks social previews
  • X-Robots-Tag blocks
  • noindex social card
  • nosnippet preview

Why it matters

noindex / none / nosnippet (and similar) on the page or X-Robots-Tag can prevent scrapers from building a rich card.

Which platforms are affected

All OGKit platform profiles, including major social, chat, and search crawlers that consume these tags.

Platform-specific crawler budgets and field priority live in the platform behaviour reference.

How to reproduce

  1. Check meta robots and the X-Robots-Tag response header for noindex / none / nosnippet.
  2. curl -sI URL | grep -i robots
OGKit