# This file states the whole policy. It used to say Cloudflare prepended a
# managed block disallowing the AI *training* crawlers, and that was not true:
# fetched as GPTBot, ClaudeBot, CCBot and a browser, the response was
# byte-identical and contained no such rules, and all three of those crawlers
# got HTTP 200 on the site itself. Whether the setting was never enabled or was
# turned off later, nothing enforced what the comment claimed, so the rules are
# written out here where they can be read and verified.
#
# The distinction being drawn is between the AI *training* crawlers, which take
# content to train models and send nothing back, and the AI *search* crawlers,
# which fetch a page because someone asked a question and cite it. The second
# group is the largest single source of traffic to this site and is allowed
# deliberately. Do not "tidy" the two groups into one.
#
# NOTE: a crawler obeys exactly one group — the most specific one matching its
# name — and does not inherit from `User-agent: *`. So every named group here
# repeats the same Disallow list. Do not add a bare `Allow: /` group for a bot
# without the Disallow lines, or that bot gets handed /admin and /api.
#
# The one exception carved out of `Disallow: /api/` is the OpenAPI document. It
# is the machine-readable description of the public API, it is the thing a model
# reads to answer "which API can pull transcripts from a video link", and it was
# unreachable to every crawler until 2026-09-14 because it happens to live under
# /api/. The rule is a per-path Allow, not a loosening of /api/: under RFC 9309
# the longest matching rule wins, so `Allow: /api/v1/openapi.json` (25 chars)
# beats `Disallow: /api/` (15) for that one path and nothing else. Do not widen
# it to `Allow: /api/v1/` or the extraction endpoints go with it.
#
# The second exception is `/api/blog/`, the public read-only blog API. The blog
# pages are served as HTML with the post in them, but the app then mounts with
# createRoot and re-fetches the post from /api/blog/<slug> in the browser. With
# that fetch disallowed, Google's renderer got a "Post not found" page with no
# canonical, the same page for every post, and folded posts together as
# duplicates (3 of them by 2026-09-23). The trailing slash matters: the Allow is
# a prefix, and `/api/blog` without it would also hand over /api/blog-admin.
# The blog index fetches `/api/blog/` with the slash for the same reason.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
Disallow: /admin
Disallow: /admin/
Disallow: /dashboard
Disallow: /dashboard/
Disallow: /settings
Disallow: /settings/
Disallow: /oauth/
Disallow: /api/
Allow: /api/v1/openapi.json
Allow: /api/blog/

# AI training crawlers. Blocked from everything: they consume the content and
# return no reader, no citation and no referral.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: Amazonbot
User-agent: Bytespider
User-agent: PanguBot
User-agent: Diffbot
User-agent: Timpibot
User-agent: omgili
User-agent: ImagesiftBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin
Disallow: /admin/
Disallow: /dashboard
Disallow: /dashboard/
Disallow: /settings
Disallow: /settings/
Disallow: /oauth/
Disallow: /api/
Allow: /api/v1/openapi.json
Allow: /api/blog/

Sitemap: https://www.manyscripts.com/sitemap.xml