# Josh Wood Colour — crawler policy # # Policy (JWC-2301, UCP-028): the site is indexable and citable by AI answer # engines, which send traffic. Its content is not offered for model training. # search=yes — index the site and surface it in results # ai-input=yes — quote and cite it when answering a user's question # ai-train=no — do not use it as model training data # # Content-Signal is declared per group because RFC 9309 gives groups no # inheritance. Spec: https://contentsignals.org/ # # Edited by hand. This file used to be generated by gatsby-plugin-robots-txt, # which could emit neither Content-Signal nor comments; the plugin was removed # in JWC-2301. # # NOTE: the rule block below is duplicated in the AI-search group. RFC 9309 # forbids inheritance, so a named bot ignores the * group entirely. Keep both # copies in sync — src/__test__/seo/robots.test.js fails if they drift. User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Disallow: /search Disallow: /account/ Disallow: /collections/v2/* Disallow: /collections/v3/* Disallow: /collections/v4/* Disallow: /collections/v5/* Disallow: /collections-faq/* Disallow: /filter-blogs Disallow: /products/miracle-system-v2 Disallow: /pages/hair-colour-care-quiz-test Disallow: /home-a Disallow: /pages/popup Disallow: /social-discount Disallow: /yoursubscription Disallow: /manage-subscription-link-expired Disallow: /pages/thankyoubrunette Disallow: /video-products Disallow: /social-range Disallow: /multi-buy-miracle-system Disallow: /collections/test Disallow: /collections/prep-page Disallow: /collections/peter-hicks-empty-collection-test Disallow: /collections/anniversary-mini-mask-1 Disallow: /cart Disallow: /bundles-offer Disallow: /anniversary Disallow: /test-new-modal Disallow: /collections/washbag-launch Disallow: /josh-wood-colour-black-friday-e-book Disallow: /collections/social-cm Disallow: /social-gloss Disallow: /social-colour Disallow: /free-gift-for-you # AI answer engines and assistant fetches — ALLOWED. # These cite sources and send traffic. Same restrictions as the * group above: # they must not reach the cart, customer accounts, A/B duplicates or test pages. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Amazonbot User-agent: Applebot User-agent: DuckAssistBot User-agent: Google-CloudVertexBot User-agent: meta-externalfetcher Content-Signal: search=yes, ai-input=yes, ai-train=no Disallow: /search Disallow: /account/ Disallow: /collections/v2/* Disallow: /collections/v3/* Disallow: /collections/v4/* Disallow: /collections/v5/* Disallow: /collections-faq/* Disallow: /filter-blogs Disallow: /products/miracle-system-v2 Disallow: /pages/hair-colour-care-quiz-test Disallow: /home-a Disallow: /pages/popup Disallow: /social-discount Disallow: /yoursubscription Disallow: /manage-subscription-link-expired Disallow: /pages/thankyoubrunette Disallow: /video-products Disallow: /social-range Disallow: /multi-buy-miracle-system Disallow: /collections/test Disallow: /collections/prep-page Disallow: /collections/peter-hicks-empty-collection-test Disallow: /collections/anniversary-mini-mask-1 Disallow: /cart Disallow: /bundles-offer Disallow: /anniversary Disallow: /test-new-modal Disallow: /collections/washbag-launch Disallow: /josh-wood-colour-black-friday-e-book Disallow: /collections/social-cm Disallow: /social-gloss Disallow: /social-colour Disallow: /free-gift-for-you # Model training crawlers and training opt-out tokens — DENIED. # GPTBot OpenAI, training corpus # ClaudeBot Anthropic, training corpus # anthropic-ai Anthropic, legacy token # Claude-Web Anthropic, deprecated; kept so old parsers still see a rule # CCBot Common Crawl, feeds most public training sets # Applebot-Extended control token: opts out of Apple Intelligence training. # Does NOT affect Applebot search, allowed above. # meta-externalagent Meta AI, training corpus # Bytespider ByteDance, training corpus # Google-Extended is deliberately NOT listed. Google uses one token for both # Gemini training and Gemini Apps grounding, so blocking it would also remove the # site from Gemini answers. Search and AI Overviews follow Googlebot either way. User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web User-agent: CCBot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Bytespider Content-Signal: ai-train=no Disallow: / Sitemap: https://joshwoodcolour.com/sitemap-index.xml Host: https://joshwoodcolour.com # AI/LLM context: https://joshwoodcolour.com/llms.txt # Agent capability manifest (ARD): https://joshwoodcolour.com/.well-known/ai-catalog.json # Kept as a comment, not an Agentmap directive: Agentmap is absent from every # Lighthouse robots.txt safelist, so the directive fails the SEO audit. The # well-known path above is ARD's primary discovery mechanism anyway.