# robots.txt for https://www.extrieve.com/ # Layout: Vite static site at the root, with THREE WordPress installs in # subdirectories: /blog/, /news/ and /jobs/. The old blog./news./jobs. # subdomains now 301 to these paths. # Last reviewed: 2026-09-20 # --------------------------------------------------------------------------- # 1. Global rules (apply to every crawler that has no more specific group below) # # Content-Signal is a machine-readable statement of how this content may be # used. Advisory, and ignored by crawlers that do not implement it, but it # records intent and supports a takedown. # search = build a search index and link back -> yes # ai-input = ground a live generative answer, with citation -> yes # ai-train = train or fine-tune a model -> no # --------------------------------------------------------------------------- User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Disallow: /blog/wp-admin/ Disallow: /blog/wp-login.php Disallow: /news/wp-admin/ Disallow: /news/wp-login.php Disallow: /jobs/wp-admin/ Disallow: /jobs/wp-login.php Disallow: /search/ Disallow: /api/ Allow: /blog/wp-admin/admin-ajax.php Allow: /blog/wp-content/uploads/ Allow: /news/wp-admin/admin-ajax.php Allow: /news/wp-content/uploads/ Allow: /jobs/wp-admin/admin-ajax.php Allow: /jobs/wp-content/uploads/ # Tracking-parameter and legacy crawl-waste URLs (from the GSC audit) Disallow: /*?trk= Disallow: /*?fbclid= Disallow: /*?source= Disallow: /*?extrieve_src= Disallow: /*?amp= Disallow: /*?*amp=1 Disallow: /*?s= Disallow: /*?replytocom= Disallow: /blog/feed/ Disallow: /blog/author/ Disallow: /blog/tag/ Disallow: /blog/page/ Disallow: /news/feed/ Disallow: /news/author/ Disallow: /news/tag/ Disallow: /news/page/ Disallow: /jobs/feed/ Disallow: /jobs/author/ Disallow: /jobs/tag/ # --------------------------------------------------------------------------- # 2. Main search engines # # NOTE (fixed 2026-07-25): this group previously contained only # "Disallow: /blog/wp-admin/" + "Allow: /". Because a crawler obeys ONLY its # most specific matching group, Googlebot and Bingbot were silently exempt from # every crawl-waste rule in group 1 above, which is exactly backwards: those # rules exist mainly for Google. The disallows are now repeated here so they # actually apply to the crawlers that matter. # --------------------------------------------------------------------------- User-agent: Googlebot User-agent: Googlebot-Image User-agent: Bingbot User-agent: DuckDuckBot User-agent: YandexBot Disallow: /blog/wp-admin/ Disallow: /blog/wp-login.php Disallow: /news/wp-admin/ Disallow: /news/wp-login.php Disallow: /jobs/wp-admin/ Disallow: /jobs/wp-login.php Disallow: /search/ Disallow: /api/ Disallow: /*?trk= Disallow: /*?fbclid= Disallow: /*?source= Disallow: /*?extrieve_src= Disallow: /*?amp= Disallow: /*?*amp=1 Disallow: /*?s= Disallow: /*?replytocom= Disallow: /blog/feed/ Disallow: /blog/author/ Disallow: /blog/tag/ Disallow: /news/feed/ Disallow: /news/author/ Disallow: /news/tag/ Disallow: /jobs/feed/ Disallow: /jobs/author/ Disallow: /jobs/tag/ Allow: /blog/wp-admin/admin-ajax.php Allow: /blog/wp-content/uploads/ Allow: /news/wp-admin/admin-ajax.php Allow: /news/wp-content/uploads/ Allow: /jobs/wp-admin/admin-ajax.php Allow: /jobs/wp-content/uploads/ Allow: / # --------------------------------------------------------------------------- # 3. AI engines: allow answers and citation, refuse training # # ALLOW the AI *search / answer* crawlers. These are what put Extrieve into # ChatGPT Search, Claude, Perplexity, Copilot and Apple Intelligence answers # with a citation and a link back. Blocking them would remove us from # generative search entirely, which is the opposite of what we want. # --------------------------------------------------------------------------- User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-User User-agent: Claude-SearchBot User-agent: Applebot User-agent: Amazonbot User-agent: DuckAssistBot User-agent: MistralAI-User User-agent: Google-CloudVertexBot Disallow: /search/ Disallow: /api/ Allow: / # BLOCK the AI training / dataset harvesters. Our content is published to be # read and cited, not to be ingested into model weights or resold as a corpus. # See /tdm-policy.json and /.well-known/tdmrep.json for the formal reservation. # These agents are also blocked server-side in .htaccess, because robots.txt is # voluntary and this list is not. User-agent: GPTBot # Google-Extended also disables grounding in Gemini Apps and Vertex AI (Google # couples training and grounding in this one token). Google Search and AI # Overviews are unaffected. Deliberate choice: we accept losing Gemini Apps # citations rather than permit training on our content. User-agent: Google-Extended User-agent: ClaudeBot User-agent: anthropic-ai User-agent: CCBot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Meta-ExternalAgent User-agent: FacebookBot User-agent: Bytespider User-agent: AI2Bot User-agent: ImagesiftBot User-agent: Omgilibot User-agent: Omgili User-agent: Diffbot User-agent: Timpibot User-agent: YouBot User-agent: PanguBot User-agent: Kangaroo Bot User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: Scrapy Disallow: / # --------------------------------------------------------------------------- # 4. Aggressive commercial SEO spiders (server load, no benefit to us) # --------------------------------------------------------------------------- User-agent: SemrushBot User-agent: AhrefsBot User-agent: MJ12bot User-agent: DotBot User-agent: BLEXBot User-agent: DataForSeoBot User-agent: Barkrowler User-agent: serpstatbot User-agent: ZoominfoBot Disallow: / # --------------------------------------------------------------------------- # 5. Sitemaps # NOTE (updated 2026-09-20): news and jobs HAVE now moved into subdirectories. # Their sitemap indexes are /news/sitemaps.xml and /jobs/sitemaps.xml (the old # /sitemap_index.xml paths 301 to these). Verified live: both return 200 and # every inside uses https://www.extrieve.com/. # --------------------------------------------------------------------------- Sitemap: https://www.extrieve.com/sitemap.xml Sitemap: https://www.extrieve.com/blog/sitemap_index.xml Sitemap: https://www.extrieve.com/news/sitemaps.xml Sitemap: https://www.extrieve.com/jobs/sitemaps.xml