40ドメイン調査:Shopifyストアの80%が/API/ucp/MCPを公開、/agents.mdは0%
40-domain study: 80% of Shopify stores expose /API/ucp/MCP, 0% /agents.md (verisource.dev)
要約
2026年11月の調査によると、40の公開ドメインのうち、50%がカスタムな/llms.txtファイルを持っており、17.5%はボイラープレート、7.5%はソフト404でした。カスタム/agents.mdファイルを持つのはわずか5%でした。Shopifyストアの80%は/api/ucp/mcpエンドポイントを公開していますが、カスタム/agents.mdファイルは0%でした。また、/robots.txtでボットをブロックしていないにも関わらず、5%のサイトがGPTBotやPerplexityBotに対してCDNレベルでHTTP 403エラーを返していました。
全文翻訳
2026 AI Search & Agent Readiness Benchmark: /llms.txt, /agents.md & Silent GPTBot WAF Blocks.
Custom /llms.txt
50.0%
20/40 curated · 7/40 boilerplate · 3/40 soft-404
Custom /agents.md
5.0%
2/40 custom (vercel.com, linear.app) · 5/40 stock Shopify
Homepage FAQPage Schema
7.5%
3/40 include FAQPage vs. 33/40 (82.5%) with base JSON-LD
Silent Edge WAF 403 Blocks
5.0%
2/40 return HTTP 403 to GPTBot despite 0/40 blocking in /robots.txt
**Cross-Vertical Readiness Matrix (N = 40 Live-Probed Public Domains)**Probed 2026-11-16 · 6–8 live HTTP requests
| Industry Cohort | Sample | Custom /llms.txt | Auto Boilerplate | Custom /agents.md | Base JSON-LD | FAQPage Schema | WAF HTTP 403 |
|---|---|---|---|---|---|---|---|
| B2B SaaS & Developer Tools | 10 | 10/10 (100%) | 0/10 (0%) | 2/10 (20%) | 9/10 (90%) | 0/10 (0%) | 0/10 (0%) |
| Shopify & D2C Commerce | 10 | 1/10 (10%) | 6/10 (60%) | 0/10 (0%) | 7/10 (70%) | 0/10 (0%) | 1/10 (10%) |
| SEO, Content & Digital Agencies | 10 | 4/10 (40%) | 1/10 (10%) | 0/10 (0%) | 8/10 (80%) | 2/10 (20%) | 0/10 (0%) |
| Multi-Location Healthcare, Legal & Home Services | 10 | 5/10 (50%) | 0/10 (0%) | 0/10 (0%) | 9/10 (90%) | 1/10 (10%) | 1/10 (10%) |
| All Cohorts Combined | 40 | 20/40 (50.0%) | 7/40 (17.5%) | 2/40 (5.0%) | 33/40 (82.5%) | 3/40 (7.5%) | 2/40 (5.0%) |
KEY ARCHITECTURAL TAKEAWAYS
Four protocol gaps separating cited brands from invisible domains
80% UCP ACTIVE VS. 0% CUSTOM /AGENTS.MD
The 80-Point Shopify Agent Truth Gap
Eight of ten probed D2C commerce brands (80.0%) expose active Shopify /api/ucp/mcp commerce endpoints, yet 0 of 10 (0.0%) have customized /agents.md and only 1 of 10 (10.0%) serves a custom /llms.txt. Six of ten serve unedited stock Shopify Shop-skill boilerplate lacking free-shipping thresholds, return windows, and bundle exclusions.
0% ROBOTS.TXT BLOCK · 5.0% WAF HTTP 403
Open /robots.txt Files Mask Silent CDN 403 Blocks
Zero of the 40 probed domains (0.0%) disallow GPTBot or PerplexityBot at root inside /robots.txt, yet 2 of 40 (5.0% — ridge.com and westerndental.com) return HTTP 403 Forbidden when probed with real GPTBot or PerplexityBot User-Agent headers at the CDN edge.
50.0% CURATED · 17.5% BOILERPLATE · 7.5% SOFT-404
One-Third of HTTP 200 /llms.txt Responses Are Uncurated or HTML Soft-404s
Thirty domains returned HTTP 200 on GET /llms.txt, but Content-Type and body inspection revealed 3 HTML SPA soft-404 pages (7.5% of the cohort) and 7 auto-generated Shopify or Rank Math boilerplate dumps (17.5%). Developer SaaS leads genuine curation at 100% (10/10), compared with 40% of SEO agencies and 10% of D2C brands.
82.5% BASE JSON-LD · ONLY 7.5% FAQPAGE
Base Organization Schema Is Common; Answer-First FAQPage Schema Is Rare
While 33 of 40 domains (82.5%) render at least one application/ld+json block on their homepage, only 3 of 40 (7.5%) include FAQPage structured Q&A pairs—leaving ChatGPT Search, Gemini, and Perplexity to infer pricing tiers, service areas, and SLAs from third-party directories.
DOMAIN-LEVEL HTTP DATASET · REPRODUCIBLE TELEMETRY
Filter all 40 probed domains by industry vertical
SHOWING 40 OF 40 DOMAINS
| Domain | Vertical | /llms.txt | /agents.md | Homepage JSON-LD | GPTBot / PerplexityBot | Shopify UCP | Live Check |
|---|---|---|---|---|---|---|---|
| stripe.com | B2B SaaS | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| vercel.com | B2B SaaS | CUSTOM CURATED | CUSTOM ENRICHED | 3 nodes | HTTP 200 OPEN | — | Re-Probe → |
| supabase.com | B2B SaaS | CUSTOM CURATED | MISSING | 4 nodes | HTTP 200 OPEN | — | Re-Probe → |
| resend.com | B2B SaaS | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| linear.app | B2B SaaS | CUSTOM CURATED | CUSTOM ENRICHED | 0 nodes | HTTP 200 OPEN | — | Re-Probe → |
| cloudflare.com | B2B SaaS | CUSTOM CURATED | MISSING | 3 nodes | HTTP 200 OPEN | — | Re-Probe → |
| posthog.com | B2B SaaS | CUSTOM CURATED | MISSING | 2 nodes | HTTP 200 OPEN | — | Re-Probe → |
| clerk.com | B2B SaaS | CUSTOM CURATED | MISSING | 2 nodes | HTTP 200 OPEN | — | Re-Probe → |
| dub.co | B2B SaaS | CUSTOM CURATED | MISSING | 2 nodes | HTTP 200 OPEN | — | Re-Probe → |
| cal.com | B2B SaaS | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| allbirds.com | Shopify D2C | AUTO BOILERPLATE | STOCK SHOPIFY | 0 nodes | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| gymshark.com | Shopify D2C | AUTO BOILERPLATE | STOCK SHOPIFY | 0 nodes | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| brooklinen.com | Shopify D2C | AUTO BOILERPLATE | STOCK SHOPIFY | 4 nodes | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| chubbies.com | Shopify D2C | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| ridge.com | Shopify D2C | AUTO BOILERPLATE | MISSING | 0 nodes | BOTH WAF 403 | UNAVAILABLE | Re-Probe → |
| cutsclothing.com | Shopify D2C | MISSING | MISSING | 2 nodes | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| ruggable.com | Shopify D2C | HTML SOFT-404 | MISSING | 1 node | HTTP 200 OPEN | UNAVAILABLE | Re-Probe → |
| trueclassictees.com | Shopify D2C | AUTO BOILERPLATE | STOCK SHOPIFY | 3 nodes | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| carawayhome.com | Shopify D2C | MISSING | MISSING | 2 nodes | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| beardbrand.com | Shopify D2C | AUTO BOILERPLATE | STOCK SHOPIFY | 2 nodes | HTTP 200 OPEN | UCP ACTIVE | Re-Probe → |
| siegemedia.com | SEO Agency | MISSING | MISSING | 2 nodes | HTTP 200 OPEN | — | Re-Probe → |
| animalz.co | SEO Agency | MISSING | MISSING | 0 nodes | HTTP 200 OPEN | — | Re-Probe → |
| omniscientdigital.com | SEO Agency | MISSING | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| singlegrain.com | SEO Agency | CUSTOM CURATED | MISSING | 3 nodes · FAQPage | HTTP 200 OPEN | — | Re-Probe → |
| directiveconsulting.com | SEO Agency | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| victorious.com | SEO Agency | MISSING | MISSING | 2 nodes · FAQPage | HTTP 200 OPEN | — | Re-Probe → |
| webmechanix.com | SEO Agency | AUTO BOILERPLATE | MISSING | 5 nodes | HTTP 200 OPEN | — | Re-Probe → |
| klientboost.com | SEO Agency | HTML SOFT-404 | MISSING | 0 nodes | HTTP 200 OPEN | — | Re-Probe → |
| powerdigitalmarketing.com | SEO Agency | CUSTOM CURATED | MISSING | 2 nodes | HTTP 200 OPEN | — | Re-Probe → |
| webfx.com | SEO Agency | CUSTOM CURATED | MISSING | 3 nodes | HTTP 200 OPEN | — | Re-Probe → |
| aspendental.com | Multi-Location | HTML SOFT-404 | MISSING | 2 nodes | HTTP 200 OPEN | — | Re-Probe → |
| hellotend.com | Multi-Location | MISSING | MISSING | 36 nodes | HTTP 200 OPEN | — | Re-Probe → |
| asktia.com | Multi-Location | MISSING | MISSING | 1 node · FAQPage | HTTP 200 OPEN | — | Re-Probe → |
| onemedical.com | Multi-Location | MISSING | MISSING | 0 nodes | HTTP 200 OPEN | — | Re-Probe → |
| forthepeople.com | Multi-Location | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| rotorooter.com | Multi-Location | CUSTOM CURATED | MISSING | 3 nodes | HTTP 200 OPEN | — | Re-Probe → |
| servpro.com | Multi-Location | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| mrrooter.com | Multi-Location | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| aireserv.com | Multi-Location | CUSTOM CURATED | MISSING | 1 node | HTTP 200 OPEN | — | Re-Probe → |
| westerndental.com | Multi-Location | MISSING | MISSING | 2 nodes | GPTBOT WAF 403 | — | Re-Probe → |
METHODOLOGY & REPRODUCIBILITY SPECIFICATION
How every domain was probed and classified
Each of the 40 public domains was probed directly over HTTPS using the same deterministic parser powering Verisource's production scanner: (1) GET /robots.txt parsed for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended directives; (2) homepage requests sent with explicit GPTBot/1.2 and PerplexityBot/1.0 User-Agent headers to catch silent CDN HTTP 403 / 429 blocks; (3) GET /llms.txt and GET /agents.md inspected for text/html SPA soft-404s and stock Shopify (shop.app/SKILL.md) or SEO plugin boilerplate; (4) homepage application/ld+json script nodes parsed for FAQPage entities; and (5) JSON-RPC tools/list handshakes tested against /api/ucp/mcp and /api/mcp on D2C commerce stores.
NOVEMBER 2026 ADDENDUM · DEVELOPER INFRASTRUCTURE COHORT (N = 10)
90% of DevTools ship /llms.txt. Only 20% ship /agents.md.
In a follow-up live HTTP probe across 10 developer infrastructure and B2B SaaS domains (trigger.dev, inngest.com, turso.tech, tinybird.co, knock.app, workos.com, stytch.com, upstash.com, highlight.io, and infisical.com), 9 of 10 (90.0%) served a valid /llms.txt directory, yet only 2 of 10 (20.0% — tinybird.co and infisical.com) published a custom /agents.md pricing and limits guardrail file, and only 1 of 10 (10.0% — upstash.com) shipped all 3 JSON-LD pillars.
| Protocol / Schema Signal | DevTools Cohort (N = 10) | Commercial Impact |
|---|---|---|
| /llms.txt (HTTP 200) | 90.0% (9 / 10) | High documentation-link adoption across modern DevTools |
| Custom /agents.md | 20.0% (2 / 10) | 80% leave pricing tiers & rate limits unguarded against AI hallucination |
| Full 3-Pillar JSON-LD | 10.0% (1 / 10) | 90% omit FAQPage or Offer pricing nodes on root pages |
BENCHMARK FAQ
Common questions about the 2026 AI Search Readiness dataset
What percentage of websites have /llms.txt and /agents.md in November 2026?
In our live HTTP readiness probe across 40 public domains spanning B2B SaaS, Shopify D2C, SEO agencies, and multi-location services, 50.0% (20/40) serve a custom-curated /llms.txt file, another 17.5% (7/40) expose auto-generated Shopify or SEO-plugin boilerplate, 7.5% (3/40) return HTML soft-404s with HTTP 200 status, and only 5.0% (2/40 — vercel.com and linear.app) publish a custom /agents.md specification.
Why do Shopify stores show 80% /api/ucp/mcp adoption but 0% custom /agents.md adoption?
Shopify rolled out platform-level /api/ucp/mcp endpoints and stock /agents.md files pointing to https://shop.app/SKILL.md across modern storefronts, which explains why 8 of 10 probed D2C stores respond on /api/ucp/mcp. However, stock Shopify files omit brand-specific free-shipping thresholds, return windows, and bundle exclusions unless the merchant overrides them via templates/agents.md.liquid or /pages/ai-facts.
Can a domain block GPTBot or PerplexityBot even when /robots.txt allows them?
Yes. Zero of the 40 probed domains disallowed GPTBot or PerplexityBot at root in /robots.txt, yet 5.0% (2/40 — ridge.com and westerndental.com) returned HTTP 403 Forbidden at the CDN/WAF layer when requested with GPTBot or PerplexityBot User-Agent headers. Testing both /robots.txt and simulated crawler User-Agent HTTP status codes is required.
BENCHMARK YOUR OWN DOMAIN IN 60 SECONDS
See how your website stacks up against the 40-domain 2026 readiness cohort.
$0 TODAY · 7-DAY FREE TRIAL