01TL;DR
shop.usaclean.com is currently invisible to AI assistants — and the cause is a switch, not a strategy problem. Cloudflare returns HTTP 403 to AI crawlers at the network edge, before robots.txt is even consulted. ChatGPT, Claude, and Perplexity cannot retrieve or cite any page on the store.
Everything else this audit found is ordinary, well-understood content work — most of it already scoped by the existing SEO playbook. The decision on the table: approve Stage 0 ("Turn AI visibility on") this week. It is hours of configuration work, and nothing else in this audit matters until it lands.
Decision of record: none yet — a client go-ahead on Stage 0 gets the next PD-NNN id in the decision log.
| Layer | What it covers | Grade | Headline finding | Fixed by |
|---|---|---|---|---|
| 1. Crawler access | Can AI engines fetch pages at all? | Blocked | Cloudflare 403s AI bots at the edge; robots.txt says two contradictory things | Stage 0 |
| 2. Structured data | Can machines parse what a page means? | B− | Product pages are strong; guides and About have zero schema | Stage 1 |
| 3. Extractable content | Is there a quotable, answer-shaped fact to lift? | C | Content hub exists but isn't answer-first; credibility stats aren't in crawlable HTML | Stage 1 |
| 4. Measurement | Would anyone know if any of this worked? | Absent | No AI-referral tracking, no citation monitoring, no re-check cadence | Stage 2 |
02What GEO/AEO is — and why it matters in July 2026
GEO (Generative Engine Optimization) and AEO (Answer Engine Optimization) are the practice of making a site visible, comprehensible, and quotable to AI answer engines — ChatGPT, Claude, Perplexity, Gemini, and Google's AI Overviews. Where classic SEO earns a ranked blue link, GEO/AEO earns a citation inside the answer — "USA-CLEAN sells this squeegee blade, ships same day" — at the moment a facilities manager asks an assistant what to buy.
Roughly 80% of GEO/AEO is executing classic SEO well: crawlable pages, clear structure, genuinely useful content. Independent analyses consistently find that the large majority of URLs cited in AI answers already rank in the traditional top 10 for the same query. The remaining 20% is genuinely new, and falls into four buckets — exactly the four layers this audit graded:
- Crawler access. A new population of bots with their own user agents and robots.txt conventions — and a new class of accidental blocking: CDN and security products (Cloudflare in particular) now ship "block AI bots" defaults that silently override a site's stated policy. Industry measurement puts accidental AI-crawler blocking at roughly 27–30% of B2B/ecommerce sites.
- Three bot classes that deserve different policies. Training crawlers (GPTBot-as-trainer, CCBot, Bytespider) collect data to train future models — blocking them is a legitimate IP-policy choice. Retrieval/search indexers (OAI-SearchBot, PerplexityBot, ClaudeBot) build the answer-engine indexes that produce citations and referral traffic — blocking them is invisibility. User-triggered fetchers (ChatGPT-User, Claude-Web) act on behalf of a live human asking about your site right now. Conflating the three is the most common GEO policy error.
- Entity clarity. AI systems extract facts, not pages. A page whose first sentence is the answer ("The Tennant T7 uses a 32-inch squeegee blade") gets quoted; a page that opens with brand framing does not.
- A new measurement surface. Rank trackers and Search Console don't capture AI referral sessions or citation rate. GA4 added a native "AI Assistant" channel grouping in May 2026; citation monitoring requires deliberately running a panel of buyer-intent prompts on a schedule.
AI referral traffic is still small — well under 2% of sessions for most sites — but it is the fastest-growing acquisition channel measured, with year-over-year growth around 10× while organic search grows single digits. The honest framing is cheap insurance on a compounding channel, not a promised conversion lift (the conversion-quality evidence is genuinely mixed — see §11).
03Method & evidence
All tests were run externally on 2026-07-08 against the live site, with no access to Google Search Console, GA4, server logs, or the Cloudflare dashboard. What was tested:
- User-agent matrix — identical requests to
https://shop.usaclean.com/with four user-agent strings (a standard Chrome browser UA, GPTBot, ClaudeBot, PerplexityBot), recording HTTP status codes. - Policy files — full capture of
robots.txt; probe forllms.txt; sitemap index enumeration (xmlsitemap.php). - Structured-data extraction — JSON-LD
@typeinventory on the homepage, a representative product page (/Battery-tray-154-2603/), a brand/category page (/parts-and-supplies/tennant/), a content-hub guide (/floor-pad-selection-guide/), and the About page. - On-page signals —
<title>, meta description, canonical, H1 on the same pages. - Redirect behavior —
/home/(the June playbook's duplicate-homepage finding) and the brand URL paths (/tennant/vs/parts-and-supplies/tennant/).
The user-agent tests were spoofed UAs from a non-bot IP. A WAF that verifies bot identity by IP range could in principle treat genuine crawler traffic differently. But a 403 on UA string alone is exactly how Cloudflare's AI-bot blocking behaves, and the robots.txt corroborates a deliberate-by-default block — so the finding stands with high confidence. Confirming from inside the Cloudflare dashboard is itself a Stage 0 action item.
04Layer 1 — AI crawler access · Blocked
The 403 test
Same URL, four user agents:
Captured 2026-07-08. Cloudflare blocks AI crawlers at the network edge — the request never reaches the site, and robots.txt never enters the picture.
robots.txt argues with itself
The file contains two contradictory policies:
Abridged from the live robots.txt, captured 2026-07-08.
The Cloudflare-managed block (injected by Cloudflare's "manage AI bots" feature) fully disallows the major AI training and retrieval crawlers. The origin block — evidently a deliberate, hand-maintained policy — welcomes the same bots with a polite crawl-delay and standard checkout exclusions. The retrieval crawlers that power ChatGPT search and Perplexity citations are allowed on paper and 403'd in practice.
Retrieval invisibility (the emergency): ChatGPT, Claude, and Perplexity cannot fetch, index, or cite any page. When a buyer asks any of them where to get a Tennant part, USA-CLEAN structurally cannot be the answer.
Gemini exclusion (a policy decision made by accident): Google-Extended is disallowed, which removes the site from Gemini's grounding corpus. Google AI Overviews are unaffected — they draw on the standard Googlebot index — which is why AI Overviews is the only AI surface where the store is currently visible.
llms.txt
https://shop.usaclean.com/llms.txt returns 404. For calibration: Ahrefs' May 2026 analysis of 137,000 domains found ~97% of llms.txt files receive zero AI-bot requests, and Google has said its systems ignore them. Ship it as a checkbox (it costs an afternoon), not as a strategy.
05Layer 2 — Structured data · B−
The schema picture is better than the June playbook assumed — several of its recommendations have been executed since:
What's good
- Product pages carry
Product+Offer(price,InStockavailability) +BreadcrumbListJSON-LD. This is the highest-value schema on the site and it's done. - Homepage carries
Organization,WebSite+SearchAction, andStore(with address) — the core entity block AI systems use to understand who the business is.
What's missing
- Guide pages have zero structured data.
/floor-pad-selection-guide/— exactly the content AI engines quote for advice queries — has noArticle, noauthor, no dates, noFAQPage. Its<title>is bare ("Floor Pad Selection Guide", no brand). - Category pages carry only
BreadcrumbList. NoFAQPage(none have Q&A content yet — see Layer 3). - The About page has no schema at all — no
AboutPage, noOrganizationreinforcement. - No
AggregateRating— correctly absent (product reviews haven't launched; playbook Short Term 4 is still open). Add it when reviews exist, not before.
Google's own May 2026 guidance says schema markup drives rich-result eligibility, not AI Overview citation. Schema helps AI systems parse meaning confidently — especially prices, availability, and authorship — but it is not a GEO silver bullet and shouldn't be sold as one.
06Layer 3 — Extractable content · C
AI engines lift answer-shaped facts. The site's content gives them surprisingly little to lift:
- PDP meta descriptions are raw part-number dumps. The audited page's description is literally
154-2603 TRAY, BATTERY. (Playbook Short Term 1 — not yet executed.) - The homepage
<title>is still just "USA-CLEAN" — four characters, no category terms. (Playbook Quick Hit 3 — not executed, though the H1 was fixed to "The Nation's Largest Selection of Cleaning Equipment Parts".) - The credibility stats are invisible. The "600 technicians / 40 years" claims — USA-CLEAN's strongest trust differentiator, and exactly what an AI engine wants when deciding whether to recommend a vendor — are not present in extractable HTML text on the homepage or About page.
- A content hub partially exists — battery guides, a floor-pad selection guide, troubleshooting guides are in the pages sitemap — but none of it is extraction-ready: no schema (§05), no bylines, no dates, and copy that isn't answer-first.
- Fixed since June, worth crediting: the brand-path duplication is resolved (
/tennant/301s to/parts-and-supplies/tennant/— the direction the addendum argued should be checked for equity, rather than the playbook's shorter-URL preference). The duplicate homepage is gone, though/home/now returns 404 rather than a 301 — the duplication is killed but any residual link equity is being dropped instead of consolidated.
07Layer 4 — Measurement · Absent
Nothing measures whether any of this works, or would warn if it regressed:
- No AI-referral segmentation. GA4 has offered a native "AI Assistant" channel grouping (ChatGPT, Gemini, Claude) since May 2026; Perplexity needs one manual channel definition. Every week this isn't configured is baseline data lost.
- No citation monitoring. Rank trackers answer "where do we sit in blue links," not "does ChatGPT recommend USA-CLEAN when a technician asks about a Tennant T7 squeegee blade." That requires a fixed panel of buyer-intent prompts run across the major engines on a schedule.
- No re-check cadence. The Cloudflare block demonstrates why this matters: bot lists and CDN defaults shift under your feet. A quarterly crawler-access check (the §04 user-agent matrix takes ten minutes) keeps a one-time fix fixed.
08Assessment of the June playbook & July addendum
Are the recommendations solid? Yes — with the addendum's caveats confirmed.
The June 2026 playbook is a sound technical SEO program: correctly sequenced (indexation → content → authority), built on site-specific findings rather than template advice, with reachable backlink targets and a correctly-framed content-hub moat. Its weaknesses are the ones the addendum identified: overconfident timelines ("within weeks"), a canonical-direction call made on URL length rather than link equity, schema implicitly oversold as an AI-visibility play, no AI-crawler access check anywhere in its quick hits, and a measurement stack that only sees Google organic.
The July addendum (the GEO/AEO review) holds up well against this audit. Its ~80/20 framing — most AI-search value comes from executing the existing playbook; the genuinely-new work is crawler access, entity clarity, and measurement — matches what this audit found on the ground. Most notably, its single most concrete warning is empirically confirmed live: it flagged Cloudflare's "block AI bots" default silently overriding origin robots.txt as the top-priority check, and that is precisely the site's current state (§04). Its skeptical calibration on llms.txt and its honest, two-sided read of the conversion evidence are the right way to present this to a client.
State of play — executed since the June playbook
| Playbook item | Status July 2026 |
|---|---|
| PDP structured data (Product/Offer/Breadcrumb) | ✓ Done |
| Brand-path canonical consolidation | ✓ Done (to /parts-and-supplies/…, with 301s) |
| Homepage H1 | ✓ Done |
Duplicate homepage /home/ | Partial Removed, but 404 instead of 301 |
| Homepage title tag | ✗ Still "USA-CLEAN" |
| PDP meta descriptions | ✗ Still part-number dumps |
| Product reviews / AggregateRating | ✗ Not launched |
| Category copy + FAQ blocks | Partial Copy exists on major brand pages; no FAQ |
| AI crawler access (addendum quick hit) | ✗ Blocked — the headline gap |
09The full recommendation set
Impact ratings are for AI visibility specifically; several items double as classic SEO fixes (marked †).
| # | Recommendation | Layer | Impact | Effort | Stage | Where it lands |
|---|---|---|---|---|---|---|
| R1 | Unblock AI retrieval crawlers at the Cloudflare edge | 1 | Critical | Hours | 0 | Cloudflare dashboard |
| R2 | Reconcile robots.txt into one deliberate policy (retrieval ≠ training) | 1 | High | Hours | 0 | Cloudflare + BigCommerce |
| R3 | Decide Google-Extended explicitly (it gates Gemini grounding) | 1 | Medium | Minutes | 0 | robots.txt |
| R4 | Ship llms.txt — budgeted as a checkbox | 1 | Low | Hours | 0 | Theme static assets |
| R5 | Entity-clarity copy pass: answer-first openings on category/guide/About pages | 3 | High | Medium | 1 | Content + Stencil theme |
| R6 | Put "600 technicians / 40 years" into crawlable HTML (homepage, About, guide footers) | 3 | Medium | Low | 1 | Stencil theme |
| R7 | Article + author + dates schema on all guides; brand the guide titles | 2 | High | Low–Med | 1 | Stencil theme |
| R8 | Q&A blocks + FAQPage schema on top 20 categories | 2+3 | High | Medium | 1 | Content + Stencil theme |
| R9 | PDP meta-description template + custom top 500 † | 3 | Medium | Medium | 1 | BigCommerce admin |
| R10 | Homepage title tag rewrite † | 3 | Medium | Minutes | 1 | BigCommerce admin |
| R11 | Convert /home/ 404 → 301 † | 3 | Low | Minutes | 1 | BigCommerce admin |
| R12 | GA4 "AI Assistant" channel group + manual Perplexity channel | 4 | High* | Hours | 2 | GA4 |
| R13 | Monthly citation panel: ~10 buyer-intent prompts across ChatGPT/Claude/Perplexity/Gemini | 4 | Medium | Low, recurring | 2 | Runbook |
| R14 | Quarterly crawler-access re-check (the §04 UA matrix) | 4 | Medium | Minutes, recurring | 2 | Runbook |
| R15 | Video transcripts + VideoObject schema, riding on the planned YouTube repurposing | 2 | Medium | Rides on existing work | 2 | Content + Stencil theme |
* High for measurement value — it's what makes every other row provable.
10The staged plan
Stage 0 — Turn AI visibility on · week 1, hours of work
R1–R4. Unblock retrieval crawlers in the Cloudflare dashboard (Security → Bots → review the AI-bot blocking setting and the managed-robots.txt injection); rewrite robots.txt as one deliberate policy; make the Google-Extended call consciously; ship llms.txt.
Retrieval is not training. Keeping Content-Signal: ai-train=no and blocking training-only crawlers (CCBot, Bytespider) is a defensible IP stance — blocking OAI-SearchBot and PerplexityBot is simply invisibility on a public commerce catalog. Nothing else in this audit matters until Stage 0 lands.
Advance trigger: the §04 UA matrix returns 200s for retrieval bots.
Stage 1 — Make content quotable · weeks 2–6
R5–R11. The copy pass leads with the flattest extractable fact ("what this category covers, which machines it fits") before brand framing; the credibility stats go into real HTML; guides get authorship, dates, and Article schema; the top 20 categories get 3–5 real buyer questions with FAQPage markup; the PDP meta-description and homepage-title fixes clear the last open SEO quick hits.
Advance trigger: guides and top categories pass schema validation and read answer-first.
Stage 2 — Measure & compound · months 2–3, then ongoing
R12–R15. GA4 channel setup first (every week without it loses baseline), then the monthly citation panel and the quarterly access re-check as standing runbook items; transcripts and VideoObject ride along when the playbook's YouTube-repurposing initiative happens. This stage never "completes" — it's the maintenance layer that keeps Stages 0–1 true.
11Measurement framework & what to honestly expect
Add three lines to the measurement stack (these extend, not replace, the playbook's Longer Term 6):
- AI referral sessions & conversions by engine — GA4 native channel group for ChatGPT/Gemini/Claude; one manual channel definition for Perplexity.
- Citation rate — the fixed prompt panel (R13), logged monthly: does the engine mention USA-CLEAN, does it link, what does it say.
- Crawler-access status — the quarterly UA matrix (R14), because CDN defaults change silently.
What to expect, stated honestly
- AI referral volume is still small in absolute terms — under 2% of sessions for most sites. One large 2025 ecommerce study measured ChatGPT referrals growing over 1,000% year-over-year while non-branded organic grew 17% — but from a base where organic still dominates discovery by roughly 47:1.
- Conversion quality is genuinely contested: Adobe's trillion-visit retail dataset (Q1 2026) found AI-referred traffic converting better than other channels; an independent academic study over ~$20B of first-party transactions (Kaiser & Schulze, Oct 2025) found ChatGPT referrals converting worse than organic. Attribution error cuts both ways (copied links strip referrers and get counted as direct).
- For B2B specifically — closer to USA-CLEAN's buyer than the DTC data — the available evidence leans toward longer sessions and somewhat higher conversion, on small samples.
This work protects and modestly compounds the existing SEO investment for a channel growing fast off a small base. Don't attach a percentage to it. The cost of Stage 0 is hours; the cost of skipping it is structural absence from the fastest-growing referral channel.
12Where the fixes land
No single owner covers this work — four surfaces, four access needs:
| Surface | What lands there | Access needed |
|---|---|---|
| Cloudflare dashboard | R1 unblock, R2 managed-robots.txt reconciliation, R14 verification | Cloudflare admin |
| BigCommerce admin | R2 origin robots.txt, R9 meta templates, R10 title, R11 301 | Store control panel |
| Stencil theme | R4 llms.txt, R6 stats markup, R7 Article schema, R8 FAQ blocks, R15 VideoObject | Theme repo + deploy |
| GA4 | R12 channels, R13 panel logging | Analytics admin |
13Catalyst / replatform note
A future move to Catalyst (headless React storefront) changes where these fixes are implemented — robots.txt and llms.txt become framework routes, JSON-LD becomes React components instead of Stencil partials — but changes nothing about whether they're needed. Every recommendation here is platform-portable: the copy, the schema shapes, the crawler policy, and the measurement stack all carry over intact. Nothing in this audit is a reason to accelerate or delay replatforming; do the fixes now and they move with you.
14Sources
Live evidence (all captured 2026-07-08)
Client documents assessed (not in this repo)
- USA-CLEAN SEO Detailed Playbook — June 2026 (16 initiatives: 4 quick hits, 6 short-term, 6 longer-term)
- USA-CLEAN SEO Strategy Addendum — GEO/AEO Review — July 2026
Industry references (as cited in the addendum, current to June 2026)
- Google Search Central — first official guidance on AI features in Search (May 2026): schema ≠ AI-citation lever; llms.txt not read by Google
- Ahrefs — llms.txt analysis across 137,000 domains (May 2026): ~97% receive zero AI-bot requests
- Cloudflare Radar — AI-crawler robots.txt analysis (2026): ~27–30% of B2B/ecommerce sites accidentally block AI crawlers
- GA4 release notes — native "AI Assistant" channel grouping (May 2026)
- Adobe Digital Insights — Q1 2026 AI traffic report (AI referrals convert better)
- Kaiser & Schulze — academic study of ChatGPT ecommerce referrals (Oct 2025; converts worse)
- Visibility Labs / Search Engine Land — 2025 ecommerce AI-referral growth study