AI Shopping Agents Are Getting Blocked: How to Let Them In

AI shopping agents can now browse, compare and buy on a person's behalf, and a growing number of stores are turning them away at the door. TechCrunch reported on October 6 that users of Meta's Muse and similar agents hit blocks on Yelp, eBay, Zillow and several airlines, and that Amazon began blocking Muse from its retail site in September.

The detail that should worry you is Walmart's. Walmart is a Muse partner, yet its own tests failed because a human-verification button broke the agent's flow, and the company says that wasn't intentional. If you sell online, your bot stack may already have made a decision about agents that nobody on your team actually made.

TL;DR

AI shopping agents are getting blocked by CAPTCHAs, anti-scraping rules and CDN defaults, often by accident rather than policy.
Cloudflare's new defaults (from September 15) block training and agent use on pages with ads for sites that never touched their settings, while still allowing search.
Decide your agent policy per page type, check your CDN and bot settings, and make product pages machine-readable so the agents you do let in can finish the job.

What happened: AI shopping agents are hitting walls

Agents such as Muse, Instinct and ChatGPT's Dots can book flights and order groceries, but many tasks fail at the website. The reasons companies gave TechCrunch are all different:

Walmart: a "confirm you're human" step broke the agent. Not intentional.
Yelp: no non-human traffic without a paid data licensing agreement.
eBay: restricts unauthorized agents and actions like scraping and model training, but says it doesn't block all shopping agents.
Zillow: allows agents acting for consumers, but admits its anti-scraping measures sometimes catch them anyway.

On top of that, some people suspect CDNs. Cloudflare announced in July that from September 15, new sites and existing free accounts that hadn't changed their settings would default to allowing search but blocking "training and agent use for pages with ads." Crawlers that mix search, agent use and training without letting owners choose get blocked on those pages too.

Meanwhile, Meta and Sierra announced the Personal Agent Protocol on October 6 with Shopify, Stripe and Walmart among the partners. A v0.1 spec is planned for later this month.

Why this matters for websites

Agent-driven sales are still small. Checkout.com's 2026 research found that only 3% of merchant-reported transactions involve AI agents, even though 89% of merchants say they're preparing for agentic commerce. So nobody is losing half their revenue to a CAPTCHA this week.

The problem is that blocks are quiet and sticky. A shopper whose agent fails on your site doesn't see an error page from you. They see "I couldn't complete that" from their assistant, and the assistant moves to a store that worked. Meta put it bluntly in the TechCrunch story: turning away a personal agent means turning away the customer behind it.

There's also a readiness gap on the other side. Rezolve reported that 97 of the 100 largest US retailers it assessed had no publicly identifiable checkout route for AI shopping agents. That leaves a real opening for smaller stores.

Where the blocks actually come from

Most site owners reach for robots.txt first. That's the wrong lever for this problem. Robots.txt was built for crawlers, and none of the blocks in the TechCrunch report were traced to it. The real gatekeepers sit further down the stack.

Layer	What blocks agents	How to check
CDN AI defaults	Category rules for training, search and agent use	Review AI crawler settings in your CDN dashboard
Bot management	JS challenges, CAPTCHAs, "are you human" buttons	Look for challenge events on product and cart URLs
WAF and rate limits	Fast, repetitive requests from one session	Check firewall logs for 403 and 429 spikes
Account security	Suspensions for automated logins	Review fraud rules that flag non-human sessions
Terms of use	Blanket "no robots" clauses	Read your own ToS with agents in mind

Robots.txt still matters, just for something else

Keep crawler rules explicit so AI search can still cite you, even if you opt out of training (User-agent: GPTBot / Disallow: /). That controls indexing and training. It does nothing for a checkout session a person's agent starts.

Verified identity is where this is heading

Cloudflare's Web Bot Auth lets an agent sign its requests using HTTP Message Signatures (RFC 9421), with keys published at /.well-known/http-message-signatures-directory:

Signature-Agent: "https://agent.example.com"
Signature-Input: sig1=("@authority" "signature-agent");created=1760400000;expires=1760400300;keyid="ba3e64...";tag="web-bot-auth"
Signature: sig1=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQ=:

If your bot layer can verify that, you can allow signed shopping agents through checkout while still challenging anonymous scrapers. PAP takes a similar line at the business level: agents identify themselves, start a guest or OAuth session, and act within permissions you set.

What it means for site owners

The practical shift is that "AI bots" is no longer one category. Cloudflare's own framing splits intent three ways: search, training and agent use. You probably want different answers for each, and maybe different answers per page.

A sensible default for most stores: allow search crawlers everywhere, because that's how you get cited in AI answers. Decide on training separately, since it's a licensing question, not a sales one. Then treat agents acting for a shopper like a slightly unusual customer: let them read product, pricing and availability pages, and gate account actions behind the same login a human uses.

Watch the ads qualifier too. If your product pages carry ads and picked up Cloudflare's new defaults, agent use may be blocked on exactly the pages you want agents to read. And make sure your "are you human" step has a path a signed or logged-in agent can pass.

How to let AI shopping agents in without opening the floodgates

Audit your CDN settings. Open your AI crawler or bot settings and confirm what search, training and agent traffic get on product, cart and checkout paths. Don't assume the defaults match your intent.
Test with a real agent. Ask an agent to add a product to a cart and stop before payment. Note where it fails.
Grep your logs. Look for 403s, 429s and challenge pages on /products/, /cart and /checkout from automated sessions.
Make product data machine-readable. Agents need price, availability and identifiers without guessing from layout:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Trail Runner 2",
  "sku": "TR2-BLK-42",
  "offers": {
    "@type": "Offer",
    "price": "129.00",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  }
}

Label your forms and buttons. Size pickers, quantity fields and "Add to cart" buttons need real labels and ARIA, not unlabeled divs. A free AgentReady scan checks exactly this: its Browser Readiness pillar flags missing JSON-LD, weak ARIA and landmarks, and forms that aren't exposed as WebMCP tools.
Publish an API route. PAP names MCP and OpenAPI as ways agents can act through your APIs instead of clicking through pages. A documented spec or a discoverable MCP server is the cleanest path.

Check your own site

Agent access touches two parts of the Agent Readiness Score. Browser Readiness (40%) measures whether an agent can read your products and use your forms once it's in, and MCP Presence (20%) checks whether agents can connect to you without scraping at all. A low score there usually means agents that get past your CDN still stall on unlabeled forms or missing product data. Our guides to schema markup for AI agents and AI browser agents cover the fixes. A single-page scan is free and takes about a minute. Scan your site free.

Key takeaways

Most agent blocks come from bot management, CAPTCHAs and CDN defaults, not from robots.txt.
Cloudflare's September 15 defaults block agent use on pages with ads for sites that never changed their settings.
Treat search, training and agent use as three separate decisions.
Verified identity (Web Bot Auth, PAP) is how you'll let good agents in and keep scrapers out.
Agents that get through still need clean product JSON-LD, labeled forms and an API or MCP route.

FAQ

Are websites blocking AI shopping agents?

Yes, some are. TechCrunch reported in October 2026 that users of agents like Meta's Muse hit blocks on Yelp, eBay, Zillow and several airlines, and Amazon began blocking Muse in September. Many blocks are side effects of anti-bot tools and CAPTCHAs, not deliberate policy, which is what Walmart said happened on its site.

Does robots.txt block AI shopping agents?

Not reliably, and that's not what it's for. Robots.txt tells crawlers what to index or collect. An agent completing a purchase for a logged-in shopper behaves more like a user session, and the blocks reported so far came from bot management, CAPTCHAs and policies. Use robots.txt for search and training crawlers, and your CDN or bot settings for agents.

Did Cloudflare start blocking AI agents by default?

For some sites, yes. Cloudflare said that from September 15, 2026, new sites and existing free accounts that hadn't changed their settings would allow search but block training and agent use on pages with ads. You can change these settings in the Cloudflare dashboard at any time, so check what your product pages currently get.

How do I let AI shopping agents buy from my store safely?

Allow agents on product and cart pages, keep account actions behind normal login, and prefer verified agents that sign their requests with Web Bot Auth. Then make the pages agent-friendly: Product JSON-LD with price and availability, labeled form fields and buttons, and a documented API or MCP server so agents don't have to scrape.

Pick one product page today, ask an AI agent to add that item to a cart, and watch where it gets stuck. Then fix the layer that stopped it, whether that's a CDN rule, a challenge page or an unlabeled button. Run a free AgentReady scan.

AI Shopping Agents Are Getting Blocked. Is Your Store Next?