Meta-ExternalAgent: robots.txt Rules for Meta's AI Bots

Meta's crawlers are now the biggest source of AI bot traffic on at least one large bot-management network. DataDome's Q3 2026 AI Traffic Report, covered by Help Net Security today, found that meta-externalagent and Meta-WebIndexer made 12.4 billion requests between July and September. That's 49.7% of all AI agent traffic DataDome recorded. If you've never written a robots.txt rule for meta-externalagent, your site has been making that call by default.

The same report flags a second problem that robots.txt can't touch. Meta's new Muse agent browses from a cloud VM, fills in forms and doesn't announce itself. Two problems, two different fixes.

TL;DR

Meta runs separate bots for AI training (meta-externalagent), Meta AI search (meta-webindexer), link previews (facebookexternalhit) and user-requested fetches (meta-externalfetcher). Make a separate decision for each one.
robots.txt controls the crawlers. It doesn't control Muse, which DataDome says sends no identifying user agent and runs from Cloudflare and Fastly infrastructure.
Most of Muse's sensitive traffic hit forms, so labelled inputs, structured data and WebMCP form tools now matter as much as your blocklist.

What the DataDome report found

The headline numbers, as reported by Help Net Security and PPC Land:

AI agent traffic rose 41.5% quarter over quarter to nearly 25 billion requests in Q3 2026.
Meta-ExternalAgent and Meta-WebIndexer together took 49.7% of it. WebIndexer overtook ExternalAgent in September.
ClaudeBot moved into fourth place for the quarter, ahead of GPTBot.
Agentic browser traffic jumped from 12.5 million requests in August to 58.2 million in September. Muse, which launched in the US in early September, was 33% of the September total.
8.9% of sampled Muse requests reached login, payment, account creation, cart or form endpoints. Forms made up 81% of that, roughly eight times the requests sent to login pages.

Keep the caveats in view. This is one vendor's network, the data isn't independently audited, and DataDome sells bot management. PPC Land also notes that Profound has claimed Muse routes through residential connections, which contradicts DataDome's infrastructure numbers. Still, the direction is clear.

Meta's crawler lineup: what meta-externalagent actually does

Meta documents its bots on its web crawlers page. Here's the short version.

User agent token	What Meta says it's for	Follows robots.txt?
`facebookexternalhit`	Link previews when people share your pages on Facebook, Instagram or Messenger	Mostly, but may bypass it for security checks
`meta-externalagent`	Training foundation AI models, or improving products by indexing content directly	Yes
`meta-webindexer`	Improving Meta AI search results, so your content can be cited and linked	Implied yes
`meta-externalads`	Advertising and business products	Yes
`meta-externalfetcher`	Fetching single links when a user asks	May bypass it

Notice the wording on meta-externalagent. It isn't a pure training opt-out. Meta says it's for training "or" product indexing, so blocking it may cost you something beyond keeping your text out of a model. Meta-WebIndexer is the one tied to Meta AI answers, and Meta's own page says allowing it helps Meta cite and link your content.

Why this matters for your website

Three costs pull in different directions.

Server load. Site owners have been complaining about Meta's crawl rate for months, including on Meta's own developer forum. If one bot family is half your AI traffic, it's worth knowing what it costs you in bandwidth and origin hits.

Visibility. Meta AI sits inside WhatsApp, Instagram and Facebook. Block meta-webindexer and you lower your chances of being the source it cites.

Conversions. Muse runs errands for real people: booking, filling in forms, buying with one-time cards through Link by Stripe, according to TechRepublic. If your quote form or checkout confuses it, that's a lead or sale you never see, and you won't find it in your robots.txt logs either.

A robots.txt policy for Meta's bots

Here's a sensible starting point for a site that wants Meta AI citations and link previews but doesn't want its content used for training:

# Link previews when people share your pages
User-agent: facebookexternalhit
Allow: /

# Meta AI search answers and citations
User-agent: meta-webindexer
Allow: /
Disallow: /cart/
Disallow: /account/

# Foundation model training and product indexing
User-agent: meta-externalagent
Disallow: /

# Ads and business products
User-agent: meta-externalads
Allow: /

A few things to know before you ship it:

Trade-offs and gotchas

Meta says its crawlers may cache robots.txt for up to 24 hours, so don't expect an instant drop.
Group names are matched case-insensitively under RFC 9309, so meta-externalagent and Meta-ExternalAgent hit the same group.
Don't write rules for meta-externalfetcher and expect them to hold. Meta says it may bypass robots.txt because a person asked for the page.
robots.txt is a request, not rate limiting. If load is the real problem, throttle by user agent at your CDN or WAF.
Happy to allow training? Delete the meta-externalagent group.

For the other big names (GPTBot, ClaudeBot, Google-Extended), our robots.txt guide for AI crawlers covers the same split between training and search bots.

Muse is a different problem: make your forms readable

Muse doesn't read robots.txt because it isn't a crawler. It's a browser doing a task for someone. DataDome's advice is to classify agents by behaviour, set explicit rules for forms, login, carts and checkout, and treat Web Bot Auth plus published IP ranges as the minimum bar for verifying any agent you plan to trust.

That covers the agents you don't want. For the ones you do, the fix is on the page. An agent that has to guess what a field means will either fill it in wrong or give up. Compare an unlabelled <input name="f2"> with this:

<form action="/quote" method="post"
      toolname="request_quote"
      tooldescription="Request a price quote for a web project">
  <label for="email">Work email</label>
  <input id="email" name="email" type="email" autocomplete="email" required>
  <label for="budget">Budget in USD</label>
  <input id="budget" name="budget" type="number" min="500">
  <button type="submit">Request quote</button>
</form>

Real labels, autocomplete hints and a submit button that says what it does help every browser agent. The toolname and tooldescription attributes come from the declarative side of WebMCP, which turns the form into a named tool agents can call directly. Our declarative vs imperative WebMCP guide explains when to use which.

What to do this week

Pull 30 days of access logs and count requests from meta-externalagent, meta-webindexer, facebookexternalhit, ClaudeBot and GPTBot.
Write an explicit robots.txt group for each Meta bot, based on what you actually want from Meta: training, AI search citations, both or neither.
Keep facebookexternalhit allowed unless you have a clear reason, or shared links lose their previews.
Move any load problem to CDN or WAF rate limits instead of robots.txt.
Set explicit rules for login, account creation, cart and checkout, and decide what happens to unidentified cloud traffic on those paths.
Audit your three most valuable forms: visible labels, autocomplete values, clear button text and error messages written as text.
Add JSON-LD to the pages those forms live on, and consider declarative WebMCP attributes on the forms themselves.

You can check steps 6 and 7 in a minute: a free AgentReady scan runs the Browser Readiness pillar, which flags missing WebMCP form tools, missing JSON-LD and weak ARIA labels or landmarks on the page you test.

Check your own site

Meta's crawlers read your robots.txt, but Muse reads your page. That page-level side is the Browser Readiness pillar of the Agent Readiness Score, worth 40% of the total: WebMCP form tools, JSON-LD and Open Graph, ARIA and landmarks, and heading structure. A low score there usually means unlabelled inputs and no structured data, so agents guess. The AI Readiness Checker guide shows what each check means. A single-page scan is free and takes a minute. Scan your site free.

Key takeaways

Meta's bots made 49.7% of AI agent traffic on DataDome's network in Q3 2026, so a missing rule for them is a real decision.
meta-externalagent covers training and product indexing; meta-webindexer covers Meta AI search and citations. Treat them separately.
meta-externalfetcher and Muse can't be managed with robots.txt. Use CDN, WAF and agent verification for those.
Forms took most of Muse's sensitive traffic, so clean labels, structured data and WebMCP tools decide whether helpful agents succeed.
Vendor bot reports are useful signals, not audited facts. Check your own logs.

FAQ

What is meta-externalagent?

Meta-externalagent is Meta's web crawler for training foundation AI models and for improving Meta products by indexing content directly. Its user agent token is meta-externalagent/1.1, and Meta says it follows robots.txt. On DataDome's network in Q3 2026, it and Meta-WebIndexer together made nearly half of all recorded AI agent requests.

Should I block meta-externalagent in robots.txt?

Block it if you don't want your content used to train Meta's models and you're comfortable with the product indexing trade-off. Add User-agent: meta-externalagent followed by Disallow: /. Keep meta-webindexer and facebookexternalhit allowed if you still want Meta AI citations and link previews. Changes can take up to 24 hours to apply.

Does blocking meta-externalagent stop Meta AI from citing my site?

Not by itself. Meta lists a separate crawler, meta-webindexer, for improving Meta AI search results, and its documentation says allowing that one helps Meta cite and link your content. If you block every Meta bot with one broad rule, you'll likely lose those citations along with the training crawl.

Can robots.txt block Meta's Muse agent?

No. Muse is a personal agent that runs a browser in a cloud VM on a user's behalf, and DataDome reports that it doesn't identify itself as an AI agent. Robots.txt only works for bots that announce a user agent and choose to obey it. Manage Muse with CDN or WAF rules, behavioural detection and agent verification such as Web Bot Auth.

Start with one file and one page: write explicit Meta groups into your robots.txt today, then test the form your business depends on most to see whether an agent could fill it in without guessing. Run a free AgentReady scan.

Meta-ExternalAgent Now Leads AI Bot Traffic: What to Allow