Vercel WAF Rules for AI Scrapers: A 2026 UA Catalog

Ten AI training and search crawlers publish a documented user-agent string in 2026: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Amazonbot, GoogleOther, Google-CloudVertexBot, meta-externalagent, and Bytespider. A Vercel WAF custom rule that matches the User Agent parameter against this list, with a deny or challenge action scoped to /tools/* and /blog/*, blocks the documented 80% of AI scraping volume without touching legitimate search indexers like Googlebot or Bingbot.

This is the operational complement to the bot protection and DDoS mitigation guide. That pillar covers the strategic decision (which layer for which route, when to consider Cloudflare in front). This post ships the actual rule, the maintained UA catalog with primary-source citations per vendor, the per-route-surface patterns, the quarterly maintenance protocol, and an honest acknowledgment that a UA-based rule will always lag the next undocumented scraper by the time-to-first-public-disclosure window.

TL;DR:

• Vercel offers two paths. The AI Bots Managed Ruleset is a one-toggle alternative that Vercel updates server-side; a custom rule with an explicit User Agent list is auditable, version-controlled, and lets you selectively allow legit indexers [1][2]. The cookbook here covers the custom-rule path; the managed ruleset is mentioned where it is the better default.
• Ten documented AI crawler UAs in 2026. GPTBot, OAI-SearchBot (OpenAI [3]); ClaudeBot, Claude-SearchBot (Anthropic [4]); PerplexityBot (Perplexity [5]); Amazonbot (Amazon [6]); GoogleOther, Google-CloudVertexBot (Google [8]); meta-externalagent (Meta [9]); Bytespider (ByteDance, third-party catalogs only).
• Five user-initiated UAs are a separate decision. ChatGPT-User, Claude-User, Perplexity-User, OAI-AdsBot, meta-externalfetcher. These fetch when a user asks the assistant a question. Blocking them hides your site from Perplexity citations, ChatGPT browsing, and Claude searches [3][4][5][9]. Allow on /blog/* if AI-search visibility matters; deny on private routes.
• Rule structure follows Vercel's documented best practice. Add as a custom rule with User Agent condition + OR-joined list, action = Log first, observe 10-minute live traffic, then upgrade to Deny or Challenge once the rule is verified not catching legitimate crawlers [1]. Persistent actions add a time-based IP block.
• The rule lags new scrapers by definition. Failure mode #2 from the pillar applies: a vendor ships a new crawler today, the UA is catalogued tomorrow, your rule fires the day after. Pair the UA rule with rate limiting on the same route as the catch-up layer, and review the catalog quarterly.

Table of contents

• When should you write a custom rule instead of toggling the Managed Ruleset?
• Which AI scraper user agents are documented in 2026?
• What does a Vercel WAF custom rule for AI scrapers actually look like?
• How should the rule differ per route surface?
• How do you maintain this catalog as new vendors ship crawlers?
• Where does this rule set lag, and what do you pair it with?
• What does SecureStartKit ship today, and what should you add?

When should you write a custom rule instead of toggling the Managed Ruleset?

Vercel ships two ways to block AI crawlers: a one-toggle AI Bots Managed Ruleset, and a hand-rolled Custom Rule with an explicit User Agent list. The managed ruleset is the right default for most teams. The custom rule wins when you need auditability, version control, selective allowlisting (Googlebot stays, GPTBot leaves), or a different action per route surface.

The managed ruleset, configured under Firewall > Rules > Bot Management > AI Bots Ruleset, exposes two actions: Log (record without blocking, useful for monitoring) and Deny (block all traffic identified as coming from AI bots) [2]. Vercel maintains the underlying list server-side. You get one site-wide on or off, and the contents of "AI bots" is whatever Vercel decides today.

The custom rule path is more verbose, but four things change:

1. The UA list is in your dashboard, visible per condition. A new operator can read it. A change shows up in the rule edit history.
2. The action can vary per condition. Deny GPTBot on /blog/*, challenge it on /tools/*, log it on /api/og. The managed ruleset cannot do that.
3. You can allowlist legitimate indexers explicitly. A managed ruleset that decides Googlebot is an AI bot tomorrow takes you out of Google Search. A custom rule lists exactly what you want blocked, full stop.
4. The rule plays nicely with the rest of your custom rule chain. Vercel executes custom rules before managed rulesets [2], so a Bypass rule for, say, your own monitoring user agent runs first.

The honest framing: if you do not need any of the four, toggle the Managed Ruleset and move on. It is one click and Vercel maintains it. The cookbook below is for teams who do need at least one of the four, which in practice is most SaaS operators who care about the difference between a search crawler that drives traffic and an AI training crawler that does not.

A pattern that works for many indie teams: enable the Managed Ruleset in Log mode as a baseline observability layer, AND add the custom rule below in Deny mode for the specific UAs and routes you want to block hard. The managed ruleset reports what is reaching your origin; the custom rule deterministically blocks the ones you care about.

Which AI scraper user agents are documented in 2026?

Ten AI crawlers publish a stable User-Agent string and are appropriate targets for a WAF block rule on a SaaS site. The list is split into training/search crawlers (block by default unless you want AI-search visibility) and user-initiated agents (block only after a deliberate decision, because they fetch when a user asks an assistant about your content). Each entry below is sourced from the vendor's own published page.

OpenAI publishes four crawlers with distinct purposes [3]:

User-Agent token	Purpose	robots.txt
GPTBot	Crawls content to train generative AI foundation models	Respects
OAI-SearchBot	Surfaces websites in ChatGPT search results	Respects
ChatGPT-User	User-initiated fetches from ChatGPT and Custom GPT Actions	"Rules may not apply"
OAI-AdsBot	Validates pages submitted as ChatGPT ads	Visits only user-submitted ad landing pages

The full GPTBot UA string is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot [3]. A WAF rule matching the substring GPTBot is sufficient; matching the version number is brittle.

Anthropic publishes three crawlers under the bot policy at the Claude help center [4]:

User-Agent token	Purpose	robots.txt
ClaudeBot	Training data collection for Anthropic models	Respects
Claude-User	User-initiated fetches when someone asks Claude a question	Respects
Claude-SearchBot	Indexes content for Claude's search results	Respects

A legacy anthropic-ai UA appears in older traffic logs but is no longer the documented identifier; Anthropic's current bot policy lists the three above [4]. The contact address claudebot@anthropic.com appears in the full UA string for ClaudeBot/1.0.

Perplexity publishes two crawlers at docs.perplexity.ai/guides/bots [5]:

User-Agent token	Purpose	robots.txt
PerplexityBot	Surfaces and links websites in Perplexity search results, not for training	Respects
Perplexity-User	User actions: visits pages "to help provide an accurate answer" when users ask questions	"Generally ignores"

Perplexity is explicit that PerplexityBot is not used for model training; it is a search indexer. A SaaS that wants AI-search citations should allow PerplexityBot on /blog/* and only consider blocking it on private routes.

Amazon publishes Amazonbot at developer.amazon.com/amazonbot [6]: the full UA is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36. Amazon's stated purpose is enhancing Amazon products "to supply customers with more precise information," and the page notes the crawler "may also contribute to training Amazon AI models" [6]. Respects robots.txt.

Apple is the exception [7]. Applebot crawls for Spotlight, Siri, and Safari search, and may train Apple's generative AI foundation models. Apple does not publish a separate UA for AI training opt-out; the control mechanism is Applebot-Extended as a robots.txt directive only. From a WAF rule perspective, you either block Applebot (which removes you from Apple search results too) or you accept it. Apple's robots.txt-only opt-out for training is the right path here; a WAF user-agent rule cannot do what Applebot-Extended does because there is no distinct UA to match.

Google publishes three relevant crawler tokens at developers.google.com/search/docs/crawling-indexing/google-common-crawlers [8]:

User-Agent token	Purpose	UA-blockable?
Google-Extended	Controls Gemini training opt-out via robots.txt	No: "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings" [8]
GoogleOther	Generic crawler for Google product teams' R&D and one-off crawls	Yes: compatible; GoogleOther in UA [8]
Google-CloudVertexBot	Crawls for site owners building Vertex AI Agents	Yes: Google-CloudVertexBot substring

Google-Extended is the Apple pattern: a robots.txt-only control, no distinct UA, so it cannot be enforced via a WAF rule. The two that CAN be UA-blocked are GoogleOther and Google-CloudVertexBot, both of which are appropriate to deny on a SaaS site that does not want its content showing up in Vertex AI agents.

Meta publishes three crawlers at developers.facebook.com/docs/sharing/webmasters/web-crawlers/ [9]:

User-Agent token	Purpose	robots.txt
meta-externalagent	Crawls for "training foundation AI models or improving products by indexing content directly"	Respects
facebookexternalhit	Caches title/description/thumbnail when a link is shared on Facebook, Instagram, or Messenger	May bypass for "security or integrity checks"
meta-externalfetcher	Fetches individual links at user request for "agentic AI capabilities"	May bypass since user-requested

meta-externalagent is the AI training crawler and the appropriate WAF block target. facebookexternalhit is the OpenGraph fetcher that powers social previews on Meta surfaces; blocking it breaks your Facebook and Instagram link previews. Do not block it unless you have a specific reason.

ByteDance ships Bytespider without an official documentation page. The UA appears verbatim in third-party crawler catalogs and in production server logs: Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com). The contact address spider-feedback@bytedance.com in the UA is your only signal that the vendor is willing to receive feedback; there is no published opt-out path beyond a UA block.

The clean catalog for a default WAF deny rule, training and search crawlers only:

GPTBot
OAI-SearchBot
ClaudeBot
Claude-SearchBot
PerplexityBot
Amazonbot
GoogleOther
Google-CloudVertexBot
meta-externalagent
Bytespider

The user-initiated set, opt-in block only when you have a specific reason:

ChatGPT-User
Claude-User
Perplexity-User
OAI-AdsBot
meta-externalfetcher

Notable absences: there is no UA-based block path for OpenAI training opt-out beyond GPTBot (no separate identifier), no UA path for Google-Extended (robots.txt only), no UA path for Applebot-Extended (robots.txt only). For those three vendors, the robots.txt directive is the operative control, not a WAF rule.

What does a Vercel WAF custom rule for AI scrapers actually look like?

The rule is a single Custom Rule with one User Agent condition per crawler, joined by OR, and a Deny or Challenge action. Vercel's documented best practice is to start with Log, observe 10-minute live traffic to confirm the rule is not catching legitimate clients, and then upgrade the action [1].

The dashboard configuration, step by step:

1. From your project's Firewall dashboard, select Configure.
2. Select Add New > Rule.
3. Name: Block documented AI scraper UAs (training and search).
4. Condition: User Agent contains GPTBot. Add condition. Joiner: OR. User Agent contains OAI-SearchBot. Repeat for the 10 UAs in the training/search list above.
5. Action: Log.
6. Save Rule, Review Changes, Publish.

Wait 10 minutes. Open the Firewall overview, group by Custom Rule, confirm the matching traffic is what you expect: scrapers from cloud IP ranges, no Googlebot or Bingbot in the matches, no internal monitoring agents. If clean, edit the rule, change action to Deny, save, publish again.

For teams who prefer infrastructure-as-code, a subset of WAF rules can be configured in vercel.json directly. The supported actions in this path are challenge and deny; log, bypass, and redirect are dashboard-only [1]. The vercel.json shape uses the standard route configuration with a has condition and a mitigate property [1]:

{
  "$schema": "https://openapi.vercel.sh/vercel.json",
  "routes": [
    {
      "src": "/(.*)",
      "has": [
        { "type": "header", "key": "user-agent", "value": "(GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot|Amazonbot|GoogleOther|Google-CloudVertexBot|meta-externalagent|Bytespider)" }
      ],
      "mitigate": { "action": "deny" }
    }
  ]
}

The header-value regex matches any of the documented UA substrings against the incoming user-agent request header. The rule sits in version control, gets reviewed in pull requests, and rolls out with deployments. The trade-off is that you lose the dashboard's Log-first workflow; the rule blocks from the first request after deploy. Mitigation: deploy to a preview environment first, send a curl with each UA to confirm the rule fires.

Vercel also exposes a natural-language rule generator. Typing "Challenge requests where user agent contains GPTBot, ClaudeBot, or PerplexityBot" into the rule form generates a Challenge rule with OR conditions for each user agent automatically [1]. The output is identical to the manual configuration above; the generator is a convenience, not a different rule class.

One Vercel-specific detail worth knowing: when you select Deny or Challenge as the action, the dashboard exposes a for dropdown (timeframe) defaulting to 1 minute, configurable to longer windows. This is the persistent actions feature [1]. With persistent actions enabled, a client matched by the rule has its IP blocked for the chosen window for all requests, even ones that would not have matched the rule. The block happens before the firewall processes the request, so persistent-action-blocked requests do not count toward CDN and traffic usage [1]. For AI scrapers that send dozens of requests per minute from the same IP, enabling persistent actions with a 1-hour window dramatically reduces the number of requests the firewall has to evaluate.

How should the rule differ per route surface?

A single deny-all rule across the entire site is the wrong default. The Vercel cost-attribution math from the pillar splits routes into four classes by what they expose and what they are worth scraping for. The WAF rule should reflect that split.

The four route classes for a typical Next.js SaaS:

Route surface	Action for documented AI UAs	Action for user-initiated AI UAs	Why
/blog/*	Deny for training, Allow for search	Allow	Content meant to be indexed by search (Google, Bing, PerplexityBot, OAI-SearchBot); blocked from training (GPTBot, ClaudeBot, Bytespider, meta-externalagent)
/tools/*	Deny for both training and search	Deny (or Challenge)	Public-but-bandwidth-heavy; scrapers crawling tool pages drive function invocations without delivering signups
/api/og	Allow all (no rule)	Allow	Image generation called by social-preview fetchers (facebookexternalhit, X bot, LinkedIn bot); blocking AI here breaks legitimate previews
Private routes (/dashboard/*, /admin/*)	Deny all (any bot)	Deny all	No crawler should be inside an authenticated surface; if one is, the auth layer is the failure, but a WAF block is cheap insurance

The asymmetry between /blog/* (split allow/deny) and /tools/* (deny both) is the key call. On a content-heavy SaaS where AI-search citations drive measurable traffic, allowing PerplexityBot and OAI-SearchBot on the blog is positive ROI; they are the channel through which a Perplexity user finds your post. On tool pages, the same crawlers cost you function invocations without delivering the same value because tool pages are not the citable layer in an AI search result.

The implementation needs two rules, not one, to encode this split:

Rule 1: Training crawlers blocked everywhere except /api/og.

{
  "src": "^(?!/api/og).*",
  "has": [
    { "type": "header", "key": "user-agent", "value": "(GPTBot|ClaudeBot|Bytespider|meta-externalagent|Amazonbot|GoogleOther|Google-CloudVertexBot)" }
  ],
  "mitigate": { "action": "deny" }
}

Rule 2: Search crawlers blocked on /tools/* only, allowed on /blog/* and elsewhere.

{
  "src": "/tools/.*",
  "has": [
    { "type": "header", "key": "user-agent", "value": "(OAI-SearchBot|PerplexityBot|Claude-SearchBot)" }
  ],
  "mitigate": { "action": "deny" }
}

The two-rule structure is more verbose than a single deny-all rule, but it preserves the AI-search visibility channel that a content-led SaaS actually wants. A team that does not care about AI-search citations can collapse the two rules into a single deny-everywhere rule on the full UA list; the configuration is shorter, the trade-off is that ChatGPT, Perplexity, and Claude search will never return your pages.

A subtle gotcha: the src regex is matched against the request path, not the full URL. The negative-lookahead pattern ^(?!/api/og).* excludes /api/og from rule 1, which is what you want when the route generates social preview images that facebookexternalhit and similar fetchers legitimately request. Verify the regex matches your routes by deploying to preview and curling each path before promoting to production.

How do you maintain this catalog as new vendors ship crawlers?

The catalog above is a snapshot of 2026 disclosed crawlers. Six of the nine vendors covered above shipped a new or renamed bot in the last 18 months. The catalog needs a quarterly review or it goes stale; pillar failure mode #2 from the bot protection pillar is that "WAF custom rules alone matching only known AI UAs" lag novel crawlers by however long it takes the vendor to disclose.

The maintenance protocol is short:

1. Quarterly, re-fetch each vendor's bot page. OpenAI: developers.openai.com/api/docs/bots. Anthropic: support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler. Perplexity: docs.perplexity.ai/guides/bots. Amazon: developer.amazon.com/amazonbot. Apple: support.apple.com/en-us/119829. Google: developers.google.com/search/docs/crawling-indexing/google-common-crawlers. Meta: developers.facebook.com/docs/sharing/webmasters/web-crawlers/. Diff against your current rule list. ByteDance: no official page; consult third-party catalogs and check live traffic for unknown compatible; substrings.

2. Weekly, check Firewall traffic for high-volume unknown User Agents. The Firewall overview groups by User Agent. Sort by request count, look at the top 20. Any UA matching a compatible; pattern that is not in your rule and not a documented browser or search engine is a candidate; check the contact URL in the UA string, decide whether to add it to the rule.

3. A/B test rule changes in Log mode before promoting to Deny. Vercel's documented best practice [1]. A misconfigured rule that catches Googlebot is a 24-hour SEO regression; a one-shot Log observation before the upgrade prevents it.

4. Keep the rule list in version control, even when configured via dashboard. The dashboard does not track per-edit history beyond a recent window. A docs/waf-rules.md (or equivalent) in your repo with the full UA list and the date it was last reviewed gives you a paper trail and a diff target for the quarterly review.

5. Maintain an allowlist for monitoring agents. If you run uptime monitoring, synthetic checks, or competitor-monitoring tools, their user agents need to be in a separate Bypass rule that executes before the AI scraper deny rule. The Vercel WAF executes custom rules in the order you specify them in the rules page [2], so place the Bypass rule above the Deny rule.

The cost of this maintenance is one to two hours per quarter for the catalog review plus 10-15 minutes per week for the traffic check. That is small enough to fit in a regular pre-launch security audit cadence and large enough to actually keep the rule current. Skipping it for a year is a slow drift, not a catastrophic failure; the rule still blocks the 2026 crawlers it was written against, you just stop blocking new ones.

Where does this rule set lag, and what do you pair it with?

A UA-based rule has a structural ceiling. It catches honest crawlers that identify themselves. It does not catch:

• New crawlers shipped without public disclosure (the failure-mode-2 lag window).
• UA-spoofed scrapers that send Mozilla/5.0 Chrome-style strings indistinguishable from a real browser.
• Distributed scraping infrastructure where each request comes from a different residential IP with a different UA, none of which matches your rule.
• Sophisticated bots running headless Chromium that present a real browser TLS fingerprint and a real-browser UA, with the actual scraping logic running in JavaScript.

For each of these, the UA rule is not the defense. The pairings:

Pair with rate limiting on the same route. The catch-up layer for unknown-UA scrapers is per-IP or per-session rate limiting. A scraper that defeats the UA rule by sending generic UAs still hits the rate limiter when it makes 500 requests in 60 seconds. The five production rate-limit failure modes in the rate-limit deep-dive cover the specific cases where this catch-up layer itself fails (Vercel's x-forwarded-for overwrite, fixed-window boundary bursts, distributed-IP attacks). The two layers compose: UA rule blocks the documented honest crawlers cheaply at the edge, rate limit catches the residual stream.

Pair with BotID on high-value Server Actions. The conversion-critical routes (/purchase, auth Server Actions) are where business logic runs that costs real money when a bot abuses it. BotID classifies the client at request time using browser-fingerprint signals, not just UA. A headless Chromium scraper that defeats both the UA rule and the rate limit gets caught by BotID before the Stripe API call or the Supabase write fires.

Pair with the OWASP Top 10 architectural defenses for the routes a scraper might eventually reach. A scraper that bypasses every perimeter layer above and reaches a Server Action still has to defeat backend-only data access, Zod validation, and rate-limited identity checks. Defense in depth. The perimeter buys you time; the architecture is what actually prevents the breach.

Do not pair with Cloudflare in front of Vercel by default. The pillar covers the trade-off in detail. Vercel's own guide recommends against it because the proxy degrades Firewall, BotID, and DDoS visibility. Cloudflare's free Bot Fight Mode catches low-sophistication bots, but the cost is that BotID accuracy drops, Firewall logs lose the originating IP, and the DDoS layer loses visibility into volumetric attacks. Reach for Cloudflare only when scraping is materially moving the invoice and the cost reduction outweighs the visibility cost.

The honest framing: a UA rule is a cheap perimeter defense for a known threat surface. It is not a defense against an unknown one. Stack it with rate limiting, BotID, and the architectural defenses, and the scraper that defeats all four is rare enough that the residual cost is acceptable. The mistake is treating the UA rule as if it were the whole defense; the mistake the pillar named, that this catalog tries not to repeat by handing you the next layers explicitly.

What does SecureStartKit ship today, and what should you add?

SecureStartKit's next.config.ts does not ship a WAF rule, because the WAF is a Vercel dashboard configuration, not a code-level one. What the template ships is the architecture that makes the WAF rule a perimeter optimization rather than a load-bearing defense: backend-only data access on Supabase reads and writes, Zod validation on every Server Action and Route Handler, rate-limited identity checks via the Map-based limiter, and Stripe Checkout calls protected behind BotID Deep Analysis.

What an operator building on SecureStartKit needs to add at the dashboard layer, in order of cost-to-value:

1. The AI Bots Managed Ruleset in Log mode, site-wide. One toggle, free, gives you observability into which AI crawlers are hitting your site without any blocking. Run it for a week to see what the traffic actually looks like before writing rules.

2. The two custom rules from this post, in Log mode first, then Deny. Training crawlers blocked everywhere except /api/og; search crawlers blocked on /tools/*. Promote to Deny after the 10-minute observation [1].

3. A Bypass rule for your own monitoring stack, placed above the Deny rules. Whatever uptime checker, synthetic test suite, or competitor monitor you run, that UA goes in a Bypass rule at the top of the chain.

4. A docs/waf-rules.md in your repo mirroring the dashboard rule list, dated, with the source URL for each vendor's bot page. The quarterly review diffs against this file.

5. Persistent actions on the AI scraper Deny rule, set to 1 hour. Cuts the number of rule evaluations the firewall does per scraping session, and the IP-blocked requests do not count toward CDN usage [1].

The SaaS security checklist tool walks the full perimeter and application-layer review where this WAF rule fits; the pre-launch security audit is the operational checklist that closes around it. Bot protection on Vercel in 2026 is a layered defense: the WAF custom rule is the cheapest layer, the managed ruleset is the easiest, BotID is the most accurate on conversion routes, and the architecture underneath is what guarantees that any single layer's failure is recoverable. The pricing page names the architectural defaults the template ships with so the WAF rule above is a perimeter optimization, not the first line of defense.