Paid Media Decoded

Anthropic · global

Claude

Claude can search for current information and cite the relevant pages. Anthropic documents separate robots.txt controls for search, user-requested page fetches and model training.

Sources checked

What to do

  1. Allow Claude-SearchBot for visibility in web search

    Check robots.txt for each public subdomain where you want Claude to index pages for search. Audit product, support and press subdomains separately.

  2. Choose whether Claude can open pages users ask about

    Review the Claude-User rule independently of Claude-SearchBot. Blocking Claude-User prevents Claude from retrieving your content in response to an individual request.

  3. Set model-training access independently

    Review the ClaudeBot rule for the subdomains whose new content you want excluded from training datasets. Allow search or user fetches separately where appropriate.

How it works

When it searches
When web search is enabled, Claude can use it for answers that benefit from current information.
Where results come from
Claude incorporates web information and attaches direct citations.
Crawlers
User agentUsed forIf you block it
Claude-SearchBotSearch indexing and relevancePrevents Anthropic from indexing the content for search optimisation
Claude-UserFetches requested by Claude usersPrevents retrieval of the content in response to user requests
ClaudeBotCollects content for model developmentSignals that future material should be excluded from training datasets
Keeping a page out
Set the relevant Anthropic bot rules in the top-level robots.txt on each subdomain you want to restrict.
Shopping
Claude can compare features, prices and reviews from web sources.
Ads
Not documented by Anthropic.
Measurement
Not documented by Anthropic.

Recent changes that matter

  • May 2025

    Claude’s web search became available globally on all Claude plans. Cited product comparisons became available to shoppers beyond the original paid US rollout.

Analytics settings

Allow Google Analytics? It helps us see which lessons people use. You can change this in Analytics settings.

Search the course

Type to find lessons, terms and sources.