Anthropic · global
Claude
Claude can search for current information and cite the relevant pages. Anthropic documents separate robots.txt controls for search, user-requested page fetches and model training.
Sources checked
What to do
Allow Claude-SearchBot for visibility in web search
Check robots.txt for each public subdomain where you want Claude to index pages for search. Audit product, support and press subdomains separately.
Choose whether Claude can open pages users ask about
Review the Claude-User rule independently of Claude-SearchBot. Blocking Claude-User prevents Claude from retrieving your content in response to an individual request.
Set model-training access independently
Review the ClaudeBot rule for the subdomains whose new content you want excluded from training datasets. Allow search or user fetches separately where appropriate.
How it works
- When it searches
- When web search is enabled, Claude can use it for answers that benefit from current information.
- Where results come from
- Claude incorporates web information and attaches direct citations.
- Crawlers
User agent Used for If you block it Claude-SearchBot Search indexing and relevance Prevents Anthropic from indexing the content for search optimisation Claude-User Fetches requested by Claude users Prevents retrieval of the content in response to user requests ClaudeBot Collects content for model development Signals that future material should be excluded from training datasets - Keeping a page out
- Set the relevant Anthropic bot rules in the top-level robots.txt on each subdomain you want to restrict.
- Shopping
- Claude can compare features, prices and reviews from web sources.
- Ads
- Not documented by Anthropic.
- Measurement
- Not documented by Anthropic.
Recent changes that matter
- May 2025
Claude’s web search became available globally on all Claude plans. Cited product comparisons became available to shoppers beyond the original paid US rollout.