xSeek/Docs/Claude
Back to documentation

Claude user agents, explained.

Anthropic ships three Claude bots. Understand what each one does, how to control it, and what it takes to get cited in Claude's answers.

Claude · AnthropicUpdated Apr 2026~7 min read3 user agents
TL;DR

Anthropic documents three Claude bots — ClaudeBot for training, Claude-User for user-initiated fetches, and Claude-SearchBot for search indexing. Control each one independently in robots.txt.

Overview

Anthropic's Claude accesses the web through several distinct user agents, each with a different purpose. Understanding them is essential if you want Claude to cite your content — or keep it out of training runs.

The three primary bots are documented in Anthropic's official help center and all respect robots.txt.

How to identify the agents

3 crawlers

Claude identifies itself with specific user-agent strings. Verify each through a published IP range.

CB

ClaudeBot

AI training · crawler
Training
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Anthropic's primary training crawler. When allowed, content may be used to improve future Claude models. Block it if you don't want your pages used for training.

VerifiableAnthropic docs
CU

Claude-User

User-initiated fetches · on-demand
User-triggered
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)

Fired when a real Claude user asks the assistant to visit a specific URL. The clearest signal that Claude is actively recommending your content to a human right now.

VerifiableAnthropic docs
CS

Claude-SearchBot

Search indexing · crawler
Search index
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)

Indexes and evaluates content quality so Claude can return better search-style answers. Not a training crawler — this feeds Claude's retrieval layer.

VerifiableAnthropic docs

How Claude accesses your content

3 access patterns, mapped to the agents above.

1

Training

ClaudeBot collects content that may feed future Claude model training.
2

Search

Claude-SearchBot indexes content to improve Claude's search answers.
3

On-demand

Claude-User fetches a URL when a human asks Claude to read it.
!
Heads up. You may still see claude-web in older logs. It's a legacy identifier — xSeek keeps detecting it for backwards compatibility.

Control Claude's access

robots.txt

You control each agent independently. Common configurations below.

Allow citations · block trainingRecommended

Stay citable in Claude, stay out of training data.
# Allow Claude search and user fetches, block training User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /

Block all Claude accessRestrictive

Keeps Anthropic out of everything — no citations, no training.
# Block every Claude crawler User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: /

Optimize content for Claude.

What actually moves the needle when Claude decides whether to cite you.

Use clean, semantic HTML. Headings, lists, tables — so Claude can parse structure without guessing.
Render content server-side. Claude-User doesn't execute complex JS for most fetches. If it needs JS to appear, assume it won't be read.
Answer the question fully. Claude cites sources that resolve the question, not pages that dance around it.
Add schema markup. Article, FAQPage, and HowTo structured data help Claude verify authority.
Split training vs search intentionally. Not every page belongs in both — use per-agent robots.txt rules.

Track Claude visits with xSeek.

See every Claude visit in real time.

Monitor ClaudeBot, Claude-User, and Claude-SearchBot. See which URLs they hit. Watch how often Claude recommends your content. Get notified when patterns shift.

Start free →

Frequently asked questions

Three: ClaudeBot (training), Claude-User (user-initiated fetches), and Claude-SearchBot (search indexing). All three are documented by Anthropic and respect robots.txt. You may also see the legacy claude-web identifier in older logs.
Source: Anthropic's official help center article on Claude bots and robots.txt controls.
PreviousOpenAINextPerplexity user agents