Cloudflare

Enhance your website's AI SEO with our Cloudflare Agent integration. Monitor AI interactions and optimize your content for better visibility in AI search results.

Overview

Our Cloudflare Agent integration provides a seamless way to track AI interactions with your website. Deploy through Cloudflare Workers for efficient, edge-based tracking of how AI systems interact with your content. This solution works with any website hosted on Cloudflare and requires minimal configuration. The edge-based approach ensures fast performance and comprehensive tracking across all pages.

Features

Deploy via Cloudflare Workers
Edge-based AI tracking
Zero impact on site performance
Global distribution
Automatic updates and maintenance
Works with any website on Cloudflare
No code changes to your actual website
Identifies all major AI bots and crawlers

Requirements

Cloudflare account
Website hosted on Cloudflare or using Cloudflare DNS
API key from xseek
Website ID from your xseek dashboard

Setup Process

1
Create a new Cloudflare Worker with our provided code
2
Configure the worker with your API key and website ID
3
Set up a route to direct traffic through your worker
4
Deploy the worker to start tracking AI visits

Integration Setup

Important Information

Make sure to replace YOUR_API_KEY and YOUR_WEBSITE_ID with your actual values in the integration code. Your API key can be found in your account settings.

1Create a Cloudflare Worker

// Cloudflare Worker script to track AI bots
export default {
  async fetch(request, env, ctx) {
    const userAgent = request.headers.get('user-agent') || '';

    const AI_BOTS = [
    { name: 'anthropic-ai', pattern: /anthropic-ai/i },
    { name: 'claudebot', pattern: /ClaudeBot/i },
    { name: 'claude-web', pattern: /claude-web/i },

    { name: 'perplexitybot', pattern: /PerplexityBot/i },
    { name: 'perplexity-user', pattern: /Perplexity-User/i },

    { name: 'grokbot', pattern: /GrokBot(?!.*DeepSearch)/i },
    { name: 'grok-search', pattern: /xAI-Grok/i },
    { name: 'grok-deepsearch', pattern: /Grok-DeepSearch/i },

    { name: 'deepseekbot', pattern: /DeepSeekBot/i },

    { name: 'GPTBot', pattern: /GPTBot/i },
    { name: 'chatgpt-user', pattern: /ChatGPT-User/i },
    { name: 'oai-searchbot', pattern: /OAI-SearchBot/i },

    { name: 'google-extended', pattern: /Google-Extended/i },

    { name: 'applebot', pattern: /Applebot(?!-Extended)/i },
    { name: 'applebot-extended', pattern: /Applebot-Extended/i },

    { name: 'meta-external', pattern: /meta-externalagent/i },
    { name: 'meta-externalfetcher', pattern: /meta-externalfetcher/i },

    { name: 'bingbot', pattern: /Bingbot(?!.*AI)/i },
    { name: 'bingpreview', pattern: /bingbot.*Chrome/i },
    { name: 'microsoftpreview', pattern: /MicrosoftPreview/i },

    { name: 'cohere-ai', pattern: /cohere-ai/i },
    { name: 'cohere-training-data-crawler', pattern: /cohere-training-data-crawler/i },

    { name: 'youbot', pattern: /YouBot/i },

    { name: 'duckassistbot', pattern: /DuckAssistBot/i },

    { name: 'semanticscholarbot', pattern: /SemanticScholarBot/i },

    { name: 'ccbot', pattern: /CCBot/i },

    { name: 'ai2bot', pattern: /AI2Bot/i },
    { name: 'ai2bot-dolma', pattern: /AI2Bot-Dolma/i },

    { name: 'aihitbot', pattern: /aiHitBot/i },

    { name: 'amazonbot', pattern: /Amazonbot/i },
    { name: 'novaact', pattern: /NovaAct/i },

    { name: 'brightbot', pattern: /Brightbot/i },

    { name: 'bytespider', pattern: /Bytespider/i },
    { name: 'tiktokspider', pattern: /TikTokSpider/i },

    { name: 'cotoyogi', pattern: /Cotoyogi/i },

    { name: 'crawlspace', pattern: /Crawlspace/i },

    { name: 'pangubot', pattern: /PanguBot/i },
    { name: 'petalbot', pattern: /PetalBot/i },

    { name: 'semrushbot-ocob', pattern: /SemrushBot-OCOB/i },
    { name: 'semrushbot-swa', pattern: /SemrushBot-SWA/i },

    { name: 'sidetrade-indexer', pattern: /Sidetrade indexer bot/i },

    { name: 'timpibot', pattern: /Timpibot/i },

    { name: 'velenpublicwebcrawler', pattern: /VelenPublicWebCrawler/i },

    { name: 'omgili', pattern: /omgili/i },
    { name: 'omgilibot', pattern: /omgilibot/i },
    { name: 'webzio-extended', pattern: /Webzio-Extended/i }
    ];

    let detectedBot = null;

    for (const bot of AI_BOTS) {
      if (bot.pattern.test(userAgent)) {
        detectedBot = bot.name;
        break;
      }
    }

    const response = await fetch(request);

    if (detectedBot) {
      try {
        await fetch('https://api.aiseotracker.com/track-ai-bot', {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
            'x-api-key': 'YOUR_API_KEY', // Replace with your actual API key
          },
          body: JSON.stringify({
            botName: detectedBot,
            userAgent: userAgent,
            url: request.url,
            ip: request.headers.get('cf-connecting-ip') || undefined,
            referer: request.headers.get('referer') || undefined,
            websiteId: 'YOUR_WEBSITE_ID', // Replace with your website ID
          }),
        });
      } catch (error) {
        console.error('Error tracking AI bot:', error);
      }
    }

    return response;
  },
};

2Deploy your Worker

After saving your worker, set up a route to direct traffic through it. Go to the "Triggers" tab and add a route pattern:

yourdomain.com/*

This will process all requests through your worker, enabling AI bot detection across your entire site.

All set!

Your Cloudflare-powered website is now tracking AI crawler visits. Data will appear in the xseek dashboard soon.

Need more help?

Check our comprehensive documentation for detailed instructions.

View documentation