AI crawler tracking: see GPTBot, ClaudeBot and PerplexityBot on your site

AI answers

Fetches made when an AI assistant opens a page to answer someone, right now.

Search indexes

Crawlers that discover and index pages for search engines and AI search.

Training crawlers

Crawlers that collect public pages that may be used to train AI models.

Other AI bots

Ad checks, link previews and other automated fetchers run by AI companies.

How AI crawler tracking works

Crawlers don't run JavaScript, so the tracking script can't see them. Your server has to report them. Copy your ingest secret and website ID from Dashboard → Setup into environment variables, then add this middleware. For each request that looks like a bot, it sends the path and user agent to POST /api/crawls. We store the visit only if the user agent matches one of the 61 crawlers below; anything else is ignored. Visits appear in the Crawlers card on your dashboard, by crawler and by page, and are kept for 25 months. Crawls never count toward your event limit. Browsers can't send crawls: the request must carry your ingest secret, so keep it on the server. Add the tracking script too (installation guide) to see human visitors alongside the bots.

Next.js

// middleware.ts
import { NextResponse, type NextRequest, type NextFetchEvent } from 'next/server';

const MAYBE_BOT = /bot|crawl|spider|-user|agent|fetcher|copilot|grok|notebooklm|read-aloud|externalhit|cohere/i;

export function middleware(request: NextRequest, event: NextFetchEvent) {
  const userAgent = request.headers.get('user-agent') ?? '';
  if (MAYBE_BOT.test(userAgent)) {
    event.waitUntil(
      fetch('https://thebizness.ai/api/crawls', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          Authorization: `Bearer ${process.env.THEBIZNESS_INGEST_SECRET}`,
        },
        body: JSON.stringify({
          websiteId: process.env.THEBIZNESS_WEBSITE_ID,
          url: request.nextUrl.pathname,
          userAgent,
        }),
      }).catch(() => {})
    );
  }
  return NextResponse.next();
}

export const config = { matcher: ['/((?!_next/|favicon.ico).*)'] };

Any server

Send websiteId, url and the request's userAgent to POST /api/crawls with your ingest secret. Add the response status if you have it, to see which pages crawlers hit that return 404.

curl -X POST https://thebizness.ai/api/crawls \
  -H "Authorization: Bearer $THEBIZNESS_INGEST_SECRET" \
  -H "Content-Type: application/json" \
  -d '{"websiteId":"YOUR_WEBSITE_ID","url":"/pricing","userAgent":"Mozilla/5.0 (compatible; GPTBot/1.2)","status":200}'