AI bots
Which AI bots read your site, which pages they read, whether they get through, which pages are read but never cited, and the firewall warnings.
The AI bots page (under More, next to Traffic) shows Which AI bots read your site, which pages they read, and whether they get through. The numbers come from one of the three tracking routes. See AI bot visits.
The date range in the header applies to every number on the page.
Before tracking is on
- Free plan. The page says AI bot visits are on paid plans. with See plans, above a faded chart marked Example layout, not your data.
- Not set up. Set up AI bot tracking shows the one route Knownwise suggests for your site, with a Suggested badge and Set up tracking.
- Waiting. Once a route is set up, the page reads Waiting for the first AI bot visit. Setup status lists each route and its state. For Cloudflare it also shows when Knownwise last checked. A route that needs attention says why, after Problem:, in the same words as the Settings card. A snippet key that never sent anything for 7 days no longer counts as set up, and the page offers the suggested route again. Test it in Settings takes you to the tracking card, where Test it sends a test visit.
The page fills in with the first real bot visit. A test visit doesn't count.
The numbers
Each group of numbers says where it came from: Reported by your site, Estimated from Cloudflare, or Reported by your site, with refusals from Cloudflare.
A visit is counted once. When your site reports visits (the GitHub file or the snippet) and Cloudflare is connected too, both see the same requests, so the visits, bots, pages, verified share and change come from your site's reports only. Cloudflare's numbers then feed only the Refused column, because Cloudflare also sees the requests it refused before they reached your site. With Cloudflare alone, its estimates are the numbers. The overview tile, the weekly report and the agent's tools follow the same rule.
The four KPIs:
- Visits. All AI bot visits in the range.
- Bots. How many different AI bots visited.
- Verified. The share of visits that came from the bot company's published addresses. Cloudflare visits are never verified, and neither are visits from a site that sends no IP. See Verified visits.
- Change. Compared with the same length of time just before. No visits the period before when there is nothing to compare.
Visits by bot
A bar chart of the top 8 bots, coloured by kind. The kind is also written next to each bar:
- Search bots fetch pages to answer questions.
- On request means a person asked an assistant to read the page.
- Training bots collect pages for AI models.
The Bots table below it lists every bot, with Kind, Visits, Verified, Refused and Last seen.
Refused is the share of visits answered with 403 (blocked or challenged) or 429 (rate limited). A 404 is a missing page, not a refusal. It needs a status code, which Cloudflare reports and the Express code can send. The Next.js and Astro code runs before your page answers, so it has no status code, and the column shows —.
Pages bots read
The Pages bots read table lists the pages with the most bot visits: Page, Visits, Bots, Cited (how many AI answers in the range cited that page) and Last seen. It shows the top 10. View all shows more.
Paths have no query string, and a trailing slash doesn't make a different page.
Read but never cited, cited but never read
Two lists compare what bots read with what AI answers cite. They come from the answers Knownwise collects for your questions.
Read but never cited. Bots read these pages, but no AI answer cited them. The bots can reach the page, so the problem is the content. Check whether the page answers the questions you track in plain words, with your product's name in the sentences an answer would quote. Opportunities may already list fixes for it.
Cited but never read. AI answers cited these pages, but no tracked bot read them in the range. Answers may be working from an old copy or a search index. Check that the page is in your sitemap, isn't blocked in robots.txt, and isn't refused by a firewall. A longer date range can also help, since some bots visit rarely.
Both lists show the top 20 pages, and only once tracking receives visits.
The firewall warnings
Firewalls and bot protection, such as Cloudflare's bot settings, Vercel's firewall or Netlify's traffic rules, sometimes turn AI bots away. Then AI assistants learn about you from other sites. Knownwise warns you in two ways.
Possibly blocked
Every site check, on every plan, asks for your page five times: once as a normal browser, then as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. If the browser gets the page and a bot gets a 403, a 429 or a challenge page, the site check shows a warning such as Your Cloudflare may be turning AI bots away. It names your host when it can tell (Cloudflare, Vercel or Netlify), or says firewall, and tells you where to look.
This is a warning, not a score. One refused test request doesn't prove the real bot is refused, because firewalls also look at IP addresses. On a site behind Cloudflare the warning says so: Cloudflare's bot protection often challenges test requests like ours, sent from a data centre, while it lets the real crawler through, so connect Cloudflare to see whether real visits are refused. The AI bots page shows the same warning in amber.
Blocked
When real visits show a bot being refused, the warning becomes Your site refuses and the bot's name, in red. That happens when more than half of a bot's visits in the last 7 days were refused (403 or 429), counting at least 20 visits that had a status code. Knownwise counts Cloudflare visits, and visits reported by your site where most of them were verified, so a fake bot can't trigger it.
A confirmed block lowers the site check's score and appears there as an issue of its own. A bot with a confirmed block drops off the amber warning.
To fix either one, find the rule that blocks or challenges the bot and allow it. On Cloudflare, look in Security → Bots, and AI Crawl Control. On Vercel, in Firewall. Elsewhere, in your firewall or bot-protection settings. See Audits for the site check.
On the overview
The brand overview has an AI bot visits tile, between the trend and the competitors:
- Visits, last 7 days, the Top bot, and a small 14-day chart that compares Last 7 days with the Week before. See AI bot visits opens this page.
- Waiting for the first AI bot visit while a route is set up but no bot has come yet.
- The suggested route and its button when nothing is set up.
The tile doesn't show on Free.
After a fix: did the bot come back?
When a fix pull request merges, the fix row on Opportunities and the fix's own page show whether a bot has read the fixed page since. For example, GPTBot re-read this page on and the date. Until then they show Waiting for an AI bot to re-read this page.
Only visits on a day after the merge day count. The line appears only when tracking was already receiving visits when the fix merged.
In the weekly report
Once a real AI bot visit is on record (a test visit doesn't count), the weekly report email adds one line with the number of AI bot visits, the change, and the bot that visited most. On Free, including after a downgrade, there is no line.