Basis: Google’s October 2026 statements that robots.txt is an “access-preference protocol”, Cloudflare’s Web Bot Auth documentation and bot trust score plans, DataDome’s Q3 2026 traffic report and Arkose Labs’ finding that 97.7% of legitimate AI agent sessions carry no verified signature.
Know what is visiting
- List the AI user agents in your logs and check them against the operators’ published tokens.
- Check for agentic browsers that do not identify themselves; DataDome says Muse “does not identify itself as an AI agent”.
- Note which flows they reach: forms, logins, carts.
Decide the rules
- Which bots may fetch public pages? Use the exact user-agent token in robots.txt, because custom descriptive rules are discarded.
- Remember robots.txt controls fetching, not what a system does with content it already crawled.
- Which flows need verification (checkout, login), and will you accept signed agents using Web Bot Auth?
- What is your position on training use, separate from search presentation and AI Overviews controls?
Review regularly
- Revisit after Cloudflare publishes bot trust bands, and after any change to Meta’s terms on crawling advertisers’ sites (effective October 30, 2026).