Developing
Google listed no AI crawler tokens and gave no guidance on opting out of training beyond separating crawl controls from training policy.
In 60 seconds
- Illyes called robots.txt "an access-preference protocol" and said it "doesn't dictate what a system does with the data after it has already crawled it".
- To block a specific AI crawler the rule must name that crawler's user-agent token; custom descriptive rules are discarded.
- User-agent tokens can only contain letters and dashes, so emojis and non-English characters cannot work; lines beginning with a hash are comments.
- Google recommends percent-encoding for emojis and non-English characters in robots.txt rule paths.
Impact Layer
- What changed
- Google restated what robots.txt can and cannot do, including for AI crawlers.
- Why it matters
- Many sites expect robots.txt to control training use, but it only controls fetching.
- Who is affected
- SEOs, publishers and developers who manage robots.txt files.
- Martech stack area
- SEO & Search, Content
- Stack impact
- Use exact user-agent tokens, percent-encode non-ASCII paths and treat training opt-outs as a separate policy question.
- Business impact
- Moderate Reported by Search Engine Roundtable. Illyes did not confirm Cherry Prommawin's claim that Google's parser automatically encodes characters, and the video gives no list of crawler tokens or which crawlers honour robots.txt.
- Related technologies
- Related vendors
- What to watch
- Google guidance on training-specific controls.
Barry Schwartz reported on October 21, 2026 on the second video in Google’s “Non-Canonical Cabinet” series, hosted by Gary Illyes and Cherry Prommawin.
AI crawlers
- Illyes called robots.txt “an access-preference protocol” that governs whether a bot may fetch a page.
- “It doesn’t dictate what a system does with the data after it has already crawled it.”
- To block a specific AI crawler, the rule must name that crawler’s user-agent token; custom descriptive rules are discarded.
Emojis and international URLs
- “User-agent tokens can only have letters and dashes”, so emojis and non-English characters cannot work in a token.
- Lines beginning with a hash are comments that crawlers ignore.
- Google’s closing recommendation: “use percent-encoding for emojis and non-English characters in robots.txt rule paths”, because other crawlers may misread raw characters and ignore the whole disallow line.
Parser behaviour
Google’s parser reads line by line and recognises only user-agent, allow, disallow and sitemap. Malformed lines are silently dropped.
Not stated
A list of AI crawler tokens, confirmation that Google’s parser automatically encodes characters, which AI crawlers honour robots.txt beyond “well-behaved” ones, and guidance on opting out of training.
Our read
Robots.txt is about fetching. If you care about training use, you need a separate policy and, where available, a separate control.
Sources
- Search Engine Roundtable: Google On robots.txt Rules For AI Crawlers, Emojis, International URLs & More (opens in a new tab)
Source for the video's statements and what is not stated.