Robots.txt
By Gareth HoyleLast updated How we research
Robots.txt is a file at the root of a website that tells automated crawlers which paths they may and may not fetch.
Robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may access. The rules were standardized as the Robots Exclusion Protocol in RFC 9309, published by the IETF in September 2022, which states that "these rules are not a form of access authorization".
How do AI companies use robots.txt?
AI companies publish separate crawler names so sites can block training while staying in search. OpenAI says disallowing GPTBot tells it content should not be used for training, while OAI-SearchBot controls ChatGPT search. Google says Google-Extended controls use for Gemini training and grounding and "does not impact a site's inclusion in Google Search".
Why does robots.txt matter for data licensing?
Robots.txt matters for licensing because content a developer cannot crawl is content it has to license. Compliance is voluntary, so a block reduces free access rather than ending it. Records behind a login, such as company systems, are not reachable by public crawlers, which is why they sell. See what data AI companies buy.
In the EU, a robots.txt rule can serve as a machine-readable reservation of rights under Article 4 of the Digital Single Market Directive.