# Robots.txt

URL: https://sellmydatatoai.net/glossary/robots-txt

Robots.txt is a file at the root of a website that tells automated crawlers which paths they may and may not fetch.

Robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may access. The rules were standardized as the Robots Exclusion Protocol in [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), published by the IETF in September 2022, which states that "these rules are not a form of access authorization".

## How do AI companies use robots.txt?

AI companies publish separate crawler names so sites can block training while staying in search. OpenAI says [disallowing GPTBot](https://developers.openai.com/api/docs/bots) tells it content should not be used for training, while OAI-SearchBot controls ChatGPT search. Google says [Google-Extended](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) controls use for Gemini training and grounding and "does not impact a site's inclusion in Google Search".

## Why does robots.txt matter for data licensing?

Robots.txt matters for licensing because content a developer cannot crawl is content it has to license. Compliance is voluntary, so a block reduces free access rather than ending it. Records behind a login, such as company systems, are not reachable by public crawlers, which is why they sell. See [what data AI companies buy](https://sellmydatatoai.net/guides/what-data-do-ai-companies-buy).

In the EU, a robots.txt rule can serve as a machine-readable reservation of rights under Article 4 of the Digital Single Market Directive.