1001 SEO Media
All posts
By 1001 SEO MediaRSLrobots.txttechnical SEOAI crawlers

What is RSL (Really Simple Licensing), and should publishers use it to control how AI uses their content? An SEO agency answers

An SEO and GEO agency on RSL: how the licensing standard sits on top of robots.txt, how it interacts with Googlebot and Google-Extended, and when to add it.

Publisher clients have started forwarding us articles about RSL with one question attached: do we need this? Our answer depends on what kind of site they run, and it always starts with the file every SEO already knows, robots.txt.

RSL in plain terms

Really Simple Licensing is an open standard, now at version 1.0 with "Recommendation" status since December 10, 2025, for telling AI systems what they may do with your content and on what terms. You publish an XML license file and announce it, most often with a License: line in robots.txt. The file can allow or forbid uses such as training (ai-train) or feeding content into an answer (ai-input), and attach a payment model such as attribution, subscription, or per-use.

So RSL does not replace robots.txt. It rides along in it. That matters for SEO, because the robots.txt file that carries your license line is also the one that decides whether Googlebot can index you.

How it fits with the Google rules we already manage

Google is the engine most of our clients care about first, and its documentation keeps search and AI training apart.

Googlebot rules govern Google Search, including AI Overviews and AI Mode. Google says those features follow normal SEO practice and need no special files or markup. If you want to limit what appears in them, the tools are nosnippet, data-nosnippet, max-snippet and noindex.

Google-Extended is a separate robots.txt token. Google says it controls whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and that it does not affect inclusion in Google Search or act as a ranking signal. It has no user agent of its own.

Google's List of Google's common crawlers page, which notes that common crawlers always obey robots.txt when crawling automatically

Google Crawling infrastructure docs, "List of Google's common crawlers", captured November 21, 2026.

Google also says it intends to support the AIPREF machine-readable training preference signal in early 2027. Nothing on Google's pages that we read mentions RSL.

The practical point: a publisher who wants to stay in Google's AI features but keep content out of Gemini training already has a documented switch. RSL adds the ability to state a price, which Google-Extended cannot.

What we tell different clients

For news and reference publishers with large archives, adding RSL is reasonable. It costs little, it puts your terms on record in an open, documented format, and it is the entry point to the Internet Licensing Collective, the nonprofit formerly called the RSL Collective, which says it is built on RSL. We are clear with them that no AI company we checked has committed to honoring it, and that charging for access means server-side work: RSL lets a server answer 401 or 402 when a license is needed, and it defines protocols for license tokens and crawler authorization.

For brands, ecommerce sites and B2B companies, our advice is usually to leave RSL for later. Their goal is to be cited. We would rather spend the hours confirming that OAI-SearchBot, Claude-SearchBot and PerplexityBot can reach the pages that matter, and that a CDN bot rule is not quietly blocking them.

For anyone, we never touch the file without checking how it interacts with Googlebot. A robots.txt edit made for AI reasons that breaks Search crawling is the most expensive mistake on this list.

Whether RSL terms are legally enforceable is outside what we can advise on. None of our sources settle it.

How we measure before and after

We run client programs on Promptwatch, and for a licensing decision the crawler logs are where we start. Agent Analytics shows which AI bots hit which pages, verified against each operator's IP ranges, so we can tell a client how much of their AI crawl comes from training bots and how much from search indexers. Promptwatch's own crawler traffic report gives OpenAI 79.8% of IP-verified AI bot requests for August 31 to September 6, 2026, which is a useful sanity check against a client's numbers.

Search Insights then puts Google Search Console and Bing Webmaster data next to AI crawls and citations per page, so a change made for AI shows up next to its effect on classic search.

If you want to see your own bot mix before deciding anything, a free seven-day Promptwatch trial on a monthly Professional plan, the tier where crawler logs begin, will show it.