New Licensing Standard Could Change How AI Uses Web Content
370Big internet companies and publishers are trying to find a way to stop AI bots from scraping their content without paying. They’re excited about a new standard called “Really Simple Licensing” (RSL). It aims to make it clear when and how AI companies can use content, and it also makes it easier for creators to get paid.
What Is the RSL Standard?
The RSL standard builds on an old web tool called robots.txt, which tells bots which parts of a website they can or can’t visit. But RSL adds a licensing layer that communicates terms for content use. It’s open and free, so any publisher can start using it right away. The idea is to tell AI crawlers exactly what’s allowed and what isn’t. If an AI bot wants to use content for training or responses, it needs to follow the rules, and if it doesn’t, it can be blocked.
The creators of RSL are a group called the RSL Collective, founded by Doug Leeds and Eckart Walther. Leeds used to run Ask.com, and Walther was a Yahoo VP and helped create RSS, the standard that made web syndication easy. RSL is based on RSS, which was originally designed to share content across sites. Now, they’re reusing that idea to give content creators more control and a way to earn money from AI.
How Does RSL Support Fair Payment?
The new standard supports different licensing models. Publishers can choose to give their content away for free, ask for attribution, or set up pay-per-crawl fees. That means they can get paid each time an AI crawler visits their site, or every time an AI uses their content to generate a response.
Leeds explained that the idea came after a talk at UC Berkeley, where he and Walther started thinking about how AI has disrupted search engines. Publishers used to make money from appearing in search results, but now AI outputs often reference their content without compensation. RSL aims to fix that by letting publishers license parts of their content to AI firms in a simple way. If AI companies want to use that content, they have to pay, just like search engines paid for snippets in the past.
Will AI Companies Play by the Rules?
Right now, tech giants like Google, Meta, OpenAI, and xAI haven’t commented on the new standard. Leeds said they didn’t consult with AI firms when developing RSL, but it’s clear that those companies rely heavily on web content. They’ll need to decide if they want to adopt RSL or find other ways to get the data they need.
It’s possible that enforcement could be tricky. RSL offers templates for publishers to add licensing language directly into their robots.txt files, which bots respect. For example, a line might say that all bots are forbidden from using content for AI training unless they have a license. Fastly, a cloud provider, is working with the RSL group to help enforce these rules by blocking unapproved bots. Cloudflare, another big player, has already started blocking some bots and could help with RSL enforcement too.
What Does This Mean for Content Creators?
For small creators and publishers, RSL could be a game-changer. It offers a way to turn their content into a revenue stream, especially as AI becomes more prevalent. Tony Stubblebine, CEO of Medium, called out how AI currently uses content without paying. He sees RSL as a way to force AI companies to pay or stop using their work.
Leeds believes that legal enforcement is also possible. If an AI firm violates licensing terms, publishers can take legal action. The recent $1.5 billion settlement with Anthropic shows there is real money involved in AI training, so companies will want to avoid legal trouble.
What’s Next for the RSL Standard?
The real test will be whether AI companies decide to adopt RSL. Early reactions are mixed. Some see it as a necessary step toward fairer use of online content. Others are waiting to see how much they need to change their crawling and training practices.
If widely adopted, RSL could help balance the power between content creators and AI firms. It could also make AI development more transparent and fair. For publishers, it offers a tool to protect their work and earn revenue in the age of AI. The next few months will show if the industry embraces this new licensing approach or if further tweaks are needed to make it work on a large scale.
What do you think?
It is nice to know your opinion. Leave a comment.