Everyone uses artificial intelligence (AI) even if they don’t know it. AI has been used by the GAFAM for more than 10 years, Google uses it to search for the best website that matches your search intent, Apple for identifying who is in your pictures etc.
Nowadays, the power of AI is becoming mainstream with ChatGPT, everyone can use this power for various tasks.
In this article we will explore how this super AI works, the impact on content creator, how to protect your Data from openAI’s crawling and the potential impact by disallowing it.
Key takeaways
- ChatGPT is trained by crawling websites and extracting content from these websites
- ChatGPT webcrawler is known as GPTbot
- ChatGPT doesn’t ask permission to crawl websites
- You can disallow chatGPT to crawl your website with robots.txt or .htcaccess
- Pros: Protect your data, copyright infringement
- Cons: impact on SEO, hindering the improvement of the AI model.
How does ChatGPT Work?
ChatGPT is a language model developed by OpenAI. It uses machine learning algorithms to generate human-like text based on the input it receives.
The model has been trained on a diverse range of internet text, enabling it to generate creative, coherent, and contextually relevant responses.
However, it’s important to note that while ChatGPT can generate impressive text, it doesn’t understand the content it produces. It merely predicts the next word in a sentence based on its training.
The Role of GPTBot in AI Training
GPTBot is a web crawler developed by OpenAI. Its primary function is to crawl the internet and collect data to train OpenAI’s language models, including ChatGPT.
By gathering a diverse range of text from the internet, GPTBot helps in creating more robust and versatile AI models like GPT-4 and GPT-5.
GPTBot can be recognized by its user agent and string.
User agent token: GPTBot Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
The Pros for Blocking GPTbot
Compete with AI
Many content creators are wary of allowing AI companies to scrape their material, fearing that their content may be used to train future models that compete with them.
Copyright Infringement
AI web crawlers collect data from various sources without the creator’s consent, so there is a risk that copyrighted material may be used.
The Inability to Remove Content from Existing Datasets
Once your content has been crawled and included in a dataset, it cannot be removed.
Protect your content accuracy and integrity
chatGPT can “hallucinate” citations and sources, there is a risk of misquoting and misidentifying users / companies when AI chatbots like ChatGPT utilize public data. This poses a unique reputation risk, as the outputs generated by these models appear highly credible and can be disseminated at scale without any accountability.
The cons for Blocking AI crawl your website
While blocking AI web crawlers can help protect your content, there are several considerations to keep in mind before taking this step.
The Potential Impact on organic traffic
More and more user use AI to get information. By having your content included in the training data, there is a higher chance that AI models will recommend your site to users seeking relevant information on AI chatBOT.
Improve AI devloppment
One major benefit of using GPTbot for web crawling is the ability to collect diverse and up-to-date information. By accessing various websites, GPTbot can gather data from different sources, helping to create a comprehensive dataset for AI training. This diversity in data ensures that AI models are exposed to a wide range of perspectives and information, enabling them to make more informed decisions.
Practical Steps to Block OpenAI’s Web Crawler
OpenAI provides some guidelines for blocking his GPTbot : https://platform.openai.com/docs/gptbot
Using Robots.txt to Block GPTBot
One of the most effective ways to block GPTBot is by using a robots.txt file. This file is in the root directory of your website, instructs web crawlers which pages they can or cannot access. By specifying certain lines of code in your robots.txt file, you can effectively block GPTBot from accessing your content.
You can add these two lines in your robots.txt file for avoiding GPTBot to crawl your website
User-agent: GPTBot Disallow: /
You can add these three lines in your robots.txt file for accepting GPTBot to crawl only certain webpages
User-agent: GPTBot Allow: /directory-1/ Disallow: /directory-2/
Blocking AI Through .htaccess Files
Another effective way to block AI web crawlers is through .htaccess files. These files can be used to block specific IP ranges associated with GPTBot.
In August 2023, the GPTbot IP ranges are
20.15.240.64/28 20.15.240.80/28 20.15.240.96/28 20.15.240.176/28 20.15.241.0/28 20.15.242.128/28 20.15.242.144/28 20.15.242.192/28 40.83.2.64/28
It’s important to note that these IP ranges can change over time, so it’s essential to regularly check and update your .htaccess files as necessary with the IP included in this file.
https://openai.com/gptbot-ranges.txt
Who block GPTBot ?
stackoverflow
Conclusion
In conclusion, while AI web crawlers like GPTBot play a crucial role in AI development, they also raise several concerns among content creators you can stop GPTBot from crawling your website.
