Robots.txt is a text file where search engine bots are given some instructions and are generally informed about which pages they can access. This file; It is a file with a txt extension that is easily accessible and can be read and understood quickly, giving information to bots that crawl websites and serve good purposes. By taking advantage of the robots.txt file, we can ensure that some of our pages and even our entire website are closed or examined by bots.
Unfortunately, this file, which can cause very risky damage to the site after a misconfiguration, can be misused by many people in a way that can negatively affect site performance.