The robots.txt file is intended for search engine bots and is located in the main folder of the site on the ftp server. In the robots.txt file, the rules for the behavior of the bot are described – for example, whether it is allowed to visit the website at all and which parts of the website it is not allowed to visit. Along with this, in the robots.txt file, the bot must find the correct path to the xml sitemap.If the bot can find the link to the xml sitemap, it will allow faster and more detailed indexing of the website and the subsequent participation of the site in the search results.


