Robots.txt can be a file that exists on the basis directory of each Site and can be used to instruct search engines like yahoo on which directories/documents of the website they will crawl and incorporate within their index. No matter if you're employed on Website positioning you, delegate it to https://news.rafeeg.ae