Robots.txt is a file that exists on the basis directory of each Internet site and may be used to instruct search engines like google on which directories/files of the web site they are able to crawl and consist of in their index. "Once you update these options, you'll likely see https://news.rafeeg.ae