How to add add Sitemap link to Robots.txt file? As you are aware that Ask, Google, Microsoft and Yahoo! have announced support of autodiscovery of Sitemaps via robots.txt file, eliminating the need to submit sitemaps to each search engine separately. Although the instruction seemed simple enough, I tried inserting the sitemap url and noticed errors in the Google Webmaster console.
Add Sitemap Link to Robots.txt File
The simple instructions are to insert the sitemap link in the robots.txt file like this
Sitemap:
I inserted the Sitemap as instructed
Sitemap:
User-agent: *
Disallow:
It all seemed fine until I checked my Google Webmaster tools dashboard and spotted errors in the robots.txt analysis and got these parsing errors.
Since I spotted this error, I checked the analysis report daily after modifying my robots.txt file in different ways (after all these top search engines had this all worked out before). I needed to wait for about a day to recheck reports since the webmaster tools check for a new robots.txt file approximately once per day. And this is what finally worked without error
Sitemap:
User-agent: *
Disallow:
The basic change that is required is that you need a empty line after the sitemap link. Now there are no errors.The lesson learnt is that you need to edit your robots.txt file with care, since it directly influences the way search engines index your site and the Google Webmaster tools is a valuable tool to see if it is configured in the correct way.