April 18th, 2007
The Correct Way to Add Sitemap Link to Robots.txt File
As you are aware that Ask, Google, Microsoft and Yahoo! have announced support of autodiscovery of Sitemaps via robots.txt file, eliminating the need to submit sitemaps to each search engine separately. Although the instruction seemed simple enough, I tried inserting the sitemap url and noticed errors in the Google Webmaster console.
The simple instructions are to insert the sitemap link in the robots.txt file like this
Sitemap: http://example.com/sitemap.xml
I inserted the Sitemap as instructed
Sitemap: http://www.quickonlinetips.com/sitemap.xml
User-agent: *
Disallow:
It all seemed fine until I checked my Google Webmaster tools dashboard and spotted errors in the robots.txt analysis and got these parsing errors.

Since I spotted this error, I checked the analysis report daily after modifying my robots.txt file in different ways (after all these top search engines had this all worked out before). I needed to wait for about a day to recheck reports since the webmaster tools check for a new robots.txt file approximately once per day. And this is what finally worked without error
Sitemap: http://www.quickonlinetips.com/sitemap.xml
User-agent: *
Disallow:
The basic change that is required is that you need a empty line after the sitemap link. Now there are no errors.The lesson learnt is that you need to edit your robots.txt file with care, since it directly influences the way search engines index your site and the Google Webmaster tools is a valuable tool to see if it is configured in the correct way.
Continue getting our new articles by RSS or email
Related articles
Add Sitemaps Autodiscovery in Robots.txt File
Google Sitemaps for Better Indexing on Google
Google Sitemap Generator for WordPress
Update Your Google Sitemaps Generator Wordpress Plugin
Share Your OPML : Sharing Outlines, Feeds and Taxonomy




Comments
RSS feed for comments on this post.1. abhishek bhatnagar | 18/04/07 #
i use a plugin called sitemap for wordpress but this article really helped me lot in understanding the entire concept….
2. Terry Howard | 19/04/07 #
Nice catch, I went back and found a few with the same issues. I guess that’s what you get for being an early adopter. I’m wondering if this can be applied for multiple sitemaps? I’d like to tell the engines about my sitemap.xml feed as well as an ror.xml feed for products.
3. km | 14/05/07 #
Nice tips… hope this robot.txt really can help increase traffic
4. Dipankar | 13/09/07 #
The wordpress plugin to create sitemaps rocks … though i cant figure out how to push in the sitemaps of two different blogs i host at desinerd.com

5. QuickOnlineTips | 13/09/07 #
Dipankar - Sitemaps for two different blogs needs 2 different robots.txt files in their root location.
6. QuickOnlineTips | 13/09/07 #
Just in case you meant to put 2 sitemaps in the same robots.txt file, you can repeat the same code twice in the file.
Sitemap: http://www.quickonlinetips.com/sitemap1.xml
Sitemap: http://www.quickonlinetips.com/sitemap2.xml
And then check in Google webmaster central if google understands the new robots.txt file.
7. Rick | 2/01/08 #
I think I have mine right, but I can’t get Google to index my site no matter what I do.
Anyone see what I have done wrong?
8. Drainedge Link Tank · Today’s Links | 18/04/07
9. Sitemaps Autodiscovery in robots.txt at Egonitron | 1/06/07
10. Top Five Posts of the Week | Million Dollars Journey Dot Com | 30/10/07
Comment on “The Correct Way to Add Sitemap Link to Robots.txt File”