The Correct Way to Add Sitemap Link to Robots.txt File
April 18th, 2007As you are aware that Ask, Google, Microsoft and Yahoo! have announced support of autodiscovery of Sitemaps via robots.txt file, eliminating the need to submit sitemaps to each search engine separately. Although the instruction seemed simple enough, I tried inserting the sitemap url and noticed errors in the Google Webmaster console.
The simple instructions are to insert the sitemap link in the robots.txt file like this
Sitemap: http://example.com/sitemap.xml
I inserted the Sitemap as instructed
Sitemap: http://www.quickonlinetips.com/sitemap.xml
User-agent: *
Disallow:
It all seemed fine until I checked my Google Webmaster tools dashboard and spotted errors in the robots.txt analysis and got these parsing errors.
Since I spotted this error, I checked the analysis report daily after modifying my robots.txt file in different ways (after all these top search engines had this all worked out before). I needed to wait for about a day to recheck reports since the webmaster tools check for a new robots.txt file approximately once per day. And this is what finally worked without error
Sitemap: http://www.quickonlinetips.com/sitemap.xml
User-agent: *
Disallow:
The basic change that is required is that you need a empty line after the sitemap link. Now there are no errors.The lesson learnt is that you need to edit your robots.txt file with care, since it directly influences the way search engines index your site and the Google Webmaster tools is a valuable tool to see if it is configured in the correct way.
Liked it? Subscribe feed and keep reading our latest articles for free.Share: Digg Delicious Stumbleupon Twitter Email to friend
Related Posts
- Add Sitemaps Autodiscovery in Robots.txt File
- Microsoft Robots Invasion Contest : Create Robots / BOTs
- Text Link Ads Calculator : Find Website Link Worth
- Robots Welcome Humans in Firefox 3
- RobotStats : Tracks Web Robots Visits for Free
i use a plugin called sitemap for wordpress but this article really helped me lot in understanding the entire concept….
Nice catch, I went back and found a few with the same issues. I guess that’s what you get for being an early adopter. I’m wondering if this can be applied for multiple sitemaps? I’d like to tell the engines about my sitemap.xml feed as well as an ror.xml feed for products.
Nice tips… hope this robot.txt really can help increase traffic
The wordpress plugin to create sitemaps rocks … though i cant figure out how to push in the sitemaps of two different blogs i host at desinerd.com
:(
Dipankar – Sitemaps for two different blogs needs 2 different robots.txt files in their root location.
Just in case you meant to put 2 sitemaps in the same robots.txt file, you can repeat the same code twice in the file.
Sitemap: http://www.quickonlinetips.com/sitemap1.xml
Sitemap: http://www.quickonlinetips.com/sitemap2.xml
And then check in Google webmaster central if google understands the new robots.txt file.
I think I have mine right, but I can’t get Google to index my site no matter what I do.
Anyone see what I have done wrong?
It is good idea to include sitemap into robots.txt I have done this. Will see couse my site very poor in index yahoo, google, MSN.