The Correct Way to Add Sitemap Link to Robots.txt File

By Updated   BloggingSEO

How to add add Sitemap link to Robots.txt file? As you are aware that Ask, Google, Microsoft and Yahoo! have announced support of autodiscovery of Sitemaps via robots.txt file, eliminating the need to submit sitemaps to each search engine separately. Although the instruction seemed simple enough, I tried inserting the sitemap url and noticed errors in the Google Webmaster console.

Add Sitemap Link to Robots.txt File

The simple instructions are to insert the sitemap link in the robots.txt file like this
Sitemap: http://example.com/sitemap.xml

I inserted the Sitemap as instructed

Sitemap: http://www.quickonlinetips.com/sitemap.xml
User-agent: *
Disallow:

It all seemed fine until I checked my Google Webmaster tools dashboard and spotted errors in the robots.txt analysis and got these parsing errors.
Sitemaps Errors

Since I spotted this error, I checked the analysis report daily after modifying my robots.txt file in different ways (after all these top search engines had this all worked out before). I needed to wait for about a day to recheck reports since the webmaster tools check for a new robots.txt file approximately once per day. And this is what finally worked without error

Sitemap: http://www.quickonlinetips.com/sitemap.xml
User-agent: *
Disallow:

The basic change that is required is that you need a empty line after the sitemap link. Now there are no errors.The lesson learnt is that you need to edit your robots.txt file with care, since it directly influences the way search engines index your site and the Google Webmaster tools is a valuable tool to see if it is configured in the correct way.


14 comments on “The Correct Way to Add Sitemap Link to Robots.txt File

  1. abhishek bhatnagar says:

    i use a plugin called sitemap for wordpress but this article really helped me lot in understanding the entire concept….

  2. Terry Howard says:

    Nice catch, I went back and found a few with the same issues. I guess that’s what you get for being an early adopter. I’m wondering if this can be applied for multiple sitemaps? I’d like to tell the engines about my sitemap.xml feed as well as an ror.xml feed for products.

  3. km says:

    Nice tips… hope this robot.txt really can help increase traffic

  4. Dipankar says:

    The wordpress plugin to create sitemaps rocks … though i cant figure out how to push in the sitemaps of two different blogs i host at desinerd.com
    :(

  5. QuickOnlineTips says:

    Dipankar – Sitemaps for two different blogs needs 2 different robots.txt files in their root location.

  6. QuickOnlineTips says:

    Just in case you meant to put 2 sitemaps in the same robots.txt file, you can repeat the same code twice in the file.
    Sitemap: http://www.quickonlinetips.com/sitemap1.xml
    Sitemap: http://www.quickonlinetips.com/sitemap2.xml

    And then check in Google webmaster central if google understands the new robots.txt file.

  7. Rick says:

    I think I have mine right, but I can’t get Google to index my site no matter what I do.

    Anyone see what I have done wrong?

  8. Alli says:

    It is good idea to include sitemap into robots.txt I have done this. Will see couse my site very poor in index yahoo, google, MSN.

  9. TheNightOwl says:

    Hey QOT, nice one!

    Thanks very much for this slick and simple — but obviously very important! — little tip.

    Much appreciated.

    TheNightOwl

    P.S. It’s my first time here and I like what I see. I’ve bookmarked your site and I’ll be sure come back and have a poke around when I have a bit more time.

  10. Ryan says:

    Thanks for the simplistic / easy reading explanation.

  11. meet-sos says:

    you saved my life…. Thank you

    Blank spaces in my theme’s functions.php was the problem…

    I had this problem with mingle forum plugin (with sitemap.php)

  12. DW says:

    In the case of multiple sitemap files – can this point to the sitemap index XML file? Or must it include each sitemap XML individually?

  13. Isaac Sarayiah says:

    Hi
    I have added my sitemap to my robots file like this:

    User-agent: *
    Disallow:
    http://www.theSarayiahpost.com/sitemap.xml
    http://www.theSarayiahpost.com/ror.xml

    I do not seem to be getting any errors and bots are crawling my site, but they were doing that anyway. Is there anything wrong with what I have done?

    Your advice is much appreciated.
    Thank you.
    Isaac

  14. comfort flats says:

    Hey,
    i think adding sitemap link at end will do the trick. I have added my site map ate the end of the robots.txt and I had no issues with it. Just try and check what you think!

    Regards,
    Andy

Leave a Reply

Your email address will not be published. Required fields are marked *




css.php