How to Stop Search Engines From Indexing Site

By Posted 2005 Updated   BloggingGoogleSEO

How to block search engines? You created your very own personal home on the web. However, the site turning up into search engines and web directories, revealing your personal details. It is very simple to block search engines from indexing your website by adding a small META code in your web pages and adding a small text file called robots.txt. Protect your privacy.

Method 1

Add meta tag to head of your
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

Method 2

Create a robots.txt (all lower-case) using any simple text editor like Notepad. Save it into your root directory of your domain to prevent search bots from accessing any page on your site. Type the details exactly as give below into the robots.txt file

To exclude all robots from the entire server

User-agent: *
Disallow: /

To allow all robots complete access

User-agent: *
Disallow:

Or create an empty “/robots.txt” file.

To exclude all robots from part of the server

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/

To exclude a single robot

User-agent: BadBot
Disallow: /

To allow a single robot

User-agent: WebCrawler
Disallow: User-agent: *
Disallow: /

More information is available here.


14 comments on “How to Stop Search Engines From Indexing Site

  1. conservatories says:

    Useful but i found some places will ignore these codes.

  2. Lisa says:

    I followed your directions and made a robot.txt file. It worked for yahoo, but not google. I’m afraid it might not be working on google because someone found my page using google before I made the robot.txt file and it is somehow archived with google. Is there a way to fix this? Google is the most common search engine of all. I really don’t want my personal page popping up on it. Thanks!

  3. QuickOnlineTips says:

    Read how to prevent content from being indexed or remove content from Google’s index here.

  4. giysi giydirme oyunlar? says:

    Lisa i think your pages will be drop from google index in time. for your security change your file names

  5. gazduire web says:

    question is: how do i block /blog/index part just?

  6. cyberhost says:

    Will this work for sites already in index ? I need to get some old pages of a website removed from google index. How can we get it removed ?

  7. majalah says:

    @cyberhost
    You can remove it via google webmaster tools. I hope this help

  8. submityoursitenow.com says:

    I have started my own phpLD directory and have found this advice very useful. I find that S.E.O friendly directories are the best to submit sites to. Also alot of people have said that no-follow links have no value… While this may be true in terms of getting link juice back to your site it doesn’t neccessarily mean a person’s site won’t get any traffic. In saying that, my directory is dofollow :).

  9. Googma Sansar says:

    Is it really important ? i would like to replicate this ideas to my blog. it will be helpful to on-page-optimization.

  10. Stelio says:

    Is it okay to put those in a single line? Shouldn’t be each directive in a separate?

    User-agent: *
    Disallow: /

  11. James says:

    To stop indexing from the wordpress reading setting itself we can discourage search engines in my opinion.It that method ok ?

  12. Ajith says:

    To stop Indexing we can mention that in the Robots.txt file in WordPress …

  13. sam says:

    Recently after that Google Webmasters notification , i have changed the wpadmin not crawl by mistake , because of that my wp folder got indexed

  14. Tony says:

    Method 2 idea of blocking search engines worked perfect on my site.

Leave a Reply

Your email address will not be published. Required fields are marked *




css.php