Block Search Engines From Indexing Site: META Tags & Robot.txt
January 16th, 2005You created your very own personal home on the web. However, the site turning up into search engines and web directories, revealing your personal details. It is very simple to block search engines from indexing your website by adding a small META code in your web pages and adding a small text file called robots.txt. Protect your privacy.
Method 1
Add meta tag to head of your
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Method 2
Create a robots.txt (all lower-case) using any simple text editor like Notepad. Save it into your root directory of your domain to prevent search bots from accessing any page on your site. Type the details exactly as give below into the robots.txt file
To exclude all robots from the entire server
User-agent: * Disallow: /
To allow all robots complete access
User-agent: * Disallow:
Or create an empty “/robots.txt” file.
To exclude all robots from part of the server
User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /private/
To exclude a single robot
User-agent: BadBot Disallow: /
To allow a single robot
User-agent: WebCrawler Disallow: User-agent: * Disallow: /
More information is available here.
Subscribe RSS feed
nice blog!
nice blog!
Useful but i found some places will ignore these codes.
I followed your directions and made a robot.txt file. It worked for yahoo, but not google. I’m afraid it might not be working on google because someone found my page using google before I made the robot.txt file and it is somehow archived with google. Is there a way to fix this? Google is the most common search engine of all. I really don’t want my personal page popping up on it. Thanks!
Read how to prevent content from being indexed or remove content from Google’s index here.
Lisa i think your pages will be drop from google index in time. for your security change your file names
question is: how do i block /blog/index part just?
This is what i have been looking for, finally i have found here: