How to Block Search Engines and Remove Your Site from Google (2026)
How do you block search engines and remove your site from search results? Maybe it’s a personal site, a staging copy, a members-only area or an old page you never meant to make public. The right method depends on whether the page is already in Google. A common mistake is using robots.txt alone, which can leave pages in the results. Here is how to block search engines properly in 2026, and how to remove pages that are already indexed.
Quick answer: Add a noindex robots meta tag (or the X-Robots-Tag: noindex header) and let search engines crawl the page so they see it. For private content, password-protect it. To hide indexed pages fast, use the Removals tool in Google Search Console (temporary, about six months) while the noindex or a 404/410 takes effect. In WordPress, tick Settings > Reading > Discourage search engines.
Table of Contents
1. Noindex meta tag (best for most pages)
Add this inside the <head> of each page:
<meta name="robots" content="noindex, nofollow">Search engines drop the page once they recrawl it. For PDFs and images, send the same instruction as an HTTP header: X-Robots-Tag: noindex. In WordPress, SEO plugins such as Rank Math or Yoast have a noindex option per post.
2. robots.txt Disallow (blocks crawling, not indexing)
The robots.txt file in your site root tells well-behaved bots which paths not to crawl. To block all bots from the whole site:
User-agent: *
Disallow: /To block just some folders:
User-agent: *
Disallow: /private/
Disallow: /tmp/Important: Disallow stops crawling, not indexing. If other sites link to a blocked page, Google may still list its URL. And if you block a page in robots.txt, Google can’t see its noindex tag. So use noindex first, and only block crawling once the pages have dropped out. See robotstxt.org (external site) for the full syntax.
3. Password protection (best for private content)
Search engines can’t index what they can’t open. Use your host’s directory password protection, a members plugin, or WordPress’s per-post password option. Here’s how to password protect a web page.
4. Block search engines in WordPress
Go to Settings > Reading and tick Discourage search engines from indexing this site. WordPress then adds a noindex directive site-wide. Great for staging sites; remember to untick it when you launch.
How to remove pages that are already in Google
- Temporary removal: in Google Search Console (external site), open Removals > New request. Pages are hidden for about six months.
- Permanent removal: before that expires, add noindex, password protection, or delete the page so it returns 404 or 410. Here’s how to force 404 errors in WordPress or use 410 Gone.
- Bing: use the Block URLs tool in Bing Webmaster Tools.
- Not your site? Use Google’s “Results about you” tool to request removal of results showing your phone, address or email, and the Refresh Outdated Content tool for pages that have changed. Here’s how to clean up your online persona.
What about AI crawlers?
You can also block AI training bots in robots.txt with user agents such as GPTBot, Google-Extended and CCBot. Blocking Google-Extended doesn’t affect normal Google Search.
FAQ
How long does it take to remove a site from Google?
The Removals tool works within a day. A noindex takes effect after Google recrawls, from days to a few weeks.
Does robots.txt hide private information?
No. It is public and only a request. Anyone can read it, and bad bots ignore it. Use passwords for private data.
Can I block only Google?
Yes. Use User-agent: Googlebot in robots.txt, or <meta name="googlebot" content="noindex">.
$4.99Free when you subscribe
Get 100 tech terms explained in plain English
8-page PDF on AI, coding, slang and SEO + one tech email a week.
Sent by email. Unsubscribe anytime. See our privacy policy.
Can update it either the Robot.txt or Google Webmasters
Method 2 idea of blocking search engines worked perfect on my site.
Recently after that Google Webmasters notification , i have changed the wpadmin not crawl by mistake , because of that my wp folder got indexed
To stop Indexing we can mention that in the Robots.txt file in WordPress …
To stop indexing from the wordpress reading setting itself we can discourage search engines in my opinion.It that method ok ?
Is it okay to put those in a single line? Shouldn’t be each directive in a separate?
User-agent: *Disallow: /
Is it really important ? i would like to replicate this ideas to my blog. it will be helpful to on-page-optimization.
I have started my own phpLD directory and have found this advice very useful. I find that S.E.O friendly directories are the best to submit sites to. Also alot of people have said that no-follow links have no value… While this may be true in terms of getting link juice back to your site it doesn’t neccessarily mean a person’s site won’t get any traffic. In saying that, my directory is dofollow :).
@cyberhost
You can remove it via google webmaster tools. I hope this help
Will this work for sites already in index ? I need to get some old pages of a website removed from google index. How can we get it removed ?
question is: how do i block /blog/index part just?
Lisa i think your pages will be drop from google index in time. for your security change your file names
Read how to prevent content from being indexed or remove content from Google’s index here.
I followed your directions and made a robot.txt file. It worked for yahoo, but not google. I’m afraid it might not be working on google because someone found my page using google before I made the robot.txt file and it is somehow archived with google. Is there a way to fix this? Google is the most common search engine of all. I really don’t want my personal page popping up on it. Thanks!