Blocked by robots.txt: How to Find and Fix Blocked URLs (2026)
In 2013, while investigating a 50% traffic drop on QuickOnlineTips after Panda, we found that robots.txt was blocking a growing number of our URLs every week. Our host, KnownHost, later suggested the CSF server firewall was intermittently blocking Googlebot and added rules to allow it; we had seen CSF block site traffic before. Two months later blocked pages were still in the thousands. If Search Console shows blocked by robots.txt for your pages, here’s how to fix it in 2026.
Quick answer: In Search Console, open Indexing > Pages and click “Blocked by robots.txt” to see affected URLs. Open Settings > robots.txt report to see the file Google fetched and any errors. Remove or narrow the Disallow rule if those pages should be indexed, make sure your firewall or CDN isn’t blocking Googlebot, then validate the fix.
Table of Contents
Why URLs get blocked
- A broad Disallow rule, like
Disallow: /left over from a staging site. - Rules blocking folders with real content or CSS/JS files.
- WordPress “Discourage search engines” setting (Settings > Reading).
- Server firewalls or bot protection returning errors to Googlebot when it fetches robots.txt.
How to fix blocked by robots.txt
- Check the file: visit yourdomain.com/robots.txt and read each Disallow line.
- Use Search Console’s robots.txt report (Settings) to see fetch status and errors.
- Test URLs with URL Inspection; it shows whether robots.txt blocks crawling.
- Edit the file: in WordPress, SEO plugins like Rank Math and Yoast include a robots.txt editor.
- Whitelist Googlebot in your firewall or CDN if it’s being blocked (verify real Googlebot by reverse DNS).
- Validate the fix in the Pages report and request indexing for key pages.
Indexed, though blocked by robots.txt
This status means Google indexed a URL from links without crawling it. If you don’t want it indexed, allow crawling and add a noindex tag instead; robots.txt alone doesn’t remove pages from the index.
Good robots.txt habits
- Keep rules minimal; block only areas with no search value.
- Don’t block CSS and JavaScript Google needs to render pages.
- List your sitemap:
Sitemap: https://example.com/sitemap_index.xml.
Read Google’s robots.txt introduction (external site), and see how to check if Google indexed your site.
FAQ
Does robots.txt remove pages from Google?
No. It stops crawling, not indexing. Use noindex or removal tools instead.
How long until Google recrawls after a fix?
Google refreshes robots.txt often, usually within a day; recrawling pages can take longer.
Can a firewall cause robots.txt errors?
Yes. If Googlebot gets errors fetching robots.txt, crawling can be paused.
Stuck on a gift? Try our free Gift Idea GeneratorIdeas for anyone by age, job, interests and budget, with Amazon.com and Amazon.in linksFind gifts →Get our free weekly email: new tech guides and tools, no spam.
New and updated tech guides, a free tool worth bookmarking and one useful pick, once a week. No spam, unsubscribe anytime.
One email a week. Unsubscribe anytime. See our privacy policy.
nice post how to check which url is blocked by robots.txt if we have not block manually in robots.txt please advise me how to check.
Robots.txt is a sensitive file and should not be touched unless you have experience.
A tip for people using WordPress, if you include this line:
Disallow: /page/
You can avoid problems with duplicate pages due to the wordpress pagination system on blogs.
But take care when modifying this file and always test with Webmaster Tools!
Thanks for the good suggestion.
Seems like there was a panda update recently. Did you see any change in your stats? My blog was hit back in Nov. of 2012 and the latest update ( 7/ 9/ 2013) hit me again.
Just wondering if yours change.
Thanks,
Hi Chandra,
Thanks for sharing info, in my robot.txt the sitemap is appearing as Sitemap: https://www.techpopular.com/. is it ?
why that .gz extension has come, any issues with that? does it require to change? please guide me
The sitemap plugins usually generate both sitemap files – sitemap.xml and the zipped smaller size sitemap.xml.gz
Hi Chandra
I dont’ use a robots.txt file as there are situations when some variables from the web server come in conflict with the commands from the robots.txt
If I want to block a specific page I use a meta line like this:
meta name=”robots” content=”noindex, nofollow”
thanks for info about Robot.txt. i would like to learn more about Robot.txt!!
Robot.txt, very powerful if any one use it carefully otherwise it is only tool which decrease so much traffic without any update :( :)
An important, but sometimes overlooked element of onsite optimisation is the robots.txt file. This file alone, usually weighing not more than a few bytes, can be responsible for making or breaking your site’s relationship with the search engines.
I’m new on these things, what I did was copy the robots.txt file of a blog that I follow, but recently I have received messages of blocking url by google. Now I just make the same changes that you and I’ll let you know if works.
Thanks
We should use robots.txt which is suitable for our sites. Other websites may have reasons for various configurations.
You can add this anywhere in the robots.txt file because the directive is independent of the user-agent line. All you have to do is specify the location of your Sitemap in the sitemap-location.xml part of the URL. If you have multiple Sitemaps you can also specify the location of your Sitemap index file. Learn more about sitemaps in our blog on XML Sitemaps .
Thanks for the tip, I checked and I am running the old style Robots.txt so I redid it. Will track the results to see if it helps or not.
I knew it!, For a moment I entered the paranoia, since I read several times that google and other search engines, make your robots.txt file only as a TIP, most do not follow it to the letter. Moreover, I put as blocked hundreds of directories, indexed however normal rule I put it, but I do not consider important things.
Cheers =)
Indeed robots.txt is a powerful file and should be used with care.
This one may be the actual reason
Disallow: /*/trackback
/*/ it means all kind of urls will be blocked which have /trackback at the end.
because trackback redirects bot to the previous url which is your post. and your post is blocked by *
Just a guess. But this could be a reason I think.
That was a good pick. Worth thinking more about this. That code was meant to block and not index trackbacks only which are just redirect to the main url. Should be more careful before getting these codes from *popular websites, as they too could be wrong.
fed up with site connection time out errors i moved my hosting from knownhost to other web-host
How did you find out about these connection time out errors?
Today I too got a mail from Google team about the same issue. I just rebuild my sitemap and checked the status in webmaster tools. Now everything is fine.
What email did you get? Can you post an excerpt?