{"id":33254,"date":"2013-04-26T22:03:12","date_gmt":"2013-04-26T16:33:12","guid":{"rendered":"https:\/\/www.quickonlinetips.com\/archives\/?p=33254"},"modified":"2026-10-13T13:41:52","modified_gmt":"2026-10-13T08:11:52","slug":"bad-robots-txt-blocked-urls","status":"publish","type":"post","link":"https:\/\/www.quickonlinetips.com\/archives\/2013\/04\/bad-robots-txt-blocked-urls\/","title":{"rendered":"Blocked by robots.txt: How to Find and Fix Blocked URLs (2026)"},"content":{"rendered":"

In 2013, while investigating a 50% traffic drop on QuickOnlineTips after Panda, we found that robots.txt was blocking a growing number of our URLs every week. Our host, KnownHost, later suggested the CSF server firewall was intermittently blocking Googlebot and added rules to allow it; we had seen CSF block site traffic<\/a> before. Two months later blocked pages were still in the thousands. If Search Console shows blocked by robots.txt<\/strong> for your pages, here’s how to fix it in 2026.<\/p>\n

\"Blocked<\/p>\n

Quick answer:<\/strong> In Search Console, open Indexing > Pages and click “Blocked by robots.txt” to see affected URLs. Open Settings > robots.txt report to see the file Google fetched and any errors. Remove or narrow the Disallow rule if those pages should be indexed, make sure your firewall or CDN isn’t blocking Googlebot, then validate the fix.<\/p>\n

\n

Table of Contents<\/p>\n