Guide SEO & Marketing

How to check Google can crawl your site

Being online and being crawlable are not the same thing. Here is how to check the difference directly rather than assume it.

iFastHosting Support Team Updated 26 August 2026 7 min read Intermediate

A page loading correctly in your own browser tells you almost nothing about whether Google's crawler can reach it. Your browser ignores robots.txt, has no idea what a noindex tag is, and is never blocked by a security rule aimed at bots. Checking crawlability means checking from the crawler's perspective directly, using the tools built for exactly that.

Start with URL Inspection

Inside Google Search Console, paste the exact URL you are checking into the URL Inspection tool. It reports the last time Google crawled the page, whether the page is currently indexed, and — if it is not — a reason. Look specifically for statuses that indicate a crawling problem rather than an indexing decision: "Blocked by robots.txt", "Blocked due to unauthorised request (401)", or "Server error (5xx)" all mean the crawler could not get the page at all, which is a different and more urgent problem than a page Google could fetch but chose not to index.

Test robots.txt against the specific URL

A general read-through of robots.txt catches obvious mistakes, but the robots.txt tester in Search Console checks one exact URL against the live file and gives a definite allowed or blocked answer, along with which specific rule caused it. This is worth doing even when you are fairly confident the file is correct — a wildcard or an overlapping rule further down the file can block a URL in a way that is not obvious from reading top to bottom. See how to write a robots.txt file for the syntax if the test reveals a rule you did not expect.

Check for a noindex tag hiding in the page

View the page's actual HTML source — not the rendered page, the raw source — and search for noindex. It can appear as a meta tag in the <head>:

<meta name="robots" content="noindex">

or as an X-Robots-Tag HTTP response header, which will not show up by reading the visible page at all and needs a browser's network tab or an online header checker to see. This second form is easy to miss and is a common leftover from a staging environment that was configured to keep search engines out before launch.

A robots.txt block and a noindex tag together cancel each other out — badly

If a URL is disallowed in robots.txt, Googlebot never fetches it, which means it never sees a noindex tag sitting inside that page either. The two are not redundant safety measures that stack; the robots.txt rule actively prevents the noindex tag from ever being read. If you genuinely want a page kept out of the index, allow it to be crawled and rely on noindex alone.

Check what Googlebot actually sees rendered

Modern crawling involves rendering JavaScript much like a browser does, but it is not identical, and a page that depends heavily on client-side scripts to display its main content can render differently — or fail to render key content at all — for a crawler. The URL Inspection tool includes a rendered screenshot and the rendered HTML it used for indexing; compare that against what a normal visitor sees. A mismatch here points to a JavaScript rendering problem rather than anything to do with robots.txt or noindex.

Rule out the server itself

Firewalls, security plugins and rate-limiting tools occasionally block Googlebot by mistake — usually as collateral damage from a rule aimed at malicious bot traffic that turns out to be broader than intended. If crawling has stopped with no change to robots.txt or any meta tag, check your security tool's access logs for blocked requests, and confirm any allowlist rule covers Google's published crawler IP ranges rather than relying on user-agent string matching alone, which can be spoofed and is sometimes deliberately ignored by security tools for that reason.

What to do once you have found the cause

Each of these has a specific fix rather than a general one, and it is worth resisting the urge to change several things at once. If robots.txt is the cause, remove or correct the single rule responsible and retest that exact URL before touching anything else — a broad rewrite of the whole file risks introducing a second problem while you are still trying to confirm the first fix worked. If a noindex tag is the cause and it was left over from a staging build, check whether your CMS has a single sitewide "discourage search engines" setting rather than a scattering of individual page tags, since that single setting is a common source of an entire site being blocked at once. If the cause is server-side blocking, adjust the specific firewall or security rule rather than disabling the tool altogether.

Working through these in order — Search Console status, robots.txt, noindex, rendering, then server-level blocking — covers the overwhelming majority of "why is this page not showing up" cases without needing to guess.

Frequently asked questions

My page loads fine in my browser. Why would Google not be able to crawl it?

Because your browser is not subject to robots.txt rules, is not blocked by any bot-specific firewall setting, and does not care whether a noindex tag is present. All three of those can silently stop Googlebot while a normal visitor sees nothing wrong at all.

Does a security plugin or firewall ever block Googlebot by mistake?

Yes, this happens more often than site owners expect. Overly aggressive bot-blocking, rate limiting, or a rule that blocks unfamiliar user agents can catch Googlebot along with genuinely malicious traffic. Check your security tool's logs for blocked requests from Google's published IP ranges if crawling seems to have stopped for no visible reason.