Guide SSL & Security

How to keep a site out of search engines

A request that well-behaved crawlers respect is very different from actually restricting access. Here is which setting does which job.

iFastHosting Support Team Updated 26 August 2026 6 min read Beginner

"Hide from search engines" actually describes two different needs that get confused with each other, and picking the wrong one is a common, avoidable mistake. One is asking search engines not to list a page that is otherwise perfectly public. The other is genuinely restricting who can see something at all. Only one of the settings below does the second thing.

The noindex tag: keep it out of results, but still public

For a page that should remain reachable by anyone with the link but should not appear in search results, add a meta tag to the page's HTML:

<meta name="robots" content="noindex">

Search engines that respect this will stop showing the page in results and, over time, remove it from their index if it was already there. The page itself remains fully accessible to anyone who visits it directly — this setting only affects whether it turns up in search results, not who can open it.

For non-HTML files, use a header instead

A PDF, an image, or any file that cannot carry a meta tag can use the equivalent X-Robots-Tag: noindex response header instead, set at the server level rather than inside the file itself.

Robots.txt: a request, not a restriction

A robots.txt file at the root of your site can ask crawlers not to visit certain paths at all:

User-agent: *
Disallow: /staging/

This is worth understanding precisely: it is a courtesy request that well-behaved crawlers choose to respect, not an access control mechanism of any kind. It does nothing to actually restrict who can view the content — anyone who has the direct address, whether from a shared link, a bookmark, or simply guessing, can still open the page normally. A poorly behaved crawler can also simply ignore the file entirely, since nothing enforces it. See how to write a robots.txt file for the full syntax.

Do not rely on this for anything that needs to be genuinely private

A disallowed path in robots.txt is, in effect, a published list of paths you would rather people did not look at — which is the opposite of hiding them. Never use robots.txt as the only protection for a staging site, an admin area, or anything containing sensitive information.

Password protection: genuinely restricting access

If content actually needs to be kept private — a staging copy of a live site, an internal tool, anything not meant for the public — the correct setting is access restriction at the server level, which requires a login before any content is served at all, to anyone, crawler or human. How to password-protect a folder covers setting this up. This is the only one of the three approaches here that actually stops an uninvited visitor from viewing the content, rather than simply asking search engines not to list it.

A common mistake: combining the two incorrectly

A frequent error is disallowing a path in robots.txt and adding a noindex tag to the pages inside it, expecting the two to work together. In practice they can undermine each other: if robots.txt tells a crawler not to visit a path at all, that crawler never actually reaches the page to read the noindex tag on it, so a page already indexed before the disallow was added can sometimes remain in results indefinitely, because the crawler is now being told to stay away before it can see the instruction to remove it. If the goal is genuinely removing a page from an index, allow crawling and rely on the noindex tag alone, rather than combining both.

Confirming it worked

For a noindex tag, view the page's source and confirm the meta tag is present exactly as written, with no typo in the attribute values. Removal from an index is not instant even once the tag is correctly in place, since it depends on the search engine revisiting the page, so allow some time before concluding it has not worked. For robots.txt, most search engines provide a tool that lets you check how they currently interpret your file, which is more reliable than assuming your syntax is being read the way you intended.

Choosing the right one

Situation	Use
Public page, just do not want it in search results	Noindex meta tag
Asking well-behaved crawlers to skip a whole section	Robots.txt, understanding it is only a request
Genuinely private content, no public access at all	Password protection

Does keeping something out of search engines affect the rest of the site?

No. Excluding one page or section with a noindex tag or a robots.txt entry has no effect on how the rest of your site is treated in search results — it applies only to the specific pages or paths targeted. If you are more broadly concerned with how the site as a whole performs in search, that is a separate topic covered in whether HTTPS affects SEO and elsewhere in the SEO section of this knowledgebase, and is unrelated to the settings covered here.

Frequently asked questions

Will robots.txt stop people finding my staging site?

No, and this is the single most common misunderstanding here. Robots.txt is a request to well-behaved crawlers, not an access restriction. Anyone with the direct address can still visit the site regardless of what robots.txt says, and a link to the page shared anywhere can still lead people there. For genuinely private content, use password protection instead.