Guide SEO & Marketing

How to write a robots.txt file

The syntax, the common mistakes, and the one thing robots.txt cannot do no matter how you write it.

iFastHosting Support Team Updated 26 August 2026 8 min read Intermediate

Robots.txt is a plain text file, sitting at the root of your domain, that tells well-behaved crawlers which parts of the site they may fetch. It is one of the oldest conventions on the web, it has no legal force, and it is frequently misunderstood in a way that causes real damage: people use it to try to keep a page out of search results, which is not what it does.

The syntax, in full

The file is a series of groups. Each group starts with one or more User-agent lines naming which crawler it applies to, followed by Allow and Disallow rules.

User-agent: *
Disallow: /admin/
Disallow: /cart.php
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

User-agent names the crawler the group applies to. * means every crawler that does not have its own named group elsewhere in the file.
Disallow names a path the crawler should not fetch. Disallow: /admin/ blocks that folder and everything inside it; Disallow: / blocks the entire site.
Allow carves out an exception inside a disallowed path — useful when you need to block a folder but keep one file inside it crawlable.
Sitemap points to your sitemap's full URL. It can appear anywhere in the file and is not tied to any particular User-agent group.

A named group replaces the wildcard group for that crawler — it does not add to it

If you write a specific User-agent: Googlebot group, Googlebot reads only that group and ignores the User-agent: * group entirely, even though other rules exist there. A named group has to repeat every rule you want that crawler to follow. Forgetting this is how a well-intentioned specific rule for one bot accidentally un-blocks everything else for it.

What robots.txt does not do

This is the single most consequential misunderstanding about this file, so it is worth stating plainly: robots.txt blocks crawling. It does not block indexing. A search engine can still list a disallowed URL in its results — typically as a bare link with no title or description pulled from the page — if enough other pages link to it. Blocking the crawl does not remove the URL from consideration; it just stops the engine from reading what is on the page.

If your actual goal is to keep a specific page out of search results entirely, the correct tool is a noindex meta tag or response header on that page, not a robots.txt rule. And there is a trap hiding inside that fix too:

A page blocked in robots.txt cannot have its noindex tag seen

A crawler that is disallowed from fetching a page never reads that page's contents — including any noindex tag sitting in its <head>. If you disallow a URL in robots.txt and add noindex to it, the noindex is never read and does nothing. To reliably remove a page from the index, let it be crawled and rely on noindex alone; use robots.txt only for pages you are content to have discovered but never fetched.

What is actually worth blocking

Resist the urge to block things defensively. A sensible robots.txt is usually short:

Admin and login areas that have no reason to be crawled.
Internal search result pages, which multiply endlessly and add nothing a crawler needs to index.
Cart, checkout and account pages behind a login, which are dead ends for a crawler and sometimes sensitive.
Staging or duplicate environments, if for some reason they are reachable on the open web at all.

Do not block CSS or JavaScript files needed to render the page. Search engines render pages much like a browser does before evaluating them, and a page that cannot load its own stylesheet or script because robots.txt disallowed it can be judged as broken or low quality.

Where the file has to live

Robots.txt is only ever read from one location: https://yourdomain.com/robots.txt. A file placed anywhere else, or served only for a subfolder, is simply never checked — there is no way to have a different robots.txt per section of a site on the same domain.

Common mistakes

Disallow: / left in from a staging build. This blocks the entire site and is the single most common cause of a launched site sitting unindexed. Check this first if a new site is not appearing anywhere.
Blocking a path you meant to noindex instead. Covered above — the fix is usually to remove the robots.txt rule and rely on the noindex tag alone.
Wildcards used incorrectly. Disallow: /product blocks /product, /products, and /product-reviews alike, because it matches anything starting with that string. Use a trailing slash — /product/ — when you mean the folder and nothing that merely starts with the same letters.
Assuming it stops bad bots. Robots.txt is a request, not an enforcement mechanism. A crawler that ignores it simply ignores it; there is nothing in the protocol that can compel compliance.

Testing before you rely on it

Search Console includes a robots.txt tester that shows exactly how Google's crawler will interpret a given URL against your live file — whether it is allowed, and which specific rule caused a block if it is not. Use it after any change rather than assuming the syntax did what you intended; a single misplaced character in a Disallow path is easy to write and easy to miss on a read-through.

Once it is right, robots.txt needs almost no ongoing attention. Revisit it only when your site's structure changes — a new section that should be excluded, or an old one that no longer needs to be.