Search
Publish a file that tells crawlers the rules
Your site has no reachable /robots.txt, so search engine crawlers get no guidance about what to index.
Why it matters
The robots file is the first thing a crawler asks for. Without it, crawlers guess: they may waste their visit on admin pages, or fail to find your sitemap, which is the list of pages you actually want indexed. A missing file is also a signal that a site is unfinished, which is not the impression you want a search engine to start with.
How you would notice it
- New pages take a long time to appear in search results.
- Internal or staging pages show up in search results.
- Search console reports that no sitemap was found.
What to do
Our robots file is deliberately boring, and that is the point. One note about what the check actually scores.
Stage 1
- Create a plain text file named robots.txt in the top level of your site, next to your index page.
- Allow normal crawlers and point them at your sitemap.
Replace the sitemap address with your own. If you do not have a sitemap yet, leave that line out rather than inventing an address.
Reachable beats clever, every time
The scored signal is reachability at your root, not how sophisticated the rules are: ours declares a plain allow for everyone because we want the whole site indexed, and it has never returned an error. When we renamed it out of the tree in a test rebuild, two checks failed at once: the file no longer survived into the built output, and the crawler rules went missing. A careful set of rules that returns a 404 scores zero, so publish the simplest file that answers your purpose and keep it reachable.
Source: Crawler reachability negative test on our own deployed origin
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlStage 2
- Block anything you do not want indexed, one path per line.
This is a request, not a lock. It keeps honest crawlers out; it does not protect anything with a password.
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /checkout/Stage 3
- Confirm the file is reachable at your root address, with no redirect in front of it.
https://www.example.com/robots.txtHow to check it worked
Confirm it worked
Visit your own address followed by /robots.txt. You should see the text of your file in the browser, not a 404. Re-run the Siege Test and confirm the robots.txt recommendation is gone.