All tutorials
Tutorial 4 of 5 · 0:41
Find pages hidden by noindex
A noindex tag tells search engines to leave a page out of their results. Left behind after a redesign or a staging setup, it can quietly remove your most important pages from Google.
Find it
- Crawl the site.
- Open the Non-indexable tab. It lists every page search engines will leave out, with the reason.
- Click a page with the reason Noindex. The details pane shows the Meta robots tag or X-Robots-Tag header that causes it.
Fix it
- Remove the noindex tag from pages that should be found.
- Check the server’s X-Robots-Tag header too; it does the same thing.
- After every release, glance at the Non-indexable tab.
<meta name="robots" content="noindex, follow">Try it yourself
The tutorial crawls our example pages, which have this problem on purpose. Start a crawl at https://crawldude.com/examples/ in CrawlDude to follow along.