Install
CrawlDude runs on macOS 11 or later. Windows and Linux versions are coming.
- Download the installer for your Mac from the download page: Apple Silicon for M1 and newer, Intel for older Macs. If you're not sure, open the Apple menu, choose About This Mac, and look for "Chip" (Apple Silicon) or "Processor" (Intel).
- Open the downloaded
.dmgfile and drag CrawlDude to the Applications folder. - Open CrawlDude from Applications.
Current builds aren't signed by Apple yet, so the first time you open CrawlDude, macOS says it can't be opened. Right-click CrawlDude in Applications, choose Open, then confirm. On macOS 15 and later, open System Settings, go to Privacy & Security, scroll down and click Open Anyway. You only need to do this once.
Your first crawl
CrawlDude opens on the projects screen. A project called "My crawls" is ready to use.
- Click the project card. An empty crawl view opens.
- Type the site's address in the bar at the top, for example
https://example.com, and press Start. - The table fills up as pages are crawled, and the issues list on the right updates every few seconds. Press Pause or Stop at any time.
- When the crawl finishes, click an issue in the sidebar to see the pages it affects, and click a row to see that page's details.
Every crawl is saved to its project automatically when it finishes or is stopped.
On a site you don't own, keep the speed down: set a time between requests in the project settings and leave "Respect robots.txt" on.
Projects and history
Everything in CrawlDude belongs to a project, usually one per website or client. Each project has its own start URL, crawl settings and crawl history.
- Projects screen: one card per project, with its site, the latest health score, URL and error counts, and the number of crawls. Click a card to open its latest crawl. Open it any time from the breadcrumb at the top left or with ⇧⌘P.
- Crawls: every crawl of the project, newest first. Open a crawl to load it, move it to another project, or delete it.
- Settings: the project's name, start URL and crawl settings. Changes apply to the next crawl.
- New crawl: opens an empty crawl view with the project's start URL filled in.
Deleting a project also deletes its saved crawls. Crawl history is stored on your Mac in ~/Library/Application Support/CrawlDude/library/.
Crawl settings
Open a project's settings from its card, from the gear button at the top right, or with ⌘,. Reset to defaults puts every crawl setting back, keeping the name and start URL.
| Setting | Default | What it does |
|---|---|---|
| Time between requests | 0 ms | The minimum gap between two requests. Use 500 ms or more on sites you don't own. If robots.txt sets a Crawl-delay and this is 0, CrawlDude uses the Crawl-delay (up to 10 seconds). |
| Concurrent requests | 5 | How many pages are fetched at the same time. |
| Timeout per request | 20,000 ms | How long to wait for a response before recording a timeout. |
| Max URLs | 10,000 | Stops the crawl after this many URLs. Raise it for big sites; your computer's memory is the only limit. |
| Max depth | 20 | How many clicks from the start URL to follow. |
| Respect robots.txt | On | Skips URLs that robots.txt disallows and reports them as blocked. Turn off only for your own sites. |
| Crawl subdomains | Off | Treats subdomains such as blog.example.com as part of the site. www and non-www always count as the same site. |
| Follow nofollow links | Off | Also crawls links marked rel="nofollow", ugc or sponsored. |
| Ignore query strings | Off | Treats /page?a=1 and /page as the same URL. |
| Check external links | Off | Requests links to other sites to get their status codes, so broken external links show up. |
| Check images | Off | Requests the site's images to get their status codes and sizes. |
| Use XML sitemaps | On | Reads the site's XML sitemaps and audits them. See XML sitemaps and orphan pages. |
| AI search checks | On | Checks llms.txt, robots.txt rules for AI crawlers and AI-related page signals. See AI search readiness. |
| Dry run | Off | A quick sample crawl to test settings: at most 25 URLs (change it with Dry run URLs) and 2 clicks deep. |
| User agent | CrawlDude | How CrawlDude identifies itself. Choose a preset (Googlebot Smartphone or Desktop, Bingbot, Chrome on Windows, macOS or Android, Safari on iPhone) or type any string. Crawling as Googlebot or Bingbot also makes CrawlDude follow that bot's rules in robots.txt. |
The crawl view
Tabs
The tabs above the table filter the crawled URLs: All, HTML, Redirects, 4xx, 5xx / no response, Non-indexable, Blocked (by robots.txt), External, Images and AI search. Each tab shows the columns that matter for it. The filter box searches URLs and titles.
The table
- Click a column header to sort, click again to reverse, and a third time to clear the sort.
- Drag the right edge of a header to resize a column. Double-click the edge to reset it. Widths are remembered.
- A warning triangle next to an address means that URL has errors or warnings. Hover it to see how many.
- Double-click a row, or press Enter, to open the URL in your browser. Use ↑ and ↓ to move between rows.
The details pane
Click a row to see everything about that URL below the table. Drag the divider to make the pane taller or shorter.
- Details: status, indexability and the reason, title, meta description, headings, canonical, robots directives, language, hreflang, Open Graph and Twitter tags, structured data, dates and author, word count, response time, size and more.
- Inlinks and Outlinks: every link to and from the page, with anchor text, status and nofollow.
- Images: each image with its alt text.
- Issues: every issue found on the page, with an explanation.
- Redirect chain: each hop when the URL redirects, including loops.
Click an address in the details pane to jump to it in the table. Hold ⌘ while clicking, or use the ↗ button, to open it in your browser.
The sidebar
The sidebar shows the site health score, the AI search readiness score, URL and indexable counts, error and warning totals, a breakdown of status codes, and every issue grouped by topic. Click an issue to filter the table to the URLs it affects; click it again, or the × on its tab, to clear the filter. "Hide zero" hides issues that weren't found.
Site health score
Every crawl gets a score from 0 to 100: the share of crawled internal URLs without an error-level issue. A site where 9 of 10 pages have no errors scores 90. Warnings and notices don't lower the score.
The score is green from 80, amber from 50 and red below that. It shows in the sidebar, on project cards and in each project's crawl history, so you can see whether fixes are working.
XML sitemaps and orphan pages
With "Use XML sitemaps" on, CrawlDude reads the sitemaps listed in robots.txt, or /sitemap.xml when robots.txt lists none. Sitemap index files and gzipped .xml.gz sitemaps are supported.
Once the link crawl is done, CrawlDude also requests the sitemap URLs that no link led to. Those are your orphan pages: listed in the sitemap, but nothing on the site links to them. It also reports sitemap URLs that return errors, redirect, are noindex, canonicalised or blocked by robots.txt, and indexable pages missing from the sitemap.
AI search readiness
AI search engines and assistants such as ChatGPT, Claude, Perplexity, Google's AI Overviews and Copilot read websites to answer questions and cite sources. These checks show whether they can read yours. They're on by default; turn them off with "AI search checks" in project settings.
The readiness score
The AI search readiness card in the sidebar shows a score from 0 to 100. Click it, or press ⇧⌘A, for the full report. The score comes from ten checks: each pass counts 1, each warning 0.5 and each failure 0, divided by the checks that apply.
| Check | What it looks at |
|---|---|
| llms.txt present | Whether /llms.txt exists. A file that returns an HTML page instead (a soft 404) counts as missing. |
| llms.txt follows the llmstxt.org format | One H1 title on the first line, a summary, and ## sections with link lists. See llms.txt below. |
| llms.txt links resolve | Every link in llms.txt. Links the crawl reached use the crawl's status; the rest (up to 300) are requested after the crawl. |
| llms-full.txt present (optional) | Whether /llms-full.txt exists. Missing is a warning, not a failure. |
| AI search & assistant bots can reach the homepage | Whether robots.txt blocks any AI search or assistant bot from the start URL. |
| No indexable pages blocked for AI search bots | How many indexable pages robots.txt blocks for at least one search or assistant bot. |
| No snippet restrictions | Pages with nosnippet or max-snippet:0. Google leaves these out of AI Overviews and AI Mode. |
| Content is in the HTML | Pages that look like they need JavaScript to show their content. Most AI crawlers don't run JavaScript. |
| Structured data on 80% of indexable pages | The share of indexable pages with JSON-LD. Passes at 80%, warns from 40%. |
| Sitemap declared in robots.txt | Whether robots.txt has a Sitemap: line. |
llms.txt
llms.txt is a Markdown file at the root of a site that gives AI assistants a short summary and a list of the most useful pages. The format is described at llmstxt.org. CrawlDude checks that:
- the first line is a single H1 title, like
# Example Shop(required) - a one-line summary in a blockquote (
> …) follows the title ## Sectionheadings contain link lists like- [Returns policy](/returns): how returns work- links are http(s) and don't return errors
- the file is served as
text/plainortext/markdown
A section called ## Optional marks links an assistant can skip when it's short on space. The report shows the file's title, summary, sections, size, every problem found and a table of its links with their status.
AI crawlers in robots.txt
CrawlDude reads robots.txt as each of 20 known AI crawlers would, using the same rules as Google: the most specific User-agent group wins, and the longest matching rule wins. robots.txt is read for this even when "Respect robots.txt" is off. Each bot has a purpose:
- Search: builds an AI search or answer index, for example OAI-SearchBot (ChatGPT Search), Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. Blocking these means your pages aren't cited.
- Assistant: fetches a page when someone asks an assistant about it, for example ChatGPT-User, Claude-User and Perplexity-User.
- Training: collects data to train models, for example GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Blocking these is a legitimate choice, so CrawlDude shows it as information, not as a problem.
The report's crawler table shows each bot's company, purpose, whether it may fetch the homepage, how many indexable pages it's blocked from, the robots.txt rule that matched, and whether robots.txt has rules just for that bot.
Page signals
The AI search tab lists every HTML page with the AI bots blocked from it, whether it seems to need JavaScript, snippet limits, word count, schema types, date and author. A page counts as needing JavaScript when it has fewer than 50 words in its HTML and either an empty app container (such as #root or #__next) or five or more scripts.
Export the whole report from its Export CSV button.
Issue reference
CrawlDude checks for 64 issues in three levels:
- Error something is broken and needs fixing. Errors lower the site health score.
- Warning likely hurts search performance; worth fixing.
- Notice worth knowing; may be fine depending on the site.
Response codes
| Issue | Level | What it means |
|---|---|---|
| Client error (4xx) | Error | Internal URLs returning 4xx. Fix or remove links pointing to them. |
| Server error (5xx) | Error | Internal URLs returning 5xx errors. |
| No response / timeout | Error | The server did not respond (DNS, connection or timeout error). |
| Internal redirect (3xx) | Warning | Internal links that redirect. Link directly to the final URL. |
| Redirect chain | Warning | Redirects that hop more than once before reaching the final URL. |
| Redirect loop | Error | Redirects that eventually point back to themselves. |
| Has broken internal links | Error | Pages linking to internal URLs that return 4xx/5xx. |
| Broken external link target | Warning | External URLs that returned an error (only when external link checking is on). |
| Blocked by robots.txt | Notice | Internal URLs the crawler was not allowed to fetch. |
Page titles
| Issue | Level | What it means |
|---|---|---|
| Missing title | Error | Pages without a <title>. |
| Duplicate title | Warning | The same title is used on more than one indexable page. |
| Title over 60 characters | Warning | Long titles are often truncated in search results. |
| Title under 30 characters | Notice | Short titles may not describe the page well. |
| Multiple title tags | Warning | Pages with more than one <title> element. |
Meta description
| Issue | Level | What it means |
|---|---|---|
| Missing meta description | Warning | Pages without a meta description. |
| Duplicate meta description | Warning | The same description is used on more than one indexable page. |
| Meta description over 155 characters | Notice | Long descriptions are often truncated. |
| Meta description under 70 characters | Notice | Short descriptions may be replaced by search engines. |
Headings
| Issue | Level | What it means |
|---|---|---|
| Missing H1 | Warning | Pages without an <h1>. |
| Multiple H1 | Notice | Pages with more than one <h1>. |
| Duplicate H1 | Notice | The same H1 is used on more than one indexable page. |
| Title same as H1 | Notice | The title and H1 are identical. Use them to target different wording. |
Directives
| Issue | Level | What it means |
|---|---|---|
| Noindex | Notice | Pages with a noindex directive (meta robots or X-Robots-Tag). |
| Canonicalised | Notice | Pages whose canonical points to a different URL. |
| Missing canonical | Notice | Indexable pages without a canonical link. |
| Canonical to non-200 URL | Error | The canonical target redirects or returns an error. |
| Noindex page linked internally | Warning | Noindex pages that still receive many internal links. |
| Multiple canonical tags | Warning | Pages with more than one rel="canonical" link. Search engines may ignore them all. |
Sitemaps
| Issue | Level | What it means |
|---|---|---|
| 4xx/5xx URL in sitemap | Error | Sitemap URLs that return an error. Remove them or fix the pages. |
| Redirect in sitemap | Warning | Sitemap URLs that redirect. List the final URL instead. |
| Noindex URL in sitemap | Warning | Sitemaps should only list pages you want indexed. |
| Canonicalised URL in sitemap | Warning | Sitemap URLs whose canonical points elsewhere. List the canonical URL instead. |
| Blocked URL in sitemap | Warning | Sitemap URLs that robots.txt does not allow crawling. |
| Orphan page | Warning | Pages in the sitemap that no crawled page links to. |
| Indexable page not in sitemap | Notice | Indexable pages missing from the XML sitemap. |
Content
| Issue | Level | What it means |
|---|---|---|
| Low word count (<200) | Notice | Thin pages with little visible text. |
| Exact duplicate content | Warning | Pages with identical visible text. |
| Images missing alt attribute | Warning | Pages containing <img> tags without alt. |
| Missing html lang | Notice | Pages without a lang attribute on <html>. |
| Missing meta viewport | Warning | Pages without <meta name="viewport">. They will not render well on mobile. |
| Mixed content | Warning | HTTPS pages loading images, scripts or styles over plain HTTP. |
Social
| Issue | Level | What it means |
|---|---|---|
| Incomplete Open Graph tags | Notice | Missing og:title, og:description or og:image, used when the page is shared. |
| Missing Twitter card | Notice | Pages without <meta name="twitter:card">. |
Links
| Issue | Level | What it means |
|---|---|---|
| Links without anchor text | Warning | Links with no text, image alt, aria-label or title. |
| Non-descriptive anchor text | Notice | Internal links like "click here" or "read more" that say nothing about the target. |
| No internal outlinks | Warning | Dead-end pages that link to no other internal page. |
URL
| Issue | Level | What it means |
|---|---|---|
| Uppercase characters | Notice | URLs containing uppercase letters. |
| Underscores | Notice | URLs containing underscores. |
| Query parameters | Notice | URLs with a query string. |
| URL over 115 characters | Notice | Very long URLs. |
| Non-HTTPS URL | Warning | Internal URLs served over plain HTTP. |
Performance
| Issue | Level | What it means |
|---|---|---|
| Slow response (>1s) | Warning | Pages taking over a second to respond. |
| Large HTML (>1 MB) | Notice | HTML documents larger than 1 MB. |
Site structure
| Issue | Level | What it means |
|---|---|---|
| Crawl depth over 3 | Notice | Pages more than 3 clicks from the start URL. |
| Only one internal inlink | Notice | Indexable pages linked from a single page. |
AI search
| Issue | Level | What it means |
|---|---|---|
| Blocked for AI search bots | Warning | robots.txt blocks at least one AI search or assistant bot, so those assistants can't read or cite the page. |
| Blocked for AI training bots | Notice | robots.txt blocks at least one AI training bot. This is a legitimate choice, shown for visibility. |
| Content needs JavaScript | Warning | Little text in the HTML and an app shell or many scripts. Most AI crawlers don't run JavaScript. |
| Snippet restriction | Warning | nosnippet or max-snippet:0 in meta robots or X-Robots-Tag. Google excludes such pages from AI Overviews and AI Mode. |
| noai / noimageai directive | Notice | A non-standard opt-out. Few AI crawlers honour it. |
| No structured data | Notice | Indexable pages without JSON-LD. Schema helps AI systems understand and attribute the content. |
| Invalid JSON-LD | Warning | A JSON-LD block that isn't valid JSON. The whole block is ignored. |
| No published or modified date | Notice | No date in JSON-LD, article meta tags or <time>. Dates are a freshness signal. |
| Article without author | Notice | Article-type schema without an author. Authorship supports E-E-A-T. |
Export and save
- Export (⌘E) saves the table as you see it, with its tab, issue filter, search and sort, as a CSV file that opens cleanly in Excel and Google Sheets.
- File → Export Issues Report as CSV saves every issue with its severity, group and affected URL.
- Export CSV in the AI search readiness report saves the checklist, the AI crawler table and the llms.txt links and problems.
- Save (⌘S) saves a copy of the crawl as a
.crawldudefile to share or archive. Open (⌘O) loads one into the current project.
Updates
CrawlDude checks for a new version a few seconds after it starts. When one is out, a bar appears at the top: click Install and restart, and CrawlDude downloads the new version, replaces itself and opens again. Your projects and crawls are kept. To check by hand, choose Help → Check for Updates.
Versions before 0.2.1 can't update themselves: download the latest installer from the download page and replace the app in Applications once.
If CrawlDude can't replace itself, for example when it runs from the disk image instead of Applications, it opens the downloaded installer so you can drag the new version over the old one.
Keyboard shortcuts
| Shortcut | Action |
|---|---|
| ⌘N | New crawl in the current project |
| ⌘Y | Current project's crawls and settings |
| ⇧⌘N | New project |
| ⇧⌘P | All projects |
| ⌘, | Project settings |
| ⇧⌘A | AI search readiness report |
| ⌘O | Open a crawl file |
| ⌘S | Save the crawl as a file |
| ⌘E | Export the current view as CSV |
| ↑ ↓ | Move between rows in the table |
| Enter | Open the selected URL in your browser |
| Esc | Go up one level on the projects screens |
Getting help
Check the FAQ first. For anything else, or to report a bug, email hello@crawldude.com. Include your CrawlDude version (CrawlDude → About CrawlDude) and the site you crawled if you can.