Crawls up to 100 internal pages · pages appear live as they're scanned · takes 15–60 s
What is a Site Crawler and why does it matter for SEO?
A site crawler (also called a web spider or bot) systematically visits every page on your website, just like Googlebot does. It follows internal links, reads HTML, and collects metadata — titles, descriptions, status codes, canonical tags and headings — to build a complete picture of your site's technical health.
Unlike manual audits, a crawler catches problems you'd never find by hand: a duplicate title buried 40 pages deep, a 404 error on a page linked from your homepage, or 30 pages missing a canonical tag. These issues silently erode your rankings and CTR every day they go unfixed.
SEOKit's Site Crawler goes beyond a simple error list. Each issue comes with a priority level (🔥 high / ⚡ medium / 🟢 low), an explanation of why it affects your rankings, and a concrete fix recommendation. High-priority pages can be sent directly to the CTR Optimizer for a full meta tag audit and title rewrite.
Frequently Asked Questions
How many pages does the crawler check?
The crawler visits up to 100 internal pages per run, starting from the URL you enter. It only follows links within the same domain — external links are detected but not crawled. For most small and medium sites this covers the entire site; for larger sites focus the crawl on the most important section (e.g. https://example.com/blog/).
Why are some of my real pages showing as errors?
Pages that require login (returning 401 or 403) are excluded from the broken-links report — they are not broken, just protected. Only genuine 404 Not Found and 5xx server errors are flagged as broken. If a working page still shows as an error, it may be blocking server-side requests via a bot-detection firewall (Cloudflare, DataDome). In that case the page itself is fine — it's just unreachable to automated crawlers.
What is keyword cannibalization and why are duplicate titles a problem?
Keyword cannibalization happens when multiple pages on your site target the same keyword. Google struggles to decide which page to rank, so it often ranks neither well — splitting ranking potential across both. Duplicate title tags make this worse: they signal to Google that the pages are interchangeable. Fix by giving each page a unique title targeting a distinct keyword or search intent.
How is the Site Health Score calculated?
The score (0–100) penalizes issues by weight: high-priority issues (missing title, broken links, duplicate titles) subtract 3× more than medium-priority issues (missing descriptions, missing H1). The penalty is normalized by the number of pages crawled so a site with 2 issues across 50 pages scores higher than one with 2 issues across 5 pages. It's a directional indicator — not a guarantee of ranking, but a reliable signal of how much technical debt your site carries.
Does crawling my site affect its performance or SEO?
No. The crawler makes standard HTTP GET requests — the same as any browser visitor. It fetches pages one at a time with a short delay and does not submit forms, modify content or affect your analytics. It will not trigger penalties from Google or harm your site in any way.
What's the difference between a redirect and a broken link?
A redirect (3xx) sends users and crawlers from one URL to another — it works, but each hop adds latency and loses a small fraction of link equity. A broken link (4xx/5xx) leads nowhere or throws a server error, wasting crawl budget and destroying the link equity passed through that link. Broken links are 🔥 high priority; redirects are 🟢 low priority unless they form chains (A→B→C→D).