How a Site Crawl Actually Works
Understanding how a crawler sees your site helps you fix what it finds. Here is the short version.
A crawler starts on a page, finds every link, follows each to the next page, and repeats, discovering your site the way a search engine does. It records the response each link returns, whether a page loads, redirects, or fails, and turns that into a fix list showing exactly where each broken link lives.
Following links, one by one
A crawler starts from a page, finds every link on it, follows each to the next page, and repeats. By walking the links, it discovers your site the same way a search engine does. Related: When to Run a Full Link Audit
Keep reading: When to Run a Full Link Audit, Reading a Redirect Chain Map, A Pre Launch Link Checklist. See how DeadLinkr helps you on demand link and redirect audits for seo.
What it records
As it goes, a crawler notes the response each link returns, whether a page loads, redirects, or fails. That record is what turns into an audit of broken links, redirects, and errors. Related: A Pre Launch Link Checklist
Why coverage matters
A crawl can only find pages that are linked. Pages with no links pointing to them, or blocked from crawling, stay invisible. Knowing this helps explain gaps and orphaned pages in a report. Related: How Broken Links Waste Crawl Budget
From crawl to action
The value of a crawl is not the data, it is the fix list it produces. A good crawl tells you not just what is broken, but where the broken link lives, so you can act quickly. Related: How do you handle broken external links on sites you do not control?
- A crawler discovers a site by following links
- It records the response each link returns
- Unlinked or blocked pages stay invisible to it
- The point of a crawl is the actionable fix list
Julien Jimenez
Julien Jimenez is an independent software builder based in Paris. He designs, ships, and operates focused SaaS products for small businesses and independent professionals. Read the full author page.
Frequently asked questions
Why do some pages stay invisible to a crawler?
A crawl can only find pages that are linked. Pages with no links pointing to them, or pages blocked from crawling, stay invisible, which is why orphaned pages show up as gaps in a report.
What is the real value of a site crawl?
Not the raw data, but the actionable fix list it produces. A good crawl tells you what is broken and where the broken link lives, so you can act quickly.
A deep link audit whenever you need one
On demand link and redirect audits for SEO. DeadLinkr is built to help you put this into practice.
Run an auditMore from the DeadLinkr blog
When to Run a Full Link Audit
Reading a Redirect Chain Map
A Pre Launch Link Checklist
Get the DeadLinkr playbook
Practical guides on link audits, straight to your inbox as we publish them. No spam, unsubscribe any time.