How a Site Crawl Actually Works
Understanding how a crawler sees your site helps you fix what it finds. Here is the short version.
Following links, one by one
A crawler starts from a page, finds every link on it, follows each to the next page, and repeats. By walking the links, it discovers your site the same way a search engine does.
What it records
As it goes, a crawler notes the response each link returns, whether a page loads, redirects, or fails. That record is what turns into an audit of broken links, redirects, and errors.
Why coverage matters
A crawl can only find pages that are linked. Pages with no links pointing to them, or blocked from crawling, stay invisible. Knowing this helps explain gaps and orphaned pages in a report.
From crawl to action
The value of a crawl is not the data, it is the fix list it produces. A good crawl tells you not just what is broken, but where the broken link lives, so you can act quickly.
- A crawler discovers a site by following links
- It records the response each link returns
- Unlinked or blocked pages stay invisible to it
- The point of a crawl is the actionable fix list
Julien Jimenez
Julien Jimenez is an independent software builder based in Paris. He designs, ships, and operates focused SaaS products for small businesses and independent professionals. Read the full author page.
A deep link audit whenever you need one
On demand link and redirect audits for SEO. DeadLinkr is built to help you put this into practice.
Run an auditMore from the DeadLinkr blog
When to Run a Full Link Audit
Reading a Redirect Chain Map
