SEO
Crawling, how it works: crawlers, spiders and robots
Introduction to crawling
When we talk about crawling, we are referring to the way search engines such as Google and Microsoft Bing find your pages, examine them and store the information in their database. Those first two steps are part of crawling. Storing the information in the database is called 'indexation' (another important concept).
Crawling and optimising it are highly technical aspects of SEO, so we will try to explain them in an accessible way.
The importance of crawling
Crawling is a fundamental part of the search engine process. Crawling enables search engines to discover new pages or include changes to existing pages in the index. If the crawler encounters many links (both internal and external) to a page, it can refresh that page in the index more often, increasing the likelihood that it will rank better in the search results or be better understood by the search engine. Optimising the crawling of your website is therefore hugely important for your SEO success!
How does a crawler work?
A crawler, also known as a spider, bot or robot, is essentially a system released onto the internet to discover it and examine the content it encounters.
When it encounters a hyperlink to another page, it will follow it and view the next page. This is how the bot crawls across the internet, like a spider in its web.
Discovering and viewing these pages is the first step towards indexation, or: 'recording these pages in the search engine's database'.
Which tools are available for crawl optimisation?
There are various tools you can use to make it as easy as possible for the search engine to crawl your website as efficiently as possible.
Robots.txt
With robots.txt, you can inform search engine robots, SEO tools and other crawlers which parts of the website must not be crawled. Broadly speaking, robots.txt is the first line of defence. You block access to irrelevant (or unwanted) pages, media files and source files.
Sitemap.xml
An XML sitemap is a file that search engines can use to obtain an overview of all pages, files and paths within the website at once. If a URL is included in the sitemap, the search engine knows the URL and the likelihood of it being crawled is considerably greater. Most CMSs can be provided with a sitemap via a plug-in.
Links (internal and external)
Both internal and external links (also known as 'hyperlinks') contribute to the crawlability of your website. It is important that the hyperlinks in the code comply with all HTML standards. The search engine can follow the links to discover new pages. If the link has good 'anchor text', this also gives the crawler some context about the page being linked to.
Meta robots
The meta-name="robots" tag has two important functions (for each individual page). With 'index' and 'noindex', you indicate whether or not a page may be included in the index. In addition, with 'follow' and 'nofollow', you indicate whether the hyperlinks on this page may be followed by the search engine. This therefore affects crawler behaviour. These tags work independently of one another.
Which crawlers are there?
There are various types of crawlers that may visit your website, but almost all of them can be divided into one of these categories:
| Crawler type | Companies | Name of bot(s) | Purpose |
| Search engines | Google
Bing |
GoogleBot, Adsbot Bingbot, AdIdxBot | Crawling and indexing |
| LLMs | OpenAI
Anthropic |
GPTBot, ChatGPT-User/2.0
ClaudeBot, anthropic-ai |
Use as a search engine, training A.I. models and carrying out A.I. tasks. |
| SEO tools | Ahrefs
Moz Semrush |
AhrefsBot
Rogerbot SemrusBot |
Reporting website signals (browser) |
| Crawl tools | Screaming Frog | Screaming Frog SEO Spider | Reporting website signals (application) |
The purpose of the different crawlers varies. While search engines are focused on finding new, relevant pages for the index, the tools are mainly focused on mapping the entire site to identify areas for improvement in crawling, indexing and optimising the website.
For most people reading this, the bots of the A.I. platforms will be new. These bots and crawlers work slightly differently from, for example, Google's bots. Because A.I. platforms also want to perform “tasks”, we see that these can be divided into bots that crawl and so-called users that perform tasks. Finally, there are crawlers intended to “train” the LLMs. For your website, you therefore need to determine what you want to make your content available for.
How does crawl budget work?
Crawl budget is essentially an indication of how often search engines want to crawl your website and how long they want to do so. There is therefore a maximum, which is logical because the internet is many times larger than the search engines' index can accommodate. The crawl budget available for your website depends, among other things, on these factors:
- The total number of pages the search engine can crawl.
- How popular your website is (external factors, in other words).
- The extent to which your website is renewed or expanded.
From this, you can already conclude, for example, that it is important for the website to remain 'fresh', for the search engine to encounter links to your website on other sites and for there to be enough information to crawl.
Influencing crawl budget
There are several ways in which you can positively (or negatively) influence the crawl budget or its use. First of all, good technical health is required. Negative response codes or slow response times mean that the crawl budget is used incorrectly or inefficiently. This can cause the search engine to carry out fewer crawl requests (see graph).
The original chart is currently unavailable at its source. View the original report link.
Adding content and updating pages creates the 'freshness' that the search engine needs to remain interested. This will have a positive effect on the crawl budget, just as an outdated website that is no longer updated will have a negative effect.
Optimisation checklist
Use the checklist below to optimise the crawling of your website as effectively as possible. Read through the steps; if you need more information or cannot work it out, please get in touch!
-
Robots.txt is up to date
You use robots.txt to prevent the search engine from visiting certain pages. This may be unnecessary for small websites, but for larger websites (+1.000 pages), you can prevent the search engine from visiting irrelevant pages. In other words, more budget for more relevant pages.
Test: Test your robots.txt file
-
Sitemap.xml file is up to date
All pages that you want to have indexed should be included in sitemap.xml. The sitemap serves as a menu for search engines, allowing them to see at a glance which pages can be found and crawled within the website. Deleted and redirected pages are (usually) best left out.
Create: Create an XML sitemap
-
4xx, 3xx and 5xx status codes checked
When crawlers encounter negative status codes (list of negative status codes for crawling), this will alter the speed at which they crawl. You can check this by carrying out a crawl yourself regularly.
-
Overview of pages in Search Console checked
In Google Search Console, you have an overview of ‘Discovered’, ‘Crawled’ and ‘Indexed’ pages. Do you notice that a new or important page does not appear in the search results? Then there is a good chance that it has not yet been discovered or indexed. In that case, you can try to request indexing manually!
-
Fast server response time
The server's response time affects the total number of pages that the crawler can crawl in a session. If you have a longer (that is, slower) Time-to-First-Bite, the crawler can visit and crawl fewer pages in a session. This therefore has a negative effect on how quickly changes or new pages are indexed. Test this with PageSpeed Insights or check the Crawl Stats in Search Console.
Crucial role of crawling during a migration
During the migration of your website, you want the search engine to crawl and index the correct pages in good time and accurately, and remove the old URLs (where necessary) from the index. For this reason, search engines such as Google have a mechanism that increases the crawl budget and frequency when site-wide changes take place, such as a migration. Very useful!
Crawling and indexing
As mentioned, crawling a page is the first step that must be taken before it can be found in the search engine. Once it has been crawled, the search engine assesses whether the page will be indexed. Usually, if the content is sufficiently unique and relevant, the search engine will include it in the index, after which it can be displayed as a result during a search. Each search engine simply follows these steps:
- Crawling
- Indexing
- Serving
It is not always quite so simple, and each step involves many processes, mathematical formulas and sometimes a few flaws, but essentially, this is how search engines work.
Frequently asked questions
What is a crawler?
A crawler, also known as a spider, bot or robot, is essentially a system released onto the internet to discover it and view the content that the crawler finds.
What is the crawl budget?
For each website, the search engine sets a certain number of pages that can be crawled per day. This is known as the crawl budget, and therefore differs from one website to another. It is based on the estimated number of pages that the search engine can do something with, how popular your website is and how often the pages and content are updated.
Why is crawling important?
A website's crawlability ensures that it and its content can be found in search engines. These search engines determine which pages are relevant to include in their index, so that they can then be displayed in the search results.
BRING YOUR VISIBILITY POTENTIAL TO LIGHT!
Discover your potential Challenge us
\ Free visibility check
Discover your organic potential
Curious how you compare in your industry? Our experts will analyse your online position for free.
Request a free scan