EN
Webmail

Will a Website Audit Slow Your Site? How Polite Crawling Works

Will a Website Audit Slow Your Site? How Polite Crawling Works

Before you run a website audit, a reasonable worry crosses your mind: a program is about to visit every page of my site, so will it slow things down for my real customers? A website audit crawler does read your pages one after another, but a well-behaved one does it gently, at a measured pace, and by the rules you set in your robots.txt file. Site AI Audit and Site SEO AI Audit, two products from the Internet Solutions team, are built that way: their crawlers make about three requests per second and follow your robots.txt. This guide explains what that means in practice, how to recognise the crawler in your logs, what can go wrong with firewalls and protection systems, and how to prepare your site so the audit sees everything it should.

We cover the worry first, because an audit you hesitate to run is an audit that never happens. Then we cover the practical details that make the result complete and reliable.

What a crawler does, in plain words

A crawler is a program that opens web pages the way a visitor’s browser does, reads them, and follows the links it finds. Search engines use crawlers to discover pages. Audit tools use them to examine pages for problems: titles, headings, links, images, indexing rules, speed and security settings. Because it works from the outside, it needs nothing installed on your server. As the product pages put it, the site is checked the way visitors and search engines see it.

StepWhat the crawler doesWhy it matters to you
1. Reads robots.txtFinds out which parts of the site it may visitYou stay in control of what is crawled
2. Reads the sitemapLearns which pages you want search engines to knowImportant pages are not missed
3. Requests pagesOpens pages at a steady, limited paceReal visitors are not crowded out
4. Follows linksDiscovers pages that the sitemap does not listOrphan and weakly linked pages become visible
5. AnalysesChecks each page against the rules of the auditYou receive findings ranked by impact

What “about three requests per second” means

Both audit crawlers state the same limit: about three requests per second. A request is one file asked of your server, such as a page, a stylesheet or an image. Three per second is a modest rate, and it is useful to compare it with other traffic.

  • A single visitor loading a typical page triggers dozens of requests at once, because every image, script and style is a separate request. A few visitors arriving together generate more instantaneous load than a polite crawler.
  • Search engine crawlers routinely visit sites, and your server handles them every day without trouble.
  • The limit is a ceiling, not a target. The product page says that the crawler makes about three requests per second, which means that at no moment does it flood the server.

The same figure also tells you how long a large audit takes. At three requests per second, a site of one thousand pages needs at least five to six minutes of crawling, and a site of ten thousand pages, which is the maximum per check, needs correspondingly longer. The speed is chosen for your safety, not for the tool’s convenience.

Why robots.txt matters

The robots.txt file sits at the root of your website and tells crawlers which paths they may visit. It is a convention, described at robotstxt.org and explained in Google’s guide to robots.txt. Well-behaved crawlers read it before anything else and obey it. Both audit products state that they follow your robots.txt rules, with two consequences worth understanding.

  1. You can keep the audit out of sensitive areas. Paths you disallow, such as an admin area or a private section, are not crawled.
  2. Blocked pages are not examined. If you disallow something that you actually want to rank, the audit cannot assess it, and the robots rule itself is a finding worth fixing.

The robots.txt file controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it, and a noindex rule on a page cannot be read by a crawler that is not allowed to open the page. These interactions are a frequent source of confusion, and our guide to why Google is not indexing your pages helps untangle them.

Recognising the crawler in your server logs

A crawler introduces itself with a name in the user agent field of each request. Site AI Audit uses the name SiteAuditBot, and Site SEO AI Audit uses SEOAuditBot. Searching your access logs for these names shows how often the audit visited and which addresses it requested. This is useful in two situations: when you want to confirm that an audit ran, and when you want to understand why a firewall reacted. Our guide to server log analysis describes how to read such logs. Remember that a user agent name can be copied by anyone, so treat it as a label rather than as proof of identity.

When protection systems get in the way

Modern websites often sit behind security layers: a web application firewall, a content delivery network with bot protection, rate-limiting rules in the server, or a security plugin. These layers are designed to stop abusive traffic, and sometimes they cannot tell a polite audit from a hostile scan. The signs are easy to recognise in a report.

Sign in the auditPossible causeWhat to check
Only a few pages crawledThe firewall blocked the crawler after the first requestsLogs for blocked requests with the crawler’s name
Many pages return errors such as 403 or 429Access denied or rate limitingBot protection and rate limit settings
Pages show a challenge or consent wallA security service asks the crawler to prove it is humanAllow rules for known, legitimate crawlers
Everything redirects to a login pageSite is in maintenance or members-only modeMaintenance mode and password protection
Zero pages foundrobots.txt disallows everything, or the site blocks all botsThe robots.txt file and the noindex setting of a staging copy

If you see these signs, ask whoever manages your hosting or security to allow the audit crawler by its user agent name for the duration of the check, and then run the audit again. If you want help configuring this, our server administration team looks after servers, security and monitoring.

Preparing your site for a complete audit

A few preparations take minutes and improve the result.

  1. Read your robots.txt. Open it in a browser and confirm that it does not block the whole site or the sections you care about.
  2. Check the sitemap. Make sure it exists, is referenced in robots.txt and lists the pages you want indexed.
  3. Remove maintenance blocks. A site under construction or behind a password cannot be audited.
  4. Run the audit at a quiet time. If your server is small, avoid the busiest hour of your business day.
  5. Tell your host. If you share hosting or use a managed security service, a short note avoids surprises.

What the audit can and cannot see

Because the audit works from the outside, it sees what any visitor or search engine can see. That is its strength: the findings reflect your public site. It is also its limit. It cannot read pages behind a login, it cannot see your database, and it cannot judge your business rules. For the quality of results, remember that the crawl covers up to 10,000 pages per check, and that the report ranks findings by impact so you can start with the ones that matter. Our articles on reading a Site AI Audit report and on the seven areas of Site SEO AI Audit describe the output in detail.

Why caching makes crawling gentler still

Most websites are built on a content management system that assembles every page from a database when it is requested. If that work has to be repeated for each request, a crawl is more expensive than a visit to a static page. Caching changes this: the server stores ready-made copies of pages and serves them quickly. A well-cached site handles a polite crawl without noticing it, and it also handles real visitors better. If your audit later reports slow server response times, caching is one of the first things to look at, as we describe in our guides on server tuning and website speed.

What if your server is small or fragile

A very small shared hosting package, or an old server running a heavy application, may feel any extra load. Three requests per second is gentle, but a server that is already close to its limit has no spare capacity. In that case a few precautions are sensible.

  • Choose a quiet hour, such as early morning, for the audit.
  • Check that your monitoring is on, so you notice if response times rise.
  • Do not run several heavy tasks at once, such as a backup and an audit together.
  • If the audit reveals slow responses, treat that as a finding, not as proof that the audit was too heavy: a server that struggles under three requests per second will struggle under a handful of real visitors too.

Telling an audit from an attack

Sometimes the first sign of an audit is an alarm from a security system, and it helps to know how to tell a legitimate crawl from hostile traffic. A polite crawler introduces itself with a stable name, makes requests at a steady, limited pace, follows links in an orderly way and respects robots.txt. A hostile scan often hides behind a browser-like name, probes paths that do not exist on your site, such as admin panels of software you do not use, and ignores robots.txt entirely. If you are unsure about a specific request, check your logs against the time of your audit and the crawler’s name. The log analysis habit pays off in both cases.

After the audit: what to do next

The crawl is the means, not the goal. Once the report is ready, the useful part begins.

  1. Start with the findings ranked highest by impact.
  2. Group related findings, so that one change in a template fixes many pages.
  3. Fix, then run a re-check to confirm that the change worked and that nothing else broke.
  4. Keep the results of each audit, so that you can see progress over time.
  5. If you prefer to hand the work over, use the option in the report to ask the team for an estimate, since Internet Solutions has built and maintained websites, servers and email since 2011.

A second look after the fixes is where the polite pace pays off: you can repeat the check as often as your plan allows without worrying about the effect on your visitors.

Frequently Asked Questions

Will a website audit slow down my site?

No. The audit crawlers make about three requests per second and follow your robots.txt rules, so they behave like a polite visitor. Real visitors are not crowded out.

Do I need to install anything?

No. The site is checked from the outside, the way visitors and search engines see it, so nothing is installed on your server.

How can I recognise the audit in my logs?

Look for the crawler’s name in the user agent field: SiteAuditBot for Site AI Audit and SEOAuditBot for Site SEO AI Audit.

Can I keep part of my site out of the audit?

Yes. Disallow those paths in robots.txt and the crawler will not visit them. Keep in mind that pages you block cannot be assessed.

My firewall blocked the audit. What should I do?

Ask your host or security administrator to allow the crawler by its user agent name for the duration of the check, then run the audit again.

How many pages can be audited?

Both products check up to 10,000 pages per check. Because the pace is limited to about three requests per second, larger sites take longer.

The Bottom Line

A polite crawler is no threat to a healthy website. The audit crawlers behind Site AI Audit and Site SEO AI Audit make about three requests per second, follow your robots.txt rules, introduce themselves by name and need nothing installed on your server. Prepare by checking robots.txt, the sitemap and any maintenance mode, expect longer runs on larger sites, and allow the crawler through any firewall that blocks it. With that done, the audit sees your site the way search engines and customers see it, and tells you what to fix first. If you want help preparing the server or acting on the findings, contact our team through the server administration page.