About our scanner (BestWebAuditBot)

Did you see BestWebAuditBot in your server logs? Someone entered your website address into Best Web Audit to check it for problems. Here is how the scanner behaves.

How it identifies itself

User-agent: BestWebAuditBot/0.1 (+https://bestwebaudit.com; passive public-page link check; respects robots.txt)

What it does

What it does not do

How gently it behaves

The free scan visits about 25 pages, at roughly one request per second to your site, and stops if your site answers with “too many requests” (HTTP 429). It obeys your robots.txt.

How to control or block it

Add this to your robots.txt to keep it out entirely:

User-agent: BestWebAuditBot
Disallow: /

Or disallow only some paths. If your robots.txt cannot be retrieved reliably, the scanner assumes it is not allowed and does not scan.

The only time the scanner ignores robots.txt is for a site whose owner has proven ownership to us (by uploading a code file or adding a DNS record). A firewall rule on the user-agent above will also block it.

Questions or complaints

If the scanner is causing a problem, tell us the website address and we will look into it promptly.