About our scanner (BestWebAuditBot)
Did you see BestWebAuditBot in your server logs? Someone entered your website address into Best Web Audit to check it for problems. Here is how the scanner behaves.
How it identifies itself
User-agent: BestWebAuditBot/0.1 (+https://bestwebaudit.com; passive public-page link check; respects robots.txt)
What it does
- Loads your public pages in a real browser to find links, images and scripts, and to notice browser errors.
- Checks that linked pages and files respond, using lightweight requests.
- Reads page titles, descriptions and a few technical settings.
What it does not do
- It never logs in, submits forms, or follows links such as “log out”.
- It does not try to find vulnerabilities, guess passwords or access private areas.
- It does not follow redirects to other websites.
How gently it behaves
The free scan visits about 25 pages, at roughly one request per second to your site, and stops if your site answers with “too many requests” (HTTP 429). It obeys your robots.txt.
How to control or block it
Add this to your robots.txt to keep it out entirely:
User-agent: BestWebAuditBot Disallow: /
Or disallow only some paths. If your robots.txt cannot be retrieved reliably, the scanner assumes it is not allowed and does not scan.
The only time the scanner ignores robots.txt is for a site whose owner has proven ownership to us (by uploading a code file or adding a DNS record). A firewall rule on the user-agent above will also block it.
Questions or complaints
If the scanner is causing a problem, tell us the website address and we will look into it promptly.
