BevizioBot

BevizioBot is the crawler behind the page watcher in Bevizio’s site log. If you saw it in your server logs, someone with access to this site’s Google Search Console turned the watcher on in their Bevizio account. It visits that site’s own pages so the log can say what changed and when, for example a new title, a page that now returns an error, or an edited robots.txt file. If you would rather it did not visit, the steps to stop it are at the bottom of this page.

How to recognize it

Every request it makes sends this user agent:

BevizioBot/1.0 (+https://www.bevizio.com/bot)

It does not run from a fixed list of addresses, and it does not yet sign its requests, so we cannot offer a way to verify a request by its address. The user agent can be copied by anyone, so treat it as a label and not as proof. If a request claims to be BevizioBot and does not carry that exact text, it is not ours. In a robots.txt file, use the name BevizioBot on its own, without the /1.0 and the rest. If your site sits behind a bot-protection service that challenges or blocks crawlers it cannot verify, BevizioBot will be blocked there and will stop; to let it through, allow that user agent, or ask us to stop watching the site.

What it visits, and how much

• Only pages on the site the owner connected to Bevizio through Search Console. If they connected a whole domain, that includes its subdomains, and it reads each subdomain’s own robots.txt before visiting it. If they connected a single section of a site, it stays inside that section. It does not follow a redirect that leaves the site.
• Your robots.txt file first, then the sitemap files it lists (or /sitemap.xml if it lists none), then the home page and other pages the owner’s Search Console data shows. It watches up to 500 pages per site and makes at most 1,500 requests a site a day.
• It runs at about 3 minutes past each hour, but a site is only visited when its pages are due. Each page is revisited at most about once a day, and your robots.txt file is read at the start of every visit.
• It reads the page’s HTML with an ordinary GET request. It does not log in, submit a form, run scripts or send cookies. It does not look at images, video or other files, and it keeps nothing from them if one is requested.

How it behaves

• It reads your robots.txt at the start of every visit and checks every request against it, including each redirect it follows on your site. It does not fetch anything robots.txt disallows for BevizioBot. If robots.txt cannot be read because your server returns an error, it does not crawl.
• It makes one request at a time and never more than 2 a second, counting robots.txt, sitemap files and redirects. If your robots.txt sets a Crawl-delay it waits that long between requests, up to 30 seconds.
• It slows down when your server answers 429, or 503 with a Retry-After header, and it waits out a Retry-After of up to a minute. If a site asks it to wait longer than that, it ends the visit and tries again the next day. It treats any other server error (500 and up) as a sign your site is struggling. If it is blocked twice in a row, or gets five errors or slow-downs in a row, it stops for that visit and tries again the next day. A site that blocks it is left alone for 24 hours, then it reads robots.txt again and carries on if it is allowed.

What it keeps

For each page it stores a few facts: the address (and the address it redirects to), the status code, the title and description, the headings, the canonical address, the robots settings, the word and link counts, and short fingerprints that change when the main text, the structured data or the internal links change. It does not keep the page itself. When a later visit shows the same difference again, an entry is added to the owner’s site log. A change to numbers alone, such as a price edited from 19 to 29, is not noticed. The owner can switch the watcher off in their Bevizio settings, which stops the visits, and can delete everything it stored by deleting their site log. How Bevizio handles this data is in the privacy policy.

How to block or slow it

Add this to the robots.txt file of the site, and the crawler will not visit any page there. It takes effect at the next visit, which is within about a day:

User-agent: BevizioBot
Disallow: /

To keep it but make it slower, set a delay in seconds between requests:

User-agent: BevizioBot
Crawl-delay: 10

If you cannot edit robots.txt, you can block the user agent above in your firewall or server settings, or email hello@bevizio.com with the address of the site and we will stop watching it. It is run by Bevizio, which is described on the about page.