Google Quietly Launches New AI Crawler via @sejournal, @martinibuster

Google softly added a caller bot to their crawler documentation that crawls connected behalf of commercialized clients of their Vertex AI product. It appears that the caller crawler whitethorn lone crawl sites controlled by the tract owners, but the documentation isn’t wholly wide connected that point.

Vertex AI Agents

Google-CloudVertexBot, the caller crawler, ingests website contented for Vertex AI clients, dissimilar different bots listed successful the Search Central documentation that are tied to Google Search oregon advertising.

The authoritative Google Cloud documentation offers the pursuing information:

“In Vertex AI Agent Builder, determination are assorted kinds of information stores. A information store tin incorporate lone 1 benignant of data.”

It goes connected to database six types of data, 1 of which is nationalist website data. On crawling the documentation says that determination are 2 kinds of website crawling with limitations circumstantial to each kind.

1. Basic website indexing
2. Advanced website indexing

Documentation Is Confusing

The documentation explains website data:

“A information store with website information uses information indexed from nationalist websites. You tin supply a acceptable of domains and acceptable up hunt oregon recommendations implicit information crawled from the domains. This information includes substance and images tagged with metadata.”

The supra statement doesn’t accidental thing astir verifying domains. The statement of Basic website indexing doesn’t accidental thing astir tract proprietor verification either.

But the documentation for Advanced website indexing does accidental that domain verification is required and besides imposes indexing quotas.

However, the documentation for the crawler itself says that the caller crawler crawls connected the “site owners’ request” truthful it whitethorn beryllium that it won’t travel crawling nationalist sites.

Now here’s the confusing part, the Changelog notation for this caller crawler indicates that the caller crawler could travel to scrape your site.

Here’s what the changelog says:

“The caller crawler was introduced to assistance tract owners place the caller crawler traffic.”

New Google Crawler

The caller crawler is called Google-CloudVertexBot.

This is the caller accusation connected it:

“Google-CloudVertexBot crawls sites connected the tract owners’ petition erstwhile gathering Vertex AI Agents.

User cause tokens

• Google-CloudVertexBot
• Googlebot”

User cause substring
Google-CloudVertexBot

Unclear Documentation

The documentation seems to bespeak that the caller crawler doesn’t scale nationalist sites but the changelog indicates that it was added truthful that tract owners tin place postulation from the caller crawler. Should you artifact the caller crawler with a robots.txt conscionable successful case? It’s not unreasonable to see fixed that the documentation is reasonably unclear connected whether it lone crawls domains that are verified to beryllium nether the power of the entity initiating the crawl.

Read Google’s caller documentation:

Google-CloudVertexBot

Featured Image by Shutterstock/ShotPrime Studio

SEJ STAFF Roger Montti Owner - Martinibuster.com astatine Martinibuster.com

I person 25 years hands-on acquisition successful SEO, evolving on with the hunt engines by keeping up with the latest ...