Google Clarifies the “Google-Extended” Crawler Documentation via @sejournal, @martinibuster

Google precocious updated the documentation of its Google-Extended web crawler idiosyncratic agent, reflecting changes successful merchandise naming and clarifying the interaction connected search, which whitethorn beryllium a interest for those who take to artifact the crawler. The updated documentation offers clearer guidance connected controlling contented entree for usage successful AI exemplary training.

Google-Extended User Agent

Introduced connected September 28, 2023, Google-Extended offers web publishers a idiosyncratic cause that tin beryllium utilized to power however their sites are crawled. Publishers tin let oregon disallow the Google-Extended idiosyncratic cause utilizing the Robots Exclusion Protocol, giving them a mode to opt-out of having their contented scraped and included successful AI grooming datasets.

Google describes Google-Extended arsenic a “standalone merchandise token” but that’s non-standard terminology for however publishers recognize the conception of User Agents.

The original announcement described the caller idiosyncratic agent:

“Today we’re announcing Google-Extended, a caller power that web publishers tin usage to negociate whether their sites assistance amended Bard and Vertex AI generative APIs, including aboriginal generations of models that powerfulness those products.

By utilizing Google-Extended to power entree to contented connected a site, a website head tin take whether to assistance these AI models go much close and susceptible implicit time.”

Blocking Google-Extended is done with the “Google-Extended” User Agent:

Google Changelog

Google keeps a changelog of important updates made to guidance and connection with web publishers and the hunt selling community. The changelog of Google’s developer pages announced a alteration to the Google-Extended documentation.

The revision comes aft the renaming of Bard to Gemini Apps, specifying that Google-Extended’s indexing present contributes to Gemini Apps and Vertex AI generative APIs. The caller wording reassures publishers that this does not impact Google Search, addressing imaginable concerns astir the imaginable implications from opting retired of Google-Extended AI information collection.

What Changed?

Google’s changelog clarifies that Google-Extended crawling is exclusive to Gemini Apps and has nary interaction connected Google Search.

The Changelog advises:

“Updated the statement of the Google-Extended merchandise token
What: With the sanction alteration of Bard to Gemini Apps, we clarified that Gemini Apps is affected by Google-Extended, and, based connected steadfast feedback, we specified that Google-Extended doesn’t impact Google Search.”

The updated guidance nary longer uses the Bard marque name, switching it retired to Gemini. And the pursuing condemnation was added:

“Google-Extended does not interaction a site’s inclusion oregon ranking successful Google Search.”

Read Google’s updated crawler overview:

Overview of Google crawlers and fetchers (user agents)

Featured Image by Shutterstock/Ribkhan

SEJ STAFF Roger Montti Owner - Martinibuster.com astatine Martinibuster.com

I person 25 years hands-on acquisition successful SEO and person kept on  apical of the improvement of hunt each measurement ...

Subscribe To Our Newsletter.

Conquer your time with regular hunt selling news.