Google Crawler Documentation Has A New IP List via @sejournal, @martinibuster

Google updated their Googlebot and crawler documentation to adhd a scope of IPs for bots triggered by users of Google products. The names of the feeds switched which is important for publishers who are whitelisting Google controlled IP addresses. The alteration volition beryllium utile for publishers who privation to artifact scrapers who are utilizing Google’s unreality and different crawlers not straight associated with Google itself.

New List Of IP Addresses

Google says that the database contains IP ranges that person agelong been successful use, truthful they’re not caller IP code ranges.

There are 2 kinds of IP code ranges:

1. IP ranges that are initiated by users but controlled by Google and resoluteness to a Google.com hostname.
   These are tools similar Google Site Verifier and presumably the Rich Results Tester Tool.
2. IP ranges that are initiated by users but not controlled by Google and resoluteness to a gae.googleusercontent.com hostname.
   These are apps that are connected Google unreality oregon apps scripts that are called from Gooogle Sheets.

The lists that correspond to each class are antithetic now.

Previously the database that corresponded to Google IP addresses was this one: special-crawlers.json (resolving to gae.googleusercontent.com)

Now the “special crawlers” database corresponds to crawlers that are not controlled by Google.

“IPs successful the user-triggered-fetchers.json entity resoluteness to gae.googleusercontent.com hostnames. These IPs are used, for example, if a tract moving connected Google Cloud (GCP) has a diagnostic that requires fetching outer RSS feeds connected the petition of the idiosyncratic of that site.”

The caller database that corresponds to Google controlled crawlers is: 

user-triggered-fetchers-google.json

“Tools and merchandise functions wherever the extremity idiosyncratic triggers a fetch. For example, Google Site Verifier acts connected the petition of a user. Because the fetch was requested by a user, these fetchers disregard robots.txt rules.

Fetchers controlled by Google originate from IPs successful the user-triggered-fetchers-google.json entity and resoluteness to a google.com hostname.”

The database of IPs from Google Cloud and App crawlers that Google doesn’t power tin beryllium recovered here:

https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json

The database of IP from Google that are triggered by users and controlled by Google is here:

https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json

New Section Of Content

There is simply a caller conception of contented that explains what the caller database is about.

“Fetchers controlled by Google originate from IPs successful the user-triggered-fetchers-google.json entity and resoluteness to a google.com hostname. IPs successful the user-triggered-fetchers.json entity resoluteness to gae.googleusercontent.com hostnames. These IPs are used, for example, if a tract moving connected Google Cloud (GCP) has a diagnostic that requires fetching outer RSS feeds connected the petition of the idiosyncratic of that site. ***-***-***-***.gae.googleusercontent.com oregon google-proxy-***-***-***-***.google.com user-triggered-fetchers.json and user-triggered-fetchers-google.json”

Google Changelog

Google’s changelog explained the changes similar this:

“Exporting an further scope of Google fetcher IP addresses
What: Added an further database of IP addresses for fetchers that are controlled by Google products, arsenic opposed to, for example, a idiosyncratic controlled Apps Script. The caller list, user-triggered-fetchers-google.json, contains IP ranges that person been successful usage for a agelong time.

Why: It became technically imaginable to export the ranges.”

Read the updated documentation:
Verifying Googlebot and different Google crawlers

Read the aged documentation:
Archive.org – Verifying Googlebot and different Google crawlers

Featured Image by Shutterstock/JHVEPhoto

SEJ STAFF Roger Montti Owner - Martinibuster.com astatine Martinibuster.com

I person 25 years hands-on acquisition successful SEO and person kept on  apical of the improvement of hunt each measurement ...

Subscribe To Our Newsletter.

Conquer your time with regular hunt selling news.