Google’s Updated Crawler Guidance Recommends ETags via @sejournal, @martinibuster

Google announced an update to their crawler documentation, adding much accusation astir caching which should assistance amended recognize however to optimize for Google’s crawler. By pursuing the caller guidelines connected implementing due HTTP caching headers, SEOs and publishers tin amended crawling ratio and optimize server resources.

Updated Crawler Documentation

The crawler documentation present has a conception that explains however Google’s crawlers usage HTTP caching mechanisms that assistance to conserve computing resources for some publishers and Google during crawling.

Additions to the documentation importantly grow connected the anterior version.

Caching Mechanisms

Google recommends enabling caching with headers similar ETag and If-None-Match, arsenic good arsenic optionally Last-Modified and If-Modified-Since, to awesome whether contented has changed. This tin assistance trim unnecessary crawling and prevention server resources, which is simply a triumph for some publishers and Google’s crawlers.

The caller documentation states:

“Google’s crawling infrastructure supports heuristic HTTP caching arsenic defined by the HTTP caching standard, specifically done the ETag response- and If-None-Match petition header, and the Last-Modified response- and If-Modified-Since petition header.”

Google’s Preference For Preference for ETag

Google recommends utilizing ETag implicit Last-Modified due to the fact that ETag is little prone to errors similar day formatting issues and provides much precise contented validation. It besides explains what happens if some ETag and Last-Modified effect headers are served:

“If some ETag and Last-Modified effect header fields are contiguous successful the HTTP response, Google’s crawlers usage the ETag worth arsenic required by the HTTP standard.”

The caller documentation besides states that different HTTP caching directives are not supported.

Variable Support Across Crawlers

The caller documentation explains that enactment for caching differs among Google’s crawlers. For example, Googlebot supports caching for re-crawling, portion Storebot-Google has constricted caching support.

Google explains:

“Individual Google crawlers and fetchers whitethorn oregon whitethorn not marque usage of caching, depending connected the needs of the merchandise they’re associated with. For example, Googlebot supports caching erstwhile re-crawling URLs for Google Search, and Storebot-Google lone supports caching successful definite conditions”

Guidance On Implementation

Google’s caller documentation recommends contacting hosting oregon CMS providers for assistance. It besides suggests (but doesn’t require) that publishers acceptable the max-age tract of the Cache-Control effect header successful bid to assistance crawlers cognize erstwhile to crawl circumstantial URLs.

Entirely New Blog Post

Google has besides published a marque caller blog post:

Crawling December: HTTP caching

Read the updated documentation:

HTTP Caching

Featured Image by Shutterstock/Asier Romero

SEJ STAFF Roger Montti Owner - Martinibuster.com astatine Martinibuster.com

I person 25 years hands-on acquisition successful SEO, evolving on with the hunt engines by keeping up with the latest ...