Google Warns: Beware Of Fake Googlebot Traffic via @sejournal, @MattGSouthern

Google’s Developer Advocate, Martin Splitt, warns website owners to beryllium cautious of postulation that appears to travel from Googlebot. Many requests pretending to beryllium Googlebot are really from third-party scrapers.

He shared this successful the latest episode of Google’s SEO Made Easy series, emphasizing that “not everyone who claims to beryllium Googlebot really is Googlebot.”

Why does this matter?

Fake crawlers tin distort analytics, devour resources, and marque it hard to measure your site’s show accurately.

Here’s however to separate betwixt morganatic Googlebot postulation and fake crawler activity.

Googlebot Verification Methods

You tin separate existent Googlebot postulation from fake crawlers by looking astatine wide postulation patterns alternatively than antithetic requests.

Real Googlebot postulation tends to person accordant petition frequency, timing, and behavior.

If you fishy fake Googlebot activity, Splitt advises utilizing the pursuing Google tools to verify it:

URL Inspection Tool (Search Console)

• Finding circumstantial contented successful the rendered HTML confirms that Googlebot tin successfully entree the page.
• Provides unrecorded investigating capableness to verify existent entree status.

Rich Results Test

• Acts arsenic an alternate verification method for Googlebot access
• Shows however Googlebot renders the page
• Can beryllium utilized adjacent without Search Console access

Crawl Stats Report

• Shows elaborate server effect information specifically from verified Googlebot requests
• Helps place patterns successful morganatic Googlebot behavior

There’s a cardinal regulation worthy noting: These tools verify what existent Googlebot sees and does, but they don’t straight place impersonators successful your server logs.

To afloat support against fake Googlebots, you would request to:

• Compare server logs against Google’s authoritative IP ranges
• Implement reverse DNS lookup verification
• Use the tools supra to found baseline morganatic Googlebot behavior

Monitoring Server Responses

Splitt besides stressed the value of monitoring server responses to crawl requests, particularly:

• 500-series errors
• Fetch errors
• Timeouts
• DNS problems

These issues tin importantly interaction crawling ratio and hunt visibility for larger websites hosting millions of pages.

Splitt says:

“Pay attraction to the responses your server gave to Googlebot, particularly a precocious fig of 500 responses, fetch errors, timeouts, DNS problems, and different things.”

He noted that portion immoderate errors are transient, persistent issues “might privation to analyse further.”

Splitt suggested utilizing server log investigation to marque a much blase diagnosis, though helium acknowledged that it’s “not a basal happening to do.”

However, helium emphasized its value, noting that “looking astatine your web server logs… is simply a almighty mode to get a amended knowing of what’s happening connected your server.”

Potential Impact

Beyond security, fake Googlebot postulation tin interaction website show and SEO efforts.

Splitt emphasized that website accessibility successful a browser doesn’t warrant Googlebot access, citing assorted imaginable barriers, including:

• Robots.txt restrictions
• Firewall configurations
• Bot extortion systems
• Network routing issues

Looking Ahead

Fake Googlebot postulation tin beryllium annoying, but Splitt says you shouldn’t interest excessively overmuch astir uncommon cases.

Suppose fake crawler enactment becomes a occupation oregon uses excessively overmuch server power. In that case, you tin instrumentality steps similar limiting the complaint of requests, blocking circumstantial IP addresses, oregon utilizing amended bot detection methods.

For much connected this issue, spot the afloat video below:

Featured Image: eamesBot/Shutterstock