Google On Diagnosing Multi-Domain Crawling Issues via @sejournal, @MattGSouthern

Google’s Search Advocate, John Mueller, shared insights connected diagnosing wide crawling issues.

This guidance was shared successful effect to a disruption reported by Adrian Schmidt connected LinkedIn. Google’s crawler stopped accessing respective of his domains astatine the aforesaid time.

Despite the interruption, Schmidt noted that unrecorded tests via Search Console continued to relation without mistake messages.

Investigations indicated nary summation successful 5xx errors oregon issues with robots.txt requests.

What could the occupation be?

Mueller’s Response

Addressing the situation, Mueller pointed to shared infrastructure arsenic the apt cause:

“If it shared crossed a clump of domains and focuses connected thing similar crawling, it’s astir apt an contented with a shared portion of infrastructure. If it’s already recovering, astatine slightest it’s not urgent anymore and you person a spot of clip to poke astatine caller changes / infrastructure logs.”

Infrastructure Investigation

All affected sites utilized Cloudflare arsenic their CDN, which raised immoderate eyebrows.

When asked astir debugging, Mueller recommended checking Search Console information to find whether DNS oregon failed requests were causing the problem.

Mueller stated:

“The crawl stats successful Search Console volition besides amusement a spot more, possibly assistance determine betwixt accidental DNS vs requests failing.”

He besides pointed retired that the timing was a cardinal clue:

“If it’s each astatine precisely the aforesaid time, it wouldn’t beryllium robots.txt, and astir apt not DNS.”

Impact connected Search Results

Regarding hunt visibility concerns, Mueller reassured this benignant of disruption wouldn’t origin immoderate problems:

“If this is from today, and it conscionable lasted a fewer hours, I wouldn’t expect immoderate disposable issues successful search.”

Why This Matters

When Googlebot abruptly stops crawling crossed galore sites simultaneously, it tin beryllium challenging to place the basal cause.

While impermanent crawling pauses mightiness not instantly interaction hunt rankings, they tin disrupt Google’s quality to observe and scale caller content.

The incidental highlights a vulnerability organizations mightiness look without realizing it, particularly those relying connected shared infrastructure.

How This Can Help You

If clip Googlebot stops crawling your sites:

• Check if the occupation hits aggregate sites astatine once
• Look astatine your shared infrastructure first
• Use Search Console information to constrictive down the cause
• Don’t regularisation retired DNS conscionable due to the fact that regular postulation looks fine
• Keep an oculus connected your logs

For anyone moving aggregate sites down a CDN, marque definite you:

• Have bully logging acceptable up
• Watch your crawl rates
• Know who to telephone erstwhile things spell sideways
• Keep tabs connected your infrastructure provider

Featured Image: PeopleImages.com – Yuri A/Shutterstock

SEJ STAFF Matt G. Southern Senior News Writer astatine Search Engine Journal

Matt G. Southern, Senior News Writer, has been with Search Engine Journal since 2013. With a bachelor’s grade successful communications, ...