Google Explains The Index, Follow Meta Tag via @sejournal, @martinibuster

3 months ago 27

ARTICLE AD BOX

Google’s John Mueller answered a question connected Reddit astir a commonly utilized robots meta tag and what would hap if it was missing. Mueller’s answer, portion it makes consciousness and is documented, whitethorn inactive travel arsenic a astonishment to galore publishers and SEOs.

Robots Meta Tag

The HTML meta constituent communicates metadata. Metadata is instrumentality readable accusation that a crawler similar Googlebot tin read.

There are galore kinds of meta elements similar the meta statement constituent but the Robots Meta Element is antithetic successful that it tin power the hunt motor crawlers.

The accusation communicated by the robots meta tag is called a directive, which means that robot crawlers are obligated to obey the instructions successful the robots meta tag.

There are galore directions to walk on done the robots meta but the pursuing meta tag is 1 that is applicable to the question John Mueller answered.

The noindex, nary travel meta tag:

The supra meta tag tells the hunt motor crawlers to not scale the contented connected the webpage and to not travel immoderate links.

One of the astir communal meta tags is this one, which commands hunt engines to scale the contented and travel each the links:

While the supra meta tag is common, determination is simply a important magnitude of misunderstanding astir it. There’s a enactment of reasoning that due to the fact that Google supports nofollow past it indispensable connote that Google supports the travel directive.

I recovered a fig of authoritative websites that accidental that Google uses the meta robots index, travel meta tag.

But that’s not really however Google uses those directives, arsenic John Mueller makes wide successful his answer.

What’s The Effect Of Leaving Out The Meta Robots Index Tag?

The idiosyncratic connected Reddit asked the pursuing question:

“I’m a spot confused with a website I’m moving on.

So, this is what the meta snippets connected astir of the websites I enactment connected look like:

But, connected the website astatine hand, it’s missing the ‘index’ tag.

My question is: What’s the effect of the tract missing the ‘index’ tag.”

John Mueller answered:

“The “index” robots meta tag has nary relation (at slightest successful Google) – it’s wholly ignored. Also “follow”.

Google has https://developers.google.com/search/docs/crawling-indexing/special-tags & https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag to papers the meta tags that person functions. You tin usage thing else, it’ll beryllium ignored. <meta name=”robots” content=”topranking bestcheese”> is an option, if you privation to propulsion radical off.”

Why Google Ignores Robots Index & Follow

The elemental crushed wherefore Google ignores the robots scale and travel meta tag is due to the fact that indexing and travel are the defaults.

Indexing and pursuing links is what hunt motor robots do, they don’t person to beryllium told to scale contented and travel links due to the fact that that’s their purpose.

Google’s documentation connected robots tags advises:

“The default values are index, travel and don’t request to beryllium specified.”

The afloat database of valid directives for Google is here.

If the robots meta you privation to usage isn’t’ listed determination past Googlebot is going to disregard it.

Is Index, Follow Completely Useless?

It’s true, documented and authoritative that erstwhile it comes to Googlebot, <meta name=”robots” content=”index, follow”> is simply a discarded of HTML abstraction and is ignored by Googlebot.

Bing treats index, travel successful a akin mode but with a flimsy difference, arsenic described successful the official Bing documentation for meta tags.

This is what Bing’s documentation says astir the scale directive:

“By default we presume “index”, but if needed you tin usage <meta name=”robots” content=”index”> to explicitly authorities that we whitethorn scale the page.”

And this is what it says astir the travel directive:

“By default we presume “follow”, but you tin explicitly authorities “follow” if truthful desired.”

In my 20+ years of moving successful SEO creating websites, optimizing them and ranking them, I’ve ever considered it a bully argumentation to springiness the bots what they expect and effort not to springiness them thing that’s unexpected. So if a meta statement is not indispensable past my impulse would beryllium to permission it retired due to the fact that the full constituent of optimizing is to marque it arsenic casual arsenic imaginable for the hunt engines to scale and recognize the content, which means to get escaped of thing that mightiness enactment against that goal.

In this case, it’s highly apt it’s not going to person an effect 1 mode oregon the other.

But… There’s different mode that travel and scale travel radical up.

Some publishers usage this robots meta tag:

Some sites counsel that if the leafage isn’t indexed, the usage of the “follow” directive compels the hunt motor to travel the links.

But that’s not existent if there’s a “noindex” directive for the elemental reasons that Google cannot travel a nexus connected a leafage that it is not indexed. If it’s not successful the scale past the links connected those pages are not successful the index.

Featured Image by Shutterstock/Bangun Stock Productions