Google On Robots.txt: When To Use Noindex vs. Disallow via @sejournal, @MattGSouthern

In a caller YouTube video, Google’s Martin Splitt explained the differences betwixt the “noindex” tag successful robots meta tags and the “disallow” bid successful robots.txt files.

Splitt, a Developer Advocate astatine Google, pointed retired that some methods assistance negociate however hunt motor crawlers enactment with a website.

However, they person antithetic purposes and shouldn’t beryllium utilized successful spot of each other.

When To Use Noindex

The “noindex” directive tells hunt engines not to see a circumstantial leafage successful their hunt results. You tin adhd this acquisition successful the HTML caput conception utilizing the robots meta tag oregon the X-Robots HTTP header.

Use “noindex” erstwhile you privation to support a leafage from showing up successful hunt results but inactive let hunt engines to work the page’s content. This is adjuvant for pages that users tin spot but that you don’t privation hunt engines to display, similar thank-you pages oregon interior hunt effect pages.

When To Use Disallow

The “disallow” directive successful a website’s robots.txt record stops hunt motor crawlers from accessing circumstantial URLs oregon patterns. When a leafage is disallowed, hunt engines volition not crawl oregon scale its content.

Splitt advises utilizing “disallow” erstwhile you privation to artifact hunt engines wholly from retrieving oregon processing a page. This is suitable for delicate information, similar backstage idiosyncratic data, oregon for pages that aren’t applicable to hunt engines.

Common Mistakes to Avoid

One communal mistake website owners marque is utilizing “noindex” and “disallow” for the aforesaid page. Splitt advises against this due to the fact that it tin origin problems.

If a leafage is disallowed successful the robots.txt file, hunt engines cannot spot the “noindex” bid successful the page’s meta tag oregon X-Robots header. As a result, the leafage mightiness inactive get indexed, but with constricted information.

To halt a leafage from appearing successful hunt results, Splitt recommends utilizing the “noindex” bid without disallowing the leafage successful the robots.txt file.

Google provides a robots.txt study successful Google Search Console to trial and show however robots.txt files impact hunt motor indexing.

Why This Matters

Understanding the due usage of “noindex” and “disallow” directives is indispensable for SEO professionals.

Following Google’s proposal and utilizing the disposable investigating tools volition assistance guarantee your contented appears successful hunt results arsenic intended.

See the afloat video below:

Featured Image: Asier Romero/Shutterstock

SEJ STAFF Matt G. Southern Senior News Writer astatine Search Engine Journal

Matt G. Southern, Senior News Writer, has been with Search Engine Journal since 2013. With a bachelor’s grade successful communications, ...