ARTICLE AD BOX
A meta robots tag is simply a portion of HTML codification that tells hunt motor robots however to crawl, index, and show a page’s content.
It goes successful the <head> conception of the leafage and tin look similar this:
<meta name="robots" content="noindex">
The meta robots tag successful the illustration supra tells each hunt motor crawlers not to scale the page.
Let’s sermon what you tin usage robots meta tags for, wherefore they’re important for SEO, and however to usage them properly.
Meta Robots vs. Robots.txt
Meta robots tags and robots.txt files person akin functions but service antithetic purposes.
A robots.txt file is simply a azygous substance record that applies to the full site. And tells hunt engines which pages to crawl.
A meta robotstag applies to lone the leafage containing the tag. And tells hunt engines however to crawl, index, and show accusation from that leafage only.
Robots meta tags assistance power however Google crawls and indexes a page's content. Including whether to:
- Include a leafage successful hunt results
- Follow the links connected a page
- Index the images connected a page
- Show cached results of the leafage connected the hunt motor results pages (SERPs)
- Show a snippet of the leafage connected the SERPs
Below, we’ll research the attributes you tin usage to archer hunt engines however to interact with your pages.
But first, let’s sermon wherefore robots meta tags are important and however they tin impact your site’s SEO.
How Do Robots Meta Tags Affect SEO?
Robots meta tags assistance Google and different hunt engines crawl and scale your pages efficiently.
Especially for ample oregon often updated sites.
After all, you apt don’t request each leafage connected your tract to rank.
For example, you astir apt don’t privation hunt engines to index:
- Pages from your staging site
- Confirmation pages, specified arsenic convey you pages
- Admin oregon login pages
- Internal hunt effect pages
- Pages with duplicate content
Combining robots meta tags with different directives and files, specified arsenic sitemaps and robots.txt, tin truthful beryllium a utile portion of your technical SEO strategy. As they tin assistance forestall issues that could different clasp backmost your website’s performance.
What Are the Name and Content Specifications for Meta Robots Tags?
Meta robots tags incorporate 2 attributes: sanction and content. Both are required.
Name Attribute
This property indicates which crawler should travel the instructions successful the tag.
Like this:
name="crawler"
If you privation to code each crawlers, insert “robots” arsenic the “name” attribute.
Like this:
name="robots"
If you privation to restrict crawling to circumstantial hunt engines, the sanction property lets you bash that. And you tin take arsenic galore (or arsenic few) arsenic you want.
Here are a fewer communal crawlers:
- Google: Googlebot (or Googlebot-news for quality results)
- Bing: Bingbot (see the list of each Bing crawlers)
- DuckDuckGo: DuckDuckBot
- Baidu: Baiduspider
- Yandex: YandexBot
Content Attribute
The “content” property contains instructions for the crawler.
It looks similar this:
content="instruction"
Google supports the pursuing “content” values:
Default Content Values
Without a robots meta tag, crawlers volition scale contented and travel links by default (unless the nexus itself has a “nofollow” tag).
This is the aforesaid arsenic adding the pursuing “all” worth (although determination is nary request to specify it):
<meta name="robots" content="all"
So, if you don’t privation the leafage to look successful hunt results oregon for hunt engines to crawl its links, you request to adhd a meta robots tag. With due contented values.
Noindex
The meta robots “noindex” worth tells crawlers not to see the leafage successful the hunt engine’s scale oregon show it successful the SERPs.
<meta name="robots" content="noindex">
Without the noindex value, hunt engines whitethorn scale and service the leafage successful the hunt results.
Typical usage cases for “noindex” are cart oregon checkout pages connected an ecommerce website.
Nofollow
This tells crawlers not to crawl the links connected the page.
<meta name="robots" content="nofollow">
Google and different hunt engines often usage links connected pages to observe those linked pages. And links tin assistance walk authorization from 1 leafage to another.
Use the nofollow regularisation if you don’t privation the crawler to travel immoderate links connected the leafage oregon walk immoderate authorization to them.
This mightiness beryllium the lawsuit if you don’t person power implicit the links placed connected your website. Such arsenic successful an unmoderated forum with mostly user-generated content.
Noarchive
The “noarchive” contented worth tells Google not to service a transcript of your leafage successful the hunt results.
<meta name="robots" content="noarchive">
If you don’t specify this value, Google whitethorn amusement a cached transcript of your leafage that searchers whitethorn spot successful the SERPs.
You could usage this worth for time-sensitive content, interior documents, PPC landing pages, oregon immoderate different leafage you don’t privation Google to cache.
Noimageindex
This worth instructs Google not to scale the images connected the page.
<meta name="robots" content="noimageindex">
Using “noimageindex” could wounded imaginable organic traffic from representation results. And if users tin inactive entree the page, they’ll inactive beryllium capable to find the images. Even with this tag successful place.
Notranslate
“Notranslate” prevents Google from serving translations of the leafage successful hunt results.
<meta name="robots" content="notranslate">
If you don’t specify this value, Google tin amusement a translation of the rubric and snippet of a hunt effect for pages that aren’t successful the aforesaid connection arsenic the hunt query.
If the searcher clicks the translated link, each further enactment is done Google Translate. Which automatically translates immoderate followed links.
Use this worth if you similar not to person your leafage translated by Google Translate.
For example, if you person a merchandise leafage with merchandise names you don’t privation translated. Or if you find Google’s translations aren’t ever accurate.
Nositelinkssearchbox
This worth tells Google not to make a hunt container for your tract successful hunt results.
<meta name="robots" content="nositelinkssearchbox">
If you don’t usage this value, Google tin amusement a hunt container for your tract successful the SERPs.
Like this:
Use this worth if you don’t privation the hunt container to appear.
Nosnippet
“Nosnippet” stops Google from showing a substance snippet oregon video preview of the leafage successful hunt results.
<meta name="robots" content="nosnippet">
Without this value, Google tin nutrient snippets of substance oregon video based connected the page’s content.
The worth “nosnippet” besides prevents Google from utilizing your contented arsenic a “direct input” for AI Overviews. But it’ll besides forestall meta descriptions, rich snippets, and video previews. So usage it with caution.
While not a meta robots tag, you tin usage the “data-nosnippet” property to forestall circumstantial sections of your pages from showing successful hunt results.
Like this:
<p>This text could be shown in a snippet
<span data-nosnippet>but this part would not be shown</span>.</p>
Max-snippet
“Max-snippet” tells Google the maximum quality magnitude it tin amusement arsenic a substance snippet for the leafage successful hunt results.
This property has 2 important cases to beryllium alert of:
- 0: Opts your leafage retired of substance snippets (as with “nosnippet”)
- -1: Indicates there’s nary limit
For example, to forestall Google from displaying a substance snippet successful the SERPs, you could use:
<meta name="robots" content="max-snippet:0">
Or, if you privation to let up to 100 characters:
<meta name="robots" content="max-snippet:100">
To bespeak there’s nary quality limit:
<meta name="robots" content="max-snippet:-1">
Max-image-preview
This tells Google the maximum size of a preview representation for the leafage successful the SERPs.
There are 3 values for this directive:
- None: Google won’t amusement a preview image
- Standard: Google whitethorn amusement a default preview
- Large: Google whitethorn amusement a larger preview image
<meta name="robots" content="max-image-preview:large">
Max-video-preview
This worth tells Google the maximum magnitude you privation it to usage for a video snippet successful the SERPs (in seconds).
As with “max-snippet,” determination are 2 important values for this directive:
- 0: Opts your leafage retired of video snippets
- -1: Indicates there’s nary limit
For example, the tag beneath allows Google to service a video preview of up to 10 seconds:
<meta name="robots" content="max-video-preview:10">
Use this regularisation if you privation to bounds your snippet to amusement definite parts of your videos. If you don’t, Google whitethorn amusement a video snippet of immoderate length.
Indexifembedded
When utilized on with noindex, this (fairly new) tag lets Google scale the page’s contented if it’s embedded successful different leafage done HTML elements specified arsenic iframes.
(It wouldn’t person an effect without the noindex tag.)
<meta name="robots" content="noindex, indexifembedded">
“Indexifembedded” has been created with media publishers successful mind:
They often person media pages that should not beryllium indexed. But they bash privation the media indexed erstwhile it’s embedded successful different page’s content.
Previously, they would person utilized “noindex” connected the media page. Which would forestall it from being indexed connected the embedding pages too. “Indexifembedded” solves this.
Unavailable_after
The “unavailable_after” worth prevents Google from showing a leafage successful the SERPs aft a circumstantial day and time.
<meta name="robots" content="unavailable_after: 2024-10-21">
You indispensable specify the day and clip utilizing RFC 822, RFC 850, oregon ISO 8601 formats. Google ignores this regularisation if you don’t specify a date/time. By default, determination is nary expiration day for content.
You tin usage this worth for limited-time lawsuit pages, time-sensitive pages, oregon pages you nary longer deem important. This functions similar a timed noindex tag, truthful usage it with caution. Or you could extremity up with indexing issues aboriginal down the line.
Combining Robots Meta Tag Rules
There are 2 ways successful which you tin harvester robots meta tag rules:
- Writing aggregate comma-separated values into the “content” attribute
- Providing 2 oregon much robots meta elements
Multiple Values Inside the ‘Content’ Attribute
You tin premix and lucifer the “content” values we’ve conscionable outlined. Just marque definite to abstracted them by comma. Once again, the values are not case-sensitive.
For example:
<meta name="robots" content="noindex, nofollow">
This tells hunt engines not to scale the leafage oregon crawl immoderate of the links connected the page.
You tin harvester noindex and nofollow utilizing the “none” value:
<meta name="robots" content="none">
But immoderate hunt engines, similar Bing, don’t enactment this value.
Two oregon More Robots Meta Elements
Use abstracted robots meta elements if you privation to instruct antithetic crawlers to behave differently.
For example:
<meta name="robots" content="nofollow"><meta name="YandexBot" content="noindex">
This operation instructs each crawlers to debar crawling links connected the page. But it besides tells Yandex specifically not to scale the leafage (in summation to not crawling the links).
The array beneath shows the supported meta robots values for antithetic hunt engines:
| Value | | Bing | Yandex |
| noindex | Y | Y | Y |
| noimageindex | Y | N | N |
| nofollow | Y | N | Y |
| noarchive | Y | Y | Y |
| nocache | N | Y | N |
| nosnippet | Y | Y | N |
| nositelinkssearchbox | Y | N | N |
| notranslate | Y | N | N |
| max-snippet | Y | Y | N |
| max-video-preview | Y | Y | N |
| max-image-preview | Y | Y | N |
| indexifembedded | Y | N | N |
| unavailable_after | Y | N | N |
How to Implement Robots Meta Tags
Adding Robots Meta Tags to Your HTML Code
If you tin edit your page’s HTML code, adhd your robots meta tags into the <head> conception of the page.
For example, if you privation hunt engines to debar indexing the leafage and to debar crawling links, use:
<meta name="robots" content="noindex, nofollow">
Implementing Robots Meta Tags successful WordPress
If you're utilizing a WordPress plugin similar Yoast SEO, unfastened the “Advanced” tab successful the artifact beneath the leafage editor.
Set the “noindex” directive by switching the “Allow hunt engines to amusement this leafage successful hunt results?” drop-down to “No.”
Or forestall hunt engines from pursuing links by switching the “Should hunt engines travel links connected this page?” to “No.”
For different directives, you person to instrumentality them successful the “Meta robots advanced” field.
Like this:
If you’re utilizing Rank Math, prime the robots directives consecutive from the “Advanced” tab of the meta box.
Like so:
Adding Robots Meta Tags successful Shopify
To instrumentality robots meta tags successful Shopify, edit the <head> conception of your theme.liquid layout file.
To acceptable the directives for a circumstantial page, adhd the codification beneath to the file:
{% if handle contains 'page-name' %}
<meta name="robots" content="noindex">
{% endif %}
This illustration instructs hunt engines not to scale /page-name/ (but to inactive travel each the links connected the page).
You indispensable make abstracted entries to acceptable the directives crossed antithetic pages.
Implementing Robots Meta Tags successful Wix
Open your Wix dashboard and click “Edit Site.”
Click “Pages & Menu” successful the left-hand navigation.
In the tab that opens, click “...” adjacent to the leafage you privation to acceptable robots meta tags for. Choose “SEO basics.”
Then click “Advanced SEO” and click connected the collapsed point “Robots meta tag.”
Now you tin acceptable the applicable robots meta tags for your leafage by clicking the checkboxes.
If you request “notranslate,” “nositelinkssearchbox,” “indexifembedded,” oregon “unavailable_after,” click “Additional tags”and “Add New Tags.”
Now you tin paste your meta tag successful HTML format.
What Is the X-Robots-Tag?
An x-robots-tag serves the aforesaid relation arsenic a meta robots tag but for non-HTML files. Such arsenic images and PDFs.
You see it arsenic portion of the HTTP header effect for a URL.
Like this:
To instrumentality the x-robots-tag, you'll request to entree your website’s header.php, .htaccess, oregon server configuration file. You tin usage the aforesaid rules arsenic those we discussed earlier for meta robots tags.
Using X-Robots-Tag connected an Apache Server
To usage the x-robots-tag connected an Apache web server, adhd the pursuing to your site's .htaccess record oregon httpd.conf file.
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>
For example, the codification supra instructs hunt engines not to scale oregon to travel immoderate links connected each PDFs crossed the full site.
Using X-Robots-Tag connected an Nginx Server
If you're moving an Nginx server, adhd the codification beneath to your site's .conf file:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
The illustration codification supra volition use noindex and nofollow values to each of the site’s PDFs.
Common Meta Robots Tag Mistakes to Avoid
Let’s instrumentality a look astatine immoderate communal mistakes to debar erstwhile utilizing meta robots and x-robots-tags:
Using Meta Robots Directives connected a Page Blocked by Robots.txt
If you disallow crawling of a leafage successful your robots.txt file, large hunt motor bots won’t crawl it. So immoderate meta robots tags oregon x-robots-tags connected that leafage volition beryllium ignored.
Ensure hunt engines tin crawl immoderate pages with meta robots tags oregon x-robots-tags.
Adding Robots Directives to the Robots.txt File
Although ne'er officially supported by Google, you were erstwhile capable to adhd a “noindex” directive to your site's robots.txt file.
This is nary longer an option, arsenic confirmed by Google.
The “noindex” regularisation successful robots meta tags is the astir effectual mode to region URLs from the scale erstwhile you bash let crawling.
Removing Pages with a Noindex Directive from Sitemaps
If you’re trying to region a leafage from the scale utilizing a “noindex” directive, permission the leafage successful your sitemap until it has been removed.
Removing the leafage earlier it’s deindexed tin origin delays successful deindexing.
Not Removing the ‘Noindex’ Directive from a Staging Environment
Preventing robots from crawling pages successful your staging tract is simply a champion practice. But it’s casual to hide to region “noindex” erstwhile the tract moves into production.
And the results tin beryllium disastrous. As hunt engines whitethorn ne'er crawl and scale your site.
To debar these issues, cheque that your robots meta tags are close earlier moving your tract from a staging level to a unrecorded environment.
Finding and fixing crawlability issues (and different method SEO errors) connected your tract tin dramatically amended performance.
If you don’t cognize wherever to start, usage Semrush’s Site Audit tool.
Just participate your domain and click “Start Audit.”
You tin configure assorted settings, similar the fig of pages to crawl and which crawler you’d similar to use. But you tin besides conscionable permission them arsenic their defaults.
When you’re ready, click “Start Site Audit.”
When the audit is complete, caput to the “Issues” tab.
In the hunt box, benignant “blocked from crawling” to spot errors regarding your meta robots tags oregon x-robots-tags.
Like this:
Click connected “Why and however to hole it” adjacent to an contented to work much astir the contented and however to hole it.
Fix each of these issues to amended your site’s crawlability. And to marque it easier for Google to find and scale your content.
FAQs
When Should You Use the Robots Meta Tag vs. X-Robots-Tag?
Use the robots meta tag for HTML pages and the x-robots-tag for different non-HTML resources. Like PDFs and images.
This is not a method requirement. You could archer crawlers what to bash with your webpages via x-robots-tags. But it’s easier to execute the aforesaid happening by implementing the robots meta tags connected a webpage.
You tin besides usage x-robots-tags to use directives successful bulk. Rather than simply connected a leafage level.
Do You Need to Use Both Meta Robots Tag and X-Robots-Tag?
You don’t request to usage some meta robots tags and x-robots-tags. Telling crawlers however to scale your leafage utilizing either a meta robots oregon x-robots-tag is enough.
Repeating the acquisition won’t summation the chances that Googlebot oregon immoderate different crawlers volition travel it.
What Is the Easiest Way to Implement Robots Meta Tags?
Using a plugin is usually the easiest mode to adhd robots meta tags to your webpages. Because it doesn’t usually necessitate you to edit immoderate of your site’s code.
Which plugin you should usage depends connected the contented absorption strategy (CMS) you’re using.
Robots meta tags marque definite that the contented you’re putting truthful overmuch effort into gets indexed. If hunt engines don’t scale your content, you can’t make immoderate integrated traffic.
So, getting the basal robots meta tag parameters close (like noindex and nofollow) is perfectly crucial.
Check that you’re implementing these tags correctly utilizing Semrush Site Audit.
This station was updated successful 2024. Excerpts from the archetypal nonfiction by Carlos Silva whitethorn remain.
English (US)