Meta Robots Tag Checker Tool & Analyser
SEO marketers commonly use meta robots tags. In simple terms, the meta robots tag is a tag that lets search engines know what to index and what not to index on SERPs.
It is a piece of code in your webpage’s <head> section and allows you to decide which specific pages are shown on results pages and which pages you might want to keep hidden. Not to be confused with robots.txt files, which relate to an entire site, meta robots can also be used to tell search engines what links going out of your website to follow and what links not to follow.
It could look like this, for example:
- <meta name=”robots” content=”noindex, nofollow”/>
The first section of the tag is ‘name=’ here, you can identify the user agent, for example, a Googlebot. The second part of the tag is ‘content=,’ where you tell the bots what you want them to do, i.e., not index a page.
The above meta robots tag tells search engines not to index the page in the results pages and not to follow any backlinks.
However, there is another type of meta robots tag: x-robots-tag. This does the same as the meta robots tag but within the headers of an HTTP response, giving you more functionality and allowing you to block images or videos from SERPs if needed.
Why is the robots meta tag important for SEO?
If it has never crossed your mind to use a meta tag checker tool before, then you should do so as it is essential for good SEO. As mentioned, the meta robots tag is usually used to stop pages from showing in search results; this is great for ensuring the pages you want to be seen are.
When used incorrectly or in the wrong places, it renders your SEO efforts useless as the code prevents your pages from appearing on SERPs, regardless of the hard work you have put in. The bigger your website, the more likely you’ll need to use meta robots to improve the crawlability and indexing of your site, so it is digestible by the search engines.
Here are some examples of the content that you might want to stop search engines from indexing:
- Thank you pages (the page users see after placing an order, for example)
- Pages with little content
- Staging pages
- Internal search results pages
- Duplicate content
- Pages that detail contests or upcoming promotions
Common Types of Meta Robots
Two attributes make up robots meta tags: ‘name’ and ‘content.’ Both need values specified. If search engines have not been told how to sort the information under ‘content’ – they will automatically index the entire site.
There are different meta robots you can apply to a web page. Below is a guide to the most common meta tags and their meanings:
- index – This lets search engines index the whole page. For example, <meta name=”robots” content=”index” />
- noindex meta tag – The meta noindex tag stops search engines from showing the page. For example, <meta name=”robots” content=”noindex” />
- all – This lets search engines index the page and follow all links. For example, <meta name=”robots” content=”all” />
- noimageindex – Stops search engines from showing an image on their search results. For example, <meta name=”robots” content=”noimageindex”>
- follow – Allows search engines to follow the links on the page.
- nofollow – The nofollow meta tag asks search engines not to follow any links from a page. For example, <meta name=”robots” content=”nofollow”>
- noarchive – Stops Google from displaying a cached version of the page.
Can I combine the meta robots tags?
Search engines will automatically index your webpage without a meta robots tag and follow links. Here are the most used combinations of tags:
- index follow –Index the webpage on SERPs and follow the links. This is the default option.
- index nofollow – Index the webpage, don’t follow the links. For example, <meta name=”robots” content=”index, nofollow” />
- meta noindex follow – Do not index the webpage but follow the links. For example, <meta name=”robots” content=”noindex, follow” />
- noindex no follow meta tag – Do not index, do not follow the links. For example, <meta name=”robots” content=”noindex, nofollow”>
When implementing the meta robots tag, you need to ensure that the code agrees with the meta robots tag you are specifying. For example, SERPs cannot see the meta tags if the robots.txt blocks it. To check this, you must use a robots.txt checker or meta tag analyser.
How To Check You Have Added Your Meta Robots Tags Correctly
It can be tricky to determine whether or not you’ve implemented the meta robots tags correctly, especially if you have a bigger website. So it is important to use a meta tag checker tool like ours to ensure that you have used the tags correctly.
This is essential for making sure that your noindex no follow meta tag is used correctly and that search engines are displaying the pages you want them to. With our meta robot checker tool, you can do just that:
- Paste your website URL into the appropriate field,
- Specify your user agent, for example, Google Bot, if you want to use the checker as a Google robots testing tool,
- Define your protocol version and let our clever tool do the rest.
Our website robot checker crawls your site and will let you know the URLs of which pages are index and no index, and which no robots meta tags are conflicting with each other. Our tool makes it easy to see which pages are hidden from relevant SERPs.
How We Can Help
If you’ve noticed that your SEO efforts are not paying off how you would like and have noticed a decline in your SERPs success, then our meta robots checker tool can help.
You can use our tool as a bulk Google index checker. If the meta robots tag isn’t listed in the HTML code, then you will need to check for other tags that could be preventing your success.
Once you have identified any issues with your website coding, our specialist SEO team can help you with the results and help fix any problems stopping your website from soaring to the top of the SERPs.
Get in touch with our team today!
