For your pages to rank well in Google search results, they need to be available for search engines to crawl. However, there are some pages on your website that should not be indexed, as they are of no use to the reader – an example may be a Tag page, or a Privacy Policy page. These pages should have a noindex meta tag applied to them.
And when you link out to other websites from your own website, this is seen by Google and other search engines as a positive sign – you can think of it as approval or an upvote for their content. However, you can apply a nofollow tag to your links, which can be used to let Google know that you’re not actually endorsing their content – an example would be for a comment on your website which links back to the author’s website or socials.
This is the basic difference between noindex and nofollow, but let’s get into things in a little more depth.
Noindex vs Nofollow: Not the same
Noindex and nofollow are sometimes confused because of their name, but they’re quite different. Although, there are some similarities – they are both meta tags that can be inserted into the HTML source code of a web page. These meta tags are designed to communicate directly with search engine programs (bots) that crawl your website.
The noindex tag instructs search engines to crawl the page, but not to index it or display it in search results. A website is set to “index” by default. So if you want to remove or hide a page from search engines, you can change the term in the robot meta tags. This gives you more control over which pages appear in the SERPs.
The nofollow attribute tells search engines to ignore the website that you’re linking to. Although links play a crucial role in search engine optimization, only external links from authoritative and high-performing websites and internal links will help boost your website’s credibility and improve your website’s ranking.
Nofollow links do not pass PageRank, and therefore do not pass authority onto the site you’re linking to (however nofollow links can still supposedly benefit SEO).
How does indexing and tracking by search engines work?
Search engines use bots to explore your web pages by trying to understand what your content is about, how it is structured and what the most valuable pages on your website are. Before they can generate results from your web pages, they have to crawl all the links on the website, index these links in their database, and then display them on the search results pages.
Google’s bots follow all the links on your website to analyze your pages and then provide searchers with the most up-to-date data – of course, it doesn’t just do this for your website, but for the more than billion websites that exist globally. So when you enter a search query into Google, you are actually searching for information in its huge index of websites and files.
Of course, it makes sense to include relevant and important pages in the index to get more clicks and increase traffic. However, having unimportant pages and links indexed and tracked can definitely hurt your SEO efforts.
Which Pages Should You Noindex?
If important pages, blog posts and landing pages are not indexed, you may be losing an important source of traffic to your website. However, there are still times where you need to exclude some pages from being indexed by Google. It’s useful for:
- Preventing duplicate content – Some of your content may appear multiple times in different versions. In this scenario, it’s pretty redundant to let Google index all these pages.
- Protect confidential content – Some of your pages that contain important sources and products should not be found and downloaded publicly on Google search pages. It’s important to keep them away from Google’s eyes.
Based on this, there’s an array of pages that you’ll want to make sure Google and other search engines don’t index (they’ll still crawl them, but not index them). This is pages like:
- Thank you page – The page is usually displayed after visitors have entered their email address in the subscription field, registering to download your software or purchase your products.
- Author archives – Most blogs have one or two authors, the author page is almost the same as the blog homepage. The content will be duplicated if you display these two types of pages in the search results. To avoid this, you can noindex them.
- Custom post types – Custom post types, like products in WooCommerce that have thin or low-quality content, can be useless for visitors clicking through from Google’s search results pages, so it’s best to noindex them.
- Administration and login pages – These pages are not typically indexed because administrators or authorized users usually log in via direct URLs. However, login pages that serve a community, like Dropbox, may be indexed.
- Internal search results on your own website – This pagerefers to searches using the search bar on your website, and it can be indexed if the tag isn’t applied.
- Community profile pages – Did you know that 75% of Moz’s website has been deindexed? Britney Muller previously found that over 56% of indexed pages were community profile pages that contain inactive or spammy profiles with bad backlinks. After she decided to stop indexing URLs of community profiles with less than 200 points, the site had a positive effect on traffic. This example proves that it is important to keep irrelevant profiles out of the search engine results pages.
- Attachment pages – WordPress creates a separate attachment page when you upload an image, and Google indexes and displays these pages to visitors. Sometimes this may be useful, but sometimes they aren’t, so it’s worth investigating.
Which Links Should You Nofollow?
Just like noindexing your pages, there’s a multitude of reasons why you may want to nofollow your links too. Things like:
- Fight blog spam – Spammers often tend to leave links to their websites on your comment page. Nofollow links can’t completely prevent spam, but they can stop spammers gaining authority from your website.
- Increase website traffic – Remember that links are not only useful for SEO purposes. Nofollow links can increase your website traffic and promote business, brand awareness or service. This is because a good, high-quality link creates a gateway for visitors to access your content. If a visitor finds your content useful, they can share it in their own posts. It can therefore be said that nofollow links indirectly lead to follow links.
- Track your link juice properly – Link juice is a ranking factor for search engines that describes the power between pages. Since a strong page passes power to the links on its content, you need to make sure that you pass the power of the page to important links. An important element of search engine optimization is making sure your links flow well and make logical sense.
As mentioned, links in blog comments are often irrelevant and used for spam. If you monetize your blog by including sponsored content on your pages, you’ll also want to nofollow paid links, along with all questionable and untrustworthy links too.
How to Make Noindex Pages
Let’s look at three effective ways you can prevent search engines from indexing specific pages on your website.
1. Use “Noindex” Meta Tag
This tag is added to the head section of a page’s HTML to instruct search engines not to index your page.
<!DOCTYPE html>
<html>
<head>
<meta name="robots" content="noindex" />
<!-- Other head elements -->
</head>
<body>
<!-- Page content -->
</body>
</html>
Pro Tip: “robots” indicates this meta tag applies to all crawlers. For specific search engines, you can customize the tag:
<meta name="googlebot" content="noindex" />
This directs only Google’s bot to exclude the page from search results while it may still appear in other search engines like Bing or Ask.com.
You can use multiple robots meta tags simultaneously:
<meta name="googlebot" content="noindex">
<meta name="googlebot-news" content="nosnippet">
One of the best ways to do this if you’re a WordPress user is by using a plugin, like Rank Math or Yoast. They typically have options that allow you to noindex content within the click of a button, as seen below:
2. Use the X-Robots-Tag HTTP Header
The X-Robots-Tag can be implemented as part of the HTTP header response:
HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
X-Robots-Tag: noindex
This method is particularly useful for non-HTML files like PDFs or images.
3. Use robots.txt File
Edit your robots.txt file through an FTP client or the “File Manager” in your WordPress hosting cPanel:
User-agent: *
Disallow: /login/
- The first line with
User-agent: *targets all search engine bots - The second line defines which URL paths to block (everything after your domain name)
Example:
To prevent Google from indexing http://example.com/login/, simply add /login/ to the Disallow line.
However, there are some important considerations using this method:
- The robots.txt file is publicly accessible, revealing which pages you’re hiding
- Incorrect instructions can negatively impact your site’s visibility
- This method is not recommended for beginners
- Pages blocked via robots.txt may still be indexed if linked from other websites
How to Nofollow Your Links
There are several ways to add the nofollow attribute to links on your WordPress site, here’s the easiest ones:
1. Add Nofollow Links in Gutenberg Editor
For WordPress sites using the Gutenberg editor, you can add them directly within the dashboard:
- Select the text or image you want to link
- Click the link button in the toolbar
- Insert your link URL
- Click the Advanced tab.
- Toggle on the Set to Nofollow option
- Click Save
2. Use Rank Math to Manage Nofollow Links
Rank Math provides good options for link management:
- Install and activate the Rank Math SEO plugin
- Edit your page or post
- In the Rank Math sidebar, click on “Advanced” tab
- Under “Robots Meta,” select the “nofollow” option
- Update or publish your page
Bear in mind that this setting will apply nofollow to all links on the page, so you’d have to manually change links that you don’t want this applied to.
3. Add the Nofollow Attribute Manually
For more control, you can add the nofollow attribute directly in HTML:
html<a href="https://example.com" rel="nofollow">anchor text</a>
Simply edit your page in HTML/Text mode and add rel="nofollow" to any link you want to nofollow.
Conclusion
The noindex and nofollow tags are important for helping search engines know which pages you want them to index, and which links you want them to ignore. Be sure to use these tags properly if you want to ensure that Google and other search engines crawl your website the way you desire.
