Hello folks, you are here because you want to take your Blogger Blog into next step. I know there are lots of things to do to get higher ranking in search engines. If you have a good knowledge about SEO you know the importance of robots.txt. Those who are new to blogging may have so many doubts regarding this, don't worry will give an exact guide on how to add SEO-friendly custom robots.txt file in blogger. Unlike other platforms blogger is not SEO-friendly compare to other platforms,so we have to do some work to overcome this problem.
Must Read: what is SEO and how it works?
Note: crawling and indexing are the two words that we are using throughout the post, so before starting I will explain what those words means.
Crawling means the search engine bots crawl or fetch your website or blog after crawling is completed they will examine how relevant it is,if it worth they will index it to show in search results.
Your blog is your hard work, studying everything before adding it to your blog is a right quality that every blogger should have. So let's start,
Current robots.txt of blogger file is not a good one, it will allow the search engine crawler to crawl and index all pages. This will lead to creating duplicate content errors because for example when the Google bots crawl all your pages including label, blog archive etc they also crawl all post which may already indexed, this can result in duplicate content errors.
This errors during indexing will reduce your page rank in search engines like Google, Bing, Yahoo etc. Decrease in page rank means a decrease in chance of getting exposure to your blog and reduction in the amount of organic traffic to your blog. So in order to avoid these problems, a quality robots.txt file is a necessary one for a blogger.
It is a text file which seen in 'search preference ' section of your dashboard. They instruct the crawler on how to crawl and index your blog pages. This will allow you to restrict pages like label, blog archive etc. You can even restrict crawler to index any pages from your blog. This will avoid indexing of unwanted pages and also reduce the chance of creating duplicate content errors while crawling.
Must read: How to SEO Optimize Your Blogger Blog Titles For Higher Search Results
Do you know the bad thing about blogger default blogger robots.txt? it allows the crawler to index only recent 25 pages. So, in order to avoid this limitation we have to add an SEO-friendly file.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&Start-index=1&max-results=500
Above code is the SEO-friendly robots.txt that you can add to your blog, replace your blog URL with given red colored URL. This is for blogs which have blog posts up to 500, if you have total posts up to 1000 use below Sitemap instead.
Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000
It is difficult to understand for those who don't have knowledge about HTML code. Don't worry I will explain what these codes means. The code is mainly three section,
1. User-agent: Mediapartners-Google
This is four Google Adsense users, this instruction will help to give ads which are suitable for your blog. If you are not using Adsense you can remove these code.
2. User-agent: *
This part has more important when it comes to SEO because you can disallow search engine crawler to index any page, for example, blogger default robot.txt does not allow the crawler to index 'labels' which is done by adding Disallow:/search. Which means a link with word search after your domain name will not be indexed. Allow:/ is for the homepage, this code will allow the search engine crawler to crawl your homepage. If you want to prevent the crawler from indexing particular pages you can do by adding below code.
Disallow: /yyyy/mm/post-url.html
Where /yyyy/mm represents the published year and month and /post-url.html is referred to particular post that you don't want to be indexed.
3. Sitemap: Given sitemap will inform the crawler to crawl every new page to get indexed.
Hope you got a little knowledge about robot.txt, now you are ready to go for next step, which is Adding Robots.txt to blogger.
Here I will give you complete step by step guide for adding robots.txt to your Blogger in few steps, follow me.
1. From your browser go to blogger.com and sign in to your account and select your blog if you have more than one blog.
2. From blogger dashboard clicks on settings > search preference > crawlers and indexing > Custom robots.txt > edit and make ti to yes.
3. Now paste above SEO-friendly robot.txt file to that box and save it. That's it, you are done!
Hope you enjoyed the post and you should try to add custom robots.txt to blogger blog. Adding wrong codes may completely affect your blog badly which may lead to a decrease in search engine rankings. So put this codes carefully after understanding it properly. If it helped you, please share it with your friends and don't forget to subscribe for my latest guides.
Must Read: what is SEO and how it works?
Note: crawling and indexing are the two words that we are using throughout the post, so before starting I will explain what those words means.
Crawling means the search engine bots crawl or fetch your website or blog after crawling is completed they will examine how relevant it is,if it worth they will index it to show in search results.
Your blog is your hard work, studying everything before adding it to your blog is a right quality that every blogger should have. So let's start,
Need for robots.txt?
Current robots.txt of blogger file is not a good one, it will allow the search engine crawler to crawl and index all pages. This will lead to creating duplicate content errors because for example when the Google bots crawl all your pages including label, blog archive etc they also crawl all post which may already indexed, this can result in duplicate content errors.
This errors during indexing will reduce your page rank in search engines like Google, Bing, Yahoo etc. Decrease in page rank means a decrease in chance of getting exposure to your blog and reduction in the amount of organic traffic to your blog. So in order to avoid these problems, a quality robots.txt file is a necessary one for a blogger.
What is robots.txt?
It is a text file which seen in 'search preference ' section of your dashboard. They instruct the crawler on how to crawl and index your blog pages. This will allow you to restrict pages like label, blog archive etc. You can even restrict crawler to index any pages from your blog. This will avoid indexing of unwanted pages and also reduce the chance of creating duplicate content errors while crawling.
Must read: How to SEO Optimize Your Blogger Blog Titles For Higher Search Results
How to Add SEO-friendly custom robots.txt in blogger?
Do you know the bad thing about blogger default blogger robots.txt? it allows the crawler to index only recent 25 pages. So, in order to avoid this limitation we have to add an SEO-friendly file.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&Start-index=1&max-results=500
Above code is the SEO-friendly robots.txt that you can add to your blog, replace your blog URL with given red colored URL. This is for blogs which have blog posts up to 500, if you have total posts up to 1000 use below Sitemap instead.
Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://myblog.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000
It is difficult to understand for those who don't have knowledge about HTML code. Don't worry I will explain what these codes means. The code is mainly three section,
1. User-agent: Mediapartners-Google
This is four Google Adsense users, this instruction will help to give ads which are suitable for your blog. If you are not using Adsense you can remove these code.
2. User-agent: *
This part has more important when it comes to SEO because you can disallow search engine crawler to index any page, for example, blogger default robot.txt does not allow the crawler to index 'labels' which is done by adding Disallow:/search. Which means a link with word search after your domain name will not be indexed. Allow:/ is for the homepage, this code will allow the search engine crawler to crawl your homepage. If you want to prevent the crawler from indexing particular pages you can do by adding below code.
Disallow: /yyyy/mm/post-url.html
Where /yyyy/mm represents the published year and month and /post-url.html is referred to particular post that you don't want to be indexed.
3. Sitemap: Given sitemap will inform the crawler to crawl every new page to get indexed.
Hope you got a little knowledge about robot.txt, now you are ready to go for next step, which is Adding Robots.txt to blogger.
Here I will give you complete step by step guide for adding robots.txt to your Blogger in few steps, follow me.
1. From your browser go to blogger.com and sign in to your account and select your blog if you have more than one blog.
2. From blogger dashboard clicks on settings > search preference > crawlers and indexing > Custom robots.txt > edit and make ti to yes.
3. Now paste above SEO-friendly robot.txt file to that box and save it. That's it, you are done!
Hope you enjoyed the post and you should try to add custom robots.txt to blogger blog. Adding wrong codes may completely affect your blog badly which may lead to a decrease in search engine rankings. So put this codes carefully after understanding it properly. If it helped you, please share it with your friends and don't forget to subscribe for my latest guides.
Comments
Post a Comment