Asense and SEO Friendly Custom Robots.txt for Blogger site with Explanation
What is Custom robots.txt?
Custom robots.txt is a simple text file with simple coding with which you can direct your crawler how it will crawl or index your blog or website for search result. That means with it you can restrict any page/pages of your website from web crawler to be indexed in the search engines like blog labels page, your demo page or or any other pages.
Remember that search engines detect your robots.txt first rather than your blog or website indexing. That means when a search engine crawler visit your site, it at first look for robots.txt file, You have the opportunities to direct search crawler which pages you want to be indexed or which are not simply using the robots.txt file.
Remember that search engines detect your robots.txt first rather than your blog or website indexing. That means when a search engine crawler visit your site, it at first look for robots.txt file, You have the opportunities to direct search crawler which pages you want to be indexed or which are not simply using the robots.txt file.
What is it's Function?
If you have a blog hosted by blogger, it has a custom robots.txt which is something look like this. See the robots.txt file. The explanation will be given hereafter.
If you have a blog hosted by blogger, it has a custom robots.txt which is something look like this. See the robots.txt file. The explanation will be given hereafter.
# Blogger Sitemap generated By www.amiodhara.blogspot.com on 2014.01.25 User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: http://yourblogslink.blogspot.com/feeds/posts/default?orderby=UPDATED
It is almost same for every blogger site. And it is adsense and SEO friendly. If it meets your demand please do not change it. Now I will tell you how to set a Custom robots.txt for your blogger sites. Follow these steps accordingly:
Setting up a robots.txt in Blogger site.
- Go to your blogger dashboard with your Gmail and password.
- Go to Settings > Search Preferences > Crawling and Indexing.
- Click 'Custom robots.txt' > click 'yes'.
- Paste your custom robots.txt. (An example is given below)
- Click 'Save changes' to save the custom robots.txt
How to add a proper custom robots.txt for your blogger blog site
Here is an example of a proper Custom robots.txt for blogger site.
# robots.txt of mysite.blogspot.com from http://www.amiodhara.blogspot.com User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Disallow: /p/* Disallow: /view/* Disallow: /?m=1 Disallow: /?m=0 Disallow: /*?m=1 Disallow: /*?m=0 Allow: / Sitemap: http://yourblogslink.blogspot.com/feeds/posts/default?orderby=UPDATED Sitemap: http://yourblogslink.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
[N.B- Please keep blank for 4 and 14 no. lines for the accuracy]
Now I will explain above robots.txt file accordingly so that you may apply these in your blogger site accurately.
- # robots.txt of mysite.blogspot.com from http://www.amiodhara.blogspot.com
This is a comment line. Here you can write anything that you like. See here I have written about my blog, similarly you can write with your own. Crawler will not give any error result for this that means crawler doesn't take it into considerations.
- User-agent: Mediapartners-GoogleDisallow:
These lines are only for adsense crawler that means when adsense crawler roams about your website it searches these lines for publishing accurate ads for your contents. For these lines adsense crawler visits all the page as per adsense guidelines and recommends accurate ads for those page. Either you are using Google Adsense on your blog or not simply leave it as it is.
- User-agent: *
This line invite all the crawler except ones for which particular rule is set, here the adsense crawler. You can define more to invite the crawler in specialized way.
- Disallow: /search
This line is a substitute of the above line and tells the crawler not to crawl and index the pages with /search That means the above line will say crawler not to crawl or index the URL which has /search at the end of the domain name. Such as ... http://amiodhara.blogspot.com/search/label/Android%20apps
This line is most effective for making your blog SEO friendly as it says the crawler to stopindexing label, search query, archive pages as they are not unique pages.
- Disallow: /p/*
The line above says the crawler not to index any pages. We know that generally pages do not have any content. So almost every blogger wants that their pages should not be indexed by the crawler. But you want the pages of your blog should be indexed, then just delete the above line and do nothing, your page will be indexed.
- Disallow: /view/*
It says the crawler not to crawl or index the URL which has %%%. Because most often these are not unique content. But if you are using the %%% in the URL of your unique content. Then just delete the above line and do nothing.
- Disallow: /?m=1
Disallow: /?m=0
Disallow: /*?m=1
Disallow: /*?m=0
These lines are used for mobile redirect pages. These lines say to crawler to stop indexing the mobile redirect pages. If you do not use them, you might see your blogs links in mobile search results have ?m=1 or ?m=0 with them. They are necessary to avoid the duplicate pages problem. If you are not facing these problems or if you don't need them you can remove them.
- Allow: /
After adding all the instructions for restricting the robots to crawl or index your website, this tells the robots that the other links in the blogs are free to crawl and index.
- Sitemap: http://yourblogslink.blogspot.com/feeds/posts/default?orderby=UPDATED
Sitemap: http://yourblogslink.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500The above lines are nothing but the sitemaps of your site. If you keep only above first line, then it will help
[N.B- Just replace the http://yourblogslink.blogspot.com with your blog's name.]
No comments