Robots.txt and Meta robots are two important factors in the SEO of any website. It has to use wisely else it may create issues in reaping required SEO benefits. This article shares a complete guide about, both, robots.txt and meta robots tag.
Before getting into details of robots file and tag, you need to understand how search engine bots work?
The above-mentioned process or behavior is called ‘Spidering’.
What is the usage of the Robots.txt and Robots Meta Tag?
This file and meta tag tells search engines what pages of the website shouldn’t get crawled and indexed.
How robots file and meta work?
Syntax:
Meta Robots tag
<meta name=”robots” content=”noindex” />
* It will instruct all search bots not to crawl a specific page in which this tag is put.
For specific search engine bot:
<meta name=”user-agent name” content=”noindex” />
Example:
<meta name=”googlebot” content=”noindex” />
* It will instruct Google organic search bot to not crawl a specific page.
Robots.txt File
User-agent: [user-agent name]
Disallow: [URL pattern not to be crawled]
Values of different parameters of Robots.txt file:
What you can pass as values in the URL?:
You can pass 2 different types of values in the URL field:
Check a complete list of pattern matching values with specific examples from Google here: https://support.google.com/webmasters/answer/6062596?hl=en
Example Robots.txt file:
User-agent: Googlebot-Image
Disallow: /profile/*
User-agent: msnbot
Crawl-delay: 200
Disallow: /*.xml$
User-agent: *
Disallow: /plugin/
Sitemap: https://digitalmarketersindia.com/sitemap.xml
Above robots.txt Decoded
The above mentioned robots.txt file means the following:
File format of Robots.txt
You must put the instruction in notepad without any rich formatting and save it as robots.txt. Make sure to give the exact name ‘robots.txt’ because the file name is case sensitive. The wrong name will ruin your efforts as the bots will ignore looking at that file.
Location of Robots.txt
Put it in the root folder. It should be accessible at the below location:
mainwebsiteurl.com/robots.txt
Use cases of Robots.txt:
We all are in the race of getting more pages indexed so you may wonder what can be the use case of robots, then, here are a few cases where you don’t want to get crawled or indexed in search engines:
Best practices:
Robots file is a very crucial and sensitive tool which has to be used with caution. If you are unsure of its usage, please don’t play with it because it is like playing with fire and can harm you badly, if you made mistakes. Below are a few important tips and guidelines to follow:
This is all you need to know about robots without getting much deep with technical details. You can learn more in-depth detail from Google here: https://support.google.com/webmasters/answer/6062596?hl=en
Please be noted that it is okay to not have robots file or tag so don’t hurt yourself with wrong details. Use it only when you need it and you are sure about its usage.