What is Robots.txt? Have you ever heard of it, if not, it is a matter Good news for you today, because today I am going to provide you some information about Robots.txt।
If you have a blog or a website, then you must have felt that sometimes all the information we don't want is also public in Internet, Do you know why this happens. Why many of our good contents would not have been indexed after many days. If you want to know about the secret behind all these things then what is this article Robots.txt, you have to read all these things till the end of article Will be aware of।
With the con to tell Search Engines, Files and Folders have to show all the public in the website and not what Robots meta tag is allowed to do. but all Search Engines Does not know how to read Meta tag so many Robots Meta tag go unnoticed without reading. The best way to do this is Robots.txt Instructing File so that Search Engines can be easily protected about their Website or Blog's Files and Folders।
So today I thought why not give you complete information about what Robots.txt is, so that you do not have any problem in understanding it further. Then what should the delay start and know that Robots.txt What is it and what is it।
What is robots.txt
Robots.txt There is a text file that you keep in your site so that you can tell Search Robots which pages they have to visit or crawl in your site And which ones not।
The thing to note here is that if we do not implement this file in the right place, then Search Engines will feel that you probably did not include robot.txt file Which may not even contain the pages of your site।
So this little file has a lot of importance if it is not done properly then it can also reduce the ranking of your website. So it is very important to have a good knowledge about it.
how does it work?
Any search engines or Web Spiders If your website or blog comes for the first time, they first crawl your robot.txt file because it contains all the information about your website What happens is not to crawl and what to do. And they index your guided pages, so that your indexed pages are displayed in search engine results।
Robots.txt files can prove to be very firey for you if :
How robots.txt file is made
If you have not yet created robots.txt file in your website or blog, then you should make it very soon, because it will prove to be very difficult for you later is. To make it you have to follow some instructions :
What is Syntax of Robots.txt
In Robots.txt we offer some syntax which we are very important to know about।
• User-Agent: Those robots follow all these rules and in which they are applicable (e.g. “Googlebot, ” etc.)
• Disallow: To destroy this means to block the pages with bots that you do not want, someone else can access it. (Here is the need to write disallow before files)
• Noindex: Search engine from its estates will not index your pages that you do not want to be indexed।
• all User-agent/disallow group To separate, a blank line should be destroyed, But note here that the two groups of the middle should not be any blank line (the bitches of the User-agent line and the last Disallow should not be gap ।
• hash symbol (#) can be used to deliver real comments within a robots.txt file, where everything that will be the symbol of # before will be ignore. They are mainly used for whole lines or end of lines।
• Directories and filenames case-sensitive Occur: “private”, “Private”, and “PRIVATE” These are completely different to all search engines.
Let's understand this with the help of the example. Here I have written about it।
• Joe robot here “Googlebot” No disallowed statement has been written in it so that it is free to go anywhere
• All site has been closed here where “ msnbot” is destroyed.
• There is no permission to see all robots (other than Googlebot) /tmp/ directory or directories or files called /logs, E.g., tmp.htm, through comments that have been printed below,
/logs or logs.php.
User-agent: googlebot
Disallow:
User-agent: msnbot
Disallow: /
# Block all robots from tmp and logs directions
User-agent: *
Disallow: /tmp/
Disallow: /logs # for directions and files called logs
Advantages of using Robots.txt
If seen by the way, many of the robots.txt have been allowed to do this, but I have told or given some very important Fa Ye De here, about whom all Must have information।
- With the help of robots.txt “canonicalization” problems can be kept away or multiple “canonical” URLs can also be kept. This problem is also called forgotten “duplicate content” problem .
- You can also help Google Bots Pages index to do.
What if we don't destroy robots.txt file then?
If we do not destroy any robots.txt file then there is no restriction on seach engines where to crawl and where not they can index everything Whatever they found in your website।
This is all for many websites but if we talk about some good practice then we should destroy robots.txt file because this search engines index your pages They are easy to do, and they do not need to go all the pages again।
What did you learn today
I hope i have you guys Robots.txt what Gave full information about Hai and I hope you guys have understood about Robots.txt. I have passed on to all of you readers, share this information in your neighborhood, relatives, your friends, Which will be our middle awareness and it will benefit everyone a lot. I need the support of you so that I can convey more information to you।
I have always tried this to always help my readers or readers from all sides, If you guys have any kind of doubt then you can ask me free. I will try to solve those Doubts।
How did you like this article Robots.txt? Let us know by writing a comment so that we too will get to learn something from your thoughts and improve something।