So you heard about someone stressing the importance of the robots.txt file, or noticed in your website’s logs that the robots.txt file is causing an error, or somehow it is on the extremely prime of the top visited pages, or, you study some write-up about the death of the robots.txt file and about how you should not bother with it ever once more. Or possibly you never heard of the robots.txt file but are intrigued by all that speak about spiders, robots and crawlers. In this article, I will hopefully make some sense out of all of the above.

There are many people out there who vehemently insist on the uselessness of the robots.txt file, proclaiming it obsolete, a point of the previous, plain dead. I disagree. The robots.txt file is in all probability not in the top rated ten methods to promote your get-rich-rapid affiliate web page in 24 hours or significantly less, but nonetheless plays a key part in the lengthy run.

Initial of all, the robots.txt file is nonetheless a pretty critical aspect in advertising and maintaining a website, and I will show you why. Second, the robots.txt file is one of the straightforward means by which you can guard your privacy and/or intellectual property. I will show you how.

Let’s attempt to figure out some of the lingo.

What is this robots.txt file?

The robots.txt file is just a extremely plain text file (or an ASCII file, as some like to say), with a incredibly very simple set of instructions that we give to a net robot, so the robot knows which pages we want scanned (or crawled, or spidered, or indexed – all terms refer to the very same factor in this context) and which pages we would like to retain out of search engines.

What is a www robot?

A robot is a pc system that automatically reads web pages and goes via just about every link that it finds. The goal of robots is to collect information. Some of the most famous robots described in this report operate for the search engines, indexing all the information out there on the internet.


The initially robot was developed by MIT and launched in 1993. It was named the Planet Wide Net Wander and its initial purpose was of a purely scientific nature, its mission was to measure the development of the internet. The index generated from the experiment’s benefits proved to be an amazing tool and correctly became the first search engine. Most of the stuff we take into consideration nowadays to be indispensable on-line tools was born as a side effect of some scientific experiment.

What is a search engine?

Generically, a search engine is a system that searches through a database. In the well-liked sense, as referred to the net, a search engine is regarded to be a system that has a user search kind, which can search by means of a repository of net pages gathered by a robot.

What are spiders and crawlers?

Spiders and crawlers are robots, only the names sound cooler in the press and inside metro-geek circles.

What are the most well-liked robots? Is there a list?

Some of the most well recognized robots are Google’s Googlebot, MSN’s MSNBot, Ask Jeeves’s Teoma, Yahoo!’s Slurp (funny). One particular of the most well-liked areas to search for active robot info is the list maintained at http://www.robots.org.

Why do I will need this robots.txt file anyway?

A wonderful reason to use a robots.txt file is truly the reality that quite a few search engines, such as Google, post recommendations for the public to make use of this tool. Why is it such a huge deal that Google teaches people about the robots.txt? Nicely, simply because today, search engines are not a playground for scientists and geeks anymore, but huge corporate enterprises. Google is a single of the most secretive search engines out there. Pretty tiny is identified to the public about how it operates, how it indexes, how it searches, how it creates its rankings, and so forth. In fact, if you do a careful search in specialized forums, or wherever else these problems are discussed, no one definitely agrees on regardless of whether Google puts much more emphasis on this or that element to produce its rankings. And when people don’t agree on things as precise as a ranking algorithm, it signifies two points: that Google continually changes its solutions, and that it does not make it really clear or pretty public. There’s only 1 factor that I think to be crystal clear. If they advocate that you use a robots.txt (“Make use of the robots.txt file on your internet server” – Google Technical Guidelines), then do it. It could not help your ranking, but it will unquestionably not hurt you.

There are other motives to use the robots.txt file. If you use your error logs to tweak and retain your website free of charge of errors, you will notice that most errors refer to someone or one thing not getting the robots.txt file. All you have to do is create a simple blank page (use Notepad in Windows, or the most simple text editor in Linux or on a Mac), name it robots.txt and upload it to the root of your server (that is where your house web page is).

On a distinct note, presently, all search engines look for the robots.txt file as quickly as their robots arrive on your internet site. There are unconfirmed rumors that some robots may even ‘get annoyed’ and leave, if they don’t locate it. Not positive how accurate that is, but hey, why not be on the protected side?

Once again, even if you don’t intend to block anything or just don’t want to bother with this stuff at all, obtaining a blank robots.txt is still a excellent concept, as it can essentially act as an invitation into your site.

Don’t I want my web-site indexed? Why quit video analytics ?

Some robots are well developed, professionally operated, bring about no harm and present worthwhile service to mankind (don’t we all like to “google”). Some robots are written by amateurs (remember, a robot is just a system). Poorly written robots can lead to network overload, safety issues, and so on. The bottom line here is that robots are devised and operated by humans and are prone to the human error factor. Consequently, robots are not inherently bad, nor inherently brilliant, and need cautious attention. This is yet another case exactly where the robots.txt file comes in handy – robot handle.

Now, I am certain your principal purpose in life, as a webmaster or site owner is to get on the 1st page of Google. Then, why in the planet would you want to block robots?

Here are some scenarios:

1. Unfinished web site

You are nevertheless building your web-site, or portions of it, and don’t want unfinished pages to appear in search engines. It is mentioned that some search engines even penalize websites with pages that have been “below construction” for a long time.

two. Safety

Constantly block your cgi-bin directory from robots. In most instances, cgi-bin consists of applications, configuration files for these application (that could truly have sensitive details), and so forth. Even if you don’t presently use any CGI scripts or programs, block it anyway, superior secure than sorry.

3. Privacy

You might have some directories on your web-site exactly where you keep stuff that you don’t want the whole Galaxy to see, such as photographs of a buddy who forgot to put clothes on, etc.

4. Doorway pages

In addition to illicit attempts to boost rankings by blasting doorways all over the net, doorway pages actually do have a pretty morally sound usage. They are equivalent pages, but every 1 is optimized for a particular search engine. In this case, you ought to make confident that individual robots do not have access to all of them. This is very critical, in order to stay clear of getting penalized for spamming a search engine with a series of really related pages.