What is robots.txt?

[Garyu73]

I dont know about robots.txt. What is robots.txt?
What only to websites that have a domain only? please explain. I dont understaind about robots.txt

[Hogward]

Robots.txt is an file which must be placed immediately after the root domain.. It is used to prevent the specific pages of the website from being crawled by the Bots.

[redchillimedia]

Robots.txt is a file to which spiders read first. It contain file exclusion property. Mainly we use it to disallow those pages to which we want to hide from search engine crawlers.

[Sherly]

robots.txt is simply a plain-text file that a Web publisher should put in the root directory of their website. The text files includes instructions that tell indexing spiders, or "robots," what content and directories on that website they may, or may not, look at.

[yamunacbs]

robot.txt file is plain text work like a web browser. It browse the all web page and index page of website. It crawl your website and then send it to indexer. It include three part spider,crawler and Indexer.

[Alan Smith]

You can block search engine spider through robots.txt file. In this file you can disallow that pages whichever you don’t want to crawl.
For getting more detail you can refer this link The Web Robots Pages

[webhost]

Robots.txt is a file which must be placed immediately after the root domain. It is used to prevent the specific pages of the website from being crawled by the Bots and it also indicate the follow links.

[semwinner]

simply saying, it is a text help search engine know your site.

[royjonesjr]

Thanks for help and for thread. Now it's very clear what is robots.txt.. dont knew that before. thanks.

[paddy]

Its a file through which you can stop SE to index & crawl your website certain pages or files. You can read more on it on google central blog.

[Ammy Tisdale]

Robots.txt is a file to which spiders read first. It contain file exclusion property. Mainly we use it to disallow those pages to which we want to hide from search engine crawlers.

Agree with you. There is also an option that you can allow to any search engine or any to not to crawl.

[idreesfarooq]

Basically a file used to allow or deny access to Bots, Spiders or any SE to your site.

[isoftx]

Robots.txt is a simple text file that tells the search which page should be crawled or not.

[googlesiterank]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

[enni]

The robot.txt is a regular text file that through its name has special meaning to the majority of honourable robot on the web.The text file include instruction that tell indexing spider or robots, what content and directories on the website they may or may not.

[cirerare]

robot.txt was warning crawler should to index url or not