I understand that robots txt are used to inform search engine spiders how to index content.
I want to know if is it really important to have a robot txt?
If yes, I need help creating one because I am a layman in this issue.
For example,
My site: mysite(dot)com
links to pages of my site: mysite (dot) com/a.html
mysite (dot) com/b.html
mysite(dot)com/c.html
mysite(dot)com/a/abcd.html
mysite(dot)com/sitemap.xml
I want to create robots txt to crawl these link. Can anyone help me with creatng it.
The meta description of robots txt should be <meta name="robots" content="noindex">
or
<meta name="robots" content="noindex,nofollow">
or any other. Please specify.
Does Yahoo, Bing and Google have their own user-agent?