Robots txt

[sjeeta]

I understand that robots txt are used to inform search engine spiders how to index content.

I want to know if is it really important to have a robot txt?


If yes, I need help creating one because I am a layman in this issue.

For example,

My site: mysite(dot)com

links to pages of my site: mysite (dot) com/a.html
mysite (dot) com/b.html
mysite(dot)com/c.html
mysite(dot)com/a/abcd.html
mysite(dot)com/sitemap.xml

I want to create robots txt to crawl these link. Can anyone help me with creatng it.

The meta description of robots txt should be <meta name="robots" content="noindex">

or

<meta name="robots" content="noindex,nofollow">

or any other. Please specify.


Does Yahoo, Bing and Google have their own user-agent?

[zibidic]

A robots.txt file can make a huge impact on your blogs traffic and search engine rank. This is an SEO optimized WordPress robots.txt file. Keep in mind that if you mess up the robots.txt file by blocking too much, you could lose all of your rank.

seo robots.txt

User-agent: *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /z/j/
Disallow: /z/c/
Disallow: /stats/
Disallow: /dh_
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /contact/
Disallow: /tag/
Disallow: /wp-content/b
Disallow: /wp-content/p
Disallow: /wp-content/themes/askapache/4
Disallow: /wp-content/themes/askapache/c
Disallow: /wp-content/themes/askapache/d
Disallow: /wp-content/themes/askapache/f
Disallow: /wp-content/themes/askapache/h
Disallow: /wp-content/themes/askapache/in
Disallow: /wp-content/themes/askapache/p
Disallow: /wp-content/themes/askapache/s
Disallow: /trackback/
Disallow: /*?*
Disallow: */trackback/

User-agent: Googlebot
# disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.cgi$
Disallow: /*.wmv$
Disallow: /*.png$
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: */trackback*
Disallow: /*?*
Disallow: /z/
Disallow: /wp-*
Allow: /wp-content/uploads/

# allow google image bot to search all images
User-agent: Googlebot-Image
Allow: /*

# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow: /*?*
Allow: /z/
Allow: /about/
Allow: /contact/
Allow: /wp-content/
Allow: /tag/
Allow: /manual/*
Allow: /docs/*
Allow: /*.php$
Allow: /*.js$
Allow: /*.inc$
Allow: /*.css$
Allow: /*.gz$
Allow: /*.cgi$
Allow: /*.wmv$
Allow: /*.cgi$
Allow: /*.xhtml$
Allow: /*.php*
Allow: /*.gif$
Allow: /*.jpg$
Allow: /*.png$

# disallow archiving site
User-agent: ia_archiver
Disallow: /

# disable duggmirror
User-agent: duggmirror
Disallow: /

http: // googleblog.blogspot.com/2007/02/robots-exclusion-protocol.html

[BigDim]

robots.txt file is svery important for a website.The biggets hosting providers like hostgator create robots.txt for your websites and you don't need to do it yourself.

[sjeeta]

A robots.txt file can make a huge impact on your blogs traffic and search engine rank. This is an SEO optimized WordPress robots.txt file. Keep in mind that if you mess up the robots.txt file by blocking too much, you could lose all of your rank.

seo robots.txt



http: // googleblog.blogspot.com/2007/02/robots-exclusion-protocol.html

I do not want the sample you sent me. I need help in creating mine as per the example in my thread description.

[wink0r]

The Web Robots Pages is the authority on robots text files.

[zibidic]

i think you wants like this:
(copied entire codes to inside your "robots.txt" file) [valid for all kind robots]

User-agent: *
Disallow: /*
Allow: /a.html
Allow: /b.html
Allow: /c.html
Allow: /abcd.html

Sitemap: mysite(dot)com/sitemap.xml

Google Says

Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it's current for your site so that you don't accidentally block the Googlebot crawler.

header.php meta seo trick

Place this in your wordpress themes header.php file, if the page is a single, page, or if its the home page then the robots will index and follow links on it. Otherwise search engines will not index the pages but will still follow the links.

<?php if(is_single() || is_page() || is_home()) { ?>
<meta name="googlebot" content="index,noarchive,follow,noodp" />
<meta name="robots" content="all,index,follow" />
<meta name="msnbot" content="all,index,follow" />
<?php } else { ?>
<meta name="googlebot" content="noindex,noarchive,follow,noodp" />
<meta name="robots" content="noindex,follow" />
<meta name="msnbot" content="noindex,follow" />
<?php }?>

UNTIL THEN, SEE YOU NEXT TIME

[sjeeta]

I am not using wordpress. anyway I am thanked to wonk0r and zibidic