Welcome to the IMTalk - Internet Marketing & SEO Forum.
  • Login:
+ Reply to Thread
Results 1 to 12 of 12
  1. #1
    davidrsdw is offline IM & SEO Weak Jaw davidrsdw is on a distinguished road
    Join Date
    Oct 2014
    Location
    India
    Posts
    156
    Thanks Given
    0
    Thanked 10 Times in 8 Posts

    Want to know about robot.txt file

    I would like to know that is there any limitation for robot.txt file for a website. Is this such a serious issue if we cross the limit. Please suggest me in this.

  2. #2
    makemoneyonline10 is offline IM & SEO Mumbler makemoneyonline10 is on a distinguished road
    Join Date
    Oct 2011
    Posts
    270
    Thanks Given
    0
    Thanked 10 Times in 6 Posts
    Googlebot's 500KB Crawl Limit On Robots.txt Files and will only read the first 500kb.

  3. #3
    angelinakcg is offline IM & SEO Whisperer angelinakcg is on a distinguished road
    Join Date
    Nov 2017
    Location
    India
    Posts
    48
    Thanks Given
    0
    Thanked 3 Times in 2 Posts
    Quote Originally Posted by makemoneyonline10 View Post
    Googlebot's 500KB Crawl Limit On Robots.txt Files and will only read the first 500kb.
    This is very interesting, is that the that the case, How news magazines pages are getting crawled and showing on top of the results.

    I thought it's fake news but just a while ago I have gone through a authoritative post in a website that shared official tweet by John Mueller who is a Google employee.

    Google's John Mueller said:

    #102 of the things to keep in mind when working on a big website: If you have a giant robots.txt file, remember that Googlebot will only read the first 500kb. If your robots.txt is longer, it can result in a line being truncated in an unwanted way. The simple solution is to limit your robots.txt files to a reasonable size :-).

    So we should be careful in this regard and limit our robots file as per our requirement that could save our goals.

  4. #4
    wink0r's Avatar
    wink0r is offline Moderator wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold
    Join Date
    Oct 2010
    Location
    East Coast, USA
    Posts
    2,183
    Thanks Given
    941
    Thanked 651 Times in 491 Posts
    500kb in a .txt file is huge!

  5. #5
    arronmattwills is offline IM & SEO Mumbler arronmattwills is on a distinguished road
    Join Date
    Jan 2014
    Location
    India
    Posts
    365
    Thanks Given
    9
    Thanked 12 Times in 11 Posts
    Quote Originally Posted by wink0r View Post
    500kb in a .txt file is huge!
    Folks, as per latest reports you can upload up to 50MBPS XML sitemap file. So you expect more updates from Google team.

  6. #6
    wink0r's Avatar
    wink0r is offline Moderator wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold
    Join Date
    Oct 2010
    Location
    East Coast, USA
    Posts
    2,183
    Thanks Given
    941
    Thanked 651 Times in 491 Posts
    An XML site map has a different purpose than a robots.txt file.

  7. #7
    sadianisar is offline Banned sadianisar is on a distinguished road
    Join Date
    Mar 2017
    Posts
    42
    Thanks Given
    0
    Thanked 0 Times in 0 Posts
    robots.txt must not cross the file size of 500kb.

  8. #8
    mahakhan is offline IM & SEO Whisperer mahakhan is on a distinguished road
    Join Date
    Aug 2018
    Location
    Pakistan
    Posts
    38
    Thanks Given
    0
    Thanked 0 Times in 0 Posts
    Robot.txt file can be use to tell search engine bots that which directories or pages the search engine should not crawl and index, in addition of this there would be several other benefits of using robots.tx for example we also mention our website sitemap url in it, so google and other search engines can crawl the content and links from sitemap too. According to google there is a size limit of 500 kilobytes (KB) for robots.txt file.

  9. #9
    rupaturkar is offline IM & SEO Whisperer rupaturkar is on a distinguished road
    Join Date
    Jan 2019
    Location
    mumbai
    Posts
    31
    Thanks Given
    0
    Thanked 1 Time in 1 Post
    Type in the URL of a page on your site in the text box at the bottom of the page.

  10. #10
    lynsofia's Avatar
    lynsofia is offline IM & SEO Whisperer lynsofia is on a distinguished road
    Join Date
    Aug 2018
    Posts
    42
    Thanks Given
    2
    Thanked 1 Time in 1 Post
    Robots.txt is a text file which use webmaster creat to instruct web robots how to crawl pages on their website. It is the part of the robots exclusion protocol (REP) that regulate how robots crawl the web access and index page content and serve that content to end users.

    "Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit."

  11. #11
    Mahwish is offline IM & SEO Quiet One Mahwish is on a distinguished road
    Join Date
    Jun 2019
    Posts
    17
    Thanks Given
    0
    Thanked 0 Times in 0 Posts
    robots.txt is a file that contains the instructions by website owner for web crawlers of search engine to not visit, crawl and index the web page.

  12. #12
    coscasino is offline IM & SEO Quiet One coscasino is on a distinguished road
    Join Date
    Jun 2019
    Posts
    6
    Thanks Given
    0
    Thanked 2 Times in 1 Post
    robots.txt is very important to hide some directory of your site and for security purpose


 

Similar Threads

  1. Mega Robot Bomber
    By anthonymcauley in forum Free Stuff
    Replies: 14
    Last Post: 11-25-2012, 07:00 PM
  2. Anyone use Article Maketing Robot?
    By vf8170 in forum Internet Marketing General Talk
    Replies: 4
    Last Post: 03-16-2012, 02:48 AM
  3. How to make changes in robot.txt
    By franberries in forum On-Page SEO & Content Creation
    Replies: 5
    Last Post: 07-09-2011, 12:48 AM
  4. Wp twin + wp robot + cpa ninja course = big profit
    By SuccessKit90 in forum IMT Main Marketplace
    Replies: 2
    Last Post: 03-02-2011, 10:33 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts