Welcome to the IMTalk - Internet Marketing & SEO Forum.
+ Reply to Thread
Results 1 to 8 of 8
  1. #1
    davidrsdw is offline IM & SEO Weak Jaw davidrsdw is on a distinguished road
    Join Date
    Oct 2014
    Posts
    148
    Thanks Given
    0
    Thanked 9 Times in 7 Posts

    Want to know about robot.txt file

    I would like to know that is there any limitation for robot.txt file for a website. Is this such a serious issue if we cross the limit. Please suggest me in this.

  2. #2
    makemoneyonline10 is offline IM & SEO Mumbler makemoneyonline10 is on a distinguished road
    Join Date
    Oct 2011
    Posts
    270
    Thanks Given
    0
    Thanked 9 Times in 6 Posts
    Googlebot's 500KB Crawl Limit On Robots.txt Files and will only read the first 500kb.

  3. #3
    angelinakcg is offline IM & SEO Whisperer angelinakcg is on a distinguished road
    Join Date
    Nov 2017
    Location
    India
    Posts
    43
    Thanks Given
    0
    Thanked 3 Times in 2 Posts
    Quote Originally Posted by makemoneyonline10 View Post
    Googlebot's 500KB Crawl Limit On Robots.txt Files and will only read the first 500kb.
    This is very interesting, is that the that the case, How news magazines pages are getting crawled and showing on top of the results.

    I thought it's fake news but just a while ago I have gone through a authoritative post in a website that shared official tweet by John Mueller who is a Google employee.

    Google's John Mueller said:

    #102 of the things to keep in mind when working on a big website: If you have a giant robots.txt file, remember that Googlebot will only read the first 500kb. If your robots.txt is longer, it can result in a line being truncated in an unwanted way. The simple solution is to limit your robots.txt files to a reasonable size :-).

    So we should be careful in this regard and limit our robots file as per our requirement that could save our goals.

  4. #4
    wink0r's Avatar
    wink0r is offline Moderator wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold
    Join Date
    Oct 2010
    Location
    East Coast, USA
    Posts
    2,178
    Thanks Given
    919
    Thanked 648 Times in 489 Posts
    500kb in a .txt file is huge!

  5. #5
    arronmattwills is offline IM & SEO Mumbler arronmattwills is on a distinguished road
    Join Date
    Jan 2014
    Location
    India
    Posts
    343
    Thanks Given
    9
    Thanked 11 Times in 10 Posts
    Quote Originally Posted by wink0r View Post
    500kb in a .txt file is huge!
    Folks, as per latest reports you can upload up to 50MBPS XML sitemap file. So you expect more updates from Google team.

  6. #6
    wink0r's Avatar
    wink0r is offline Moderator wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold wink0r is a splendid one to behold
    Join Date
    Oct 2010
    Location
    East Coast, USA
    Posts
    2,178
    Thanks Given
    919
    Thanked 648 Times in 489 Posts
    An XML site map has a different purpose than a robots.txt file.

  7. #7
    sadianisar is offline Banned sadianisar is on a distinguished road
    Join Date
    Mar 2017
    Posts
    42
    Thanks Given
    0
    Thanked 0 Times in 0 Posts
    robots.txt must not cross the file size of 500kb.

  8. #8
    mahakhan is offline IM & SEO Whisperer mahakhan is on a distinguished road
    Join Date
    Aug 2018
    Location
    Pakistan
    Posts
    38
    Thanks Given
    0
    Thanked 0 Times in 0 Posts
    Robot.txt file can be use to tell search engine bots that which directories or pages the search engine should not crawl and index, in addition of this there would be several other benefits of using robots.tx for example we also mention our website sitemap url in it, so google and other search engines can crawl the content and links from sitemap too. According to google there is a size limit of 500 kilobytes (KB) for robots.txt file.
    pk domain Cheap .pk Domain Registration in Pakistan


 

Similar Threads

  1. Mega Robot Bomber
    By anthonymcauley in forum Free Stuff
    Replies: 14
    Last Post: 11-25-2012, 07:00 PM
  2. Anyone use Article Maketing Robot?
    By vf8170 in forum Internet Marketing General Talk
    Replies: 4
    Last Post: 03-16-2012, 02:48 AM
  3. How to make changes in robot.txt
    By franberries in forum On-Page SEO & Content Creation
    Replies: 5
    Last Post: 07-08-2011, 11:48 PM
  4. Wp twin + wp robot + cpa ninja course = big profit
    By SuccessKit90 in forum IMT Main Marketplace
    Replies: 2
    Last Post: 03-02-2011, 10:33 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts