What's the best crawler?
Gsitecrawler is fine but there was some problems about big websites.
A1 Sitemap generator great , but you cant pause crawler only stop and finish. Sometimes needs to pause cause u cant stay online for a week.
xml-sitemaps script ; I tried and nearly my apache will blowed up ! It doesn't know where to stop :cool:
