Thread: Google, Sitemaps and Unaccessible Pages

Hi there fellow IM'ers!

On one of my sites I currently have a sitemap with 20 or so pages - that are all in the sitemap that is fed to Google.

As 15 of these pages are all user only pages, Google recognises these but doesn't index due to not being able to process these with gbot:

"When we tested a sample of the URLs from your Sitemap, we found that some URLs were not accessible to Googlebot due to network timeouts. If this problem persists, please check the network availability of your DNS and web servers. All accessible URLs will still be submitted."

This subsequently brings up a warning in Google Webmaster Tools.

Keeping the links in might perhaps demonstrate to Google that the website is a large site - on the other hand, as Google can't view these, it might negatively impact rankings. I can see arguments for both.

I'd be very interested to hear your views and experiences on this, whether these should be kept in the sitemap or removed completely.

Cheers,

Lee

Are the pages denied via robots.txt?

It sounds like the pages are in a password protected directory, if so just deny the directory via robots.txt. Of course, Google will then complain about not being able to access the denied pages.

But if they're also complaining about doing this, then this wouldn't really solve the issue would it? Would it not be simpler to remove the pages from the sitemap?

Probably do both. Removing them from the sitemap does not necessarily mean that Google will not try to access the pages if they can find a link. A sitemap is a helper file, but bots will still find most pages if there are links on or off site.

un-accessible links are very bad for your website, you need to make sure they are marked as "no-index" or it will hurt your rankings.
remove the links from your sitemap, and mark all links to those pages as "nofollow"

If you are using wordpress or another CMS this should be pretty easy.

Cheers guys - your feedback's much appreciated

For a site on the smaller side as you've described here, I wouldn't worry too much about it. However, if the error bothers you, just edit the robot.txt and sitemap inclusion settings as mentioned above. Not sure which SEO plugin you're using, but Yoast's Wordpress SEO can edit both settings on the post edit area.