Sitemap is "pending" in Webmaster Tools after submitting an an unusually large number of links with it

Question

I recently submitted to Google 17,000 links to businesses in my database. In the past I've only ever submitted 2 or 3 hundred.

The sitemap has been listed in Webmaster Tools for the past five days as: Submitted but indexed still remains at pending. Is this a normal thing?

Solution 1

Yes this is normal for a site of your size. Google throttles the indexing of pages to prevent the manipulation of their search results. As your site becomes more seasoned (domain age, external links, spider visits. etc) more of your pages will show in their index. If you are properly using the sitemap and internally linking your pages together via site wide, categorical sub navigation then the Google crawlers and spiders will find the pages just fine. Just throttle the indexing to prevent abuse.

Some popular tricks to get attention to these pages faster is to get Google bots to visit sooner. Using Google Analytics on every page and or Google adsense.

Solution 2

As the comment from #closetnoc indicated, the sitemap is really a reference point only. The Google bot will crawl your site, so the best thing you can do to speed up the process is to ensure that there are no blocks on indexing in your robot.txt or site settings, and ensure your site's navigation links provide a path to all relevant pages, and you have a few internal links to other internal pages from each page to keep the bot going. Use Search Console to test these things.

To index a lot of pages will take time, since the bot follows paths and isn't going to get every path first time, but your site doesn't have to be ages old before a lot of it will be linked. While the sitemap can help, for deep pages, Google tends to prefer those pages which are hyperlinked from multiple sources (ie. hence the importance of internal linking).

Sidenote: If your site is created using a CMS, as Google starts indexing your site, you might want to check that you are not getting multiple links for the same page, particularly if you are using many different categories. https://joomla.stackexchange.com/questions/10046/how-can-i-get-rid-of-category-pages-in-joomla/14557#14557

Comments

[closetnoc]

Yes. It can be. Google does not guarantee the use of the sitemap. Mostly, because search engines are rather old-school, it will prefer to index your site by crawling it. It will use the sitemap to know if it can properly crawl your site. Other than that, a sitemap is largely useless short of extremely large sites or sites with a paywall or login. The advice to submit a sitemap is largely a bogus one. In your case, it has read your sitemap, however, the process of comparing what pages Google knows about and exist in your sitemap has not fully happened yet. It is doing that right now.