How to get Google to crawl AJAX pages without using #! URLs?

seo search-engines google-search ajax

5,844

Solution 1

check this thread in Google forums, AJAX Crawling working with hash but not meta tag. There is an issue with google Fetch tool.

Solution 2

In the preceeding sections on allowing AJAX content to be crawled, Google states:

3.Handle pages without hash fragments

Some of your pages may not have hash fragments. For example, you might want your home page to be www.example.com, rather than www.example.com#!home. For this reason, we have a special provision for pages without hash fragments.

Note: Make sure you use this option only for pages that contain dynamic, Ajax-created content. For pages that have only static content, it would not give extra information to the crawler, but it would put extra load on your and Google's servers.

In order to make pages without hash fragments crawlable, you include a special meta tag in the head of the HTML of your page.

The meta tag takes the following form:

<meta name="fragment" content="!">

This indicates to the crawler that it should crawl the ugly version of this URL. As per the above agreement, the crawler will temporarily map the pretty URL to the corresponding ugly URL. In other words, if you place into the page www.example.com, the crawler will temporarily map this URL to www.example.com?_escaped_fragment_= and will request this from your server. Your server should then return the HTML snapshot corresponding to www.example.com. Please note that one important restriction applies to this meta tag: the only valid content is "!". In other words, the meta tag will always take the exact form: <meta name="fragment" content="!">, which indicates an empty hash fragment, but a page with AJAX content.

Therefore, you still need to enable the mapping of the URL to the 'ugly' version containing ?_escaped_fragment_= using the headless browser.

They follow-up with the warning:

Should the content for www.example.com?_escaped_fragment_= return a 404 code, no content will be indexed for www.example.com!

So you should test for that as well.

5,844

Yoga

Updated on September 18, 2022

Comments

Yoga over 1 year

In Google's advice on "Making AJAX Applications Crawlable", they advise making AJAX URLs that aren't using hash-bang fragments (#!) crawlable by adding <meta name="fragment" content="!"> to the page <head>.

Has anyone had success with this? I can't find Googlebot taking HTML snapshots of the pages when using 'Fetch as Googlebot' in Webmaster Tools.
- Su' over 12 years
  
  I think this might do better at SO.
- Stefan Koenen over 12 years
  
  interesting. Perhaps google does not take snapshots of fragments. Are the fragment urls indexed?