How do search engines deal with AngularJS applications?

Question

I see two issues with AngularJS application regarding search engines and SEO:

1) What happens with custom tags? Do search engines ignore the whole content within those tags? i.e. suppose I have

<custom>
  <h1>Hey, this title is important</h1>
</custom>

would <h1> be indexed despite being inside custom tags?

2) Is there a way to avoid search engines of indexing {{}} binds literally? i.e.

<h2>{{title}}</h2>

I know I could do something like

<h2 ng-bind="title"></h2>

but what if I want to actually let the crawler "see" the title? Is server-side rendering the only solution?

Solution 1

Update May 2014

Google crawlers now executes javascript - you can use the Google Webmaster Tools to better understand how your sites are rendered by Google.

Original answer
If you want to optimize your app for search engines there is unfortunately no way around serving a pre-rendered version to the crawler. You can read more about Google's recommendations for ajax and javascript-heavy sites here.

If this is an option I'd recommend reading this article about how to do SEO for Angular with server-side rendering.

I’m not sure what the crawler does when it encounters custom tags.

Solution 2

(2022) Use Server Side Rendering if possible, and generate URLs with Pushstate

Google can and will run JavaScript now so it is very possible to build a site using only JavaScript provided you create a sensible URL structure. However, pagespeed has become a progressively more important ranking factor and typically pages built clientside perform poorly on initial render.

Serverside rendering (SSR) can help by allowing your pages to be pre-generated on the server. Your html containst the div that will be used as the page root, but this is not an empty div, it contains the html that the JavaScript would have generated if it were allowed to run.

The client downloads the HTML and renders it giving a very fast initial load, then it executes the JavaScript replacing the content of the root div with generated content in a process known as hydration.

Many newer frameworks come with SSR built in, notably NextJS.

(2015) Use PushState and Precomposition

The current (2015) way to do this is using the JavaScript pushState method.

PushState changes the URL in the top browser bar without reloading the page. Say you have a page containing tabs. The tabs hide and show content, and the content is inserted dynamically, either using AJAX or by simply setting display:none and display:block to hide and show the correct tab content.

When the tabs are clicked, use pushState to update the URL in the address bar. When the page is rendered, use the value in the address bar to determine which tab to show. Angular routing will do this for you automatically.

Precomposition

There are two ways to hit a PushState Single Page App (SPA)

1. Via PushState, where the user clicks a PushState link and the content is AJAXed in.
2. By hitting the URL directly.

The initial hit on the site will involve hitting the URL directly. Subsequent hits will simply AJAX in content as the PushState updates the URL.

Crawlers harvest links from a page then add them to a queue for later processing. This means that for a crawler, every hit on the server is a direct hit, they don't navigate via Pushstate.

Precomposition bundles the initial payload into the first response from the server, possibly as a JSON object. This allows the Search Engine to render the page without executing the AJAX call.

There is some evidence to suggest that Google might not execute AJAX requests. More on this here:

https://web.archive.org/web/20160318211223/http://www.analog-ni.co/precomposing-a-spa-may-become-the-holy-grail-to-seo

Search Engines can read and execute JavaScript

Google has been able to parse JavaScript for some time now, it's why they originally developed Chrome, to act as a full featured headless browser for the Google spider. If a link has a valid href attribute, the new URL can be indexed. There's nothing more to do.

If clicking a link in addition triggers a pushState call, the site can be navigated by the user via PushState.

Search Engine Support for PushState URLs

PushState is currently supported by Google and Bing.

Google

Here's Matt Cutts responding to Paul Irish's question about PushState for SEO:

http://youtu.be/yiAF9VdvRPw

Here is Google announcing full JavaScript support for the spider:

http://googlewebmastercentral.blogspot.de/2014/05/understanding-web-pages-better.html

The upshot is that Google supports PushState and will index PushState URLs.

See also Google webmaster tools' fetch as Googlebot. You will see your JavaScript (including Angular) is executed.

Bing

Here is Bing's announcement of support for pretty PushState URLs dated March 2013:

http://blogs.bing.com/webmaster/2013/03/21/search-engine-optimization-best-practices-for-ajax-urls/

Don't use HashBangs #!

Hashbang URLs were an ugly stopgap requiring the developer to provide a pre-rendered version of the site at a special location. They still work, but you don't need to use them.

Hashbang URLs look like this:

domain.example/#!path/to/resource

This would be paired with a metatag like this:

<meta name="fragment" content="!">

Google will not index them in this form, but will instead pull a static version of the site from the escaped_fragments URL and index that.

Pushstate URLs look like any ordinary URL:

domain.example/path/to/resource

The difference is that Angular handles them for you by intercepting the change to document.location dealing with it in JavaScript.

If you want to use PushState URLs (and you probably do) take out all the old hash style URLs and metatags and simply enable HTML5 mode in your config block.

Testing your site

Google Webmaster tools now contains a tool which will allow you to fetch a URL as Google, and render JavaScript as Google renders it.

https://www.google.com/webmasters/tools/googlebot-fetch

Generating PushState URLs in Angular

To generate real URLs in Angular, rather than # prefixed ones, set HTML5 mode on your $locationProvider object.

$locationProvider.html5Mode(true);

Server Side

Since you are using real URLs, you will need to ensure the same template (plus some precomposed content) gets shipped by your server for all valid URLs. How you do this will vary depending on your server architecture.

Sitemap

Your app may use unusual forms of navigation, for example hover or scroll. To ensure Google is able to drive your app, I would probably suggest creating a sitemap, a simple list of all the URLs your app responds to. You can place this at the default location (/sitemap or /sitemap.xml), or tell Google about it using webmaster tools.

It's a good idea to have a sitemap anyway.

Browser support

Pushstate works in IE10. In older browsers, Angular will automatically fall back to hash style URLs

A demo page

The following content is rendered using a pushstate URL with precomposition:

http://html5.gingerhost.com/london

As can be verified, at this link, the content is indexed and is appearing in Google.

Serving 404 and 301 Header status codes

Because the search engine will always hit your server for every request, you can serve header status codes from your server and expect Google to see them.

Solution 3

Let's get definitive about AngularJS and SEO

Google, Yahoo, Bing, and other search engines crawl the web in traditional ways using traditional crawlers. They run robots that crawl the HTML on web pages, collecting information along the way. They keep interesting words and look for other links to other pages (these links, the amount of them and the number of them come into play with SEO).

So why don't search engines deal with javascript sites?

The answer has to do with the fact that the search engine robots work through headless browsers and they most often do not have a javascript rendering engine to render the javascript of a page. This works for most pages as most static pages don't care about JavaScript rendering their page, as their content is already available.

What can be done about it?

Luckily, crawlers of the larger sites have started to implement a mechanism that allows us to make our JavaScript sites crawlable, but it requires us to implement a change to our site.

If we change our hashPrefix to be #! instead of simply #, then modern search engines will change the request to use _escaped_fragment_ instead of #!. (With HTML5 mode, i.e. where we have links without the hash prefix, we can implement this same feature by looking at the User Agent header in our backend).

That is to say, instead of a request from a normal browser that looks like:

http://www.ng-newsletter.com/#!/signup/page

A search engine will search the page with:

http://www.ng-newsletter.com/?_escaped_fragment_=/signup/page

We can set the hash prefix of our Angular apps using a built-in method from ngRoute:

angular.module('myApp', [])
.config(['$location', function($location) {
  $location.hashPrefix('!');
}]);

And, if we're using html5Mode, we will need to implement this using the meta tag:

<meta name="fragment" content="!">

Reminder, we can set the html5Mode() with the $location service:

angular.module('myApp', [])
.config(['$location', 
function($location) {
  $location.html5Mode(true);
}]);

Handling the search engine

We have a lot of opportunities to determine how we'll deal with actually delivering content to search engines as static HTML. We can host a backend ourselves, we can use a service to host a back-end for us, we can use a proxy to deliver the content, etc. Let's look at a few options:

Self-hosted

We can write a service to handle dealing with crawling our own site using a headless browser, like phantomjs or zombiejs, taking a snapshot of the page with rendered data and storing it as HTML. Whenever we see the query string ?_escaped_fragment_ in a search request, we can deliver the static HTML snapshot we took of the page instead of the pre-rendered page through only JS. This requires us to have a backend that delivers our pages with conditional logic in the middle. We can use something like prerender.io's backend as a starting point to run this ourselves. Of course, we still need to handle the proxying and the snippet handling, but it's a good start.

With a paid service

The easiest and the fastest way to get content into search engine is to use a service Brombone, seo.js, seo4ajax, and prerender.io are good examples of these that will host the above content rendering for you. This is a good option for the times when we don't want to deal with running a server/proxy. Also, it's usually super quick.

For more information about Angular and SEO, we wrote an extensive tutorial on it at http://www.ng-newsletter.com/posts/serious-angular-seo.html and we detailed it even more in our book ng-book: The Complete Book on AngularJS. Check it out at ng-book.com.

Solution 4

You should really check out the tutorial on building an SEO-friendly AngularJS site on the year of moo blog. He walks you through all the steps outlined on Angular's documentation. http://www.yearofmoo.com/2012/11/angularjs-and-seo.html

Using this technique, the search engine sees the expanded HTML instead of the custom tags.

Solution 5

This has drastically changed.

http://searchengineland.com/bing-offers-recommendations-for-seo-friendly-ajax-suggests-html5-pushstate-152946

If you use: $locationProvider.html5Mode(true); you are set.

No more rendering pages.

View more solutions

Comments

[Codemonkey]

all of these "solutions" just make me want to steer away from technologies like AngularJS, at least until google et all have more intelligent crawlers.

[Roy M J]

@Codemonkey : Yes one would wonder why of all AngularJS which is a product of Google has not come up with a built-in solution for this.. Wierd actually..

[superluminary]

Actually, Misko wrote Angular before he worked for Google. Google now sponsors the project, but they aren't the originators.

[linojon]

Perhaps someone here can/should update the Wikipedia article on SPA which states "SPAs are commonly not used in a context where search engine indexing is either a requirement, or desirable." en.wikipedia.org/wiki/Single-page_application [# Search engine optimization] Theres a huge paragraph about an (obscure) java-based framework called IsNat but no suggestion that SEO has been addressed by the likes of Angularjs.

[Toolkit]

Just an update from April 2016 - NONE of my AngularJS sites were indexed. I know others are having luck by seems like google bot doesn't understand sites with angular-ui-router

[ccnokes]

@Toolkit I can confirm this as well. Interestingly, a site I made in React and react-router is fully indexed, no problem. I really wish I knew what the differentiating factor was between Angular sites and my React one.

[Toolkit]

@ccnokes i kindof having second thoughts, because Google started to index my pages. But Chrome hangs when i try to open cache. Quite screwed up..

[Steven Ventimiglia]

@Roy M J - Why does no one see the intent? PageSpeed, Angular, etc. are all enemies of natural, organic listings on the SERPs. Purposely. When you have a huge business model based on Pay-Per-Clicks... how better to force people to pay for their listings than creating an entire toolbox that will give them no option, but to do so? Instead of building quality web sites filled with valuable content, this industry is now overflowing with cheats and solutions that don't achieve or solve squat diddly.

[luisfarzati]

Thanks for the article! I was aware of this. If there's no other option, then I guess I'll have to do it this way...

[Christoph]

@Brad Green even so the question was closed (for whatever reasons) you might be the position to answer it. I guess I must be missing something: stackoverflow.com/questions/16224385/…

[mackmack]

custom tags shouldn't matter because they get rendered into whatever the directive transcludes it as (try looking at the actual html for any rendered page). so far only oldIEs have a problem with custom tags.

[check_ca]

SEO4Ajax is also a good example of paid service (free during the beta). Unfortunately, it looks like I'm not allowed to edit this response to add it in the list.

[user276648]

The code for prerender.io is on github (github.com/collectiveip/prerender) so anyone can run it on its own servers.

[Ketan]

This answer is quite old now. Please see my below answer for the latest in Angular SEO.

[Les Hazlewood]

This is now outdated as well. See @user3330270's answer below.

[Stan]

This should be top answer now. We are in 2014 and answer by @joakimbl is no longer optimal.

[Prerender.io]

This is incorrect. That article (from March 2013) says nothing about Bing executing javascript. Bing simply gives a recommendation to use pushstate instead of their previous recommendation to use #!. From the article: "Bing tells me that while they still support the #! version of crawlable AJAX originally launched by Google, they’re finding it’s not implemented correctly much of the time, and they strongly recommend pushState instead." You still have to render the static HTML and serve it for _escaped_fragment_ URLs. Bing/Google will not execute the javascript/AJAX calls.

[Prerender.io]

This is not outdated. @user3330270's answer is incorrect. The article they link to simply says to use pushstate instead of the #!. You still have to render static pages for the crawlers because they do not execute javascript.

[superluminary]

This is no longer current. You should now use pushState instead. There is no need to serve a separate static version of the site.

[tschiela]

even with the google update, ng-view will not be rendered correctly, as i can see in Google Webmaster tools

[jvv]

I have to look into this - thanks for the explanation. One thing I keep wondering is, does google now run the javascript before indexing the page?

[superluminary]

According to Google they do execute the JavaScript and index the result. They do this to prevent cloaking, and to let them spider single page apps.

[Jamie.Good]

But will Google crawl the actual angular links?

[Agamemnus]

.pushState by itself does not actually seem to do anything. There must be something else that triggers it. (I am not using Angular) See my flower painting site as an example: tinyurl.com/mau55bn .. the text doesn't change at all from the default description meta tag. Any ideas? Edit: perhaps the description tag trumps generated content?

[superluminary]

I'm not sure I understand. Pushstate changes the address in the address bar. After that it's up to you.

[superluminary]

@Jamie, According to Google they will, provided you have told Angular to generate real links using pushState.

[tester]

Yeah just because they execute javascript doesn't mean that your page will be indexed properly. The safest way is to detect the google bot useragent, use a headless browser like phantomjs, obtain page.content and return static html.

[atconway]

"PushState changes the URL in the top browser bar without reloading the page... When the tabs are clicked, use pushState to update the url in the address bar. When the page is rendered, use the value in the address bar to determine which tab to show. Angular routing will do this for you automatically." Lightbulb!

[Stan]

You still need _escaped_fragment_ and render pure html pages. This solves nothing mate.

[Sray]

@superluminary, could you, please, explain the subject a little bit deeper? Especially the 'Server side' section. I'm using angularjs + angularjs-route + locationProvider.html5Mode + api + dynamic navigation (not the static one like on html5.gingerhost.com. URLs are displayed well, however the content does not seem to be indexed. Do I have to serve somehow a static content while accessing a page by a direct url? I'm actually confused by this message: >>you will need to ensure the same template gets shipped by your server for all valid URLs. Could you please explain it? Thanks in advance.

[superluminary]

@sray, yes I can. Visit your home page, now navigate to a sub page by clicking a link that's handled by Angular. PushState causes the URL to change, and Angular notices the change and responds appropriately. Your server is not hit, except possibly by an AJAX call. Angular has handled it all without troubling the browser. - OK, now copy that URL into another tab and press enter, what do you see?

[superluminary]

@sray - If every URL on your site is serving the same template, the browser will be able to grab the template, and Angular will be able to take it from there, by inspecting the URL and rendering the correct content. If hitting that URL directly on the server returns a 404 or a 500 then you have a problem, direct links will not work, bookmarks will not work and you will not be indexed. Do you see now?

[Sray]

@superluminary, thank you for your reply. Are you saying that it is enough to configure the server to server the same static page and no need to feed pre-rendered content for search bots like escaped_content? If so, I'm confused, I configured server, it generates a dynamic pages and dynamic urls, but they're not getting fixed. Please check my ticket with details: stackoverflow.com/questions/26200740/… I will be glad if you could help me. Thanks in advance.

[Pikachu]

Google webdev tools (fetch as google), still sees empty content of my site, dynamic content are being indexed or not?

[Pikachu]

Still google robot can't see dynamic content of my site, only empty page.

[superluminary]

@calmbird - no, if Google webmaster tools is seeing a blank page you have an issue with your site and will not be indexed. It could be configuration, it could be JavaScript, it could even be robots.txt.

[Pikachu]

no :) fetch as google doesn't interprate <meta tags>. I can see blank page in fetch as google, but my page is still being indexed.

[superluminary]

@calmbird - you need to fix the blank page issue. It's hard to say what is causing it without more information.

[Pikachu]

@superluminary - as I said, fetch as google doesn't interprate metatag that says webpage has dynamic content, thats why fetch as google see empty side. To check if everything is ok just put ?_escaped_fragment_, in fetch as google url.

[superluminary]

@calmbird - no, forget escaped fragments, take that stuff out and the metatag that supports it. Reread my post, use pushstate with real urls, not escaped fragments and hashbangs. There must be no # characters in your url, except for in page links.

[Yagiz Ozturk]

@superluminary dynamic titles and description tags are not indexed. But content does. How will we do the titles for SEO?

[Stephen Watkins]

I realize this question is specific to SEO, but keep in mind that other crawlers (Facebook, Twitter, etc.) aren't yet able to evaluate JavaScript. Sharing pages on social media sites, for example, would still be a problem without a server-side rendering strategy.

[Toolkit]

My SPA content is loaded via AJAX. I can see in search results only {{something.property}} as if JS is disabled. And also all the pages I submitted via sitemap the same {{event.description}} instead of actual value.

[Toolkit]

search site:mysite.com shows {{staff}}, not the content loaded via AngularJS. As if Google crawler never heard of JavaScript. What can I do?

[superluminary]

Hi @toolkit - it sounds like Google is not correctly executing your script. What does it look like in Webmaster Tools?

[Toolkit]

@superluminary in GWT it displays correctly when i do Fetch and Render. Should I preload pages myself and detect Googlebot and other bots?

[superluminary]

No, don't do any of that. Can you share the url of your site?

[Toolkit]

well better not. But why can't I preload?

[Toolkit]

again me - $locationProvider.html5Mode(true) doesn't work, google index is showing {{....}}

[Jeff Whelpley]

Just a note that you should not abandon server rendering if you care about SEO. Read my comment here: reddit.com/r/angularjs/comments/2wcwt9/…

[AncientSwordRage]

Googling 'Angularjs seo' gets me this: prerender.io Any use to this answer?

[user3339411]

How do i implement this on my SPA hosted in Github? it includes only one page with no links, only anchors to jump through different places in a long page. Can some refer me to github docs, if any?

[superluminary]

@user3339411 - On Github static pages you will need to create an actual page for every URL your site will respond to. Painful if you have more than one page.

[user3339411]

I don't see how that would help. My AngularJS pages would still be uncrawlable.

[superluminary]

@user3339411 - I'm sorry, can you say more, why would your Angular pages be uncrawlable? Google will hit URLs directly, you just have to make sure there's content there for it to get. Pushstate URLs are essentially real URLs, and the crawler will treat them as such.

[user3339411]

I understand from you that Google will crawl JS on my page. Since my entire content is served using Angular on one long page (reading JSON data), My content will be crawled. I don't even need to split to pages. Is this correct?

[superluminary]

@user3339411 - You should have one URL for every page your site will respond to. If your site only needs to respond to one URL with one set of content, you don't need any routing at all. This is fine for a simple site. If however your site brings in different data (via JSON) for different URLs it would make sense to use routing. Because Github static pages is file based, you would need an actual html file backing each URL in this instance. There is no rule that a website has to be file based though, and if you use an alternative platform you can serve the same template for multiple URLs.

[Lycha]

@auser Do you still recommend this approach? The newer top voted comment seems to discourage this approach.

[check_ca]

Please, can someone give an example of AngularJS site correctly indexed without implementing the Google crawling scheme specification?

[RPDeshaies]

FYI googlewebmastercentral.blogspot.ca/2015/10/…

[Allan Bogh]

@check_ca: Without escaped fragment -- with escaped fragment. Example search on MixCloud. Example search on illegalcartoon.

[check_ca]

@AllanBogh The indexed content of MixCloud is rendered on server-side.

[Toolkit]

Google doesn't parse Angular websites

[Thor]

@Toolkit what do you mean?

[Toolkit]

Googlebot is NOT able to parse Angular websites

[twigg]

@Toolkit you're talking absolute hoop, my full Angular site has been index by google with dynamic meta data without any issues

[Toolkit]

@twigg you have faulty logic, you mean if one (your) Angular website was indexed, all were. Well i have a surprise for you. NONE of mine were indexed. May be because i use angular ui router or who knows why. Not even the main pages without any ajax data

[Kevin]

Link is down in the listed blog.ajaxsnapshots.com

[phil294]

@Toolkit If not even your static html pages are indexed, this has nothing to do with googles ability of crawling JS files. If you are saying that google can't crawl anyting properly.. well I think you are wrong

[Ravinder Payal]

I am also following this solution and thought like that prior to this but i want to ask that will search engines read contents of custom tags.

[Ravinder Payal]

angular is for easing the work or just making the operations costlier and time taking

[Harkirat Saluja]

facing same issue with reactjs spa too, no indexing.

[Kaspar L. Palgi]

Sitelinks show in Google search results page JSON: google.co.uk/… - see the screenshot: i.imgur.com/fr1PAgm.png Is there a way to fix that?

[tmsppc]

This is a great example about why we should never say things like "definitive guide" in CS :). Major search engines now execute Javascript, so this answer need to be rewritten or deleted altogether.

[adriendenat]

@seb this is still needed for let's say open graph tags that need to be in the page when robots are crawling it. For example Facebook or Twitter cards need it. But this answer should be updated to focus on HTML5 pushstate instead of hashbang that is deprecated now.

[tmsppc]

@Grsmto you are right! Then I guess it should get rewritten because it says that major search engines don't execute JS, which is not true anymore.

[vijay]

@RavinderPayal can you check this solution with seoreviewtools.com/html-headings-checker

[belyid]

I've managed to setup my Angular SPA without hasbangs but I'm unable to index both title and meta description. I've explained the problem here Can you please help?

[Erik Reppen]

This is outdated info. Google no longer recommends. Consider this approach deprecated at best.

[Evgenia Karunus]

Some pages of mine can be fetch by google, and some can't. It's due to a particular library (Draft JS). So no, google can't render all js like modern browsers.

[Fery Kaszoni]

This is no longer true as of Oct 2017, this income tax calculator income-tax.co.uk is built with pure AngularJs (even the titls are like <title>Tax Calculator for £{{earningsSliders.yearly | number : 0 }} salary</title> that renders like "tax calculator for £30000 salary), and Google indexes them ranks them on first page for hundreds of keywords. Just build your websites for humans, mae them awesome, and Google will take care of the rest ;)

[Cerbrus]

This link is dead: analog-ni.co/…

[Jonathan Muszkat]

orchardmile.com is served without pre-rendered content to Google. However we are using a pre-rendered service (prerender.io) in order to work with other bots like Slack, Skype, Facebook, etc.

[Nono]

@JonathanMuszkat, Then how you are managing Google SEO, if you are not serving pre-rendered content to Google crawlers? How you have got site indexing in Google search. I am very excited to hear from you :) and any small lead would help me to understand.

[Jonathan Muszkat]

@NeerajSingh Google is SPA capable. So for Google you don't need to do anything else for SEO apart for a simple code to change the metadata based on the routing.

[Nono]

@JonathanMuszkat, Thanks John for your comment :). I was wondering, if there are some another ways. Because Google is not fully ready to understand complex SPA - especially which has slow page speed and lots of dynamic data rendering. As it's use headless browsers which is the not real scenario. But yes! most the medium level SPA could be crawled by Google itself without any issues.