What is the point of staging?

Question

I thought I'd worked this out, but after reading Continuous Delivery (excellent book) I'm a little confused. They talk about having servers for:

• development
• various forms of automated tests
• user acceptance testing (UAT) - ie sitting down with the client and demonstrating it to them, and letting them do exploratory testing. The in-house testers could also use this setup for exploratory testing.
• staging
• production.

I'd always thought of staging as providing the UAT function, but they seem to have staging as a separate level. So in that scheme, what function would the staging servers provide?

Solution 1

Staging would be putting the full product systems in place, but not actually using them yet. When they go into use would be "production". You should put everything in place as it will be used, test, then flip the switch.

The UAT commonly uses "testing" environment that are significantly different from the hardware/software/configuration that will be used in production.

For example, where I work we have the customers test everything in a VM environment running on our servers. When their system goes live it will be running on their hardware, at their facility, probably integrating with their existing systems; it will have absolutely nothing to do with our servers or test environment (except that the code and some configuration has been copied from there...)

Solution 2

I work on the release management team at a very large internet company. We use essentially the process you've outlined above, and we've chosen that process on purpose. In our methodology, staging serves as a branching mechanism for a final level of testing in production.

Obviously you want to do all testing before you go to production, but in a large, complex environment with lots of users, that's a very difficult goal to reach. In particular, it is virtually impossible to adequately load test software in QA. Functional testing is a lot easier to automate than load testing. When you have many thousands of users hitting your servers, things fail in weird and hard to predict ways.

So here's what we do:

• Development
  • includes continuous integration and automated testing
• release testing
  • my group analyzes the release itself
  • reviewing install logs
  • testing rollback
• QA
  • user acceptance testing

That's the point at which we branch between staging and production. We use a train model for releases, with a new train starting every few weeks. Even numbered trains go to the staging servers (which are in production). Odd numbered trains do not.

In between the even trains, the developers have the ability to push individual changes to the staging servers (after those changes have been tested by QA of course). This allows them to validate that their software performs as expected in a real production environment. This is generally reserved for the components which are deemed higher risk, we don't push every little piece to staging.

Then, everyone understands that when the next even train starts, it will wipe out what's on the staging servers and set them back to the train baseline. Developers either ensure that their changes got on the train, or decide they aren't ready for general use yet, in which case those changes just get erased on the staging servers.

To sum up, the short answer (at least for us) is that it is impossible to completely test complex systems in QA. Staging provides a safe way to do limited production testing.

On a related note, here's my slides from a presentation I just gave on how our release process works.

Solution 3

The simplest explanation for staging is testing your deployment process and test using the real data source. Some systems combine staging with their testing environments, but for large scale systems the deployment process could be very complex or they may be extra testing steps required once you connect to the live/production data source. In these case a staging environment lets you test your deployment process and check for last minute bugs using live data, and then once things have been verified as working you can quickly switch the stage environment into the production environment.

An example of this would be Windows Azure, which requires 5-25 minutes to deploy a new version, but you can deploy to a staging environment, perform tests and then instantly swap the production and staging environments.

Comments

[MrGigu]

I can't say I agree with that methodology. UAT should always be done on as close specs to the live system as possible (i.e. staging). I can't count the number of times we've done UAT and had everyone complain of speed issues, to which we have to explain a thousand times that "The live system will be faster". And then when the live system ISN'T faster, because of a bug in the code, or an SQL query, you have to eat your own words.

[jftuga]

More testing usually occurs on the staging server as well, not just at UAT -- just before going into production.

[Philip]

@jftuga, see last sentence of first paragraph...

[iconoclast]

@Chris S: If I understand you correctly, then there is no such thing as "a staging server" just a production server that's out of the pool, not currently serving to the end users. That makes sense, and is a methodology I follow, but I don't call those servers "staging servers", just production servers (that are not in the pool). Since everywhere I've worked that uses staging servers has them as separate servers, I don't think your description of a staging server is the standard use of that term. A good idea, but not what is normally meant by "staging server" (in my experience, anyway).

[iconoclast]

It seems to me to vary by organization. I've seen it used as a UAT server, as a server for developers to put the app in an environment supposedly identical to production, and probably other things. (I personally think the only good strategy is to use a real production server for staging.) As different organizations develop their own culture, I think they also develop their own lexicon, and so words that ought to have a standard meaning across the industry often unfortunately do not.

[iconoclast]

@ChrisS: btw, what are the regional differences of the term "bouncing"? I've always understood it to mean taking down either hardware or software and bringing it back up.

[iconoclast]

@ChrisS: to continue re: differences across companies: the author of this question was confused about the purpose of staging servers even after reading a book on the subject, so it's no wonder that people across organizations are confused (and don't know they're confused) and end up using the word very differently. Also, different organizations have different problems to solve, so their needs are different, so even if everyone had a common understanding of the canonical purpose of a staging server they would often end up using it differently in practice.