How to find performance bottlenecks in C++ code

Question

I have a server application written in C++ and deployed in Cent OS. I haven't wrote any part of its code but i need to optimize its performance. Its current performance is acceptable for few amount of users but when the number of users increase the server's performance decrease dramatically.

Are there any tools, techniques or best practices to find out the bottlenecks?

Solution 1

People typically use profilers to determine performance bottlenecks. Earlier SO questions asking for C++ profilers are here and here (depending on the operating system and compiler you use). For Linux, people typically use gprof, just because it comes with the system.

Solution 2

You'll start by building a performance test environment if you don't have one

• Production-grade hardware. If you do not have the budget for this, you may as well give up.
• Driver program(s) or hardware devices which throw production-like traffic at it at a high rate - as fast or faster than production. Depending on your protocol and use-case this may be easy or difficult. One technique is to sample some requests from production and replay them - but this may be give unrealistic results as it will give higher cache hit rates.
• Surrounding infrastructure as similar to production as you can reasonably get

Then reproduce the problem, as it exists in production. Once you've done that, then use a profiler etc, as others have suggested.

Solution 3

This works, without fail.

Solution 4

I like, MIke Dunlavey's answer above (so uptick his if you uptick mine)

I'd like to elaborate for someone in a hurry with two methods:

• A quick way for gcc users to sample in that gstack
• self inspection with SIGALRM combined with backtrace (driven by you own timer).

---

Just a few days ago I did something like this

# while true; do gstack $MYPID; sleep 2; done | logger $PARAMS

using PARAMS that go with my syslog routing rules so that my app logs were intermixed with stacks (not a perfect line-up with the events)

The results were on the nose, they pointed me to an area that I thought could be an issue at all but were my bottleneck due to misuse of reference in a tr1::bind

In the alarm method be careful what you do in the signal, don`t use anything that allocates memory (no cout/cerr/boost, and use just simple formats (i.e. "%08X" with printf)

View more solutions