Solaris 10- high kernel utilization (%sys)

Question

Our production server shows a high %sys utilization.
This is a Solaris 10 x86 box with Oracle 10g R2 database

Configuration:
CPU - Intel Xeon 4 CPU (8 socket each)
Memory - 128GB (80GB SGA, 10GB PGA, 30GB for System)

This server shows high CPU utilization in 'sys'.
How can i determine the culprit?

Solution 1

On Solaris there's only one answer to that - Dtrace.

Solution 2

Well, Dtrace is a good option if you know how to use it. The DtraceToolKit can help tremendously if you don't know how to use it.

Other programs that can help you find the culprit:

• prstat -> (try prstat -amL)
• apptrace
• iostat
• vmstat
• mpstat

And, even though it's an older book now, Adrian Cockroft's Sun Performance Tuning: Sparc & Solaris is still a must have for anyone that works with Solaris, and for pure insight reasons, I would recommend it to users of any flavor of Unix/BSD/Linux.