Why is the system CPU time (% sy) high?

Question

I am running a script that loads big files. I ran the same script in a single core OpenSuSe server and quad core PC. As expected in my PC it is much more faster than in the server. But, the script slows down the server and makes it impossible to do anything else.

My script is

for 100 iterations
Load saved data (about 10 mb)

time myscript (in PC)

real    0m52.564s
user    0m51.768s
sys    0m0.524s

time myscript (in server)

real    32m32.810s
user    4m37.677s
sys    12m51.524s

I wonder why "sys" is so high when i run the code in server. I used top command to check the memory and cpu usage. It seems there is still free memory, so swapping is not the reason. % sy is so high, its probably the reason for the speed of server but I dont know what is causing % sy so high. The process that is using highest percent of CPU (99%) is "myscript". %wa is zero in the screenshot but sometimes it gets very high (50 %).

When the script is running, load average is greater than 1 but have never seen to be as high as 2.

I also checked my disc:

strt:~ # hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   16480 MB in  2.00 seconds = 8247.94 MB/sec
 Timing buffered disk reads:   20 MB in  3.44 seconds =   5.81 MB/sec

john@strt:~> df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       245G  102G  131G  44% /
udev            4.0G  152K  4.0G   1% /dev
tmpfs           4.0G   76K  4.0G   1% /dev/shm

I have checked these things but I am still not sure what is the real problem in my server and how to fix it. Can anyone identify a probable reason for the slowness? What could be the solution? Or is there anything else I should check?

Thanks!

Solution

You're getting a high sys activity because the load of the data you're doing takes system calls that happen in kernel. To resolve your slowness problems without upgrading the server might be possible. You can modify scheduling priority. See the man pages for nice and renice. See here and especially:

Niceness values range from -20 (the highest priority, lowest niceness) and 19 (the lowest priority, highest niceness).

$ ps -lp 941
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
4 S 0 941 1 0 70 -10 - 1713 poll_s ? 00:00:00 sshd

$ nice -n 19 ./test.sh 
My niceness value is 19

$ renice -n 10 -p 941
941 (process ID) old priority -10, new priority 10

Comments

[Chris Stratton]

What "problem"? If you ask the system to do a lot of work, it's going to be quite busy until it's done. With a typical script, almost all of the "work" will take place in child processes (or kernel space tasks on their behalf, such as I/O), not in the shell process interpreting the script.

[Abhishek]

When the script is running, other users in the server get slow response. I want to run the script faster and also not disturb other users. If adding some resources is solution, i could do that. But, I dont know what might help.

[Chris Stratton]

If you are asking what sort of system to buy / upgrades to make for better performance, that's probably a superuser question. If you want to try to make the task itself more efficient (cobbled together solutions may have a lot of room for improvement), you are going to have to be a lot more specific about what it does and how - actual code, or at least a close functional description.

[Abhishek]

I may have chosen the wrong forum. I am sorry for that. But, before thinking about what resources to buy, I am interested to know what causes system time high? Is it disc space (should not be as I have enough free space)? Or is it the speed of disc? or something else? So, I want to first explore the reason (if that makes any sense)

[askb]

Your script is spending most of the time in the kernel (sys 12m51.524s), possibly doing some I/O operation. As it looks like the script does lot of I/O and swapping. I would recommend you to look into some of the kernel tunables like swapiness and writing dirty pages. Also you may want to install and configure sysstat package for analysing the sar resports, which will give you more in information on what happened during that time. Additionally you many also need to look into kernel turntables `sysctl -a | egrep -i '(swap|dirty)'