Linux slows down after long uptime

Question

I noticed a strange behavior of a for cycle...

Case 1:

Let us have an internal initialization of i variable

for ( int i=10; i <=10; i++)
{
    std::cout << i;
}
return 0;

Output:

10

Case 2:

Now we initialize i outside the cycle

int i = 10;

for ( ; i <=10; i++)
{
    std::cout << i;
}
return 0;

Output:

Nothing will be print

Case 3:

Initialization any variable not related to i:

int i = 10;

for ( int k = 0 ; i <=10; i++)
{
    std::cout << i;
}
return 0;

Output:

10

Question

How can depend a result of a cycle condition on the place where we initialize variable i?

Updated results

I traced a code using degugger and VS 2010 really jumps over a cycle in case 2. However g++ works well. Maybe a bug in compiler?

Fianal results After reboot it works correcly. Something between heaven and earth. Sorry for the useless question....

Solution 1

I have noticed this too, more so with systems I used over 6 years ago, and had forgotten about it until your question.

Upon returning to a system in the morning or worse still after a weekend, a key depression in a terminal or clicking on a tab in Firefox, would trigger an incredible amount of disk activity.

My theory for what was happening, is that the activity was due to processes being swapped in again, and the cause was due to the kernel's aggressive nature of swapping out idle processes to optimise the amount of "free" memory available, which which is probably a smart idea on an active system when a new process might be started, but has this one particular down-side.

One way to confirm this would be to have a terminal with vmstat 1 1 running, unfortunately having something running makes the system less idle, so this has some affect on the experiment, so you need to be carefull to start with a different terminal or another program when you start in the morning and ensure that that window is visible so you can see the stats.

You should see the SI Swap in column shoot up when you start.

How do I get around it? I let the computer shut down to suspend mode (on my laptop), or shut down completely on a desktop. I never use hibernate since I find it takes longer to wake up from hibernate, than simply booting from scratch and boot time has been so much improved compared to how it was, and disk drives are faster, memory is cheaper. (at the time my system had 1GB of memory whilst today it has 4GB or more) and I suspect that there is a memory usage factor as when the swapouts are triggered as well. So more memory helps.

There is a kernel tuning parameter that affects the swapiness

To see what your current setting is :

cat /proc/sys/vm/swappiness

The value can be between 0 and 100, 100 means most aggressive swap out from physical memory policy, zero least aggressive.

You can change it by simply echoing a value to that pseudo file.

eg

echo 20 > /proc/sys/vm/swappiness

There is a good article and some performance links related to swapiness here.

edit 1 to answer @Gilles

I too have had machines run for years, only to shut down because I've actually moved premises, I am not insinuating there is anything like software rot but only that I too also experienced exactly what I have described, it's not obvious with servers, when you login via ssh, but it is on say my asus eeepc 901, when it does not sleep, pressing shift there is noticeable disk activity, before the system responds. I do not make things up, and because you haven't noticed it doesn't mean to say that this behaviour it's not there. I'm not knocking linux or trying to be disparaging, at all.

edit 2 @mabalenk

You are confirming exactly the circumstances, 60 is the default swapiness, and SI count will be zero when nothing is happening. vmstat 1 prints statistics every second, so apart from the first line printed, each line is the delta for the last second, so when you see it rise, swap-ins are occurring. Having said this, your figures of 60 or so do not seem to be very high, maybe my system was particularly low on memory, I would see much higher figures 120-250 and lasting for between 20-50 seconds, (sometimes a couple of minutes) and then settling down to zero and I'd get a response from the original key depression in the terminal.

Yes, the kernel will tend to show a 90% memory usage, (it tries not to waste any. ;-) ). The 30% swap space used, doesn't mean much since it is the amount of reserved swap space used and depends on how much has been allocated. Your amount of RAM and swap space allocation would be usefull figures to add to your original post (which you can always edit.).

I don't understand why you surmise that there is something wrong with kde ? If I were you, I would change nothing but the swapiness and drop it down to the ridiculous, say 5 before you leave at night, and check again to see if you notice any difference in the stats, when you start again in the morning.

I don't know if you read any of the links I posted but there was an interesting comment from one of the long standing Kernel developers Andrew Morton. He suggests that anything other than using a swapiness less than 100 is not allowing the kernel to do it's job properly. Now this makes sense if one looks at the system as a whole, why not swap out inactive pages and allow more elbow room for the active part to function ? But this can have the detrimental affect, when the next bit of memory you will use has been swapped out, so when you start again in the morning, you have to wait for those pages to be swapped in again, before you get a response. There's no one size fits all.

Remember though, this is a tuning parameter, and if you are running with the usual 4 times swap size to actual RAM, then 30% SWAP used means you are using more than twice as much memory that you actually have, so your ultimate solution would be to get more RAM, since changing the swapiness will slow down swap-outs, but not change the behaviour entirely. you might find that after 16 hours idle time, you are swapped out anyway, due to cron jobs starting at night, changing entirely your active/passive memory page footprint.

Solution 2

I have the similar case. My box has 12GB RAM and over the nigth all desktop applications were swapped out for the cache and such. The main reason I think is that cache (dirty pages) can take up to 60% RAM by default and when this limit is reached, the kernel suddenly want to flash 6GB dirty pages to disk which effectively kills the performance.

I cured with the following parameters in /etc/sysctl.conf

# use only 10% of RAM for the cache (dirty pages).
vm.dirty_ratio=10 
# start flashing with 1% dirty pages
vm.dirty_background_ratio=1

Comments

[Andreas Brinck]

Both 2 and 3 should print 10.

[nullpotent]

Second one should print 10. Are you sure you didn't overlook something?

[PlasmaHH]

Please review the output of your case 2: ideone.com/alc3j

[Some programmer dude]

You're doing something else not shown in the code snippets here.

[mpu]

I think you did not run your compiler properly or you did not show us the full source, see this program, it seems to be your 'case 2' but it prints 10.

[Some programmer dude]

@iccthedral And indeed it does. Just tested in VC++2010 ("only" Express version though) and it works just fine. With or without newlines or manual flushes.

[Andreas Brinck]

@justik Can you post a complete program (including main) that reproduces this problem?

[the.malkolm]

Try again with flushing.

[justik]

@ Andreas Brinck: This is a complete program, such a simple construction :-)

[nullpotent]

Would you just c/p it, as it is, so we could get this done. This doesn't make any sense if the outputs are consistent. Then again, I've seen trolls wandering near c++ woods.

[Andrey]

It does print 10 in MSVC 2010.

[Andreas Brinck]

@justik I want the complete code that can be pasted into a single translation unit and compiled into an executable which reproduces this problem, the above doesn't.

[Admin]

What is running on your system, other than KDE?

[Admin]

Nothing really, there is Firefox with a few open tabs and Yakuake (Terminal emulator). I have four processors. Their average load is 5-10%.

[Admin]

Is there anything interesting in your system logs?

[Admin]

I have just checked /var/log/messages and /var/log/boot.log. There are no warnings or errors. Interesting is that /var/log/messages contains the info for today only. There is no history for the past days. Should I check anything else?

[nullpotent]

VS 2010 has a mature compiler, it should print 10. Perhaps you should follow Andreas' advice and flush the output.

[Some programmer dude]

Actually, just adding '\n' doesn't flush the output, as I discovered in this question. However, the program terminating should definitely flush the output.

[Neel Basu]

But If it flushes when i is initilized at loop and it doesn't flush when i is initilized before loop is also a bug in compiler. But I don't think VS will do that.

[justik]

@ Andreas it looks like a VS 2010 compiler bug, g++ works well

[Andreas Brinck]

I would be very surprised if this was a VS 2010 compiler bug, it's not like you're at the frontier of C++ features :)

[WiSaGaN]

@justik VS2010 works as expected on my windows 7 x64 machine.

[vonbrand]

While the kernel should certainly write modified pages to disk when idle, it should never evict potentially useful data from the memory. The only explanation is that the machine really isn't idle, something is running and dirtying memory, and the useful contents are being forced out by this activity. Note that it could very well be something triggered e.g. at midnight, which is long finished when you return to the machine. Indexing being done in the "background" by KDE perhaps?

[Gilles 'SO- stop being evil']

I'm used to leaving machines running for months (it usually takes about a year until there's a power failure or a hardware or kernel upgrade I can't put off any longer), and I haven't noticed any such performance degradation over time. Nor do I see how changing the swappiness would help.

[mabalenk]

My default swappiness setting is 60. When I return to work in the morning I notice that swap is occupied upto ~30%. I believe there is no need to change the default setting. On the contrary the RAM is fully occupied (~95--100%). But I think this is also normal. I have also tried to launch and see the output of vmstat 1 (this is the correct command if you want to see the live update of the stats). The si column is usually 0, but when I start the values fire up to 16, 32 and sometimes 64. Potentially, something is wrong with the KDE. But how do I check that?