kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"

Question

I got this errors in syslog. Any idea what does it mean? The system is running on Ubuntu 12.04, kernel: 3.8.0-35-generic #52~precise1

It looks to me like failure when trying to write to disk...

 [151850.317166] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 [151850.318428]  ffff8803640f5be8 0000000000000082 0000000034c733da ffff88040e233ec0
 [151850.318444]  ffff880408149740 ffff880403c6c5c0 0000000000000001 ffff8803edae80a8
 [151850.318468]  [<ffffffff816f41c9>] schedule+0x29/0x70
 [151850.318481]  [<ffffffff816f30b7>] __mutex_lock_slowpath+0xd7/0x150
 [151850.318497]  [<ffffffff816f2cca>] mutex_lock+0x2a/0x50
 [151850.318510]  [<ffffffff811aa006>] path_lookupat+0x236/0x7a0
 [151850.318523]  [<ffffffff811aade0>] ? getname_flags.part.31+0x30/0x150
 [151850.318537]  [<ffffffff811aaf6e>] ? getname_flags+0x6e/0x80
 [151850.318552]  [<ffffffff813141f4>] ? apparmor_inode_getattr+0x54/0x60
 [151850.318565]  [<ffffffff811bb929>] ? mntput_no_expire+0x49/0x160
 [151850.318578]  [<ffffffff811ab711>] user_path_at+0x11/0x20
 [151850.318588]  [<ffffffff811a0c3e>] vfs_lstat+0x1e/0x20
 [151850.318602]  [<ffffffff816fdf5d>] system_call_fastpath+0x1a/0x1f
 [151850.319634] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 [151850.376552]  [<ffffffff816f41c9>] schedule+0x29/0x70
 [151850.376565]  [<ffffffff816f30b7>] __mutex_lock_slowpath+0xd7/0x150
 [151850.376579]  [<ffffffff816f2cca>] mutex_lock+0x2a/0x50
 [151850.376590]  [<ffffffff811aa006>] path_lookupat+0x236/0x7a0
 [151850.376602]  [<ffffffff811aade0>] ? getname_flags.part.31+0x30/0x150
 [151850.376616]  [<ffffffff811aaf6e>] ? getname_flags+0x6e/0x80
 [151850.376680]  [<ffffffff813141f4>] ? apparmor_inode_getattr+0x54/0x60
 [151850.376691]  [<ffffffff811bb929>] ? mntput_no_expire+0x49/0x160
 [151850.376704]  [<ffffffff811ab711>] user_path_at+0x11/0x20
 [151850.406554] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 [151850.465132] sftp-server     D ffff880403bfdf60     0 15199      1 0x00000000
 [151850.465158]  ffff8803d2c87fd8 ffff8803d2c87fd8 ffff8803d2c87fd8 0000000000013ec0
 [151850.465173]  ffffffff81c15440 ffff88040583ae80 0000000000000001 ffff8803edae80a8
 [151850.465263]  [<ffffffff811aa006>] path_lookupat+0x236/0x7a0
 [151850.465269]  [<ffffffff81186441>] ? kmem_cache_alloc+0x31/0x140
 [151850.465275]  [<ffffffff811aade0>] ? getname_flags.part.31+0x30/0x150
 [151850.465283]  [<ffffffff811aa5a4>] filename_lookup+0x34/0xc0
 [151850.465289]  [<ffffffff811aaf6e>] ? getname_flags+0x6e/0x80
 [151850.465296]  [<ffffffff811ab6b9>] user_path_at_empty+0x59/0xa0
 [151850.465303]  [<ffffffff813141f4>] ? apparmor_inode_getattr+0x54/0x60
 [151850.465309]  [<ffffffff81087e9a>] ? lg_local_unlock+0x1a/0x20
 [151850.465315]  [<ffffffff811bb929>] ? mntput_no_expire+0x49/0x160
 [151850.465320]  [<ffffffff811a0946>] ? cp_new_stat+0x116/0x130
 [151850.465327]  [<ffffffff811ab711>] user_path_at+0x11/0x20
 [151850.465332]  [<ffffffff811a0bc1>] vfs_fstatat+0x51/0xb0
 [151850.465337]  [<ffffffff811a0c3e>] vfs_lstat+0x1e/0x20
 [151850.465343]  [<ffffffff811a0dea>] sys_newlstat+0x1a/0x40
 [151850.465350]  [<ffffffff816fdf5d>] system_call_fastpath+0x1a/0x1f

Solution

The message appears in case that certain process (in this case sftp-server) doesn't get CPU for 120s (default limit). This could be caused by high load on the system. Generally this could be caused waiting on any resource, most likely candidates are CPU, disk and network.

When debugging such problems you can test writing speed on disk:

$ dd if=/dev/zero of=/tmp/output conv=fdatasync bs=384k count=1k; rm -f /tmp/output
1024+0 records in
1024+0 records out
402653184 bytes (403 MB) copied, 2753.13 s, 146 kB/s

You should expect values in MB/s, e.g. 50-230 MB/s depending on hard drive. In this case throughput 146 kB/s is extremely low and can cause such kernel messages.

You could also use iostat -x 5 to monitor disk utilization.

Comments

[iTag]

Is this a VM by any chance? Disk load on the hardware from other VMs can cause this kind of problem.

[Tombart]

Sorry, I'm pretty sure I wrote there kernel version, just the text got messed up. It's a standalone server running on Ubuntu 3.8.0-35-generic #52~precise1.

[phunehehe]

Seems similar blackmoreops.com/2014/09/22/…

[phunehehe]

Also related: unix.stackexchange.com/q/45906/250