Java process (OpenDS) consumes all cpu (futex flood) - how to debug futex?

Question

Today I noticed that my LDAP server (OpenDS) consumes 100% CPU on every available core. Fast diagnose showed me that there is a lot of ETIMEDOUT during the futex lock.

How Can I debug it to figure out what is happening and how to fix it?

futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779077, {1342114766, 309244206}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779079, {1342114766, 309528293}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779081, {1342114766, 309812186}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779083, {1342114766, 310107537}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779085, {1342114766, 310406075}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779087, {1342114766, 310705339}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779089, {1342114766, 311018839}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779091, {1342114766, 311315446}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779093, {1342114766, 311632617}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779095, {1342114766, 311951779}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779097, {1342114766, 312278610}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f7ecf9053a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1540779099, {1342114766, 312601929}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f7ecf9053d0, FUTEX_WAKE_PRIVATE, 1) = 0

Not sure if its related - but I noticed also that Redmine (Ruby app) have similar "feature"... Help :)

Solution

Wow, I spent additional time googling and seems leap second was a reson:

http://blog.wpkg.org/2012/07/01/java-leap-second-bug-30-june-1-july-2012-fix/

Solution:

date `date +"%m%d%H%M%C%y.%S"`

Comments

[dusan]

Thanks a lot, today JBoss was using a lot of CPU and this fixed it :)