Memcache 'stats' reports non-zero 'curr_connections' - but lsof shows no socket connections
I think you're correct in your logic that stat curr_connections is the number of connections that are current.
curr_connections - Number of open connections to this Memcached server, should be the same value on all servers during normal operation. This is something like the count of mySQL's "SHOW PROCESSLIST" result rows.
Source: Memcached statistics (stats command)
When I setup memcached I noticed that it always maintained 10 as the least amount of curr_connections too.
$ echo stats | nc localhost 11211 | grep curr_conn
STAT curr_connections 10
But why 10?
If you run memcached in verbose mode you'll notice the following output:
$ memcached -vv
...
<26 server listening (auto-negotiate)
<27 server listening (auto-negotiate)
<28 send buffer was 212992, now 268435456
<28 server listening (udp)
<28 server listening (udp)
<28 server listening (udp)
<29 send buffer was 212992, now 268435456
<28 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)
If you tally up the listening servers (8) + 2 servers (auto-negotiated) you'll discover why there are 10 base servers to start, at least that's what I thought at first. But you need to dig deeper to better understand what's going on.
It would appear that memcached is multi-threaded and so the output that you were taking notice of isn't really how one would tally up the connections.
Notice the threads
The output of ps -eLf shows the threads:
$ ps -eLf | grep memc
saml 20036 4393 20036 0 6 20:07 pts/7 00:00:00 memcached -vv
saml 20036 4393 20037 0 6 20:07 pts/7 00:00:00 memcached -vv
saml 20036 4393 20038 0 6 20:07 pts/7 00:00:00 memcached -vv
saml 20036 4393 20039 0 6 20:07 pts/7 00:00:00 memcached -vv
saml 20036 4393 20040 0 6 20:07 pts/7 00:00:00 memcached -vv
saml 20036 4393 20041 0 6 20:07 pts/7 00:00:00 memcached -vv
There is one master node and 5 worker threads.
Here's what the output of lsof looks like when I've made 3 connections to memcached using the same method as you, telnet localhost 11211.
So it would appear that each thread is maintaining a connection (or a reference to each connection) as they're made and kept open. As soon as the telnet connections are closed they go away from this list.
So how can we count the connections?
So if you wanted to tally up the connections you could either subtract 10 from the curr_connections result, or you could run lsof and count the number of connections that are associated with the main PID. Notice this output:
$ lsof |grep memcac | grep IPv
memcached 20036 saml 26u IPv4 7833065 0t0 TCP *:memcache (LISTEN)
memcached 20036 saml 27u IPv6 7833066 0t0 TCP *:memcache (LISTEN)
memcached 20036 saml 28u IPv4 7833069 0t0 UDP *:memcache
memcached 20036 saml 29u IPv6 7833070 0t0 UDP *:memcache
memcached 20036 saml 30u IPv6 7962078 0t0 TCP localhost:memcache->localhost:38728 (ESTABLISHED)
That last line is an active connection. So we could count them like this:
$ lsof -p $(pgrep memcached) | grep "memcache->" | wc -l
1
But your output shows IPv4 & IPv6, what's up with that?
To simplify the example even more let's start memcached up and force it to only listen on a single IPv4 address, 192.168.1.20. In the above examples we were starting memcached up on all interfaces and connecting to it using localhost, so let's take another look.
$ memcached -vv -l 192.168.1.20
...
<26 server listening (auto-negotiate)
<27 send buffer was 212992, now 268435456
<27 server listening (udp)
<27 server listening (udp)
<27 server listening (udp)
<27 server listening (udp)
Notice that we're only getting 1/2 as many? Previously we had 2 auto-negotiated servers and 8 UDP, this time we have 1 auto and 4 UDP. Why? Well we've told memcached to only listen on the IPv4 interface, previously it was listening on everything, the IPv4 and localhost. We can convince ourselves of this by attempting to connect to the server on localhost:
$ telnet localhost 11211
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
See we can't connect. But we can using the IPv4 address:
$ telnet 192.168.1.20 11211
Trying 192.168.1.20...
Connected to 192.168.1.20.
Escape character is '^]'.
What does stats say now?
STAT curr_connections 5
See? The curr_connections is showing the number 5 (1 auto + 4 UDP).
4 UDPs?
Why are we running 4 UDPs? This would appear to be a default setting in memcached. You can see the settings using the stats settings command when youtelnet` to the server:
stats settings
STAT maxbytes 67108864
STAT maxconns 1024
STAT tcpport 11211
STAT udpport 11211
STAT inter 192.168.1.20
STAT verbosity 2
...
STAT num_threads 4
STAT num_threads_per_udp 4
...
Can we change that value? Sure it's the -t # switch to memcached.
$ memcached -vv -l 192.168.1.20 -t 1
...
<11 server listening (auto-negotiate)
<12 send buffer was 212992, now 268435456
<12 server listening (udp)
So now we only have the main listener (auto) and 1 UDP thread. If we check stats now:
STAT curr_connections 2
Incidentally, we can't set the number of threads to 0.
$ memcached -vv -l 192.168.1.20 -t 0
Number of threads must be greater than 0
So the lowest we can get curr_connections to go is 2. If we open up 6 telnets (5 from ourself - greeneggs and 1 from another remote server named skinner) we'll see the following in our lsof output:
$ lsof -p $(pgrep memcached) | grep ":memcache"
memcached 24949 saml 11u IPv4 847365 0t0 TCP greeneggs.bubba.net:memcache (LISTEN)
memcached 24949 saml 12u IPv4 847366 0t0 UDP greeneggs.bubba.net:memcache
memcached 24949 saml 13u IPv4 855914 0t0 TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48273 (ESTABLISHED)
memcached 24949 saml 14u IPv4 872538 0t0 TCP greeneggs.bubba.net:memcache->skinner.bubba.net:41352 (ESTABLISHED)
memcached 24949 saml 15u IPv4 855975 0t0 TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48298 (ESTABLISHED)
memcached 24949 saml 16u IPv4 855992 0t0 TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48305 (ESTABLISHED)
memcached 24949 saml 17u IPv4 859065 0t0 TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48316 (ESTABLISHED)
memcached 24949 saml 18u IPv4 859077 0t0 TCP greeneggs.bubba.net:memcache->greeneggs.bubba.net:48326 (ESTABLISHED)
So we still have the auto + the 1 UDP and 6 other connections. Our stats command shows this:
STAT curr_connections 8
So no surprise there.
Related videos on Youtube
02 : 36
01 : 31 : 30
30 : 46
02 : 29
ttsiodras
Summary From 2002 to 2012, I was part-owner of a startup ( Semantix S.A. ), in the role of the company's Senior Software Engineer and Technical Lead. The company was acquired by Neuropublic S.A in June 2012. Tech I mostly code in Python and C/C++, targeting Linux, Windows and embedded development. In some of my projects I had to optimize for speed using CUDA and OpenMP/TBB. I can also code in x86/SSE asm if necessary. Scripting: Python mostly ; Perl in the past; daily one-liners with bash/awk/sed. SQL-wise: Oracle, MySQL and PostgreSQL (I've also written native apps over direct APIs: OCI for Oracle, psycopg2 for PostgreSQL). I begun my career 11 years ago, successfully coding Windows device drivers for 7 different FPGA designs that stress-tested Siemens 3G switches. I mostly use VIM these days but have no fear of IDEs (that's where I begun). That being said, I prefer Makefiles (recently, tup) to Eclipse-sized monsters. I value strong type systems and functional-style thinking (OCaml/F#). I am not an extremist in this regard, sometimes mutable state is the way to go (translation: I think Haskell takes it too far). I love ZFS - in general, data checksums in the filesystem. When they are not there, I use my own. I have a soft spot for Lisps. When my work in my startup demanded it, I skimmed over C#, Java and Windows administration. My programming/admin blog (Slashdotted, Reddit-ed, HN-ed, etc) Here ( mirrored on: http://ttsiodras.github.com/ ). My CV is here: https://www.thanassis.space/cv.pdf ( mirrored on: http://ttsiodras.github.io/cv.pdf )
Updated on September 18, 2022Comments
-
ttsiodras over 1 year
Our memcache daemon reports non-zero 'curr_connections'...
$ telnet memcache-server 11211 Escape character is '^]'. stats ... STAT curr_connections 12 ......and yet,
lsofshows no socket connections:$ ssh memcache-server # lsof -P -i -n | grep memcache memcached 1759 memcached 26u IPv4 11638 0t0 TCP *:11211 (LISTEN) memcached 1759 memcached 27u IPv6 11639 0t0 TCP *:11211 (LISTEN) memcached 1759 memcached 28u IPv4 11642 0t0 UDP *:11211 memcached 1759 memcached 29u IPv6 11643 0t0 UDP *:11211I am guessing 'curr_connections' does not mean what I think it does...
-
ttsiodras about 10 yearsThanks for investigating this. It's still not clear what exactly is happening... for example, in my case subtracting 10 would still leave 2 - which are not there in my lsof output. IMHO this looks like a bug in memcached - why would
statsreport connections... from the memcached threads themselves? And what kind of connects are these? Also, in your screenshot you appear to be connected over IPv6 - which is not the case for me. All in all, I find memcached stats to be... weird, at the very least. -
slm about 10 years@ttsiodras - No I'm definitely not using IPv6, I only have IPv4, the memcached daemon out of the box sets up listeners on IPv6, but I have no IP address there so ignore that. -
slm about 10 years@ttsiodras - can you get the full output from
lsof -p $(pgrep memcached) | grep "memcache->"? This should show you all the active connections. I do not believe it's a bug, as I said it spins up threads. -
slm about 10 years@ttsiodras - I've added another example, does that help?