pam_unix generates a lot of open/close sessions for my domain user

Question

I setup a new VPS with ubuntu 18.04, including virtualmin/usermin. In auth.log I see a lot of

su[12936]: Successful su for domain by root
su[12936]: + ??? root:domain 
systemd-logind[148]: New session c315 of user domain .
su[12936]: pam_unix(su:session): session opened for user domain by (uid=0)
su[12936]: pam_unix(su:session): session closed for user domain 

in syslog, I see a lot of

systemd[1]: Started Session c314 of user domain.
systemd[1]: Started Session c315 of user domain.

domain is the user of my virtual server defined in the VPS. c314/c315 increased by 1 each time... It used to appear every 2-3 minutes, now it's every 5 minutes.

Reading on the internet about this, all the "solutions" were how to remove this logging from the log but nothing was explaining what are all those open/close sessions in the first place.

Also, when running loginctl list-sessions those sessions are accumulated in "active=yes" and "state=closing" mode and never disappear from the list. At the moment there are 95 such sessions.

What is happening on my VPS, who is opening/closing sessions so many times and why? Also, why those sessions never disappear from the sessions list?

Thanks

update

loginctl session-status c315
c315 - domain (1000)
           Since: Sat 2020-02-08 20:27:08 UTC; 23h ago
          Leader: 12936
             TTY: ???
          Remote: user root
         Service: su; type tty; class user
           State: closing
            Unit: session-c315.scope

Unit user-1000.slice (/user.slice/user-1000.slice):
└─session-2691929.scope
├─19035 sshd: domain [priv]
├─19051 sshd: domain@pts/0
├─19052 -bash
├─20124 sudo systemd-cgls -u user-1000.slice
├─20125 systemd-cgls -u user-1000.slice
└─20126 pager

Solution

pam_unix sessions exit normally, as seen in the logs. Those increasing number of sessions are systemd-logind sessions, which for some reason remain open, even when they don't contain any processes.

A workaround you might try would be to force systemd-logind to kill all the session processes, when the session leader exits. You can do it by modifying the KillUserProcesses and KillOnlyUsers setting in /etc/systemd/logind.conf:

KillUserProcesses=yes
KillOnlyUsers=domain

and restarting systemd-logind:

systemctl restart systemd-logind

However, this does not answer the question, why the sessions are not closing by themselves, since the session scopes are empty.

Edit: About the difference between pam_unix and systemd-logind sessions:

• pam_unix sessions consist in a small record added to or removed from /var/run/utmp. You can list them with w or who,
• systemd-logind sessions are more heavy, as explained in the manpage of pam_systemd. Dangling systemd-logind consume much more resources. They are listed with loginctl list-sessions

Since you identified a possible culprit (in a comment to this answer), you can apply another workaround: replace

@include common-session

in /etc/pam.d/su with:

@include common-session-noninteractive

which does not contain pam_systemd. When you modify PAM files, the usual precautions apply: keep a root shell active (e.g. sudo -i) until you tested the new config, in case you break something.

Comments

[Piotr P. Karwasz]

pam_unix sessions exit normally: who will show you only a couple of records. systemd-logind (through pam_systemd) doesn't close its session: you should see systemd-logind[148]: Session c315 logged out. Waiting for processes to exit. and systemd-logind[148]: Removed session c315. when it does. Add the status of the unclosed sessions: loginctl session-status c315 to show which processes are still running under it.

[Piotr P. Karwasz]

As for who is starting those session, IIRC your previous question, neither systemd nor cron can do it, but add the result of systemctl list-timers and cat /etc/cron.d/* /etc/crontab /var/spool/cron/crontabs/* | grep -v '^#'

[Amos]

I updated my question with more info, the open/close is happening every 5 minutes, nothing is scheduled for 5 minutes neither in cron nor timers.... I do wonder what is causing all this but I'm more surprised that no one else encountered it. Those who wrote about it, were satisfied solely by knowing how to remove it from the logging...

[Piotr P. Karwasz]

The session status you added looks like an ssh session, although it was opened by su. Are all the sessions similar?

[Amos]

All sessions remain state=closing. I did connect with user domain to ssh (instead of root) so you probably see this but that also means, nothing else is happening with user domain. Maybe this is how ubuntu is handling a user that was sudo'ed? I don't know.

[Piotr P. Karwasz]

One of the advantages of systemd-logind is that it isolates every session into its own cgroup. The ssh connection, bash and systemd-cgls command you see, where launched inside session c315. Until they exit, the session will be in a 'closing' state.

[Amos]

Perhaps there's a misunderstanding. I selected c315 randomly and ran the slice command on user domain just to see the output so the output of slice has nothing to do with c315 session. If the above does give you an idea on what I can do to dig further, do let me know :)

[Piotr P. Karwasz]

A misunderstanding indeed: usually the output of loginctl session-status contains the list of all the processes in the session. Your list is empty, so the mystery of never-closing sessions persists.

[Amos]

As you can see in auth.log, it is opening and closing immediately. A real mystery. Do you have any idea where I can ask this question? I'm lost... It's over 400 sessions now... I suddenly found this: github.com/systemd/systemd/issues/8598 does it make any sense?

[Amos]

FWIW: in /etc/systemd/logind.conf everything is in comment

[Amos]

Here is the source! cloudmin.com/node/54674 question remains about why session remain open....

[Amos]

w/who show only my own session but loginctl list-sessions shows 652 (and counting) sessions... what are those sessions in loginctl list-sessions?

[Amos]

oh, one more thing: I connected as root using ssh, exited and re-connected with root and waited. Both sessions were in loginctl list-sessions....

[Amos]

what do you mean by "keep a root shell active"? I understand that the change simply won't list those sessions, it won't really "fix" the problem itself, right?

[Piotr P. Karwasz]

I mean: keep a shell running as root open, not just a shell that can sudo or su to root, because breaking PAM will stop you from becoming root again. The second workaround will stop the creation of systemd-logind sessions, it doesn't fix what's wrong with those already created.

[Amos]

Virtualmin has a reason for opening those sessions and I prefer not to break them without knowing what I'm doing. I posted a question in virtualmin forum, but it's the community so I hope I will get some answers

[Piotr P. Karwasz]

Virtualmin just wants to execute a command, (openssl according to your link), it doesn't need any kind of session. A session is just a way to keep common processes together.

[Amos]

I admit that I'm a bit afraid of doing that, that's a production server...

[Amos]

I got a test vps, installed centos7 + virtualmin and I see 9 sessions, 7 of them are of the domain user, 2 of root. There is no increase over time but those 9 are there with no intention to go away. I see a session created for root (not for the domain user!) every 5 minutes but it’s not on the list. I wonder if it’s a bug with ubuntu 18.04.4 or a switch I forgot to turn on or off. ssh sessions also remain in the list for good. I setup this vps quickly to test it and not the comprehensive setup I did with my real VPS.

[Amos]

After several hours I have 14 sessions in the list (5 new are of root and not the domain user). I remind you that this is one difference from ubuntu, the open/close was of root and not of the domain user. Looks like the issue is happening also with centos but with much less effect. The question remains whether it's an ubuntu problem or a setting of some kind.

[Piotr P. Karwasz]

The question is whether those sessions contain processes or are empty.

[Amos]

I'm back to my server, will a pstree -p output help here?