Failed to get D-Bus connection: Connection refused

Question

A sudo user named my-host is getting the following error message when trying to reload systemctl on an Amazon Linux 2 EC2 image.

[my-host@ip-12-3-4-56 ~]$ systemctl --user daemon-reload
Failed to get D-Bus connection: Connection refused

What specifically needs to be done to resolve and remove this error?

The other postings I have found on this error message are either related to Debian with an apt-get solution that does not apply here, or are related to Docker images, which also do not apply here.

If the problem has to do with user privileges, what would need to be done to remediate the problem?

This machine has a requirement that some systemctl commands like this must not run as sudo.

TRY AGAIN WITH DIRECT LOGIN USER PER @Stewart's and @JdeBP's SUGGESTIONS:

We then tried using a sudo user that has direct ssh ability from the outside world, but still got the following results:

[ssh-login-user@ip-12-3-4-567 ~]$ systemctl --user daemon-reload
Failed to get D-Bus connection: No such file or directory
[ssh-login-user@ip-12-3-4-567 ~]$ echo $XDG_RUNTIME_DIR $UID
/run/user/1001 1001
[ssh-login-user@ip-12-3-4-567 ~]$ systemctl --user status
Failed to get D-Bus connection: No such file or directory
[ssh-login-user@ip-12-3-4-567 ~]$ sudo install -d -o ssh-login-user /run/user/`id -u ssh-login-user`
[sudo] password for ssh-login-user:
[ssh-login-user@ip-12-3-4-567 ~]$ sudo systemctl start user@`id -u ssh-login-user`
Failed to start [email protected]: Unit not found.
[ssh-login-user@ip-12-3-4-567 ~]$ sudo -u ssh-login-user DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/`id -u ssh-login-user`/bus systemctl --user
Failed to get D-Bus connection: No such file or directory

To confirm the state of systemctl, we then also did the following, which reported one degraded service as follows:

● ip-12-3-4-567.aws-region-n.compute.internal
    State: degraded
     Jobs: 0 queued
   Failed: 1 units
    Since: Wed 2020-10-28 23:40:50 UTC; 7min ago
   CGroup: /
           ├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
           ├─user.slice
           │ └─user-1001.slice
           │   └─session-2.scope
           │     ├─18736 sshd: ssh-login-user [priv]
           │     ├─18738 sshd: ssh-login-user@pts/0
           │     ├─18739 -bash
           │     ├─18792 sudo systemctl status
           │     └─18793 systemctl status
           └─system.slice
             ├─rngd.service
             │ └─2692 /sbin/rngd -f
             ├─amazon-ssm-agent.service
             │ ├─3810 /usr/bin/amazon-ssm-agent
             │ └─4428 /usr/bin/ssm-agent-worker
             ├─libstoragemgmt.service
             │ └─2694 /usr/bin/lsmd -d
             ├─systemd-udevd.service
             │ └─1968 /usr/lib/systemd/systemd-udevd
             ├─system-serial\x2dgetty.slice
             │ └─[email protected]
             │   └─3273 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220
             ├─chronyd.service
             │ └─2685 /usr/sbin/chronyd
             ├─auditd.service
             │ └─2646 /sbin/auditd
             ├─systemd-journald.service
             │ └─1931 /usr/lib/systemd/systemd-journald
             ├─atd.service
             │ └─3239 /usr/sbin/atd -f
             ├─sshd.service
             │ └─3486 /usr/sbin/sshd -D
             ├─crond.service
             │ └─3529 /usr/sbin/crond -n
             ├─gssproxy.service
             │ └─2711 /usr/sbin/gssproxy -D
             ├─rsyslog.service
             │ └─3229 /usr/sbin/rsyslogd -n
             ├─rpcbind.service
             │ └─2683 /sbin/rpcbind -w
             ├─network.service
             │ ├─2919 /sbin/dhclient -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
             │ └─3034 /sbin/dhclient -6 -nw -lf /var/lib/dhclient/dhclient6--eth0.lease -pf /var/run/dhclient6-eth0.pid eth0
             ├─lvm2-lvmetad.service
             │ └─1953 /usr/sbin/lvmetad -f
             ├─postfix.service
             │ ├─3177 /usr/libexec/postfix/master -w
             │ ├─3178 pickup -l -t unix -u
             │ └─3179 qmgr -l -t unix -u
             ├─acpid.service
             │ └─3456 /usr/sbin/acpid
             ├─dbus.service
             │ └─2681 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
             ├─system-getty.slice
             │ └─[email protected]
             │   └─3274 /sbin/agetty --noclear tty1 linux
             └─systemd-logind.service
               └─2686 /usr/lib/systemd/systemd-logind

What else can we do to resolve this problem?

ANSWER: We decided to continue running systemctl as sudo and instead to just specify the user in the .service file so that the service will run as the specified user and not as root or sudo. We chose this approach because the answers below did not resolve the problem.

Solution 1

How are you logging in as my-host?

If it's with su, I found that /run/user/$UID isn't created and XDG_RUNTIME_DIR isn't updated.

user ~ $ echo $XDG_RUNTIME_DIR $UID
/run/user/1000 1000

user ~ $ su my-host

my-host ~ $ echo $XDG_RUNTIME_DIR $UID
/run/user/1000 1001                     <-- Mismatch here! 

my-host ~ $ systemctl --user status
Failed to connect to bus: Permission denied

Because $XDG_RUNTIME_DIR isn't changed, systemd is trying to connect to the user runtime of your original user which my-host doesn't have permissions for.

Per the systemd devs:

"su" is a tool for changing user identities and very few other process credentials temporarily. It's not a tool for opening a completely new login session. A new login session has a very well defined, pristine setup, not inheriting anything from any other session, but this is really not the case for "su" uid changes: most of the execution environment is inherited over, in numerous and non-obvious ways, for example MAC contexts, audit contexts, cgroup contexts, namespace contexts, scheduling, timer granularity,

The answer is to login as my-host with a totally fresh session. Ways to do that:

1. reboot
2. Exit to your display manager and login again
3. machinectl login
4. ssh [email protected]

I've seen other posts which do things like mkdir /run/user/$(id -u) and manually set environment variables such as XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS, but the most reliable way is to simply change how you login. A clean login will do all of that for you.

Solution 2

The other answer I wrote is valid for cases where your user has a login. Your comment to that answer suggested that you don't have remote login. It sounds like my-host is a system user.

The reasons we'd often want to use the user bus is so that the user can control their own units with systemctl and review their own logs with journalctl. If the user is a system user with no login, then there is no one to control the bus and so it doesn't really make sense.

Instead of using the systemd user bus, consider using the systemd system bus.

If you need to run an application as a specific user, just add User= to your [Service] section. Then it will run as part of the system bus, but execute as your unprivileged system user. Maybe that's what you were trying to do with systemctl --user in the first place.

If you are running an application which includes <systemd/sd-bus.h> or a script which calls systemctl and you are want to run that as my-host, it's still possible to grant access to the system bus without sudo. This requires some polkit rules described in this answer or in this answer. If you also want to use the system's journal, add my-host to group systemd-journal or follow instructions here.

Solution 3

Stewart is right for the XDG_RUNTIME_DIR and UID mismatch using the su command from another account.

Sometimes also, the user manager instance is not running for the user UID (i.e [email protected]).

Therefore, you get this error message:

Failed to get D-Bus connection: No such file or directory

Check that these exists:

ls -l /usr/lib/systemd/system/user{,-runtime-dir}@.service

If so, start the user manager instance for USER_UID as root:

systemctl start user@USER_UID.service

Otherwise read more at https://www.freedesktop.org/software/systemd/man/[email protected]



Some useful links:

https://serverfault.com/a/1026914

https://bugs.centos.org/view.php?id=8767

Comments

[Stewart]

I just reproduced this on a debian machine with a secondary user. At first I thought maybe it was because ~/.config/systemd/user/ didn't exist, but that doesn't appear to the be the problem. It works fine with my primary user.

[JdeBP]

Related questions are unix.stackexchange.com/q/423632/5132 and unix.stackexchange.com/q/431896/5132 .

[RabT]

@JdeBP This OP specifies Amazon Linux 2, which is derived from RHEL.

[RabT]

@JdeBP I just added the results of the suggestions from your links to the OP. Do you have specific suggestions here?

[RabT]

$XDG_RUNTIME_DIR is not defined for my-host in the EC2 instance in question. Since Ansible is being used to create my-host, it may take a little time to get remote login set up for my-host. Our model was for Ansible to login as user and then change to become my-host. I will be able to work on this Monday and get back to you with more results. Thank you.

[RabT]

We just added the results of your suggestion to the OP. We are happy to use a user with remote login abilities per your suggestion, but this still leaves the problem shown above. What else should we try?

[RabT]

Just added additional updates to the OP. Do you have more suggestions?

[Community]

Please provide additional details in your answer. As it's currently written, it's hard to understand your solution.