“configured resource limit was exceeded” after trying to completely reinstall icinga2

Question

I was trying to remove icinga2 completely from my Ubuntu Server 16.04 by running apt-get purge --auto-remove icinga2 and also removing anything related to icinga2 i could find after locate icinga2

My second move turned out to be rather stupid because no matter what i do I don't seem to be able to reinstall icinga2 properly. I noticed, that after a reinstall the service was still missing (nothing in /etc/init.d/)

I just copied the file from another host and now, when I try to start icinga2 I get:

Job for icinga2.service failed because a configured resource limit was exceeded. See "systemctl status icinga2.service" and "journalctl -xe" for details.

systemctl status icinga2.service shows:

    ● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/lib/systemd/system/icinga2.service; disabled; vendor preset: enabled)
   Active: failed (Result: resources)

Jul 23 20:53:41 srvweb systemd[1]: icinga2.service: Failed to load environment files: No such file or directory
Jul 23 20:53:41 srvweb systemd[1]: icinga2.service: Failed to run 'start-pre' task: No such file or directory
Jul 23 20:53:41 srvweb systemd[1]: Failed to start Icinga host/service/network monitoring system.
Jul 23 20:53:41 srvweb systemd[1]: icinga2.service: Failed with result 'resources'.
Jul 23 20:55:25 srvweb systemd[1]: Stopped Icinga host/service/network monitoring system.
Jul 24 21:44:02 srvweb systemd[1]: Stopped Icinga host/service/network monitoring system.
Jul 24 21:44:02 srvweb systemd[1]: icinga2.service: Failed to load environment files: No such file or directory
Jul 24 21:44:02 srvweb systemd[1]: icinga2.service: Failed to run 'start-pre' task: No such file or directory
Jul 24 21:44:02 srvweb systemd[1]: Failed to start Icinga host/service/network monitoring system.
Jul 24 21:44:02 srvweb systemd[1]: icinga2.service: Failed with result 'resources'.

and journalctl -xe shows the same.

Shouldn't there be a way to fix this? Please help me learn from my mistake. Thanks a lot in advance :)

EDIT:

more info:

cat /lib/systemd/system/icinga2.service
[Unit]
Description=Icinga host/service/network monitoring system
After=postgresql.service mariadb.service carbon-cache.service mysql.service syslog.target network.target

[Service]
Type=simple
UMask=0007
EnvironmentFile=/usr/lib/icinga2/icinga2
EnvironmentFile=/etc/default/icinga2
ExecStartPre=/usr/lib/icinga2/prepare-dirs /usr/lib/icinga2/icinga2
ExecStart=/usr/sbin/icinga2 daemon -e ${ICINGA2_ERROR_LOG}
ExecReload=/usr/lib/icinga2/safe-reload /usr/lib/icinga2/icinga2
PIDFile=/run/icinga2/icinga2.pid

[Install]
WantedBy=multi-user.target

but /etc/default/icinga2 does not exist. On my other hosts the file looks exactly the same and the path does exist so I assume that this needs to be present.

Solution 1

Solution:

I was lucky enough here that I had multiple hosts with icinga2 installed (for agent based checks).

I copied /lib/systemd/system/icinga2.service and the missing /etc/default/icinga2 via scp from a working machine to my host.

After there, I got icinga2 errors because there where still config files missing.

I copied those to my machine and got it all working again. Now I can concentrate on my initial problem was gdb.

Solution 2

Looks like you are missing a lot of files. Likely you didn't remove all the packages icinga uses files from in your apt-get purge and then when you deleted files manually, you left broken packages.

Failed to load environment files is a reference in the SystemD icinga unit file. Find it (under /etc/systemd/system or /lib/systemd/system) and see if it has an EnvironmentFile setting and that file exists. If it doesn't exist you'd need to figure out which package it belongs to (using dpkg -S <file>) and do an apt-get install --reinstall <package>.

Comments

[JdeBP]

That's only some of what systemctl status shows. You have hidden from potential answerers the part that actually helps them to answer the question, the part that explicitly tells them that there's an error in your question. You should edit your question and fix this to show all of the systemctl status output for starters.

[mike.b93]

@JdeBP I've added the complete output now. That's all I get from systemctl status icinga2.service

[JdeBP]

The icinga2 doco on the WWW and this out of date mailing list message will help you to eliminate the guesswork from this answer and make it better.

[mike.b93]

I really appreciate your answers but I don't think I understand fully what all this means. I've updated my main post with more info. Can you try to explain what needs to be done and why? Thanks a lot

[Guillaume Boudreau]

Its weird that a reinstall of icinga2 didn't bring /etc/default/icinga2 (the fact that it was missing is what caused your Failed to load environment files error). If I were you I would check which package it came from and reinstall that using dpkg -S /etc/default/icinga2 and apt-get install --reinstall <package reported by dpkg>.

[mike.b93]

@Guss it's from icinga2-common which i actually did reinstall a few times.. strange that it did not bring the file back