How to restart prometheus?

Question

I have set-up prometheus in my Ubuntu machine and it is running at localhost:9090 now. But, when I run the following command, I get a failed status.

systemctl status prometheus

Output:

● prometheus.service - Prometheus
Loaded: loaded (/lib/systemd/system/prometheus.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-11-06 14:58:36 +0530; 8s ago
Main PID: 7046 (code=exited, status=1/FAILURE)

නෙවැ 06 14:58:36 ayesh systemd[1]: prometheus.service: Service hold-off time over, scheduling restart
නෙවැ 06 14:58:36 ayesh systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 5
නෙවැ 06 14:58:36 ayesh systemd[1]: Stopped Prometheus.
නෙවැ 06 14:58:36 ayesh systemd[1]: prometheus.service: Start request repeated too quickly.
නෙවැ 06 14:58:36 ayesh systemd[1]: prometheus.service: Failed with result 'exit-code'.
නෙවැ 06 14:58:36 ayesh systemd[1]: Failed to start Prometheus.

I tried to restart prometheus using;

killall -HUP prometheus
sudo systemctl daemon-reload
sudo systemctl restart prometheus

and using;

curl -X POST http://localhost:9090/-/reload

but they did not work for me. I have checked for syntax errors of prometheus.yml using 'promtool' and it passed successfully.

Is there any other way to fix this problem?

Solution 1

Check if it still running on your task manager and then kill it's task from there, that will work.

Solution 2

The output shows a failed start of prometheus. So you shouldn't be able to kill anything. Just check your processes with:

ps -ef | [p]rometheus   # the [p] is used to hide the grep process itself 

Use the following command to see more log content about prometheus:

journalctl -t prometheus

There might also be more information in your log files in the directory /var/log, especially in /var/log/messages and/or /var/log/syslog.

For debugging purposes just start prometheus in the foreground by executing the following:

$(which prometheus)

This will help to find additional information about the failed start.