letsencrypt/certbot systemd timer/service not working on Ubuntu 18.04

Question

I installed certbot for nginx and the autorenew script set itself up automatically, but this is the email I receive, whenever it runs:

/home/foobar/certbot-renew.sh: 1: /home/foobar/certbot-renew.sh: /usr/bin/certbot: not found

What confuses me is, there's no certbot-renew.sh in my home dir...?

Further information from systemctl:

# /lib/systemd/system/certbot.timer
[Unit]
Description=Run certbot twice daily

[Timer]
OnCalendar=*-*-* 00,12:00:00
RandomizedDelaySec=43200
Persistent=true

[Install]
WantedBy=timers.target

and

# /lib/systemd/system/certbot.service
[Unit]
Description=Certbot
Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
Documentation=https://letsencrypt.readthedocs.io/en/latest/
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true

I'm new to using systemd, help is appreciated!

Edit:

As @grawity suggested, I checked my crontabs instead and found this:

foo@bar:~$ cat /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

Also this:

foo@bar:~$ cat /etc/cron.d/certbot
# /etc/cron.d/certbot: crontab entries for the certbot package
#
# Upstream recommends attempting renewal twice a day
#
# Eventually, this will be an opportunity to validate certificates
# haven't been revoked, etc.  Renewal will only occur if expiration
# is within 30 days.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

To clarify:

foo@bar:~$ which certbot
/usr/bin/certbot

foo@bar:~$ certbot --version
certbot 0.26.1

Solution 1

Certbot has its own service to renew things, you do not need no additional crontab lines. In theory.

Practically I've seed certbot.timer stopping seeming for no reason at all:

$ sudo systemctl status certbot.timer
● certbot.timer - Run certbot twice daily
   Loaded: loaded (/lib/systemd/system/certbot.timer; enabled; vendor preset: enabled)
   Active: inactive (dead) since .....; 1 months 7 days ago

In that case restarting it helps:

sudo systemctl enable certbot.timer
sudo systemctl start certbot.timer

The reason why it stops functioning is yet to be found.

Solution 2

Systemd services do not generate email notifications. Cron jobs do.

All of this indicates that your shown systemd units are not related to the problem (it's possible that they already work without problems) – but the same task is being done from another place; most likely a job defined in your crontab.

Use crontab -l to list cron jobs for your user account and crontab -e to edit them.

Make sure to check your own crontab, root's crontab (via sudo), and the system-wide /etc/crontab file (no special command for that one).

Comments

[moritzjacobs]

You're right, I didn't know that. I still don't know what's wrong, so I edited my question above with some additional info. Thanks for your input!

[jakethedog]

This happened to me recently, where auto-renew did not run, though the system has been doing so for years. Maybe a package update broke something. Anyway, kicking the timer fixed it.

[Amit Tandel]

Note; I use the Python PIP version of certbot on a few machines. If you ever installed certbot via APT and later remove it, the certbot.timer may still exist, but is in a 'masked' state. Please be aware you will need to create your own renew script in this case regardless of the presence of a systemd timer. Just edit the apt /etc/cron.d/certbot version to your liking (remove the systemd test check).