How urgent is a *** System restart required *** for security?

Question

To learn a bit of server administration I've set up a simple Ubuntu 14.04 server on which I run a personal website. I've set it to automatically install security updates, but leave out the other updates. This seems to work pretty fine. Occasionally I get a message when logging into the server (with ssh) saying:

*** System restart required ***

The times this happened I simple rebooted Ubuntu and all was fine. This is ok because it's a simple personal website. What I wonder about though, is how this works for webservers which should be up 99.9999etc% of the time? Do they simply not restart and risk the security being breached because security updates are not installed (which I cannot imagine)? Or do they take the downtime for granted (which I cannot imagine either)?

How should I handle this if this were a very important production server which I want to keep up and running? All tips are welcome!

[EDIT] I know I can do cat /var/run/reboot-required.pkgs to list the packages which cause the reboot. The command currently yields the following:

linux-image-3.13.0-36-generic
linux-base
dbus
linux-image-extra-3.13.0-36-generic
linux-base

but how do I know if the updates are little things of whether I have a serious security vulnerability if I don't do the restart?

[EDIT2] Okay, I now combined the commands I've found to be useful into one:

xargs aptitude changelog < /var/run/reboot-required.pkgs | grep urgency=high

If this doesn't output anything, there don't seem to be security issues with a high urgency.

One last question though: are low, medium, and high the only urgency possibilities, or are there any more like for example critical or extremelyimportant?

Solution 1

The is no simple answer as it depends on the updates made. If the kernel had a serious security problem then it is good to restart as soon as possible. If the kernel had only minor fixes then the restart could be postponed.

If you guarantee an availability > 99.9% then you will almost always have a clustered system where you can reboot the nodes one by one without interrupting the service.

So you reboot the first system and reatach it to the cluster. Then the second and so on. Then the service will never become unavailable.

Solution 2

addon for the topic solution

I perform similar check for 'reboot requirement' for zabbix monitoring system

I see 2 issue in 'Topic' solution:

1. aptitude usually works badly in scripts. I kill a few hours but still didn't make it work with zabbix
2. if only 1 changelog includes urgent update - your check will always show positive results

My logic is:

1. Check last change only in changelog for every package which requires system reboot
2. As an output show only highest priority update

Using Debian documentation I found 5 possible values for 'urgency' and also fact that it can followed by equal("=") or semicolon(":") characters. Also there're can be upper and lower case characters

So I ended up with following:

#!/bin/bash
##################################
# Zabbix monitoring script
#
# Checking urgency in changelog 
# for updates which require system restart
#
##################################
# Contact:
#  [email protected]
##################################
# ChangeLog:
#  20151205    initial creation
#  20151208    check uniq packages only 
##################################

case "$1" in

status)
    if [ -f /var/run/reboot-required ]; then
      echo 1
    else
      echo 0
    fi 
    ;;

urgency)
    if [ -f /var/run/reboot-required.pkgs ]; then
      while read pkg; do
        tmp=`/usr/bin/apt-get changelog $pkg | \
             /bin/grep -m1 -ioP '(?<=[Uu]rgency[=:])(low|medium|high|emergency|critical)' | \
             tr '[:upper:]' '[:lower:]'`
        if [ -n $tmp ]; then
          if   [ "$tmp" == "low" ] && \
               [ "$urgency" != "medium" ] && \
               [ "$urgency" != "high" ] && \
               [ "$urgency" != "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=low
          elif [ "$tmp" == "medium" ] && \
               [ "$urgency" != "high" ] && \
               [ "$urgency" != "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=medium
          elif [ "$tmp" == "high" ] && \
               [ "$urgency" != "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=high
          elif [ "$tmp" == "emergency" ] && \
               [ "$urgency" != "critical" ]; then 
            urgency=emergency
          elif [ "$tmp" == "critical" ]; then 
            urgency=critical
            break
          fi
        fi 
      done < <(sort -u /run/reboot-required.pkgs)
    else
      urgency=none
    fi

    case "$urgency" in
        none)      urgency=0 ;;
        low)       urgency=1 ;;
        medium)    urgency=2 ;;
        high)      urgency=3 ;;
        emergency) urgency=4 ;;
        critical)  urgency=5 ;;
        *)         urgency=42 ;;
    esac

    echo $urgency
    ;;
esac
exit 0

As a result:

• reboot_required_check.sh status returns 1 if reboot is required, 0 if isn't
• reboot_required_check.sh urgency returns highest 'urgency' level or '0' if reboot is not required

Hope it helps someone to save a time ;)

Comments

[Ramhound]

I don't understand the question. Websites with larger traffic simply schedule this downtime during a period of time with less traffic. How urgent it is depends on what's being updated exactly.

[David Richerby]

I wonder how many people came here because they saw the question in the "Hot Network Questions" list and wondered what the expletives were... *raises hand*

[Lightness Races in Orbit]

@Ramhound: Ehm, no, they transparently switch over to a secondary server for the duration of the maintenance.

[KajMagnus]

Re the last question: I'm having in mind to filter out low and medium and consider all other / unknown levels urgent: | grep 'urgency=' | egrep -v '=(low|medium)'

[kramer65]

Thanks for your answer. I added a little piece to my initial question; I know I can do cat /var/run/reboot-required.pkgs to get the packages which require the reboot. But how do I know if these are only minor fixes, or whether it is a serious security vulnerability?

[Uwe Plonus]

@kramer65 each package has a changelog. E.g. the changlog for the kernel can be found here.

[kramer65]

Alright, so then it is up to the sysadmin (i.e.: in this case myself) to determine whether those changes are important? I have far too little knowledge to determine this for the Linux kernel, let alone for all the zillion other packages. Is there no central place where I can find a determination whether the update is absolutely needed for security?

[Uwe Plonus]

@kramer65 security patches are marked as those in ubuntu.

[kramer65]

I know they are security patches, but my question is how urgent they are? Is it extremely import to apply a certain patch, or are they patches for extremely rare cases. That's what I'm after. Would you have any idea how I can find this out?

[nyuszika7h]

@kramer65 Run aptitude changelog <package>, here is an example output: paste.ubuntu.com/8410798 (This is on a Debian system, not Ubuntu, but the same will work on Ubuntu too.)

[kramer65]

@nyuszika7h - Awesome! That gives some insight! One more question though. When I view this for the linux kernel, there is a massive list of commits which I stopped scrolling after about 5 minutes. They are grouped into security risks being low, medium, and I suppose high (I have only seen low and medium). Is there a way to only output the high (and maybe critical?) commits?

[Robbietjuh]

@kramer65 You could pipe it through grep. It doesn't really matter what command you want to 'filter', it works for almost all of them. In your case you might want to do something like aptitude changelog <package> | grep high

[kramer65]

Thanks for all the help here. I finally combined all the things I've learned here into one command: xargs aptitude changelog < /var/run/reboot-required.pkgs | grep urgency=high (added it to the initial question as well) which gives some output as to which packages have highly urgent patches. After that, individual packages can of course be inspected. Thanks a million for all the answers and ideas!

[itsmejoeeey]

While this is theoretically correct, Big web servers run custom versions of Linux. They won't see a System restart required dialogue, they update what they need to stay secure. In most cases, many if not all of the updates can be done while the system is running (I believe it is even possible to update a Linux kernel on a running system without a reboot).

[rom]

Interesting. I have a server on Amazon and I often restart it because of this message... I am running Ubuntu on my server. How to customize it so I don't have to reboot it every now and then?

[itsmejoeeey]

I don't have any experience with Amazon servers. Big web servers are run on dedicated servers and VPS's. Because of this, the system administrator has more control over the software. Does Amazon give you root shell access to your server?

[rom]

Yes it's possible to have root access.

[itsmejoeeey]

Then updating packages manually, and then restarting the affected services, and using something like Ksplice for the kernel updates would be one way. It is worth noting that Ksplice freezes execution of a computer so it is the only program running when applying a patch, so there may still be a little bit of downtime (due to the web server process being 'frozen'). This is where the answer by @Uwe Plonus comes in.

[Tarnay Kálmán]

@joejoe31b, a few milliseconds of freezing is unlikely to cause dropped requests or a noticeable downtime.

[Matiullah Khan]

@Robbietjuh, @kramer65: note that the aptitude changelog <pkg> prints the full changelog for pkg, so greping will produce a hit if the package has ever had a high urgency update. What you really want to know is if the updates since the last reboot were high urgency, but grep isn't telling you that here.

[Timo]

@UwePlonus do you know if I can postpone the restart? I am not in hurry to restart, and prefer to continue working. There is too much restart in the IT world;)