How to setup linux server for headless use?

Question

Consider a headless server like this: A typical x86 box at a remote location, which you can remotely initialize with a stock - say - Ubuntu image. After it is initialized you only can login via ssh - or remotely reset it, i.e. you can't access the BIOS or the boot manager prompt (say Grub 1).

Perhaps some kind of KVM is available, but the use of KVM is very expensive and you have to book it on a hourly basis.

Given this scenario, one can get paranoid about boot problems. For example:

1. What if a kernel upgrade fails?
2. what about a fsck-prompt in the early boot process? Probably, ssh is not available yet ...

Are there other gotchas to watch out for?

For kernel upgrades I configure grub (the legacy one) such that the menu.lst preamble contains

default saved
fallback 2  # counts from 0

and the first entry ends with:

savedefault fallback

The first grub entry is the upgraded kernel, and the third is a known working one. See also the grub manual section on fallback boot.

I changed the startup script /etc/rc.local (on a Debian-like system) to the effect that the default-entry setting is reset in case of a successful boot:

grub-set-default 0

This grub-setup works, but e.g. on Ubuntu this is not the default and one has to manually adjust the menu.lst after each kernel update.

I supply

panic=60

as kernel parameter such that, e.g. in case of a wrong root= parameter or broken kernel, the system automatically reboots in case of an error.

About the fsck issue I am not sure what the best way is. On Debian-like systems you can set

FSCKFIX=yes

in /etc/default/rcS, which tells fsck to auto-repair by default.

But if the auto-repair fails, perhaps I still get a prompt I can't access remotely?

Alternatively I could just disable fsck-checking via a zero in the sixth column of /etc/fstab - in case of a fs-error would then just re-initialize the system and restore the backups - thus avoiding all the fsck trouble?

Solution 1

Seriously, if your provider does not offer free (or at least cheap) manual assistance for extreme cases, it's time to switch. Otherwise, I think that you are pretty much OK with your setup.

When your system is so broken that fsck can't fix it, there isn't much else to do, other than a complete reinstall. I actually haven't seen this happen unless there was a fatal hardware failure.

One thing to note. For a machine like this, choose a stable distribution (Debian, RHEL, SLES), and definitely upgrade only after a suitably long period (new version stabilized for at least 6 months).

Solution 2

You should be looking for a hosting provider who will provider serial-over-ssh access and configure your Linux installation to use the (relevant) serial port as the the console (how you do this depends on whether the system uses upstart or sysV type initialization). Note that there are BIOS available which will talk to a serial port rather than the built in screen device. But usually they only come on expensive hardware.

You also need to tell grub to use the serial port if you want to control it via a DTE.

Solution 3

Something that you could look into is making a custom initrd that will include dropbear (running on another port, of course), enough logic to get your networking going and perhaps a way to load some recovery tools if required. Based on this you could then make a recovery kernel config which will load up with network capability and allow you to ssh in, allowing you to get back onto the system and attempt a recovery.

Comments

[Rufo El Magufo]

Very good question. A bare metal server without a remote console or one expensive is a big problem.

[maxschlepzig]

Yes, sound like quite a project. I could even imagine creating a small Linux system that is always booted first and acts like a boot manager (while providing ssh-access and screen) - it could then boot the real kernel via techniques like en.wikipedia.org/wiki/Kexec. Or one can look into servers that come with coreboot.org instead of some crappy BIOS from the 80ies. But sure, all this is nothing you can reliably setup and maintain in a few hours - at this point of time with a stable distribution.

[froztbyte]

Looks like you'll be able to skimp some of the effort by looking at this page