mdadm raid 1 grub only on sda

Question

I just finished setting up a CentOS 6.3 64bit server with mdadm however a lightbulb went on and I realised grub would only be installed on the first drive which is about as much use and an ashtray on a motorbike.

I had a look to confirm my suspicion:

grub> find /grub/stage1
find /grub/stage1
 (hd0,0)

So I updated my device map to look like:

(fd0)   /dev/fd0
(hd0)   /dev/sda
(hd1)   /dev/sdb

(Note the (hd1) entry was added by me

so then I tried to install grub on /dev/sdb

And I get:

grub> root (hd1,0)
root (hd1,0)
 Filesystem type is ext2fs, partition type 0x83
grub> setup (hd1)
setup (hd1)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... no

Error 15t: File not found

So I did some googling (Sadly google has just done a good job and picked up 100's of grub install examples which dont help here)

After finding a few clues I tried:

# grub-install --recheck /dev/sdb

Probing devices to guess BIOS drives. This may take a long time.
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.

(fd0)   /dev/fd0
(hd0)   /dev/sda
(hd1)   /dev/sdb

# grub-install /dev/sdb
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.

(fd0)   /dev/fd0
(hd0)   /dev/sda
(hd1)   /dev/sdb

Which sort of suggests grub is now installed on /dev/sdb too, however if I take another look I still get:

grub> find /grub/stage1
find /grub/stage1
 (hd0,0)

parted outputs for the 2 drives:

SDA

Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
 1      17.4kB  500MB   500MB   ext3         1     boot
 2      500MB   81.0GB  80.5GB               2     raid
 3      81.0GB  85.0GB  4000MB               3     raid
 4      85.0GB  3001GB  2916GB               4     raid

SDB

Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
 1      17.4kB  500MB   500MB   ext3         1
 2      500MB   81.0GB  80.5GB               2     raid
 3      81.0GB  85.0GB  4000MB               3     raid
 4      85.0GB  3001GB  2916GB               4     raid

And mdadm mdstat:

Personalities : [raid1]
md1 : active raid1 sdb3[1] sda3[0]
      3905218 blocks super 1.1 [2/2] [UU]

md2 : active raid1 sdb4[1] sda4[0]
      2847257598 blocks super 1.1 [2/2] [UU]

md0 : active raid1 sda2[0] sdb2[1]
      78612189 blocks super 1.1 [2/2] [UU]

Is anyone able to throw some light on the situation, it feels like I am 99% there at the moment and missing something obvious.

Thanks.

edit update:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0               74G   18G   53G  25% /
tmpfs                 580M     0  580M   0% /dev/shm
/dev/sda1             462M   98M  341M  23% /boot
xenstore              580M   64K  580M   1% /var/lib/xenstored

/ is on md0 which is made up of sda2 and sdb2 swap is md1 which is sda3 and sdb3 md2 is LVM however /boot is only on /sda1

I suppose that is the problem, would the resolution be to create md4 and have it contain sda1 and sdb1

Perhaps I have things mixed up a little in my head but I assumed grub was not installed on a partition but the first few blocks of the drive i.e. sda or hd0/1

Any clarification and advice is appreciated.

Solution 1

This should be your problem

root (hd1,0)
 Filesystem type is ext2fs, partition type 0x83

Take the following steps:

• Create the 2 /boot partitions on /dev/sda1 and /dev/sdb1 - type fd(Linux autodetect raid) - use your favorite tool(fdisk, cfdisk, gparted,...) (fd00 for GPT)
• Remember to turn on the bootable flag on both partitions, sda1 and sdb1 (not for GPT)

• Force the disks to be a brand new raid:

  mdadm --zero-superblock /dev/sda1 
  mdadm --zero-superblock /dev/sdb1
  

• While creating the raid metadata that will be your /boot partition, use the version 0.9. Linux cannot autodetect newer versions (without a ramdisk).

  mdadm --create /dev/md0 --level=1 --raid-disks=2 /dev/sda1 /dev/sdb1 --metadata=0.9
  

• Format using ext2 or ext3

• Install your Linux of choice, WITHOUT formating the /boot

After your distro first boot:

• Fix your /etc/fstab to point /boot to /dev/md0(maybe it will not be necessary)

• Install grub on the 2 disks MBR

  # grub /dev/sda
   grub> root (hd0,0)
   grub> setup (hd0)
   grub> quit
   quit
  
  # grub /dev/sdb
   grub> root (hd1,0)
   grub> setup (hd1)
   grub> quit
   quit
  

• Edit your bootloader(instructions to Grub1)

• Search the "default" line and add the "fallback" option bellow

  vi /boot/grub/menu.lst
  default 0
  fallback 1
  

• Add another entry to your bootloader(again, in my case i've choosen grub1 since its less complicated and it's good enough to my needs), one of each pointing to the different boot partitions that are members of the raid:

  title           Debian GNU/Linux, kernel 2.6.32-5-686  (default)
  root            (hd0,0)
  kernel          /vmlinuz-2.6.32-5-686 root=/dev/mapper/vg-root ro quiet
  initrd          /initrd.img-2.6.32-5-686
  
  title           Debian GNU/Linux, kernel 2.6.32-5-686  (fallback)
  root            (hd1,0)
  kernel          /vmlinuz-2.6.32-5-686 root=/dev/mapper/vg-root ro quiet
  initrd          /initrd.img-2.6.32-5-686 
  

• Note that in my case, i have a LVM layer on my / md raid.

Done. This should be enough to you to have a "redundant" bootloader.

Solution 2

Whether you want to map those two drives onto another raid1 array is up to you. It's a legitimate choice because you would then be able to install grub to md4 directly and avoid the hustle of managing both installations manually. I've done it™.

In my experience, the device map file is largely irrelevant and its behavior (how grub reads it) is at best unpredictable, if not outright arbitrary.

The device command from the grub shell is a lot more reliable. You can read the help info on it, but the basic syntax speaks for itself:

grub> device (hd0) /dev/md4

After that, /dev/md4 will be mapped to hd0 in the currently running grub session, disregarding the device map file. From here, one would proceed as usual with root (hd0) and setup (hd0,x). The reason for installing to a partition is explained by Henry S.

AFAIK, the only thing apart from the "boot code" (stage1) that is written to the first 512 bytes of the drive (MBR) is the number of partition where that stage should look for the menu.

It's possible to mess that up. One would be greeted with a prompt instead of a menu after POST, but grub provides commands for "initializing" that menu from a different partition (file) than the once specified under installation. It generally works out without intervention though, because the "first boot drive" in bios will be detected as "hd0".

Comments

[Henry S.]

Can you confim, that sda1 and sdb1 do contain the same files (incl. kernel, initrd, grub stages etc.), please?

[Backtogeek]

Perhaps the initial setup on single disk is the cause, I will update the original post with more info

[Henry S.]

IIRC grub would also load from a partition marked as raid. So if you create this md4, you keep the two partitions in sync. But i think you still have to run grub-install /dev/sdX for the separate disks. Because there is no md device at boot time. The bootfloader of grub is installed in the first blocks of the disk but the later stages are stored in the /boot _partition. Because all those fancy menues do not fit in the space before the partition table.

[244boy]

Or, since /boot is pretty static, you could just create 2 partitions(sda1 and sdb1), sync them daily with rsync. install grub on both disks manually and use the default/fallback trick.

[GeoSword]

Rsync?!! That's a horrible idea! Let raid 1 do its thing! It sounds like the problem here is setting up raid1 AFTER the centos install has completed. Ive just finished setting up gentoo with raid 1 on the boot partition, but because raid is setup BEFORE the installation occurs, grub sees one device, and Linux RAID installs the bootloader to both disks.