Creating Persistent Drive Labels With UDEV Using /dev/disk/by-path
My Solution
I'm not sure it's the best solution, and I'm still interested in other answers, but here's what I did.
I could not figure out how to force it to map based on the SCSI path (e.g. pci-0000:02:04.0-scsi-0:0:0:0). So instead I am mapping based on the UUID, which is the drive models and serial numbers. I just have to remember that when I swap drives I have to update the UDEV entries.
Finding the UUIDs
Used entries from the following commands to identify the current drive names, what SCSI PATH they were mapped to, and what their UUIDs were.
# ls -l /dev/disk/by-path
# /lib/udev/scsi-id -gu /dev/sdX
Created UDEV entries
I put entries like this into a new file I called /etc/udev/rules.d/59-persistent-disk.rules. There are three rows of Three backplanes each so I did something like this.
# ROW 1 BACKPLANE 1
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0076306", SYMLINK+="pci24scsi00"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0073437", SYMLINK+="pci24scsi01"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0073085", SYMLINK+="pci24scsi02"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0072374", SYMLINK+="pci24scsi03"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0071564", SYMLINK+="pci24scsi04"
# ROW 1 BACKPLANE 2
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0077571", SYMLINK+="pci24scsi20"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0077346", SYMLINK+="pci24scsi21"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0074773", SYMLINK+="pci24scsi22"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0071238", SYMLINK+="pci24scsi23"
KERNEL=="sd*", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id --whitelisted --replace-whitespace /dev/$name", RESULT=="1ATA_WDC_WD30EFRX-68AX9N0_WD-WMC1T0076264", SYMLINK+="pci24scsi24"
.......etc
Creating the Arrays
Now that I have persistent device names I created five 9 drive arrays (RAID5 with one spare) as follows. I used 1 drive per backplane in each array so I could lose an entire backplane (or even two) and the arrays would stay running. Performance may suffer for doing it this way, but I bought this for archiving and offsite backups so performance isn't the primary concern.
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=8 --spare-devices=1 /dev/pci24scsi00 /dev/pci24scsi20 /dev/pci24scsi30 /dev/pci44scsi00 /dev/pci44scsi20 /dev/pci44scsi30 /dev/pci64scsi00 /dev/pci64scsi20 /dev/pci64scsi30
mdadm --create --verbose /dev/md1 --level=5 --raid-devices=8 --spare-devices=1 /dev/pci24scsi01 /dev/pci24scsi21 /dev/pci24scsi31 /dev/pci44scsi01 /dev/pci44scsi21 /dev/pci44scsi31 /dev/pci64scsi01 /dev/pci64scsi21 /dev/pci64scsi31
mdadm --create --verbose /dev/md2 --level=5 --raid-devices=8 --spare-devices=1 /dev/pci24scsi02 /dev/pci24scsi22 /dev/pci24scsi32 /dev/pci44scsi02 /dev/pci44scsi22 /dev/pci44scsi32 /dev/pci64scsi02 /dev/pci64scsi22 /dev/pci64scsi32
mdadm --create --verbose /dev/md3 --level=5 --raid-devices=8 --spare-devices=1 /dev/pci24scsi03 /dev/pci24scsi23 /dev/pci24scsi33 /dev/pci44scsi03 /dev/pci44scsi23 /dev/pci44scsi33 /dev/pci64scsi03 /dev/pci64scsi23 /dev/pci64scsi33
mdadm --create --verbose /dev/md4 --level=5 --raid-devices=8 --spare-devices=1 /dev/pci24scsi04 /dev/pci24scsi24 /dev/pci24scsi34 /dev/pci44scsi04 /dev/pci44scsi24 /dev/pci44scsi34 /dev/pci64scsi04 /dev/pci64scsi24 /dev/pci64scsi34
Using LVM to Create VGs and LVs
These were my LVM steps to create the volumes so they could be iSCSI mounted remotely.
PVCREATE
pvcreate /dev/md0
pvcreate /dev/md1
pvcreate /dev/md2
pvcreate /dev/md3
pvcreate /dev/md4
VGCREATE
vgcreate VgArr0 /dev/md0
vgcreate VgArr1 /dev/md1
vgcreate VgArr2 /dev/md2
vgcreate VgArr3 /dev/md3
vgcreate VgArr4 /dev/md4
vgcreate VgArr5 /dev/md5
LVCREATE for iSCSI
Create the LVs but do not format or mount the volumes. They get formatted/mounted remotely
lvcreate -L100%FREE VgArr0 -n LvISCSI0
iSCSI
Edit the /etc/iet/ietd.conf and /etc/iet/initiators.conf
Map the iSCSI LUNs remotely on the other host
Matt Mencel
Updated on September 18, 2022Comments
-
Matt Mencel almost 2 years
I have a new BackBlaze Pod (BackBlaze Pod 2.0).
It has 45 3TB drives and they when I first set it up they were labeled /dev/sda through /dev/sdz and /dev/sdaa through /dev/sdas.
I used mdadm to setup three really big 15 drive RAID6 arrays.
However, since first setup a few weeks ago I had a couple of the hard drives fail on me. I've replaced them but now the arrays are complaining because they can't find the missing drives. When I list the the disks...
ls -l /dev/sd*
I see that
/dev/sda /dev/sdf /dev/sdk /dev/sdp
no longer appear and now there are 4 new ones...
/dev/sdau /dev/sdav /dev/sdaw /dev/sdax
I also just found that I can do this...
ls -l /dev/disk/by-path/ total 0 lrwxrwxrwx 1 root root 10 Sep 19 18:08 pci-0000:02:04.0-scsi-0:0:0:0 -> ../../sdau lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-0:1:0:0 -> ../../sdb lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-0:2:0:0 -> ../../sdc lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-0:3:0:0 -> ../../sdd lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-0:4:0:0 -> ../../sde lrwxrwxrwx 1 root root 10 Sep 19 18:08 pci-0000:02:04.0-scsi-2:0:0:0 -> ../../sdae lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-2:1:0:0 -> ../../sdg lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-2:2:0:0 -> ../../sdh lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-2:3:0:0 -> ../../sdi lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-2:4:0:0 -> ../../sdj lrwxrwxrwx 1 root root 10 Sep 19 18:08 pci-0000:02:04.0-scsi-3:0:0:0 -> ../../sdav lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-3:1:0:0 -> ../../sdl lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-3:2:0:0 -> ../../sdm lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-3:3:0:0 -> ../../sdn lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:02:04.0-scsi-3:4:0:0 -> ../../sdo lrwxrwxrwx 1 root root 10 Sep 19 18:08 pci-0000:04:04.0-scsi-0:0:0:0 -> ../../sdax lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:04:04.0-scsi-0:1:0:0 -> ../../sdq lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:04:04.0-scsi-0:2:0:0 -> ../../sdr lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:04:04.0-scsi-0:3:0:0 -> ../../sds lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:04:04.0-scsi-0:4:0:0 -> ../../sdt lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:04:04.0-scsi-2:0:0:0 -> ../../sdu lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:04:04.0-scsi-2:1:0:0 -> ../../sdv lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:04:04.0-scsi-2:2:0:0 -> ../../sdw lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:04:04.0-scsi-2:3:0:0 -> ../../sdx lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:04:04.0-scsi-2:4:0:0 -> ../../sdy lrwxrwxrwx 1 root root 9 Sep 19 18:08 pci-0000:04:04.0-scsi-3:0:0:0 -> ../../sdz
I didn't list them all....you can see the problem above. They're sorted by scsi id here but sda is missing...replaced by sdau...etc...
So obviously the arrays are complaining. Is it possible to get Linux to reread the drive labels in the correct order or am I screwed?
My initial design with 15 drive arrays is not ideal. With 3TB drives the rebuild times were taking 3 or 4 days....maybe more. I'm scrapping the whole design and I think I am going to go with 6 x 7 RAID5 disk arrays and 3 hot spares to make the arrays a bit easier to manage and shorten the rebuild times. But I'd like to clean up the drive labels so they aren't out of order. I haven't figured out how to do this yet.
Does anyone know how to get this straightened out?
Thanks,
Matt
-
Matt Mencel almost 12 yearsLooks like I can do this using persistent naming in udev. I'm trying to find the right way to do this using the scsi id....so no matter what...if I replace a new drive in the /dev/sda scsi slot....it is still labled /dev/sda.
-