Replacing hard drive

I noticed one of the hard drives on my main Hetzner box was throwing some new errors, indicating bad blocks that couldn’t be reallocated.

I must say I was rather impressed with Hetzner in the process, and the whole thing was very straight forward

Preparatory work

Though the drive was failing, it was still part of the RAID array. So to remove it from the array, steps were

# First, fail it out of the array
mdadm --manage /dev/md0 --fail /dev/sdb1
mdadm --manage /dev/md1 --fail /dev/sdb2
mdadm --manage /dev/md2 --fail /dev/sdb3
mdadm --manage /dev/md3 --fail /dev/sdb4
# Next, remove it from the RAID array
mdadm /dev/md0 -r /dev/sdb1
mdadm /dev/md1 -r /dev/sdb2
mdadm /dev/md2 -r /dev/sdb3
mdadm /dev/md3 -r /dev/sdb4

Hetzner has some good resources that you’re linked to while making the online support request:

• Serial number retrieval and general advice
• Working with software RAID and HDDs

# Backup partition tables (gpt in this example)
sgdisk --backup=sda_parttable_gpt.bak /dev/sda
sgdisk --backup=sdb_parttable_gpt.bak /dev/sdb

At which point I notified Hetzner to replace the disk, which they had done in about half an hour (during the night time!). After the system is rebooted, I could see it was the same make/model of drive as originally, so easy enough to proceed…

# Restore the old partition table.
sgdisk --load-backup=sdb_parttable_gpt.bak /dev/sdb
# Randomize the disk's GUIDs
sgdisk -G /dev/sdb
# Add the new partitions into the RAID array
mdadm /dev/md0 -a /dev/sdb1
mdadm /dev/md1 -a /dev/sdb2
mdadm /dev/md2 -a /dev/sdb3
mdadm /dev/md3 -a /dev/sdb4
# Then bootloader stuff
grub-mkdevicemap -n
grub-install /dev/sdb

After that, I leave it working away in the background while the new partitions are sequentially updated into the RAID array. All very straightforward!