Rebuilding mdadm RAID array after OS drive failure

Rebuilding mdadm RAID array after OS drive failure

[Steve Kolk]

I have a system that was running Ubuntu 18.04 server with a 5 drive
RAID 6 array using mdadm. I had a hard drive failure, but instead of
the array it was the OS hard drive that completely failed. I also do
not have access to the original mdadm.conf file.

In an attempt to get my array back up, I booted Ubuntu 22.04 from a
USB and tried to assemble and mount the array.  It did not work.
Trying some diagnostic steps I found online, it seems to be claiming
that there is no md superblock on any of the drives.

I've included the commands I ran and their output below. Does anyone
have any advice on what next steps I can take? I have no reason to
think these drives have failed mechanically (much less all of them). I
also tried with Ubuntu 18.04 and got the same results.

ubuntu@ubuntu:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0   2.2G  1 loop /rofs
loop1    7:1    0  32.3M  1 loop /snap/snapd/12883
loop2    7:2    0  55.4M  1 loop /snap/core18/2128
loop3    7:3    0  61.8M  1 loop /snap/core20/1081
loop4    7:4    0  65.1M  1 loop /snap/gtk-common-themes/1515
loop5    7:5    0   219M  1 loop /snap/gnome-3-34-1804/72
loop6    7:6    0 241.4M  1 loop /snap/gnome-3-38-2004/70
loop7    7:7    0   2.5M  1 loop /snap/gnome-calculator/884
loop8    7:8    0   704K  1 loop /snap/gnome-characters/726
loop9    7:9    0   548K  1 loop /snap/gnome-logs/106
loop10   7:10   0   2.5M  1 loop /snap/gnome-system-monitor/163
sda      8:0    0   9.1T  0 disk
├─sda1   8:1    0   9.1T  0 part
└─sda9   8:9    0     8M  0 part
sdb      8:16   0   9.1T  0 disk
├─sdb1   8:17   0   9.1T  0 part
└─sdb9   8:25   0     8M  0 part
sdc      8:32   0   9.1T  0 disk
├─sdc1   8:33   0   9.1T  0 part
└─sdc9   8:41   0     8M  0 part
sdd      8:48   0   9.1T  0 disk
├─sdd1   8:49   0   9.1T  0 part
└─sdd9   8:57   0     8M  0 part
sde      8:64   0   9.1T  0 disk
├─sde1   8:65   0   9.1T  0 part
└─sde9   8:73   0     8M  0 part
sdf      8:80   1   7.6G  0 disk
└─sdf1   8:81   1   7.6G  0 part /cdrom

ubuntu@ubuntu:~$ sudo mdadm --assemble /dev/sda1 /dev/sdb1 /dev/sdc1
/dev/sdd1 /dev/sde1
mdadm: device /dev/sda1 exists but is not an md array.

ubuntu@ubuntu:~$ sudo mdadm --examine --verbose /dev/sd[a-e]1
mdadm: No md superblock detected on /dev/sda1.
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdc1.
mdadm: No md superblock detected on /dev/sdd1.
mdadm: No md superblock detected on /dev/sde1.

ubuntu@ubuntu:~$ sudo mdadm --assemble --scan --verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/loop10
mdadm: no recogniseable superblock on /dev/loop9
mdadm: no recogniseable superblock on /dev/loop8
mdadm: Cannot assemble mbr metadata on /dev/sdf1
mdadm: Cannot assemble mbr metadata on /dev/sdf
mdadm: no recogniseable superblock on /dev/sdb9
mdadm: no recogniseable superblock on /dev/sdb1
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: no recogniseable superblock on /dev/sde9
mdadm: no recogniseable superblock on /dev/sde1
mdadm: Cannot assemble mbr metadata on /dev/sde
mdadm: no recogniseable superblock on /dev/sdd9
mdadm: no recogniseable superblock on /dev/sdd1
mdadm: Cannot assemble mbr metadata on /dev/sdd
mdadm: no recogniseable superblock on /dev/sdc9
mdadm: no recogniseable superblock on /dev/sdc1
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: no recogniseable superblock on /dev/sda9
mdadm: no recogniseable superblock on /dev/sda1
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: no recogniseable superblock on /dev/loop7
mdadm: no recogniseable superblock on /dev/loop6
mdadm: no recogniseable superblock on /dev/loop5
mdadm: no recogniseable superblock on /dev/loop4
mdadm: no recogniseable superblock on /dev/loop3
mdadm: no recogniseable superblock on /dev/loop2
mdadm: no recogniseable superblock on /dev/loop1
mdadm: no recogniseable superblock on /dev/loop0
mdadm: No arrays found in config file or automatically

Steve

Re: Rebuilding mdadm RAID array after OS drive failure

[anthony]

On 16/10/2022 22:38, Steve Kolk wrote:
> I have a system that was running Ubuntu 18.04 server with a 5 drive
> RAID 6 array using mdadm. I had a hard drive failure, but instead of
> the array it was the OS hard drive that completely failed. I also do
> not have access to the original mdadm.conf file.
> 
> In an attempt to get my array back up, I booted Ubuntu 22.04 from a
> USB and tried to assemble and mount the array.  It did not work.
> Trying some diagnostic steps I found online, it seems to be claiming
> that there is no md superblock on any of the drives.

 From this site? https://raid.wiki.kernel.org/index.php/Linux_Raid

Try https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
> 
> I've included the commands I ran and their output below. Does anyone
> have any advice on what next steps I can take? I have no reason to
> think these drives have failed mechanically (much less all of them). I
> also tried with Ubuntu 18.04 and got the same results.
> 
If you've got python 2.7, can you run lsdrv? Hopefully that will give us 
some more clues.

Provided the data itself isn't damaged, you should be able to retrieve 
your array. Read up the section on overlays, that will protect your data 
from being overwritten while you play around trying to recover it.

The other thing I would do here, is five drives is a lot to try and get 
right. I'd pick just three drives, and try to get them to assemble 
correctly (using overlays for safety!). Once you've got three drives to 
give you a functional array, you can then try and get the other two 
correct before the final recovery.

What can you remember about the system? What version of superblock? And 
something I've noticed that seems weird, why do your drives have 
partitions 1 & 9, but not 2-8? That is well odd ... Have you rebuilt the 
array in any way, or is it exactly as originally created? Etc etc the 
more information you can give us the better.

Let us know how you get on with this, and then we'll come back with more.

Cheers,
Wol

Re: Rebuilding mdadm RAID array after OS drive failure

[Steve Kolk]

On Sun, Oct 16, 2022 at 6:22 PM anthony <antmbox@youngman.org.uk> wrote:
>
> On 16/10/2022 22:38, Steve Kolk wrote:
> > I have a system that was running Ubuntu 18.04 server with a 5 drive
> > RAID 6 array using mdadm. I had a hard drive failure, but instead of
> > the array it was the OS hard drive that completely failed. I also do
> > not have access to the original mdadm.conf file.
> >
> > In an attempt to get my array back up, I booted Ubuntu 22.04 from a
> > USB and tried to assemble and mount the array.  It did not work.
> > Trying some diagnostic steps I found online, it seems to be claiming
> > that there is no md superblock on any of the drives.
>
>  From this site? https://raid.wiki.kernel.org/index.php/Linux_Raid
>
> Try https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn

I set up 3 of the drives with overlays and tried forcing the assembly,
but I still get the same error. I haven't tried the steps under
"Irreversible mdadm failure recovery", is there where I need to go
next?

ubuntu@ubuntu:~$ echo $OVERLAYS
/dev/mapper/sda1 /dev/mapper/sdb1 /dev/mapper/sdc1
ubuntu@ubuntu:~$ sudo mdadm --assemble --force /dev/md1 $OVERLAYS
mdadm: no recogniseable superblock on /dev/mapper/sda1
mdadm: /dev/mapper/sda1 has no superblock - assembly aborted

> >
> > I've included the commands I ran and their output below. Does anyone
> > have any advice on what next steps I can take? I have no reason to
> > think these drives have failed mechanically (much less all of them). I
> > also tried with Ubuntu 18.04 and got the same results.
> >
> If you've got python 2.7, can you run lsdrv? Hopefully that will give us
> some more clues.

ubuntu@ubuntu:~$ ./dev/lsdrv/lsdrv
PCI [ata_piix] 00:1f.2 IDE interface: Intel Corporation 6 Series/C200
Series Chipset Family Desktop SATA Controller (IDE mode, ports 0-3)
(rev 05)
├scsi 0:0:0:0 ATA      WDC WD100EFAX-68
│└sda 9.10t [8:0] Empty/Unknown
│ ├sda1 9.10t [8:1] Empty/Unknown
│ │└dm-2 9.10t [253:2] Empty/Unknown
│ └sda9 8.00m [8:9] Empty/Unknown
├scsi 0:0:1:0 ATA      WDC WD100EFAX-68
│└sdb 9.10t [8:16] Empty/Unknown
│ ├sdb1 9.10t [8:17] Empty/Unknown
│ │└dm-0 9.10t [253:0] Empty/Unknown
│ └sdb9 8.00m [8:25] Empty/Unknown
├scsi 1:0:0:0 ATA      WDC WD100EFAX-68
│└sdc 9.10t [8:32] Empty/Unknown
│ ├sdc1 9.10t [8:33] Empty/Unknown
│ │└dm-1 9.10t [253:1] Empty/Unknown
│ └sdc9 8.00m [8:41] Empty/Unknown
└scsi 1:0:1:0 ATA      WDC WD100EFAX-68
 └sdd 9.10t [8:48] Empty/Unknown
  ├sdd1 9.10t [8:49] Empty/Unknown
  └sdd9 8.00m [8:57] Empty/Unknown
PCI [ata_piix] 00:1f.5 IDE interface: Intel Corporation 6 Series/C200
Series Chipset Family Desktop SATA Controller (IDE mode, ports 4-5)
(rev 05)
└scsi 2:0:0:0 ATA      WDC WD100EFAX-68
 └sde 9.10t [8:64] Empty/Unknown
  ├sde1 9.10t [8:65] Empty/Unknown
  └sde9 8.00m [8:73] Empty/Unknown
PCI [pata_via] 03:00.0 IDE interface: VIA Technologies, Inc. VT6415
PATA IDE Host Controller
└scsi 4:x:x:x [Empty]
USB [usb-storage] Bus 002 Device 003: ID 0930:6545 Toshiba Corp.
Kingston DataTraveler 102/2.0 / HEMA Flash Drive 2 GB / PNY Attache
4GB Stick {000AEBFFB4C15B8A130501A6}
└scsi 6:0:0:0 Kingston DataTraveler 2.0
 └sdf 7.57g [8:80] Empty/Unknown
  └sdf1 7.57g [8:81] Empty/Unknown
   └Mounted as /dev/sdf1 @ /cdrom
Other Block Devices
├loop0 2.13g [7:0] Empty/Unknown
│└Mounted as /dev/loop0 @ /rofs
├loop1 61.96m [7:1] Empty/Unknown
│└Mounted as /dev/loop1 @ /snap/core20/1587
├loop2 4.00k [7:2] Empty/Unknown
│└Mounted as /dev/loop2 @ /snap/bare/5
├loop3 163.29m [7:3] Empty/Unknown
│└Mounted as /dev/loop3 @ /snap/firefox/1635
├loop4 91.69m [7:4] Empty/Unknown
│└Mounted as /dev/loop4 @ /snap/gtk-common-themes/1535
├loop5 400.80m [7:5] Empty/Unknown
│└Mounted as /dev/loop5 @ /snap/gnome-3-38-2004/112
├loop6 46.96m [7:6] Empty/Unknown
│└Mounted as /dev/loop6 @ /snap/snapd/16292
├loop7 284.00k [7:7] Empty/Unknown
│└Mounted as /dev/loop7 @ /snap/snapd-desktop-integration/14
├loop8 45.86m [7:8] Empty/Unknown
│└Mounted as /dev/loop8 @ /snap/snap-store/582
├loop9 9.09t [7:9] Empty/Unknown
│└dm-2 9.10t [253:2] Empty/Unknown
├loop10 9.09t [7:10] Empty/Unknown
│└dm-0 9.10t [253:0] Empty/Unknown
├loop11 9.09t [7:11] Empty/Unknown
│└dm-1 9.10t [253:1] Empty/Unknown
├loop12 0.00k [7:12] Empty/Unknown
└loop13 0.00k [7:13] Empty/Unknown

> Provided the data itself isn't damaged, you should be able to retrieve
> your array. Read up the section on overlays, that will protect your data
> from being overwritten while you play around trying to recover it.
>
> The other thing I would do here, is five drives is a lot to try and get
> right. I'd pick just three drives, and try to get them to assemble
> correctly (using overlays for safety!). Once you've got three drives to
> give you a functional array, you can then try and get the other two
> correct before the final recovery.
>
> What can you remember about the system? What version of superblock? And
> something I've noticed that seems weird, why do your drives have
> partitions 1 & 9, but not 2-8? That is well odd ... Have you rebuilt the
> array in any way, or is it exactly as originally created? Etc etc the
> more information you can give us the better.

I do not know the version of superblock, after the disks were
partitioned I just used "sudo mdadm --create ..." and everything went
smoothly. I didn't use any special configuration as far as I know, so
I'm not sure why the partitioning is odd. I have not rebuilt the
array, it still has the same disks as when it was created and it has
not suffered any failures.

>
> Let us know how you get on with this, and then we'll come back with more.
>
> Cheers,
> Wol

Steve

Re: Rebuilding mdadm RAID array after OS drive failure

[Wols Lists]

On 17/10/2022 23:46, Steve Kolk wrote:
> I set up 3 of the drives with overlays and tried forcing the assembly,
> but I still get the same error. I haven't tried the steps under
> "Irreversible mdadm failure recovery", is there where I need to go
> next?
> 
> ubuntu@ubuntu:~$ echo $OVERLAYS
> /dev/mapper/sda1 /dev/mapper/sdb1 /dev/mapper/sdc1
> ubuntu@ubuntu:~$ sudo mdadm --assemble --force /dev/md1 $OVERLAYS
> mdadm: no recogniseable superblock on /dev/mapper/sda1
> mdadm: /dev/mapper/sda1 has no superblock - assembly aborted

Sounds like you haven't understood what you're supposed to do.

The whole point of overlays is it is meant to protect the underlying 
device while you do stuff that could be destructive.

What you should be doing is setting up the overlays, then running "mdadm 
--create". It sounds like if you run exactly the same command as when 
you created the array, you'll get the original back.

Read up carefully on overlays to make sure you really understand what 
you're doing. And then when you create the array with the overlays, if 
you're doing as I suggested with three drives, make sure you tell it 
it's a five drive array, and tell it two of the drives are missing.

I'm just going to work, but whatever you do, DO NOT DO ANYTHING THAT 
WILL ACTUALLY WRITE TO THE DRIVES.

Cheers,
Wol