Superblocks lost on all disks in array.

Superblocks lost on all disks in array.

[Jean-Pierre Human]

Hi

I have a 4 disk Raid10 array in an Ubuntu16.04.2 Server (/dev/md2)
which seems to have lost the superblocks on all 4 disks.

The server has two other arrays in it a 3 disk Raid5 (/dev/md1) and
another 4 disk Raid10 (/dev/md0), these arrays are fine. This is not
due to mechanical failure or disk issues. The array did not come up
after a reboot of the server.

A bit more info, the failed array /dev/md2 was 100% assigned to a
logical volume which was shared via ISCSI to a Microsoft server. I
mention this as I have a feeling this could somehow be the reason the
superblocks disappeared? These disks were part of a ZFS pool before
this, I see lsdrv picks this up...

As there are no superblocks I cannot assemble the array. I feel my
only option is to create --assume-clean the array to get new
superblocks written. The server is fairly new and software versions
etc have not changed. I have the original commands I used to create
the array which have the disk order in them.

What I am looking for is any advice on how I should proceed.
Presuming I must --create,  Should I use the "file overlay" method to
allow me to roll back? I do not really understand how this works and
if it is relevant for my situation.

Would anyone know what could have caused this?

System info below:
Ubuntu 16.04.2 LTS

The array was setup with the below commands:
#mdadm --create --verbose /dev/md2 --level=10 --raid-devices=4
/dev/sdj /dev/sdk /dev/sdl /dev/sdi
#update-initramfs -u

smartctl --xall produced too much output please let me know if you
need this. I see no errors or issues in this output.

root@store02:~# mdadm --examine /dev/sd[iklj]
/dev/sdi:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdj:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdk:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdl:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)

root@store02:~# mdadm -Q /dev/sd[iklj]
/dev/sdi: is not an md array
/dev/sdj: is not an md array
/dev/sdk: is not an md array
/dev/sdl: is not an md array

root@store02:~# cat /proc/mdstat
Personalities : [raid10] [raid6] [raid5] [raid4] [linear] [multipath]
[raid0] [raid1]
md0 : active raid10 sdd[3] sdb[2] sde[1] sdc[0]
      7813774336 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      bitmap: 2/59 pages [8KB], 65536KB chunk

md1 : active raid5 sdf[1] sdg[3] sda[0]
      3906766848 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>


root@store02:~/lsdrv# ./lsdrv
PCI [mpt3sas] 02:00.0 Serial Attached SCSI controller: LSI Logic /
Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
├scsi 0:0:0:0 ATA      WDC WD20EFRX-68E {WD-WCC4M2702612}
│└sda 1.82t [8:0] MD raid5 (0/3) (w/ sdf,sdg) in_sync 'rep01:1'
{de8e78f6-509b-1457-fdd9-a86760d19963}
│ └md1 3.64t [9:1] MD v1.2 raid5 (3) clean, 512k Chunk
{de8e78f6:509b1457:fdd9a867:60d19963}
│                  ext4 {50ee08e4-fa03-49a5-a1c8-e123322dfd12}
├scsi 0:0:1:0 ATA      WDC WD4000F9YZ-0 {WD-WMC5D0D0W9D9}
│└sdb 3.64t [8:16] MD raid10,near2 (2/4) (w/ sdc,sdd,sde) in_sync
'store02:0' {5552ee9a-b20b-3ad2-0e72-f2011a114ae7}
│ └md0 7.28t [9:0] MD v1.2 raid10,near2 (4) clean, 512k Chunk
{5552ee9a:b20b3ad2:0e72f201:1a114ae7}
│  │               ext4 {bd712370-a142-4e7a-8539-0d0e60c464c3}
│  └Mounted as /dev/md0 @ /mnt/md0
├scsi 0:0:2:0 ATA      WDC WD4000F9YZ-0 {WD-WMC1F0D9LMDJ}
│└sdc 3.64t [8:32] MD raid10,near2 (0/4) (w/ sdb,sdd,sde) in_sync
'store02:0' {5552ee9a-b20b-3ad2-0e72-f2011a114ae7}
│ └md0 7.28t [9:0] MD v1.2 raid10,near2 (4) clean, 512k Chunk
{5552ee9a:b20b3ad2:0e72f201:1a114ae7}
│                  ext4 {bd712370-a142-4e7a-8539-0d0e60c464c3}
├scsi 0:0:3:0 ATA      WDC WD4000F9YZ-0 {WD-WMC5D0D7WP6N}
│└sdd 3.64t [8:48] MD raid10,near2 (3/4) (w/ sdb,sdc,sde) in_sync
'store02:0' {5552ee9a-b20b-3ad2-0e72-f2011a114ae7}
│ └md0 7.28t [9:0] MD v1.2 raid10,near2 (4) clean, 512k Chunk
{5552ee9a:b20b3ad2:0e72f201:1a114ae7}
│                  ext4 {bd712370-a142-4e7a-8539-0d0e60c464c3}
├scsi 0:0:4:0 ATA      WDC WD4000F9YZ-0 {WD-WMC5D0D1R7E0}
│└sde 3.64t [8:64] MD raid10,near2 (1/4) (w/ sdb,sdc,sdd) in_sync
'store02:0' {5552ee9a-b20b-3ad2-0e72-f2011a114ae7}
│ └md0 7.28t [9:0] MD v1.2 raid10,near2 (4) clean, 512k Chunk
{5552ee9a:b20b3ad2:0e72f201:1a114ae7}
│                  ext4 {bd712370-a142-4e7a-8539-0d0e60c464c3}
├scsi 0:0:5:0 ATA      WDC WD20EFRX-68A {WD-WCC1T0684483}
│└sdf 1.82t [8:80] MD raid5 (1/3) (w/ sda,sdg) in_sync 'rep01:1'
{de8e78f6-509b-1457-fdd9-a86760d19963}
│ └md1 3.64t [9:1] MD v1.2 raid5 (3) clean, 512k Chunk
{de8e78f6:509b1457:fdd9a867:60d19963}
│                  ext4 {50ee08e4-fa03-49a5-a1c8-e123322dfd12}
├scsi 0:0:6:0 ATA      WDC WD20EFRX-68A {WD-WCC1T0681728}
│└sdg 1.82t [8:96] MD raid5 (2/3) (w/ sda,sdf) in_sync 'rep01:1'
{de8e78f6-509b-1457-fdd9-a86760d19963}
│ └md1 3.64t [9:1] MD v1.2 raid5 (3) clean, 512k Chunk
{de8e78f6:509b1457:fdd9a867:60d19963}
│                  ext4 {50ee08e4-fa03-49a5-a1c8-e123322dfd12}
└scsi 0:x:x:x [Empty]
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
├scsi 1:0:0:0 ATA      Maximus-16GB     {152403701000}
│└sdh 14.75g [8:112] zfs_member
│ ├sdh1 14.34g [8:113] ext4 {5b7ca57a-9540-47fa-9146-c413d649c5ba}
│ │└Mounted as /dev/sdh1 @ /
│ └sdh2 416.00m [8:114] swap {bfe8ede9-1ca4-4519-9916-3a894708f6a2}
├scsi 2:x:x:x [Empty]
├scsi 3:0:0:0 ATA      ST6000NM0115-1YZ {ZAD0RGY5}
│└sdi 5.46t [8:128] Partitioned (gpt)
├scsi 4:0:0:0 ATA      ST6000NM0115-1YZ {ZAD0RGCH}
│└sdj 5.46t [8:144] zfs_member
├scsi 5:0:0:0 ATA      ST6000NM0115-1YZ {ZAD0RHN3}
│└sdk 5.46t [8:160] zfs_member
└scsi 6:0:0:0 ATA      ST6000NM0115-1YZ {ZAD0RGQ6}
 └sdl 5.46t [8:176] Partitioned (gpt)
Other Block Devices
├loop0 0.00k [7:0] Empty/Unknown
├loop1 0.00k [7:1] Empty/Unknown
├loop2 0.00k [7:2] Empty/Unknown
├loop3 0.00k [7:3] Empty/Unknown
├loop4 0.00k [7:4] Empty/Unknown
├loop5 0.00k [7:5] Empty/Unknown
├loop6 0.00k [7:6] Empty/Unknown
└loop7 0.00k [7:7] Empty/Unknown

Any help or advice is appreciated.

Regards
J-P Human

Re: Superblocks lost on all disks in array.

[Andreas Klauer]

On Sat, Jul 15, 2017 at 07:09:08PM +0200, Jean-Pierre Human wrote:
> The array was setup with the below commands:
> #mdadm --create --verbose /dev/md2 --level=10 --raid-devices=4
> /dev/sdj /dev/sdk /dev/sdl /dev/sdi

So you didn't use a partition table?

> root@store02:~# mdadm --examine /dev/sd[iklj]
> /dev/sdi:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> /dev/sdj:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> /dev/sdk:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> /dev/sdl:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)

And then "something" created one. Is that partition table empty? 
Or did it also create and format partitions, that would be worse.

GPT partition table overwrites a bunch of sectors at both start and end. 
So that's where you'll find corruption, depending which mdadm metadata 
version you were using (which can also be located either start or end).

To recover, you'll have to determine the correct RAID level / layout / 
order / data offset. It's best to do this with overlays 

https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file

and only write to the real disks once you've found the setting that works.

In the future, consider always using a partition table. Linux doesn't care *
but the partition table is the most standard way to declare a disk is 
already in use and for what. Without a partition table, any software not 
md-raid aware will see your drive as free, unused, and might format it.

(*) it will happily run anything you like on bare disks
    but it won't do anything to protect you, either

Regards
Andreas Klauer

Re: Superblocks lost on all disks in array.

[Jean-Pierre Human]

On Sat, Jul 15, 2017 at 7:26 PM, Andreas Klauer
<Andreas.Klauer@metamorpher.de> wrote:
> On Sat, Jul 15, 2017 at 07:09:08PM +0200, Jean-Pierre Human wrote:
>> The array was setup with the below commands:
>> #mdadm --create --verbose /dev/md2 --level=10 --raid-devices=4
>> /dev/sdj /dev/sdk /dev/sdl /dev/sdi
>
> So you didn't use a partition table?
>
>> root@store02:~# mdadm --examine /dev/sd[iklj]
>> /dev/sdi:
>>    MBR Magic : aa55
>> Partition[0] :   4294967295 sectors at            1 (type ee)
>> /dev/sdj:
>>    MBR Magic : aa55
>> Partition[0] :   4294967295 sectors at            1 (type ee)
>> /dev/sdk:
>>    MBR Magic : aa55
>> Partition[0] :   4294967295 sectors at            1 (type ee)
>> /dev/sdl:
>>    MBR Magic : aa55
>> Partition[0] :   4294967295 sectors at            1 (type ee)
>
> And then "something" created one. Is that partition table empty?
> Or did it also create and format partitions, that would be worse.
>
> GPT partition table overwrites a bunch of sectors at both start and end.
> So that's where you'll find corruption, depending which mdadm metadata
> version you were using (which can also be located either start or end).
>
> To recover, you'll have to determine the correct RAID level / layout /
> order / data offset. It's best to do this with overlays
>
> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>
> and only write to the real disks once you've found the setting that works.
>
> In the future, consider always using a partition table. Linux doesn't care *
> but the partition table is the most standard way to declare a disk is
> already in use and for what. Without a partition table, any software not
> md-raid aware will see your drive as free, unused, and might format it.
>
> (*) it will happily run anything you like on bare disks
>     but it won't do anything to protect you, either
>
> Regards
> Andreas Klauer

Hi Andreas

Thanks for the prompt response,

Yes I did not use a partition table. I have always in the past used
partitions but recently noticed a trend to use the device, wont do
that again.

I am really not sure what wrote those partitions there. I only see
free space in them if I check with fdisk.

I am sure it was metadata version 1.2

I will attempt to recover using the overlay, will post my results.

Thanks again for your help.

Regards
J-P Human

Re: Superblocks lost on all disks in array.

[Anthony Youngman]

On 15/07/17 19:05, Jean-Pierre Human wrote:
> On Sat, Jul 15, 2017 at 7:26 PM, Andreas Klauer
> <Andreas.Klauer@metamorpher.de> wrote:

> Hi Andreas
> 
> Thanks for the prompt response,
> 
> Yes I did not use a partition table. I have always in the past used
> partitions but recently noticed a trend to use the device, wont do
> that again.
> 
> I am really not sure what wrote those partitions there. I only see
> free space in them if I check with fdisk.
> 
> I am sure it was metadata version 1.2
> 
> I will attempt to recover using the overlay, will post my results.

Good call. The whole point of overlays is to enable you to write-protect 
the physical disks. That means (of course) that you can't make matters 
worse. That's always the worry - that an attempt at a fix will damage 
the content of the disks and make a recovery harder/impossible.
> 
> Thanks again for your help.
> 
A little more (hopefully) help - on the wiki the last entry in the "When 
things go wrogn" section is about recovering a trashed array. As it 
says, it's a work in progress, but it gives you various hints about how 
to search the disk for clues as to where your data is, and hence what 
all those things like offsets are.

Have you ever added disks to your array? Rebuilt and changed its 
configuration? This can mess about with the superblock values changing 
them from the defaults. This is information the list will want.

If you have trouble getting overlays to work, search for clues like this 
on your list and get back here for help. ON NO ACCOUNT do anything that 
will actually write to your disks unless you are absolutely confident 
that you've found the correct magic incantation.

There are plenty of people here who will help you get your data back - 
and they've got a good track record! Unfortunately, I'm "learning by 
documenting" so I'm quite happy to give you hints and guidance, but I 
can't say for certain what is likely to work.

Cheers,
Wol

Re: Superblocks lost on all disks in array.

[Jean-Pierre Human]

On Sun, Jul 16, 2017 at 1:03 AM, Anthony Youngman
<antlists@youngman.org.uk> wrote:
>
>
> On 15/07/17 19:05, Jean-Pierre Human wrote:
>>
>> On Sat, Jul 15, 2017 at 7:26 PM, Andreas Klauer
>> <Andreas.Klauer@metamorpher.de> wrote:
>
>
>> Hi Andreas
>>
>> Thanks for the prompt response,
>>
>> Yes I did not use a partition table. I have always in the past used
>> partitions but recently noticed a trend to use the device, wont do
>> that again.
>>
>> I am really not sure what wrote those partitions there. I only see
>> free space in them if I check with fdisk.
>>
>> I am sure it was metadata version 1.2
>>
>> I will attempt to recover using the overlay, will post my results.
>
>
> Good call. The whole point of overlays is to enable you to write-protect the
> physical disks. That means (of course) that you can't make matters worse.
> That's always the worry - that an attempt at a fix will damage the content
> of the disks and make a recovery harder/impossible.
>>
>>
>> Thanks again for your help.
>>
> A little more (hopefully) help - on the wiki the last entry in the "When
> things go wrogn" section is about recovering a trashed array. As it says,
> it's a work in progress, but it gives you various hints about how to search
> the disk for clues as to where your data is, and hence what all those things
> like offsets are.
>
> Have you ever added disks to your array? Rebuilt and changed its
> configuration? This can mess about with the superblock values changing them
> from the defaults. This is information the list will want.
>
> If you have trouble getting overlays to work, search for clues like this on
> your list and get back here for help. ON NO ACCOUNT do anything that will
> actually write to your disks unless you are absolutely confident that you've
> found the correct magic incantation.
>
> There are plenty of people here who will help you get your data back - and
> they've got a good track record! Unfortunately, I'm "learning by
> documenting" so I'm quite happy to give you hints and guidance, but I can't
> say for certain what is likely to work.
>
> Cheers,
> Wol

Hello List

I am happy to say I managed to recover all the data and it was perfectly intact.

To address some of the above questions, the array and server install
are relatively new 3 months or so. There have been no disk additions,
replacements or failures. I had issues getting the overlay working
mainly due to my lack of understanding  and time restraints (used dd
and sfdisk to kinda make backups of the important bits).

Using the same disk order, mdadm version and settings when it was
initially installed it started perfectly with a --create
--assume-clean. The existing LVM was also still intact and once the
ISCSI connection was back online the data was there. I feel I got very
lucky mainly due to the server being in a roughly constant error free
state since install.

The odd thing was that after I had the array running and before I had
started the ISCSI, I had to reboot the server, when it came up the
newly written superblocks were gone again and I had to --create
--assume-clean to get it running again. Once I have secured this data
I will find out why this is happening.

The documentation on the wiki is very helpful and the most concise I
came across. The overlay feature I will spend sometime with to
understand as in the future I may need this. It just left me asking a
lot of questions like if I have LVM and no filesystem on the device
will it work or how should I adopt the setup for that etc. This
exercise has opened my eyes to just how resilient and flexible mdadm
can be.

Thanks again for all the help and advice.

J-P Human

Re: Superblocks lost on all disks in array.

[Wols Lists]

On 16/07/17 17:46, Jean-Pierre Human wrote:
> The documentation on the wiki is very helpful and the most concise I
> came across. The overlay feature I will spend sometime with to
> understand as in the future I may need this. It just left me asking a
> lot of questions like if I have LVM and no filesystem on the device
> will it work or how should I adopt the setup for that etc. This
> exercise has opened my eyes to just how resilient and flexible mdadm
> can be.

As the editor of the wiki, I do my best :-)

Thing is, I don't have lvm or anything like that on my setup. So I'm
rather chary of writing about it because I don't have the experience - I
do feel it's a major failing of the wiki. When I rebuild my computer,
I'm going to set it up as "lvm over raid 5/6", but that's probably going
to be a little way away :-(

Once I get the opportunity, of course it'll get written up and put there
as a reference for others :-)

Cheers,
Wol

Re: Superblocks lost on all disks in array.

[Jean-Pierre Human]

On Sun, Jul 16, 2017 at 7:25 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 16/07/17 17:46, Jean-Pierre Human wrote:
>> The documentation on the wiki is very helpful and the most concise I
>> came across. The overlay feature I will spend sometime with to
>> understand as in the future I may need this. It just left me asking a
>> lot of questions like if I have LVM and no filesystem on the device
>> will it work or how should I adopt the setup for that etc. This
>> exercise has opened my eyes to just how resilient and flexible mdadm
>> can be.
>
> As the editor of the wiki, I do my best :-)
>
> Thing is, I don't have lvm or anything like that on my setup. So I'm
> rather chary of writing about it because I don't have the experience - I
> do feel it's a major failing of the wiki. When I rebuild my computer,
> I'm going to set it up as "lvm over raid 5/6", but that's probably going
> to be a little way away :-(
>
> Once I get the opportunity, of course it'll get written up and put there
> as a reference for others :-)
>
> Cheers,
> Wol

Hi Wol

If you would like a VM to help with that, contact me off list.

Thanks again
J-P Human

Re: Superblocks lost on all disks in array.

[Adam Goryachev]

On 17/07/17 02:46, Jean-Pierre Human wrote:
> On Sun, Jul 16, 2017 at 1:03 AM, Anthony Youngman
> <antlists@youngman.org.uk> wrote:
>>
>> On 15/07/17 19:05, Jean-Pierre Human wrote:
>>> On Sat, Jul 15, 2017 at 7:26 PM, Andreas Klauer
>>> <Andreas.Klauer@metamorpher.de> wrote:
>>
>>> Hi Andreas
>>>
>>> Thanks for the prompt response,
>>>
>>> Yes I did not use a partition table. I have always in the past used
>>> partitions but recently noticed a trend to use the device, wont do
>>> that again.
>>>
>>> I am really not sure what wrote those partitions there. I only see
>>> free space in them if I check with fdisk.
>>>
>>> I am sure it was metadata version 1.2
>>>
>>> I will attempt to recover using the overlay, will post my results.
>>
>> Good call. The whole point of overlays is to enable you to write-protect the
>> physical disks. That means (of course) that you can't make matters worse.
>> That's always the worry - that an attempt at a fix will damage the content
>> of the disks and make a recovery harder/impossible.
>>>
>>> Thanks again for your help.
>>>
>> A little more (hopefully) help - on the wiki the last entry in the "When
>> things go wrogn" section is about recovering a trashed array. As it says,
>> it's a work in progress, but it gives you various hints about how to search
>> the disk for clues as to where your data is, and hence what all those things
>> like offsets are.
>>
>> Have you ever added disks to your array? Rebuilt and changed its
>> configuration? This can mess about with the superblock values changing them
>> from the defaults. This is information the list will want.
>>
>> If you have trouble getting overlays to work, search for clues like this on
>> your list and get back here for help. ON NO ACCOUNT do anything that will
>> actually write to your disks unless you are absolutely confident that you've
>> found the correct magic incantation.
>>
>> There are plenty of people here who will help you get your data back - and
>> they've got a good track record! Unfortunately, I'm "learning by
>> documenting" so I'm quite happy to give you hints and guidance, but I can't
>> say for certain what is likely to work.
>>
>> Cheers,
>> Wol
> Hello List
>
> I am happy to say I managed to recover all the data and it was perfectly intact.
>
> To address some of the above questions, the array and server install
> are relatively new 3 months or so. There have been no disk additions,
> replacements or failures. I had issues getting the overlay working
> mainly due to my lack of understanding  and time restraints (used dd
> and sfdisk to kinda make backups of the important bits).
>
> Using the same disk order, mdadm version and settings when it was
> initially installed it started perfectly with a --create
> --assume-clean. The existing LVM was also still intact and once the
> ISCSI connection was back online the data was there. I feel I got very
> lucky mainly due to the server being in a roughly constant error free
> state since install.
>
> The odd thing was that after I had the array running and before I had
> started the ISCSI, I had to reboot the server, when it came up the
> newly written superblocks were gone again and I had to --create
> --assume-clean to get it running again. Once I have secured this data
> I will find out why this is happening.

I can't be sure, but I suspect what happened is this:
1) setup the write overlay
2) Do the rapairs/etc
3) Export drive and test
4) Reboot - everything is lost

The reason for this is that the write overlay has been lost by the 
reboot, which is the point of the write overlay, it can easily be thrown 
away if you accidentally messed up.
What you need to verify now is that:
1) You are not using a write overlay for your current/live data, or else 
you will lose it after the next reboot
2) Reboot and make sure everything works properly, so you are not left 
in a similar issue next time

Regards,
Adam


-- 
-- 
Adam Goryachev Website Managers www.websitemanagers.com.au
-- 
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.