Fail to assemble raid4 with replaced disk

Fail to assemble raid4 with replaced disk

[Santiago DIEZ]

Hi Raiders,

I had a raid5 array md10 with sd[abcd]10.
Eventually, sdd10 failed.

I did NOT do any mdadm --fail NOR mdadm --remove command.
What I did is comment out the line "ARRAY /dev/md10 ..." in
/etc/mdadm/mdadm.conf.

Then I powered off the server, replaced the disk sdd with a new one
and booted the system.

I examined the status with:
# cat /proc/mdstat
md10 : inactive sdb10[1]
      1926247296 blocks

I stopped the array with:
# mdadm --stop /dev/md10

I tried to assemble the array with the 3 original disks like this
# mdadm --assemble /dev/md10 --verbose /dev/sda10 /dev/sdb10 /dev/sdc10
mdadm: looking for devices for /dev/md10
mdadm: /dev/sda10 is identified as a member of /dev/md10, slot 0.
mdadm: /dev/sdb10 is identified as a member of /dev/md10, slot 1.
mdadm: /dev/sdc10 is identified as a member of /dev/md10, slot 2.
mdadm: added /dev/sda10 to /dev/md10 as 0 (possibly out of date)
mdadm: added /dev/sdc10 to /dev/md10 as 2 (possibly out of date)
mdadm: no uptodate device for slot 3 of /dev/md10
mdadm: added /dev/sdb10 to /dev/md10 as 1
mdadm: /dev/md10 assembled from 1 drive - not enough to start the array.

I examined the status again with:
# cat /proc/mdstat
md10 : inactive sdb10[1](S) sdc10[2](S) sda10[0](S)
      5778741888 blocks

Now I'm SCARED!
What does the (S) mean?
How do I reassemble my array and add the new sdd10 partition?

Thanks for your help

Regards
-------------------------
Santiago DIEZ
Quark Systems & CAOBA
23 rue du Buisson Saint-Louis, 75010 Paris
-------------------------

Re: Fail to assemble raid4 with replaced disk

[Mikael Abrahamsson]

On Tue, 25 Oct 2016, Santiago DIEZ wrote:

> # mdadm --assemble /dev/md10 --verbose /dev/sda10 /dev/sdb10 /dev/sdc10
> mdadm: looking for devices for /dev/md10
> mdadm: /dev/sda10 is identified as a member of /dev/md10, slot 0.
> mdadm: /dev/sdb10 is identified as a member of /dev/md10, slot 1.
> mdadm: /dev/sdc10 is identified as a member of /dev/md10, slot 2.
> mdadm: added /dev/sda10 to /dev/md10 as 0 (possibly out of date)
> mdadm: added /dev/sdc10 to /dev/md10 as 2 (possibly out of date)
> mdadm: no uptodate device for slot 3 of /dev/md10
> mdadm: added /dev/sdb10 to /dev/md10 as 1
> mdadm: /dev/md10 assembled from 1 drive - not enough to start the array.

This means sda10 and sdc10 most likely have a lower event count than 
sdb10.

> I examined the status again with:
> # cat /proc/mdstat
> md10 : inactive sdb10[1](S) sdc10[2](S) sda10[0](S)
>      5778741888 blocks
>
> Now I'm SCARED!
> What does the (S) mean?
> How do I reassemble my array and add the new sdd10 partition?

Check with mdadm -E /dev/sd[abc]10, check the event count, if it differs 
just a little (5-10 perhaps), then you can use --assemble --force to start 
it even though the event count is not exactly the same on each drive.

The event count is increased every time a drive is written to, when there 
is an unclean shutdown mdadm won't auto-assemble drives without operator 
intervention to understand the situation and act accordingly.


-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Fail to assemble raid4 with replaced disk

[Wols Lists]

On 25/10/16 18:08, Santiago DIEZ wrote:
> Hi Raiders,

This looks like a fairly simple recovery job - but you will probably
lose a little data - fsck will moan about a few new files being corrupted.

Firstly, DON'T DO ANYTHING WITH THE RAID.

Secondly, go to the linux raid wiki
https://raid.wiki.kernel.org/index.php/Linux_Raid and read section 4
"When things go wrogn". You've messed up replacing the failed drive, and
are now at "My raid won't assemble/run". But as I say, it doesn't look
particularly serious.
> 
> I had a raid5 array md10 with sd[abcd]10.
> Eventually, sdd10 failed.
> 
> I did NOT do any mdadm --fail NOR mdadm --remove command.
> What I did is comment out the line "ARRAY /dev/md10 ..." in
> /etc/mdadm/mdadm.conf.

mdadm.conf is somewhat of a relic from a bygone age, I believe. It used
to be necessary, in the new world of raid superblocks it is mostly
ignored and redundant.
> 
> Then I powered off the server, replaced the disk sdd with a new one
> and booted the system.
> 
> I examined the status with:
> # cat /proc/mdstat
> md10 : inactive sdb10[1]
>       1926247296 blocks
> 
> I stopped the array with:
> # mdadm --stop /dev/md10
> 
> I tried to assemble the array with the 3 original disks like this
> # mdadm --assemble /dev/md10 --verbose /dev/sda10 /dev/sdb10 /dev/sdc10
> mdadm: looking for devices for /dev/md10
> mdadm: /dev/sda10 is identified as a member of /dev/md10, slot 0.
> mdadm: /dev/sdb10 is identified as a member of /dev/md10, slot 1.
> mdadm: /dev/sdc10 is identified as a member of /dev/md10, slot 2.
> mdadm: added /dev/sda10 to /dev/md10 as 0 (possibly out of date)
> mdadm: added /dev/sdc10 to /dev/md10 as 2 (possibly out of date)
> mdadm: no uptodate device for slot 3 of /dev/md10
> mdadm: added /dev/sdb10 to /dev/md10 as 1
> mdadm: /dev/md10 assembled from 1 drive - not enough to start the array.

Okay. It's got three drives. When you've done what "Asking for help"
says, you should have event counts for all those three drives -
sd[abc]10. Hopefully they're all pretty much the same. If they are, a
simple "--assemble --force" should get your array up and running again.

The complaint about slot 3 is because you haven't removed the old sdd10,
and the new sdd10 isn't part of the array, it has no superblock.
> 
> I examined the status again with:
> # cat /proc/mdstat
> md10 : inactive sdb10[1](S) sdc10[2](S) sda10[0](S)
>       5778741888 blocks
> 
> Now I'm SCARED!
> What does the (S) mean?
> How do I reassemble my array and add the new sdd10 partition?
> 
> Thanks for your help
> 
Okay. That leaves your recovery path neatly mapped out. Get the event
count of the three remaining drives and post them here. Wait for an
expert to muck in and say it all looks good. Then

Assemble the array with --force
Remove the old sdd10
Add the new sdd10
Run a fsck.

And your array should all be back fine. One thing - the wiki bangs on
about the timeout problem. Is that your problem? Because if it is you
will have grief trying to get the array back unless you fix that as your
very first step.

Cheers,
Wol

Re: Fail to assemble raid4 with replaced disk

[Santiago DIEZ]

Hi,

Indeed, here is what I had in terms of event count:
/dev/sda10: 81589
/dev/sdb10: 81626
/dev/sdc10: 81589

Then the following procedure worked quite straightforward:
--------------------------------------------------------------------------------
# mdadm --assemble /dev/md10 --verbose --force /dev/sda10 /dev/sdb10 /dev/sdc10
# mdadm --manage /dev/md10 --add /dev/sdd10
--------------------------------------------------------------------------------

And 6h+ later:
--------------------------------------------------------------------------------
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md10 : active raid5 sdd10[3] sda10[0] sdc10[2] sdb10[1]
      5778741888 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
--------------------------------------------------------------------------------

Then I ran:
--------------------------------------------------------------------------------
# e2fsck -f -n -t -v /dev/md10
e2fsck 1.42.5 (29-Jul-2012)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

    15675837 inodes used (4.34%, out of 361177088)
      188798 non-contiguous files (1.2%)
       14751 non-contiguous directories (0.1%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 15626455/47037/15
  1281308341 blocks used (88.69%, out of 1444685472)
           0 bad blocks
         101 large files

    15311457 regular files
      361754 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
        2607 symbolic links (2310 fast symbolic links)
          10 sockets
------------
    15675828 files
Memory used: 50976k/1912k (20541k/30436k), time: 1304.00/334.06/ 8.00
I/O read: 4891MB, write: 0MB, rate: 3.75MB/s
--------------------------------------------------------------------------------

Does it look OK enough to launch the mount?

Regards and thanks for your help
-------------------------
Santiago DIEZ
Quark Systems & CAOBA
23 rue du Buisson Saint-Louis, 75010 Paris
-------------------------

Re: Fail to assemble raid4 with replaced disk

[Wols Lists]

On 27/10/16 15:11, Santiago DIEZ wrote:
> Hi,
> 
> Indeed, here is what I had in terms of event count:
> /dev/sda10: 81589
> /dev/sdb10: 81626
> /dev/sdc10: 81589
> 
> Then the following procedure worked quite straightforward:
> --------------------------------------------------------------------------------
> # mdadm --assemble /dev/md10 --verbose --force /dev/sda10 /dev/sdb10 /dev/sdc10
> # mdadm --manage /dev/md10 --add /dev/sdd10
> --------------------------------------------------------------------------------
> 
> And 6h+ later:
> --------------------------------------------------------------------------------
> # cat /proc/mdstat
> Personalities : [raid1] [raid6] [raid5] [raid4]
> md10 : active raid5 sdd10[3] sda10[0] sdc10[2] sdb10[1]
>       5778741888 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
> --------------------------------------------------------------------------------
> 
> Then I ran:
> --------------------------------------------------------------------------------
> # e2fsck -f -n -t -v /dev/md10
> e2fsck 1.42.5 (29-Jul-2012)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> 
>     15675837 inodes used (4.34%, out of 361177088)
>       188798 non-contiguous files (1.2%)
>        14751 non-contiguous directories (0.1%)
>              # of inodes with ind/dind/tind blocks: 0/0/0
>              Extent depth histogram: 15626455/47037/15
>   1281308341 blocks used (88.69%, out of 1444685472)
>            0 bad blocks
>          101 large files
> 
>     15311457 regular files
>       361754 directories
>            0 character device files
>            0 block device files
>            0 fifos
>            0 links
>         2607 symbolic links (2310 fast symbolic links)
>           10 sockets
> ------------
>     15675828 files
> Memory used: 50976k/1912k (20541k/30436k), time: 1304.00/334.06/ 8.00
> I/O read: 4891MB, write: 0MB, rate: 3.75MB/s
> --------------------------------------------------------------------------------
> 
> Does it look OK enough to launch the mount?
> 
sorry - I've been away for the weekend - daughter's wedding :-)

But yes, that looks great. No errors on fsck either, I think :-)

I think your array looks fine. Just look at the output from smartctl for
your old drives and make sure that it doesn't look like another drive is
going to fail soon. I'm not quite sure what to look for, mostly bad
blocks and relocates, I think, but if you compare it with your new drive
and stuff looks dodgy, you can always ask for help.

Cheers,
Wol

Re: Fail to assemble raid4 with replaced disk

[Santiago DIEZ]

Hi,

I never said THANKS.

Never too late ;o)

-------------------------
Santiago DIEZ
-------------------------
Quark Systems & CAOBA
23 rue du Buisson Saint-Louis, 75010 Paris
-------------------------


On Mon, Oct 31, 2016 at 4:57 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 27/10/16 15:11, Santiago DIEZ wrote:
>> Hi,
>>
>> Indeed, here is what I had in terms of event count:
>> /dev/sda10: 81589
>> /dev/sdb10: 81626
>> /dev/sdc10: 81589
>>
>> Then the following procedure worked quite straightforward:
>> --------------------------------------------------------------------------------
>> # mdadm --assemble /dev/md10 --verbose --force /dev/sda10 /dev/sdb10 /dev/sdc10
>> # mdadm --manage /dev/md10 --add /dev/sdd10
>> --------------------------------------------------------------------------------
>>
>> And 6h+ later:
>> --------------------------------------------------------------------------------
>> # cat /proc/mdstat
>> Personalities : [raid1] [raid6] [raid5] [raid4]
>> md10 : active raid5 sdd10[3] sda10[0] sdc10[2] sdb10[1]
>>       5778741888 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>> --------------------------------------------------------------------------------
>>
>> Then I ran:
>> --------------------------------------------------------------------------------
>> # e2fsck -f -n -t -v /dev/md10
>> e2fsck 1.42.5 (29-Jul-2012)
>> Pass 1: Checking inodes, blocks, and sizes
>> Pass 2: Checking directory structure
>> Pass 3: Checking directory connectivity
>> Pass 4: Checking reference counts
>> Pass 5: Checking group summary information
>>
>>     15675837 inodes used (4.34%, out of 361177088)
>>       188798 non-contiguous files (1.2%)
>>        14751 non-contiguous directories (0.1%)
>>              # of inodes with ind/dind/tind blocks: 0/0/0
>>              Extent depth histogram: 15626455/47037/15
>>   1281308341 blocks used (88.69%, out of 1444685472)
>>            0 bad blocks
>>          101 large files
>>
>>     15311457 regular files
>>       361754 directories
>>            0 character device files
>>            0 block device files
>>            0 fifos
>>            0 links
>>         2607 symbolic links (2310 fast symbolic links)
>>           10 sockets
>> ------------
>>     15675828 files
>> Memory used: 50976k/1912k (20541k/30436k), time: 1304.00/334.06/ 8.00
>> I/O read: 4891MB, write: 0MB, rate: 3.75MB/s
>> --------------------------------------------------------------------------------
>>
>> Does it look OK enough to launch the mount?
>>
> sorry - I've been away for the weekend - daughter's wedding :-)
>
> But yes, that looks great. No errors on fsck either, I think :-)
>
> I think your array looks fine. Just look at the output from smartctl for
> your old drives and make sure that it doesn't look like another drive is
> going to fail soon. I'm not quite sure what to look for, mostly bad
> blocks and relocates, I think, but if you compare it with your new drive
> and stuff looks dodgy, you can always ask for help.
>
> Cheers,
> Wol
>