Need help increasing raid scan efficiency.

Need help increasing raid scan efficiency.

[David Niklas]

Hello,
My PC suffered a rather nasty case of HW failure recently where the MB
would break the CPU and RAM. I ended up with different data on different
members of my RAID6 array.

I wanted to scan through the drives and take some checksums of various
files in an attempt to ascertain which drives took the most data
corruption damage, to try and find the date that the damage started
occurring (as it was unclear when exactly this began), and to try and
rescue some of the data off of the good pairs.

So I setup the array into read-only mode and started the array with only
two of the drives. Drives 0 and 1. Then I proceeded to try and start a
second pair, drives 2 and 3, so that I could scan them simultaneously.
With the intent of then switching it over to 0 and 2 and 1 and 3, then 0
and 3 and 1 and 2.


This failed with the error message:
# mdadm --assemble -o --run /dev/md128 /dev/sdc /dev/sdd
mdadm: Found some drive for array that is already active: /dev/md127
mdadm: giving up.
# mdadm --detail /dev/md127
           Version : 1.2
     Creation Time : XXX
        Raid Level : raid6
        Array Size : XXX
     Used Dev Size : XXX
      Raid Devices : 4
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : XXX
             State : clean, degraded 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : XXX
              UUID : XXX
            Events : 3826931

    Number   Major   Minor   RaidDevice State
       7       9        0        0      active sync   /dev/md0
       -       0        0        1      removed
       -       0        0        2      removed
       6       9        1        3      active sync   /dev/md1



Any ideas as to how I can get mdadm to run the array as I requested
above? I did try --force, but mdadm refused to listen.

Thanks,
David

Re: Need help increasing raid scan efficiency.

[Wol]

On 03/06/2025 02:05, David Niklas wrote:
> So I setup the array into read-only mode and started the array with only
> two of the drives. Drives 0 and 1. Then I proceeded to try and start a
> second pair, drives 2 and 3, so that I could scan them simultaneously.
> With the intent of then switching it over to 0 and 2 and 1 and 3, then 0
> and 3 and 1 and 2.

BACKUP! BACKUP!! BACKUP!!!

Is your array that messed up that it won't assemble? If you can just get 
it to assemble normally that's your best bet by far. Trying to assemble 
it as two pairs is throwing away the whole point of a raid 6!

And make sure you know the order of the drives in the array! I hope you 
haven't lost that infof.

If your event counts are all similar, then you'll hopefully recover most 
of your data. Your biggest worry will be the mobo and ram having 
trashing an in-flight write that corrupts the disk.

Then once you've got the array assembled, I can't remember the command, 
but there is a command that will read the entire stripe, check the 
paritIES - both of them, and recreate the data. If that fails, your data 
is probably toast, and nothing you can do will be able to retrieve much :-(

Cheers,
Wol

Re: Need help increasing raid scan efficiency.

[David Niklas]

On Tue, 3 Jun 2025 17:46:01 +0100
Wol <antlists@youngman.org.uk> wrote:
> On 03/06/2025 02:05, David Niklas wrote:
> > So I setup the array into read-only mode and started the array with
> > only two of the drives. Drives 0 and 1. Then I proceeded to try and
> > start a second pair, drives 2 and 3, so that I could scan them
> > simultaneously. With the intent of then switching it over to 0 and 2
> > and 1 and 3, then 0 and 3 and 1 and 2.  
> 
> BACKUP! BACKUP!! BACKUP!!!

It's when I was trying to make my yearly backup that I found out it was
corrupting. I have HDDs I backup to. When I backup, I erase the
previous year (as I don't have enough room otherwise), then backup the new
year. As a system, it worked up until now.

> Is your array that messed up that it won't assemble? If you can just
> get it to assemble normally that's your best bet by far. Trying to
> assemble it as two pairs is throwing away the whole point of a raid 6!

It assembles fine with all the disks, the problem is the data corruption
that has occurred across the members.

> And make sure you know the order of the drives in the array! I hope you
> haven't lost that infof.

Everything is written down on paper.

> If your event counts are all similar, then you'll hopefully recover
> most of your data. Your biggest worry will be the mobo and ram having
> trashing an in-flight write that corrupts the disk.

Yes, that's my problem. I wanted to try and isolate the disk pairs so
that I could try and figure out if there is any pattern or differing
copies that would allow me to restore the corrupted data.

> Then once you've got the array assembled, I can't remember the command,
> but there is a command that will read the entire stripe, check the
> paritIES - both of them, and recreate the data. If that fails, your
> data is probably toast, and nothing you can do will be able to retrieve
> much :-(
> 
> Cheers,
> Wol
> 

Searching online turned up raid6check.
https://unix.stackexchange.com/questions/137384/raid6-scrubbing-mismatch-repair

But the people there also pointed out that Linux's raid repair operation
only recalculates the parity. I would have thought that it did a best of
3 option. I mean, that's a big part of why we have RAID6 instead of RAID5,
right?

I think you misunderstood my original question, how do I assemble the
RAID6 pairs (RO mode) into two different arrays such that I can read from
them simultaneously?

If I have to do some coding with respect to the mdadm utility, I'm
willing. But for all I know, the Linux Kernel might destroy all of my data
if I try something like that.

Thanks,
David

Re: Need help increasing raid scan efficiency.

[David Niklas]

On Tue, 3 Jun 2025 17:46:01 +0100
Wol <antlists@youngman.org.uk> wrote:
> On 03/06/2025 02:05, David Niklas wrote:
> > So I setup the array into read-only mode and started the array with
> > only two of the drives. Drives 0 and 1. Then I proceeded to try and
> > start a second pair, drives 2 and 3, so that I could scan them
> > simultaneously. With the intent of then switching it over to 0 and 2
> > and 1 and 3, then 0 and 3 and 1 and 2.  
> 
> BACKUP! BACKUP!! BACKUP!!!

It's when I was trying to make my yearly backup that I found out it was
corrupting. I have HDDs I backup to. When I backup, I erase the
previous year (as I don't have enough room otherwise), then backup the new
year. As a system, it worked up until now.

> Is your array that messed up that it won't assemble? If you can just
> get it to assemble normally that's your best bet by far. Trying to
> assemble it as two pairs is throwing away the whole point of a raid 6!

It assembles fine with all the disks, the problem is the data corruption
that has occurred across the members.

> And make sure you know the order of the drives in the array! I hope you
> haven't lost that infof.

Everything is written down on paper.

> If your event counts are all similar, then you'll hopefully recover
> most of your data. Your biggest worry will be the mobo and ram having
> trashing an in-flight write that corrupts the disk.

Yes, that's my problem. I wanted to try and isolate the disk pairs so
that I could try and figure out if there is any pattern or differing
copies that would allow me to restore the corrupted data.

> Then once you've got the array assembled, I can't remember the command,
> but there is a command that will read the entire stripe, check the
> paritIES - both of them, and recreate the data. If that fails, your
> data is probably toast, and nothing you can do will be able to retrieve
> much :-(
> 
> Cheers,
> Wol
> 

Searching online turned up raid6check.
https://unix.stackexchange.com/questions/137384/raid6-scrubbing-mismatch-repair

But the people there also pointed out that Linux's raid repair operation
only recalculates the parity. I would have thought that it did a best of
3 option. I mean, that's a big part of why we have RAID6 instead of RAID5,
right?

I think you misunderstood my original question, how do I assemble the
RAID6 pairs (RO mode) into two different arrays such that I can read from
them simultaneously?

If I have to do some coding with respect to the mdadm utility, I'm
willing. But for all I know, the Linux Kernel might destroy all of my data
if I try something like that.

Thanks,
David

Re: Need help increasing raid scan efficiency.

[anthony]

On 03/06/2025 21:04, David Niklas wrote:
> Searching online turned up raid6check.
> https://unix.stackexchange.com/questions/137384/raid6-scrubbing- 
> mismatch-repair
> 
> But the people there also pointed out that Linux's raid repair operation
> only recalculates the parity. I would have thought that it did a best of
> 3 option. I mean, that's a big part of why we have RAID6 instead of RAID5,
> right?

 From what I remember of raid6check, it actually does a proper raid 6 
calculation to recover the damaged data.

Raid 5 certainly just recalculates the parity, but it doesn't have any 
choice. Because it can only reconstruct ONE piece of information, it can 
detect the corruption, but it has no idea WHAT is corrupted. So it 
assumes (with good reason) that it's the parity and re-calculates it. 
Where raid 5 scores is if you lose a block, or a drive, or whatever, it 
is told what has been lost and can recreate it. If your data is 
corrupted, however, it has two pieces of missing information ("what" and 
"where"), and can only reconstruct one, so it assumes it's the parity 
that's been lost.

Because raid-6 has two levels of redundancy, if ONE block is damaged, it 
can work out both what and where, which is what raid6check does.

I've done it slightly differently, I've got raid-5 sat on top of 
dm-integrity, so if a disk gets corrupted dm-integrity will simply 
return a read failure, and the raid doesn't have to work out what's been 
corrupted. I've got a different problem at the moment - my array has 
assembled itself as three spares, so I've got to fix that ... :-(

Cheers,
Wol

Re: Need help increasing raid scan efficiency.

[Jani Partanen]

On 03/06/2025 23.04, David Niklas wrote:
> I think you misunderstood my original question, how do I assemble the
> RAID6 pairs (RO mode) into two different arrays such that I can read from
> them simultaneously?

I dont think there is any other way to do what you want to do than use 
overlayfs. You may find some ideas from here:

https://archive.kernel.org/oldwiki/raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recovery.html

Re: Need help increasing raid scan efficiency.

[Reindl Harald]

Am 03.06.25 um 03:05 schrieb David Niklas:
> My PC suffered a rather nasty case of HW failure recently where the MB
> would break the CPU and RAM. I ended up with different data on different
> members of my RAID6 array.
> 
> I wanted to scan through the drives and take some checksums of various
> files in an attempt to ascertain which drives took the most data
> corruption damage, to try and find the date that the damage started
> occurring (as it was unclear when exactly this began), and to try and
> rescue some of the data off of the good pairs.

forget it and restore a backup

Re: Need help increasing raid scan efficiency.

[Reindl Harald]

Am 03.06.25 um 22:04 schrieb David Niklas:
> On Tue, 3 Jun 2025 17:46:01 +0100
> Wol <antlists@youngman.org.uk> wrote:
>> On 03/06/2025 02:05, David Niklas wrote:
>>> So I setup the array into read-only mode and started the array with
>>> only two of the drives. Drives 0 and 1. Then I proceeded to try and
>>> start a second pair, drives 2 and 3, so that I could scan them
>>> simultaneously. With the intent of then switching it over to 0 and 2
>>> and 1 and 3, then 0 and 3 and 1 and 2.
>>
>> BACKUP! BACKUP!! BACKUP!!!
> 
> It's when I was trying to make my yearly backup

jesus - that's by far not enough

> When I backup, I erase the previous year

so the first you do is delete the last known good state

> as I don't have enough room otherwise

seriously?

an external 10 TB disk costs around 200 EUR
an external 20 TB disk costs around 400 EUR

with rsnapshot you can make daily, weekly and montly backups
until the doctor comes because anything unchanged are only hard links

> then backup the new year. As a system, it worked up until now.

no, it did not as you found out the first time you needed it

> It assembles fine with all the disks, the problem is the data corruption
> that has occurred across the members

and what makes you think you can fix that manually

Re: Need help increasing raid scan efficiency.

[John Stoffel]

>>>>> "Reindl" == Reindl Harald <h.reindl@thelounge.net> writes:

> Am 03.06.25 um 03:05 schrieb David Niklas:
>> My PC suffered a rather nasty case of HW failure recently where the MB
>> would break the CPU and RAM. I ended up with different data on different
>> members of my RAID6 array.
>> 
>> I wanted to scan through the drives and take some checksums of various
>> files in an attempt to ascertain which drives took the most data
>> corruption damage, to try and find the date that the damage started
>> occurring (as it was unclear when exactly this began), and to try and
>> rescue some of the data off of the good pairs.

> forget it and restore a backup

Or if your data is static, use last years backup and look for
un-changed files and them compare the data inside them?  

but yes, using a better backup system is a good idea.  I'm using
'burp' for my home backups.  And big disks are cheap.  How much is
your data worth to you?  

Re: Need help increasing raid scan efficiency.

[John Stoffel]

>>>>> "David" == David Niklas <simd@vfemail.net> writes:

> My PC suffered a rather nasty case of HW failure recently where the
> MB would break the CPU and RAM. I ended up with different data on
> different members of my RAID6 array.

Ouch, this is not good.  But you have RAID6, so it should be ok...

> I wanted to scan through the drives and take some checksums of
> various files in an attempt to ascertain which drives took the most
> data corruption damage, to try and find the date that the damage
> started occurring (as it was unclear when exactly this began), and
> to try and rescue some of the data off of the good pairs.

What are you comparing the checksums too?  Just because you assemble
drives 1 and 2 and read the filesystem, then assemble drives 3 and 4
into another array, how do you know which checksum is correct if they
differ?  

> So I setup the array into read-only mode and started the array with
> only two of the drives. Drives 0 and 1. Then I proceeded to try and
> start a second pair, drives 2 and 3, so that I could scan them
> simultaneously.  With the intent of then switching it over to 0 and
> 2 and 1 and 3, then 0 and 3 and 1 and 2.

I'm not sure this is really going to work how you think.... 

> This failed with the error message:
> # mdadm --assemble -o --run /dev/md128 /dev/sdc /dev/sdd
> mdadm: Found some drive for array that is already active: /dev/md127

This is not un-expected.  You already have md127 setup using the same
UUID, and mdadm is doing the right thing to refuse to assemble a
different array name with the same underlying UUID.  

But if you have four drives, you've got four sets of checksums to
calculate for each file, which is going to take time.  And I think
just doing it one pair of disks at a time is the safest way.  Your
data is important to you, obviously, but how much is it worth?  

Can you afford to get some replacement disks, or even just a single
large disk and them dump all your files onto a new single disk to try
and save what you have, even if it's corrupted?  

> Any ideas as to how I can get mdadm to run the array as I requested
> above? I did try --force, but mdadm refused to listen.

And for good reason.  You might be able to do an overlayfs on each
pair, then go in and change the UUID of the second pair to something
different, and then start the array with a new name and disk member
UUIDs.  

But it's alot of hacking for probably not much payout.

Have you found a file with corruption?  If so, have you done a quick
test where you do the four pairs of the array assembled and checked
just that one single file to see if the checksum differs?  

And again, if it does differ, how do you decide what is the correct
data?  

I would strongly suspect that the data is corrupted no matter what.  

In any case, good luck!  Maybe the raid6check tool will help, but I'd
rather try to at least use your most recent backup as a check.  

John

Re: Need help increasing raid scan efficiency.

[Reindl Harald]

Am 04.06.25 um 14:34 schrieb John Stoffel:
>>>>>> "David" == David Niklas <simd@vfemail.net> writes:
> 
>> My PC suffered a rather nasty case of HW failure recently where the
>> MB would break the CPU and RAM. I ended up with different data on
>> different members of my RAID6 array.
> 
> Ouch, this is not good.  But you have RAID6, so it should be ok...
it can't be OK no matter what RAID level when the hardware decides to 
write garbage on disks

that's why you need a good backup strategy which really works and not 
only when the sun shines

Re: Need help increasing raid scan efficiency.

[David Niklas]

I'm replying to everyone in the same email.

On Wed, 4 Jun 2025 08:34:35 -0400
"John Stoffel" <john@stoffel.org> wrote:
> >>>>> "David" == David Niklas <simd@vfemail.net> writes:  
> 
> > My PC suffered a rather nasty case of HW failure recently where the
> > MB would break the CPU and RAM. I ended up with different data on
> > different members of my RAID6 array.  
> 
> Ouch, this is not good.  But you have RAID6, so it should be ok...
> 
> > I wanted to scan through the drives and take some checksums of
> > various files in an attempt to ascertain which drives took the most
> > data corruption damage, to try and find the date that the damage
> > started occurring (as it was unclear when exactly this began), and
> > to try and rescue some of the data off of the good pairs.  
> 
> What are you comparing the checksums too?  Just because you assemble
> drives 1 and 2 and read the filesystem, then assemble drives 3 and 4
> into another array, how do you know which checksum is correct if they
> differ?  

Once I find some files whose checksums differ, I can perform some
automated data tests to find which file is the intact one.

> > So I setup the array into read-only mode and started the array with
> > only two of the drives. Drives 0 and 1. Then I proceeded to try and
> > start a second pair, drives 2 and 3, so that I could scan them
> > simultaneously.  With the intent of then switching it over to 0 and
> > 2 and 1 and 3, then 0 and 3 and 1 and 2.  
> 
> I'm not sure this is really going to work how you think.... 
<snip>

I just think that I'll be able to read from all 4 drives but doing it in
2 arrays of 2 drives. Basically, I'll get a 2x speed increase over doing
it as 2 drives at a time.


On Wed, 4 Jun 2025 07:22:15 +0300
Jani Partanen <jiipee@sotapeli.fi> wrote:
> On 03/06/2025 23.04, David Niklas wrote:
> > I think you misunderstood my original question, how do I assemble the
> > RAID6 pairs (RO mode) into two different arrays such that I can read
> > from them simultaneously?  
> 
> I dont think there is any other way to do what you want to do than use
> overlayfs. You may find some ideas from here:
> 
> https://archive.kernel.org/oldwiki/raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recovery.html

Thanks for the idea. I'm not following why we setup the overlay but then
use mapper devices (which came from where?), with the mdadm commands.


On Tue, 3 Jun 2025 21:27:35 +0100
anthony <antmbox@youngman.org.uk> wrote:
> On 03/06/2025 21:04, David Niklas wrote:
> > Searching online turned up raid6check.
> > https://unix.stackexchange.com/questions/137384/raid6-scrubbing- 
> > mismatch-repair
> > 
> > But the people there also pointed out that Linux's raid repair
> > operation only recalculates the parity. I would have thought that it
> > did a best of 3 option. I mean, that's a big part of why we have
> > RAID6 instead of RAID5, right?  
> 
>  From what I remember of raid6check, it actually does a proper raid 6
> calculation to recover the damaged data.
> 
<snip>
> I've done it slightly differently, I've got raid-5 sat on top of
> dm-integrity, so if a disk gets corrupted dm-integrity will simply
> return a read failure, and the raid doesn't have to work out what's
> been corrupted. I've got a different problem at the moment - my array
> has assembled itself as three spares, so I've got to fix that ... :-(
> 
> Cheers,
> Wol

Good to know. Thanks Wol. I hope you're able to get your drives up and
running again.


On Wed, 4 Jun 2025 10:59:21 +0200
Reindl Harald <h.reindl@thelounge.net> wrote:
<snip>
> > as I don't have enough room otherwise  
> 
> seriously?
> 
> an external 10 TB disk costs around 200 EUR
> an external 20 TB disk costs around 400 EUR
<snip>

Every time I upgraded the size of my array, I'd take the old disks and
use them as backup disks. Over time, it became a matter of not having
enough SATA ports, not a matter of costing too much. I was trying to
reuse disks instead of the disks being tossed out or collecting dust. I've
learned better now.


Thanks,
David