Kernel mistakenly "starts" resync on fully-degraded, newly-created raid10 array

Kernel mistakenly "starts" resync on fully-degraded, newly-created raid10 array

[Omari Stephens]

I tried asking on Reddit, and ended up resolving the issue myself:
https://www.reddit.com/r/linuxquestions/comments/1lh9to0/kernel_is_stuck_resyncing_a_4drive_raid10_array/

I run Debian SID, and am using kernel 6.12.32-amd64

#apt-cache policy linux-image-amd64
linux-image-amd64:
   Installed: 6.12.32-1
   Candidate: 6.12.32-1
   Version table:
  *** 6.12.32-1 500
         500 http://mirrors.kernel.org/debian unstable/main amd64 Packages
         500 http://http.us.debian.org/debian unstable/main amd64 Packages
         100 /var/lib/dpkg/status

#uname -r
6.12.32-amd64

To summarize the issue and my diagnostic steps, I ran this command to create a new raid10 array:

|#mdadm --create md13 --name=media --level=10 --layout=f2 -n 4 /dev/sdb1 missing /dev/sdf1 missing|

|At that point, /proc/mdstat showed the following, which makes no sense:|

md127 : active raid10 sdb1[2] sdc1[0]
       23382980608 blocks super 1.2 512K chunks 2 far-copies [4/2] [U_U_]
       [>....................]  resync =  0.0% (8594688/23382980608) finish=25176161501.3min speed=0K/sec
       bitmap: 175/175 pages [700KB], 65536KB chunk

With 2 drives present and 2 drives absent, the array can only start if the present drives are considered in sync.  The kernel spent most of a day in this state.  The
"8594688" count increased very slowly over time, but after 24 hours, it was only up to 0.1%.  During that time, I had mounted the array and transfered 11TB of data onto it.

Then when power-cycled, swapped SATA cables, and added the remaining drives, they were marked as spares and weren't added to the array (likely because the array was considered to be already resyncing):

#mdadm --detail /dev/md127
/dev/md127:
[...]
     Number   Major   Minor   RaidDevice State
        0       8       33        0      active sync   /dev/sdc1
        -       0        0        1      removed
        2       8       17        2      active sync   /dev/sdb1
        -       0        0        3      removed

        4       8        1        -      spare   /dev/sda1
        5       8       65        -      spare   /dev/sde1


I ended up resolving the issue by recreating the array with --assume-clean:

#mdadm --create md19 --name=media3 --assume-clean --readonly --level=10 --layout=f2 -n 4 /dev/sdc1 missing /dev/sdb1 missing
To optimalize recovery speed, it is recommended to enable write-indent bitmap, do you want to enable it now? [y/N]? y
mdadm: /dev/sdc1 appears to be part of a raid array:
        level=raid10 devices=4 ctime=Sun Jun 22 00:51:33 2025
mdadm: /dev/sdb1 appears to be part of a raid array:
        level=raid10 devices=4 ctime=Sun Jun 22 00:51:33 2025
Continue creating array [y/N]? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/md19 started.

#cat /proc/mdstat
Personalities : [raid1] [raid10] [raid0] [raid6] [raid5] [raid4]
md127 : active (read-only) raid10 sdb1[2] sdc1[0]
       23382980608 blocks super 1.2 512K chunks 2 far-copies [4/2] [U_U_]
       bitmap: 175/175 pages [700KB], 65536KB chunk

At which point, I was able to add the new devices and have the array (start to) resync as expected:

#mdadm --manage /dev/md127 --add /dev/sda1 --add /dev/sde1
mdadm: added /dev/sda1
mdadm: added /dev/sde1

#cat /proc/mdstat
Personalities : [raid1] [raid10] [raid0] [raid6] [raid5] [raid4]
md127 : active raid10 sde1[5] sda1[4] sdc1[0] sdb1[2]
       23382980608 blocks super 1.2 512K chunks 2 far-copies [4/2] [U_U_]
       [>....................]  recovery =  0.0% (714112/11691490304) finish=1091.3min speed=178528K/sec
       bitmap: 0/175 pages [0KB], 65536KB chunk

#mdadm --detail /dev/md127
/dev/md127:
[...]
     Number   Major   Minor   RaidDevice State
        0       8       33        0      active sync   /dev/sdc1
        5       8       65        1      spare rebuilding   /dev/sde1
        2       8       17        2      active sync   /dev/sdb1
        4       8        1        3      spare rebuilding   /dev/sda1

--xsdg

Re: Kernel mistakenly "starts" resync on fully-degraded, newly-created raid10 array

[Wol]

On 22/06/2025 02:39, Omari Stephens wrote:
> I tried asking on Reddit, and ended up resolving the issue myself:
> https://www.reddit.com/r/linuxquestions/comments/1lh9to0/ 
> kernel_is_stuck_resyncing_a_4drive_raid10_array/
> 
> I run Debian SID, and am using kernel 6.12.32-amd64
> 
> #apt-cache policy linux-image-amd64
> linux-image-amd64:
>    Installed: 6.12.32-1
>    Candidate: 6.12.32-1
>    Version table:
>   *** 6.12.32-1 500
>          500 http://mirrors.kernel.org/debian unstable/main amd64 Packages
>          500 http://http.us.debian.org/debian unstable/main amd64 Packages
>          100 /var/lib/dpkg/status
> 
> #uname -r
> 6.12.32-amd64
> 
> To summarize the issue and my diagnostic steps, I ran this command to 
> create a new raid10 array:
> 
> |#mdadm --create md13 --name=media --level=10 --layout=f2 -n 4 /dev/sdb1 
> missing /dev/sdf1 missing|
> 
> |At that point, /proc/mdstat showed the following, which makes no sense:|
> 
Why doesn't it make any sense?

Don't forget a raid-10 in linux is NOT a two raid-0s in a raid-1, it's 
its own thing entirely.

> md127 : active raid10 sdb1[2] sdc1[0]
>        23382980608 blocks super 1.2 512K chunks 2 far-copies [4/2] [U_U_]
>        [>....................]  resync =  0.0% (8594688/23382980608) 
> finish=25176161501.3min speed=0K/sec
>        bitmap: 175/175 pages [700KB], 65536KB chunk
> 
> With 2 drives present and 2 drives absent, the array can only start if 
> the present drives are considered in sync.  The kernel spent most of a 
> day in this state.  The
> "8594688" count increased very slowly over time, but after 24 hours, it 
> was only up to 0.1%.  During that time, I had mounted the array and 
> transfered 11TB of data onto it.

I can't see any mention of drive size, so your % complete is 
meaningless, but I would say my raid with about 12TB of disk takes a a 
couple of days to sort itself out ...
> 
> Then when power-cycled, swapped SATA cables, and added the remaining 
> drives, they were marked as spares and weren't added to the array 
> (likely because the array was considered to be already resyncing):
> 
I think you're right - if the array is already rebuilding, it can't 
start a new, different rebuild half way through the old one ...

> #mdadm --detail /dev/md127
> /dev/md127:
> [...]
>      Number   Major   Minor   RaidDevice State
>         0       8       33        0      active sync   /dev/sdc1
>         -       0        0        1      removed
>         2       8       17        2      active sync   /dev/sdb1
>         -       0        0        3      removed
> 
>         4       8        1        -      spare   /dev/sda1
>         5       8       65        -      spare   /dev/sde1
> 
> 
> I ended up resolving the issue by recreating the array with --assume-clean:
> 
Bad idea !!! It's okay, especially with new drives and a new array, but 
it will leave the array in a random state. Not good ...

> #mdadm --create md19 --name=media3 --assume-clean --readonly --level=10 
> --layout=f2 -n 4 /dev/sdc1 missing /dev/sdb1 missing
> To optimalize recovery speed, it is recommended to enable write-indent 
> bitmap, do you want to enable it now? [y/N]? y
> mdadm: /dev/sdc1 appears to be part of a raid array:
>         level=raid10 devices=4 ctime=Sun Jun 22 00:51:33 2025
> mdadm: /dev/sdb1 appears to be part of a raid array:
>         level=raid10 devices=4 ctime=Sun Jun 22 00:51:33 2025
> Continue creating array [y/N]? y
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md/md19 started.
> 
> #cat /proc/mdstat
> Personalities : [raid1] [raid10] [raid0] [raid6] [raid5] [raid4]
> md127 : active (read-only) raid10 sdb1[2] sdc1[0]
>        23382980608 blocks super 1.2 512K chunks 2 far-copies [4/2] [U_U_]
>        bitmap: 175/175 pages [700KB], 65536KB chunk
> 
> At which point, I was able to add the new devices and have the array 
> (start to) resync as expected:
> 
Yup. Now the two-drive array is not resyncing, the new drives can be 
added and will resync.

> #mdadm --manage /dev/md127 --add /dev/sda1 --add /dev/sde1
> mdadm: added /dev/sda1
> mdadm: added /dev/sde1
> 
> #cat /proc/mdstat
> Personalities : [raid1] [raid10] [raid0] [raid6] [raid5] [raid4]
> md127 : active raid10 sde1[5] sda1[4] sdc1[0] sdb1[2]
>        23382980608 blocks super 1.2 512K chunks 2 far-copies [4/2] [U_U_]
>        [>....................]  recovery =  0.0% (714112/11691490304) 
> finish=1091.3min speed=178528K/sec
>        bitmap: 0/175 pages [0KB], 65536KB chunk
> 
> #mdadm --detail /dev/md127
> /dev/md127:
> [...]
>      Number   Major   Minor   RaidDevice State
>         0       8       33        0      active sync   /dev/sdc1
>         5       8       65        1      spare rebuilding   /dev/sde1
>         2       8       17        2      active sync   /dev/sdb1
>         4       8        1        3      spare rebuilding   /dev/sda1
> 
> --xsdg
> 
> 
Now you have an array where anything you have written will be okay 
(which I guess is what you care about), but the rest of the disk is 
uninitialised garbage that will instantly trigger a read fault if you 
try to read it.

You need to set off a scrub, which will do those reads and get the array 
itself (not just your data) into a sane state.

https://archive.kernel.org/oldwiki/raid.wiki.kernel.org/

Ignore the obsolete content crap. Somebody clearly thinks that replacing 
USER documentation by double-dutch programmer documentation (aimed at a 
completely different audience) is a good idea ...

Cheers,
Wol

Re: Kernel mistakenly "starts" resync on fully-degraded, newly-created raid10 array

[Omari Stephens]

On 6/22/25 09:30, Wol wrote:
> On 22/06/2025 02:39, Omari Stephens wrote:
>> I tried asking on Reddit, and ended up resolving the issue myself:
>> https://www.reddit.com/r/linuxquestions/comments/1lh9to0/ 
>> kernel_is_stuck_resyncing_a_4drive_raid10_array/
>>
>> I run Debian SID, and am using kernel 6.12.32-amd64
>>
>> #apt-cache policy linux-image-amd64
>> linux-image-amd64:
>>    Installed: 6.12.32-1
>>    Candidate: 6.12.32-1
>>    Version table:
>>   *** 6.12.32-1 500
>>          500 http://mirrors.kernel.org/debian unstable/main amd64 
>> Packages
>>          500 http://http.us.debian.org/debian unstable/main amd64 
>> Packages
>>          100 /var/lib/dpkg/status
>>
>> #uname -r
>> 6.12.32-amd64
>>
>> To summarize the issue and my diagnostic steps, I ran this command to 
>> create a new raid10 array:
>>
>> |#mdadm --create md13 --name=media --level=10 --layout=f2 -n 4 /dev/ 
>> sdb1 missing /dev/sdf1 missing|
>>
>> |At that point, /proc/mdstat showed the following, which makes no sense:|
>>
> Why doesn't it make any sense?
> 
> Don't forget a raid-10 in linux is NOT a two raid-0s in a raid-1, it's 
> its own thing entirely.

Understood.  I've been using Linux MD raid10 for over a decade.  I've 
read through this (and other references) in depth:
https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10

My question is this: Suppose you create a 4-drive array.  2 drives are 
missing.  What data is there to synchronize?  What should get copied 
where, or what should get recomputed and written where?

To my understanding, in that situation, each block in the array only 
appears in one place on the physical media, and there is no redundancy 
or parity for any block that could be out of sync.

When you read from the array, yes, you're going to get interleaved bits 
of whatever happened to be on the physical media to start with, but 
that's basically the same as reading directly from any new physical 
media -- it's not initialized until it's initialized, and until it is, 
you don't know what you're going to read.

> 
>> md127 : active raid10 sdb1[2] sdc1[0]
>>        23382980608 blocks super 1.2 512K chunks 2 far-copies [4/2] [U_U_]
>>        [>....................]  resync =  0.0% (8594688/23382980608) 
>> finish=25176161501.3min speed=0K/sec
>>        bitmap: 175/175 pages [700KB], 65536KB chunk
>>
>> With 2 drives present and 2 drives absent, the array can only start if 
>> the present drives are considered in sync.  The kernel spent most of a 
>> day in this state.  The
>> "8594688" count increased very slowly over time, but after 24 hours, 
>> it was only up to 0.1%.  During that time, I had mounted the array and 
>> transfered 11TB of data onto it.
> 
> I can't see any mention of drive size, so your % complete is 
> meaningless, but I would say my raid with about 12TB of disk takes a a 
> couple of days to sort itself out ...

The rate of resync completion was ~0.  The estimated time to completion 
was 17483445 _years_.  Again, my hypothesis is that this is because the 
system was confused and wasn't actually doing anything meaningful. 
(Although the md127_resync process was sitting at 100% cpu usage the 
entire time; no clue what it was spending those cycles on)

Here's my disk layout, currently, after successfully adding the last two 
drives:
$lsblk /dev/sda /dev/sdb /dev/sdc /dev/sde
NAME              MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINTS
sda                 8:0    0  10.9T  0 disk
└─sda1              8:1    0  10.9T  0 part
   └─md127           9:127  0  21.8T  0 raid10
     ├─media_crypt 253:0    0  21.8T  0 crypt  /mnt/home_media
     └─md127p1     259:0    0 492.2G  0 part
sdb                 8:16   0  10.9T  0 disk
└─sdb1              8:17   0  10.9T  0 part
   └─md127           9:127  0  21.8T  0 raid10
     ├─media_crypt 253:0    0  21.8T  0 crypt  /mnt/home_media
     └─md127p1     259:0    0 492.2G  0 part
sdc                 8:32   0  10.9T  0 disk
└─sdc1              8:33   0  10.9T  0 part
   └─md127           9:127  0  21.8T  0 raid10
     ├─media_crypt 253:0    0  21.8T  0 crypt  /mnt/home_media
     └─md127p1     259:0    0 492.2G  0 part
sde                 8:64   0  10.9T  0 disk
└─sde1              8:65   0  10.9T  0 part
   └─md127           9:127  0  21.8T  0 raid10
     ├─media_crypt 253:0    0  21.8T  0 crypt  /mnt/home_media
     └─md127p1     259:0    0 492.2G  0 part

>>
>> Then when power-cycled, swapped SATA cables, and added the remaining 
>> drives, they were marked as spares and weren't added to the array 
>> (likely because the array was considered to be already resyncing):
>>
> I think you're right - if the array is already rebuilding, it can't 
> start a new, different rebuild half way through the old one ...
> 
>> #mdadm --detail /dev/md127
>> /dev/md127:
>> [...]
>>      Number   Major   Minor   RaidDevice State
>>         0       8       33        0      active sync   /dev/sdc1
>>         -       0        0        1      removed
>>         2       8       17        2      active sync   /dev/sdb1
>>         -       0        0        3      removed
>>
>>         4       8        1        -      spare   /dev/sda1
>>         5       8       65        -      spare   /dev/sde1
>>
>>
>> I ended up resolving the issue by recreating the array with --assume- 
>> clean:
>>
> Bad idea !!! It's okay, especially with new drives and a new array, but 
> it will leave the array in a random state. Not good ...
> 
>> #mdadm --create md19 --name=media3 --assume-clean --readonly -- 
>> level=10 --layout=f2 -n 4 /dev/sdc1 missing /dev/sdb1 missing
>> To optimalize recovery speed, it is recommended to enable write-indent 
>> bitmap, do you want to enable it now? [y/N]? y
>> mdadm: /dev/sdc1 appears to be part of a raid array:
>>         level=raid10 devices=4 ctime=Sun Jun 22 00:51:33 2025
>> mdadm: /dev/sdb1 appears to be part of a raid array:
>>         level=raid10 devices=4 ctime=Sun Jun 22 00:51:33 2025
>> Continue creating array [y/N]? y
>> mdadm: Defaulting to version 1.2 metadata
>> mdadm: array /dev/md/md19 started.
>>
>> #cat /proc/mdstat
>> Personalities : [raid1] [raid10] [raid0] [raid6] [raid5] [raid4]
>> md127 : active (read-only) raid10 sdb1[2] sdc1[0]
>>        23382980608 blocks super 1.2 512K chunks 2 far-copies [4/2] [U_U_]
>>        bitmap: 175/175 pages [700KB], 65536KB chunk
>>
>> At which point, I was able to add the new devices and have the array 
>> (start to) resync as expected:
>>
> Yup. Now the two-drive array is not resyncing, the new drives can be 
> added and will resync.
> 
>> #mdadm --manage /dev/md127 --add /dev/sda1 --add /dev/sde1
>> mdadm: added /dev/sda1
>> mdadm: added /dev/sde1
>>
>> #cat /proc/mdstat
>> Personalities : [raid1] [raid10] [raid0] [raid6] [raid5] [raid4]
>> md127 : active raid10 sde1[5] sda1[4] sdc1[0] sdb1[2]
>>        23382980608 blocks super 1.2 512K chunks 2 far-copies [4/2] [U_U_]
>>        [>....................]  recovery =  0.0% (714112/11691490304) 
>> finish=1091.3min speed=178528K/sec
>>        bitmap: 0/175 pages [0KB], 65536KB chunk
>>
>> #mdadm --detail /dev/md127
>> /dev/md127:
>> [...]
>>      Number   Major   Minor   RaidDevice State
>>         0       8       33        0      active sync   /dev/sdc1
>>         5       8       65        1      spare rebuilding   /dev/sde1
>>         2       8       17        2      active sync   /dev/sdb1
>>         4       8        1        3      spare rebuilding   /dev/sda1
>>
>> --xsdg
>>
>>
> Now you have an array where anything you have written will be okay 
> (which I guess is what you care about), but the rest of the disk is 
> uninitialised garbage that will instantly trigger a read fault if you 
> try to read it.

Because these happen to be brand new (but pre-zeroed) hard disks, it 
happens that the initialized state is all zeroes.  Where would the read 
fault come from, though?  What is there to be out of sync?  raid10 has 
no parity, and there's no redundancy across 2 disks in a 4-disk raid10.

#dd if=/dev/md127 bs=1 skip=20T count=1M | hd
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
*
00100000
1048576+0 records in
1048576+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.37091 s, 765 kB/s

And to be clear, that's way past where the resync status is right now:
$cat /proc/mdstat
Personalities : [raid1] [raid10] [raid0] [raid6] [raid5] [raid4]
md127 : active raid10 sde1[5] sda1[4] sdc1[0] sdb1[2]
       23382980608 blocks super 1.2 512K chunks 2 far-copies [4/2] [U_U_]
       [=============>.......]  recovery = 69.8% 
(8170708928/11691490304) finish=415.6min speed=141166K/sec
       bitmap: 13/175 pages [52KB], 65536KB chunk

--xsdg

> 
> You need to set off a scrub, which will do those reads and get the array 
> itself (not just your data) into a sane state.
> 
> https://archive.kernel.org/oldwiki/raid.wiki.kernel.org/
> 
> Ignore the obsolete content crap. Somebody clearly thinks that replacing 
> USER documentation by double-dutch programmer documentation (aimed at a 
> completely different audience) is a good idea ...
> 
> Cheers,
> Wol

Re: Kernel mistakenly "starts" resync on fully-degraded, newly-created raid10 array

[Wol]

On 22/06/2025 19:03, Omari Stephens wrote:
> On 6/22/25 09:30, Wol wrote:
>> On 22/06/2025 02:39, Omari Stephens wrote:

>>> |At that point, /proc/mdstat showed the following, which makes no 
>>> sense:|
>>>
>> Why doesn't it make any sense? to)
>>
>> Don't forget a raid-10 in linux is NOT a two raid-0s in a raid-1, it's 
>> its own thing entirely.
> 
> Understood.  I've been using Linux MD raid10 for over a decade.  I've 
> read through this (and other references) in depth:
> https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10
> 
> My question is this: Suppose you create a 4-drive array.  2 drives are 
> missing.  What data is there to synchronize?  What should get copied 
> where, or what should get recomputed and written where?

Ah - I get that. But presumably raid *always* fires up a resync (unless 
told not to) when an array is created - and see below ...
> 
> To my understanding, in that situation, each block in the array only 
> appears in one place on the physical media, and there is no redundancy 
> or parity for any block that could be out of sync.

No. It's trying to resync a *broken* array. Unless you had explicitly 
asked for just one mirror (which I don't think you did). A raid-10 needs 
"no of mirrors plus one". I can see why you would think "why doesn't it 
just fire up as a plain raid-1?", but I guess it just doesn't. I've 
never tried to create an array in a broken state.
> 
> When you read from the array, yes, you're going to get interleaved bits 
> of whatever happened to be on the physical media to start with, but 
> that's basically the same as reading directly from any new physical 
> media -- it's not initialized until it's initialized, and until it is, 
> you don't know what you're going to read.
> 
Yup. Read-before-write is going to return random junk - not particularly 
a good idea - but I don't think you can assume that --assume-clean will 
actually create a clean array. My fu isn't up to guessing what will 
happen when, in particular for higher raids even with zeroed data the 
parity will be one. (And I would never assume a new drive was zeroed. To 
me it has as much evidence behind it as an old wive's tale. Other people 
may have real evidence -I don't.)

What you did worked, and in hindsight you can see how, but with just two 
drives, I personally wouldn't even have tried. Anyways, all's well that 
ends well, as they say ...

Cheers,
Wol