* Strange behavior when I grow raid0
@ 2021-12-10 22:25 Jani Partanen
2022-01-03 17:20 ` Mariusz Tkaczyk
0 siblings, 1 reply; 2+ messages in thread
From: Jani Partanen @ 2021-12-10 22:25 UTC (permalink / raw)
To: linux-raid
Hello,
I recently migrated from btrfs raid-5 to md raid-5 and because I didn't
have enough drives to go directly md raid-5, I basicly did this:
- 2 disks raid-0
- added 3rd disk with grow when I had moved data, still raid-0
- added 4th disk with grow when I had moved more data, still raid-0
- added 5th disk with grow and converted to raid-5
Everything went fine other than I could not add internal bitmap because
array was created as raid-0 originally but thats not an issue, I'll
store bitmap to my /boot, it's raid-1 on SSD and ext4.
Today I noticed that my array parity layout is parity-last and I started
to wonder why is that, then I did remember that there was some strange
things happening when I did add 4th drive and I made testing and was
able to replicate what happened.
[root@nas ~]# mdadm -C /dev/md/testraid -l 0 -n 2 -N testraid
/dev/loop21 /dev/loop22
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/testraid started.
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid0
Array Size : 98304 (96.00 MiB 100.66 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Fri Dec 10 22:52:01 2021
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : -unknown-
Chunk Size : 512K
Consistency Policy : none
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 0
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
[root@nas ~]# mdadm --grow /dev/md/testraid -n 3 -l 0 --add /dev/loop23
mdadm: level of /dev/md/testraid changed to raid4
mdadm: added /dev/loop23
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid0
Array Size : 147456 (144.00 MiB 150.99 MB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Fri Dec 10 22:53:33 2021
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Chunk Size : 512K
Consistency Policy : none
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 18
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
3 7 23 2 active sync /dev/loop23
[root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0 --add /dev/loop24
mdadm: level of /dev/md/testraid changed to raid4
mdadm: added /dev/loop24
mdadm: Need to backup 6144K of critical section..
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid4
Array Size : 196608 (192.00 MiB 201.33 MB)
Used Dev Size : 49152 (48.00 MiB 50.33 MB)
Raid Devices : 5
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Fri Dec 10 22:59:56 2021
State : clean, degraded
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Chunk Size : 512K
Consistency Policy : resync
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 39
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
3 7 23 2 active sync /dev/loop23
4 7 24 3 active sync /dev/loop24
- 0 0 4 removed
[root@nas ~]# mdadm --grow /dev/md/testraid -n 5 -l 5 --add /dev/loop25
mdadm: level of /dev/md/testraid changed to raid5
mdadm: added /dev/loop25
[root@nas ~]# mdadm -D /dev/md/testraid
/dev/md/testraid:
Version : 1.2
Creation Time : Fri Dec 10 22:52:01 2021
Raid Level : raid5
Array Size : 196608 (192.00 MiB 201.33 MB)
Used Dev Size : 49152 (48.00 MiB 50.33 MB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Fri Dec 10 23:00:39 2021
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : parity-last
Chunk Size : 512K
Consistency Policy : resync
Name : nas:testraid (local to host nas)
UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
Events : 59
Number Major Minor RaidDevice State
0 7 21 0 active sync /dev/loop21
1 7 22 1 active sync /dev/loop22
3 7 23 2 active sync /dev/loop23
4 7 24 3 active sync /dev/loop24
5 7 25 4 active sync /dev/loop25
Notice how array stay at raid4 degraded after I add 4th drive. Why is
this happening? It doesn't happen when I added 3rd drive, it did reshape
it through raid-4 but when it was finished it returned raid-0.
I also did another test where I made extra step after adding 4th disk:
[root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0
mdadm: level of /dev/md/testraid changed to raid0
[root@nas ~]# cat /proc/mdstat
Personalities : [raid1] [raid0] [linear] [raid6] [raid5] [raid4]
md118 : active raid0 loop24[4] loop23[3] loop22[1] loop21[0]
196608 blocks super 1.2 512k chunks
And when I did add 5th drive and converted array to raid-5, it correctly
defaults layout to left-symmetric.
Is this some sort of bug or working as intented? Also why is this "Need
to backup 6144K of critical section.." happening when I add 4th disk but
not when I add 3rd disk?
distro: Fedora 35
kernel: 5.15.6
mdadm: v4.2-rc2
// JiiPee
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: Strange behavior when I grow raid0
2021-12-10 22:25 Strange behavior when I grow raid0 Jani Partanen
@ 2022-01-03 17:20 ` Mariusz Tkaczyk
0 siblings, 0 replies; 2+ messages in thread
From: Mariusz Tkaczyk @ 2022-01-03 17:20 UTC (permalink / raw)
To: Jani Partanen; +Cc: linux-raid
Hi Jani,
On Sat, 11 Dec 2021 00:25:14 +0200
Jani Partanen <jiipee@sotapeli.fi> wrote:
> Hello,
>
> I recently migrated from btrfs raid-5 to md raid-5 and because I
> didn't have enough drives to go directly md raid-5, I basicly did
> this:
> - 2 disks raid-0
> - added 3rd disk with grow when I had moved data, still raid-0
> - added 4th disk with grow when I had moved more data, still raid-0
> - added 5th disk with grow and converted to raid-5
>
> Everything went fine other than I could not add internal bitmap
> because array was created as raid-0 originally but thats not an
> issue, I'll store bitmap to my /boot, it's raid-1 on SSD and ext4.
>
> Today I noticed that my array parity layout is parity-last and I
> started to wonder why is that, then I did remember that there was
> some strange things happening when I did add 4th drive and I made
> testing and was able to replicate what happened.
>
> [root@nas ~]# mdadm -C /dev/md/testraid -l 0 -n 2 -N testraid
> /dev/loop21 /dev/loop22
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md/testraid started.
>
>
> [root@nas ~]# mdadm -D /dev/md/testraid
> /dev/md/testraid:
> Version : 1.2
> Creation Time : Fri Dec 10 22:52:01 2021
> Raid Level : raid0
> Array Size : 98304 (96.00 MiB 100.66 MB)
> Raid Devices : 2
> Total Devices : 2
> Persistence : Superblock is persistent
>
> Update Time : Fri Dec 10 22:52:01 2021
> State : clean
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : -unknown-
> Chunk Size : 512K
>
> Consistency Policy : none
>
> Name : nas:testraid (local to host nas)
> UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
> Events : 0
>
> Number Major Minor RaidDevice State
> 0 7 21 0 active sync /dev/loop21
> 1 7 22 1 active sync /dev/loop22
>
>
> [root@nas ~]# mdadm --grow /dev/md/testraid -n 3 -l 0 --add
> /dev/loop23 mdadm: level of /dev/md/testraid changed to raid4
> mdadm: added /dev/loop23
>
>
> [root@nas ~]# mdadm -D /dev/md/testraid
> /dev/md/testraid:
> Version : 1.2
> Creation Time : Fri Dec 10 22:52:01 2021
> Raid Level : raid0
> Array Size : 147456 (144.00 MiB 150.99 MB)
> Raid Devices : 3
> Total Devices : 3
> Persistence : Superblock is persistent
>
> Update Time : Fri Dec 10 22:53:33 2021
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 0
> Spare Devices : 0
>
> Chunk Size : 512K
>
> Consistency Policy : none
>
> Name : nas:testraid (local to host nas)
> UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
> Events : 18
>
> Number Major Minor RaidDevice State
> 0 7 21 0 active sync /dev/loop21
> 1 7 22 1 active sync /dev/loop22
> 3 7 23 2 active sync /dev/loop23
>
>
> [root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0 --add
> /dev/loop24 mdadm: level of /dev/md/testraid changed to raid4
> mdadm: added /dev/loop24
> mdadm: Need to backup 6144K of critical section..
>
>
> [root@nas ~]# mdadm -D /dev/md/testraid
> /dev/md/testraid:
> Version : 1.2
> Creation Time : Fri Dec 10 22:52:01 2021
> Raid Level : raid4
> Array Size : 196608 (192.00 MiB 201.33 MB)
> Used Dev Size : 49152 (48.00 MiB 50.33 MB)
> Raid Devices : 5
> Total Devices : 4
> Persistence : Superblock is persistent
>
> Update Time : Fri Dec 10 22:59:56 2021
> State : clean, degraded
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 0
>
> Chunk Size : 512K
>
> Consistency Policy : resync
>
> Name : nas:testraid (local to host nas)
> UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
> Events : 39
>
> Number Major Minor RaidDevice State
> 0 7 21 0 active sync /dev/loop21
> 1 7 22 1 active sync /dev/loop22
> 3 7 23 2 active sync /dev/loop23
> 4 7 24 3 active sync /dev/loop24
> - 0 0 4 removed
>
>
> [root@nas ~]# mdadm --grow /dev/md/testraid -n 5 -l 5 --add
> /dev/loop25 mdadm: level of /dev/md/testraid changed to raid5
> mdadm: added /dev/loop25
>
>
> [root@nas ~]# mdadm -D /dev/md/testraid
> /dev/md/testraid:
> Version : 1.2
> Creation Time : Fri Dec 10 22:52:01 2021
> Raid Level : raid5
> Array Size : 196608 (192.00 MiB 201.33 MB)
> Used Dev Size : 49152 (48.00 MiB 50.33 MB)
> Raid Devices : 5
> Total Devices : 5
> Persistence : Superblock is persistent
>
> Update Time : Fri Dec 10 23:00:39 2021
> State : clean
> Active Devices : 5
> Working Devices : 5
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : parity-last
> Chunk Size : 512K
>
> Consistency Policy : resync
>
> Name : nas:testraid (local to host nas)
> UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371
> Events : 59
>
> Number Major Minor RaidDevice State
> 0 7 21 0 active sync /dev/loop21
> 1 7 22 1 active sync /dev/loop22
> 3 7 23 2 active sync /dev/loop23
> 4 7 24 3 active sync /dev/loop24
> 5 7 25 4 active sync /dev/loop25
>
>
> Notice how array stay at raid4 degraded after I add 4th drive. Why is
> this happening? It doesn't happen when I added 3rd drive, it did
> reshape it through raid-4 but when it was finished it returned raid-0.
When you are starting reshape of raid0 then array level is temporarily
changed to level4 for simple reason, reshape algorithm is already
implemented there. After migration it goes back to raid0. raid4 is
degraded because we don't have parity disk.
Reshape is managed by userspace. The process is working in background
(fork or systemd service). After the reshape it should switch
level back to raid0 but it failed in your case.
Please find out how reshape is managed in all cases (fork or
systemd service: mdadm-grow-continue). I think that there should
be only one grow process waiting/working to reshape end.
>
> I also did another test where I made extra step after adding 4th disk:
>
> [root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0
> mdadm: level of /dev/md/testraid changed to raid0
> [root@nas ~]# cat /proc/mdstat
> Personalities : [raid1] [raid0] [linear] [raid6] [raid5] [raid4]
> md118 : active raid0 loop24[4] loop23[3] loop22[1] loop21[0]
> 196608 blocks super 1.2 512k chunks
>
>
> And when I did add 5th drive and converted array to raid-5, it
> correctly defaults layout to left-symmetric.
If arrays stays as degraded raid4 then it is same as raid5 with
'parity-last' consistency policy, this is why you have "parity-last"
layout. It is caused by wrong level.
> Is this some sort of bug or working as intented? Also why is this
> "Need to backup 6144K of critical section.." happening when I add 4th
> disk but not when I add 3rd disk?
>
It could be related with first problem, if mdadm decides to run fork,
then it could produce this message. As above, please check first how
reshape is managed in all cases.
> distro: Fedora 35
> kernel: 5.15.6
> mdadm: v4.2-rc2
>
Thanks,
Mariusz
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2022-01-03 17:20 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-12-10 22:25 Strange behavior when I grow raid0 Jani Partanen
2022-01-03 17:20 ` Mariusz Tkaczyk
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.