* Can't replace drive in imsm RAID 5 array, spare not shown
@ 2024-10-06 6:00 19 Devices
2024-10-09 10:09 ` Mariusz Tkaczyk
0 siblings, 1 reply; 4+ messages in thread
From: 19 Devices @ 2024-10-06 6:00 UTC (permalink / raw)
To: linux-raid
Hi, I have a 4 drive imsm RAID 5 array which is working fine. I want to remove one of the drives, sda, and replace it with a spare, sdc. From man mdadm I understand that add - fail - remove is the way to go but this does not work.
Before:
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md124 : active raid5 sdd[3] sdb[2] sda[1] sde[0]
2831155200 blocks super external:/md126/0 level 5,
128k chunk, algorithm 0 [4/4] [UUUU]
md125 : active raid5 sdd[3] sdb[2] sda[1] sde[0]
99116032 blocks super external:/md126/1 level 5, 1
28k chunk, algorithm 0 [4/4] [UUUU]
md126 : inactive sda[3](S) sdb[2](S) sdd[1](S) sde[0](S)
14681 blocks super external:imsm
unused devices: <none>
I can add (or add-spare) which increases the size of the container and though I can't see any spare drives listed by mdadm, it appears as SPARE DISK in the Intel option ROM after a reboot.
$ sudo mdadm --zero-superblock /dev/sdc
$ sudo mdadm /dev/md/imsm1 --add-spare /de
v/sdc
mdadm: added /dev/sdc
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md124 : active raid5 sdd[3] sdb[2] sda[1] sde[0]
2831155200 blocks super external:/md126/0 level 5,
128k chunk, algorithm 0 [4/4] [UUUU]
md125 : active raid5 sdd[3] sdb[2] sda[1] sde[0]
99116032 blocks super external:/md126/1 level 5, 1
28k chunk, algorithm 0 [4/4] [UUUU]
md126 : inactive sdc[4](S) sda[3](S) sdb[2](S) sdd[1](S) sde[0](S)
15786 blocks super external:imsm
unused devices: <none>
$
No spare devices listed here:
$ sudo mdadm -D /dev/md/imsm1
/dev/md/imsm1:
Version : imsm
Raid Level : container
Total Devices : 5
Working Devices : 5
UUID : bdb7f495:21b8c189:e496c216:6f2d6c4c
Member Arrays : /dev/md/md1_0 /dev/md/md0_0
Number Major Minor RaidDevice
- 8 64 - /dev/sde
- 8 32 - /dev/sdc
- 8 0 - /dev/sda
- 8 48 - /dev/sdd
- 8 16 - /dev/sdb
$
Trying to remove sda fails.
$ sudo mdadm --fail /dev/md126 /dev/sda
mdadm: Cannot remove /dev/sda from /dev/md126, array will be failed.
sda is 2TB, the others are 1TB - is that a problem?
smartctl shows 2 drives don't support SCT and it's disabled on the other 3.
There's a very similar question here from Edwin in 2017:
https://unix.stackexchange.com/questions/372908/add-hot-spare-drive-to-intel-rst-onboard-raid#372920
The only reply points to an Intel doc which uses the standard command to add a drive but doesn't show the result.
$ uname -a
Linux Intel 6.9.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 26
May 2024 01:30:29 +0000 x86_64 GNU/Linux
$ mdadm --version
mdadm - v4.3 - 2024-02-15
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: Can't replace drive in imsm RAID 5 array, spare not shown 2024-10-06 6:00 Can't replace drive in imsm RAID 5 array, spare not shown 19 Devices @ 2024-10-09 10:09 ` Mariusz Tkaczyk 2024-10-10 8:25 ` 19 Devices 0 siblings, 1 reply; 4+ messages in thread From: Mariusz Tkaczyk @ 2024-10-09 10:09 UTC (permalink / raw) To: 19 Devices; +Cc: linux-raid On Sun, 06 Oct 2024 07:00:18 +0100 19 Devices <19devices@gmail.com> wrote: > Hi, I have a 4 drive imsm RAID 5 array which is working fine. I want to > remove one of the drives, sda, and replace it with a spare, sdc. From man > mdadm I understand that add - fail - remove is the way to go but this does > not work. > > Before: > $ cat /proc/mdstat > Personalities : [raid6] [raid5] [raid4] > md124 : active raid5 sdd[3] sdb[2] sda[1] sde[0] > 2831155200 blocks super external:/md126/0 level 5, > 128k chunk, algorithm 0 [4/4] [UUUU] > > md125 : active raid5 sdd[3] sdb[2] sda[1] sde[0] > 99116032 blocks super external:/md126/1 level 5, 1 > 28k chunk, algorithm 0 [4/4] [UUUU] > > md126 : inactive sda[3](S) sdb[2](S) sdd[1](S) sde[0](S) > 14681 blocks super external:imsm > > unused devices: <none> > > > I can add (or add-spare) which increases the size of the container and though > I can't see any spare drives listed by mdadm, it appears as SPARE DISK in the > Intel option ROM after a reboot. > > $ sudo mdadm --zero-superblock /dev/sdc > > $ sudo mdadm /dev/md/imsm1 --add-spare /de > v/sdc > mdadm: added /dev/sdc > > $ cat /proc/mdstat > Personalities : [raid6] [raid5] [raid4] > md124 : active raid5 sdd[3] sdb[2] sda[1] sde[0] > 2831155200 blocks super external:/md126/0 level 5, > 128k chunk, algorithm 0 [4/4] [UUUU] > > md125 : active raid5 sdd[3] sdb[2] sda[1] sde[0] > 99116032 blocks super external:/md126/1 level 5, 1 > 28k chunk, algorithm 0 [4/4] [UUUU] > > md126 : inactive sdc[4](S) sda[3](S) sdb[2](S) sdd[1](S) sde[0](S) > 15786 blocks super external:imsm > > unused devices: <none> > $ > > > No spare devices listed here: > > $ sudo mdadm -D /dev/md/imsm1 > /dev/md/imsm1: > Version : imsm > Raid Level : container > Total Devices : 5 > > Working Devices : 5 > > > UUID : bdb7f495:21b8c189:e496c216:6f2d6c4c > Member Arrays : /dev/md/md1_0 /dev/md/md0_0 > > Number Major Minor RaidDevice > > - 8 64 - /dev/sde > - 8 32 - /dev/sdc > - 8 0 - /dev/sda > - 8 48 - /dev/sdd > - 8 16 - /dev/sdb > $ > Hello, I know. It is fine. From container point of view these all are spares. Nobody ever complained about that so we did not fixed it :) The most important is that all drives are here. To detect spares you must compare this list with list from #mdadm --detail /dev/md124 (member array). Drives that are not used in member array are spares. > > Trying to remove sda fails. > > $ sudo mdadm --fail /dev/md126 /dev/sda > mdadm: Cannot remove /dev/sda from /dev/md126, array will be failed. It might be an issue in mdadm, we added this and later we added fixes: Commit: https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=fc6fd4063769f4194c3fb8f77b32b2819e140fb9 Fixes: https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=b3e7b7eb1dfedd7cbd9a3800e884941f67d94c96 https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=461fae7e7809670d286cc19aac5bfa861c29f93a but your release is mdadm-4.3, all fixes should be there. It might be a new bug. Try: #mdadm -If sda but please do not abuse it (just use it one time because it may fail your array). According to mdstat it should be safe in this case. If you can do some investigation, I would be tankful, I expect issues in enough() function. Thanks, Mariusz > > sda is 2TB, the others are 1TB - is that a problem? > > smartctl shows 2 drives don't support SCT and it's disabled on the other 3. > > There's a very similar question here from Edwin in 2017: > https://unix.stackexchange.com/questions/372908/add-hot-spare-drive-to-intel-rst-onboard-raid#372920 > > The only reply points to an Intel doc which uses the standard command to add > a drive but doesn't show the result. > > $ uname -a > Linux Intel 6.9.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 26 > May 2024 01:30:29 +0000 x86_64 GNU/Linux > > $ mdadm --version > mdadm - v4.3 - 2024-02-15 > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Can't replace drive in imsm RAID 5 array, spare not shown 2024-10-09 10:09 ` Mariusz Tkaczyk @ 2024-10-10 8:25 ` 19 Devices 2024-10-10 10:42 ` Mariusz Tkaczyk 0 siblings, 1 reply; 4+ messages in thread From: 19 Devices @ 2024-10-10 8:25 UTC (permalink / raw) To: Mariusz Tkaczyk; +Cc: linux-raid On 9 October 2024 11:09:40 BST, Mariusz Tkaczyk <mariusz.tkaczyk@linux.intel.com> wrote: >On Sun, 06 Oct 2024 07:00:18 +0100 >19 Devices <19devices@gmail.com> wrote: > >> Hi, I have a 4 drive imsm RAID 5 array which is working fine. I want to >> remove one of the drives, sda, and replace it with a spare, sdc. From man >> mdadm I understand that add - fail - remove is the way to go but this does >> not work. >> >> Before: >> $ cat /proc/mdstat >> Personalities : [raid6] [raid5] [raid4] >> md124 : active raid5 sdd[3] sdb[2] sda[1] sde[0] >> 2831155200 blocks super external:/md126/0 level 5, >> 128k chunk, algorithm 0 [4/4] [UUUU] >> >> md125 : active raid5 sdd[3] sdb[2] sda[1] sde[0] >> 99116032 blocks super external:/md126/1 level 5, 1 >> 28k chunk, algorithm 0 [4/4] [UUUU] >> >> md126 : inactive sda[3](S) sdb[2](S) sdd[1](S) sde[0](S) >> 14681 blocks super external:imsm >> >> unused devices: <none> >> >> >> I can add (or add-spare) which increases the size of the container and though >> I can't see any spare drives listed by mdadm, it appears as SPARE DISK in the >> Intel option ROM after a reboot. >> >> $ sudo mdadm --zero-superblock /dev/sdc >> >> $ sudo mdadm /dev/md/imsm1 --add-spare /de >> v/sdc >> mdadm: added /dev/sdc >> >> $ cat /proc/mdstat >> Personalities : [raid6] [raid5] [raid4] >> md124 : active raid5 sdd[3] sdb[2] sda[1] sde[0] >> 2831155200 blocks super external:/md126/0 level 5, >> 128k chunk, algorithm 0 [4/4] [UUUU] >> >> md125 : active raid5 sdd[3] sdb[2] sda[1] sde[0] >> 99116032 blocks super external:/md126/1 level 5, 1 >> 28k chunk, algorithm 0 [4/4] [UUUU] >> >> md126 : inactive sdc[4](S) sda[3](S) sdb[2](S) sdd[1](S) sde[0](S) >> 15786 blocks super external:imsm >> >> unused devices: <none> >> $ >> >> >> No spare devices listed here: >> >> $ sudo mdadm -D /dev/md/imsm1 >> /dev/md/imsm1: >> Version : imsm >> Raid Level : container >> Total Devices : 5 >> >> Working Devices : 5 >> >> >> UUID : bdb7f495:21b8c189:e496c216:6f2d6c4c >> Member Arrays : /dev/md/md1_0 /dev/md/md0_0 >> >> Number Major Minor RaidDevice >> >> - 8 64 - /dev/sde >> - 8 32 - /dev/sdc >> - 8 0 - /dev/sda >> - 8 48 - /dev/sdd >> - 8 16 - /dev/sdb >> $ >> >Hello, > >I know. It is fine. From container point of view these all are spares. >Nobody ever complained about that so we did not fixed it :) >The most important is that all drives are here. > >To detect spares you must compare this list with list from #mdadm --detail >/dev/md124 (member array). Drives that are not used in member array are spares. >> >> Trying to remove sda fails. >> >> $ sudo mdadm --fail /dev/md126 /dev/sda >> mdadm: Cannot remove /dev/sda from /dev/md126, array will be failed. > >It might be an issue in mdadm, we added this and later we added fixes: > >Commit: >https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=fc6fd4063769f4194c3fb8f77b32b2819e140fb9 > >Fixes: >https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=b3e7b7eb1dfedd7cbd9a3800e884941f67d94c96 >https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=461fae7e7809670d286cc19aac5bfa861c29f93a > >but your release is mdadm-4.3, all fixes should be there. It might be a new bug. > >Try: >#mdadm -If sda >but please do not abuse it (just use it one time because it may fail your >array). According to mdstat it should be safe in this case. > >If you can do some investigation, I would be tankful, I expect issues >in enough() function. > >Thanks, >Mariusz > >> >> sda is 2TB, the others are 1TB - is that a problem? >> >> smartctl shows 2 drives don't support SCT and it's disabled on the other 3. >> >> There's a very similar question here from Edwin in 2017: >> https://unix.stackexchange.com/questions/372908/add-hot-spare-drive-to-intel-rst-onboard-raid#372920 >> >> The only reply points to an Intel doc which uses the standard command to add >> a drive but doesn't show the result. >> >> $ uname -a >> Linux Intel 6.9.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 26 >> May 2024 01:30:29 +0000 x86_64 GNU/Linux >> >> $ mdadm --version >> mdadm - v4.3 - 2024-02-15 >> > --------------------------------------- Thank you Mariusz, that (--incremental --fail) worked: # mdadm -If sda mdadm: set sda faulty in md124 mdadm: set sda faulty in md125 mdadm: hot removed sda from md126 # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md124 : active raid5 sdc[4] sdd[3] sdb[2] sde[0] 2831155200 blocks super external:/md126/0 level 5, 128k chunk, algorithm 0 [4/3] [UU_U] [>....................] recovery = 0.2% (2275456 /943718400) finish=222.5min speed=70515K/sec md125 : active raid5 sdc[4] sdd[3] sdb[2] sde[0] 99116032 blocks super external:/md126/1 level 5, 1 28k chunk, algorithm 0 [4/3] [UU_U] resync=DELAYED md126 : inactive sdc[4](S) sdb[2](S) sdd[1](S) sde[0](S) 10585 blocks super external:imsm unused devices: <none> # # journalctl -f kernel: md/raid:md124: Disk failure on sda, disabling device. kernel: md/raid:md124: Operation continuing on 3 devices. kernel: md/raid:md125: Disk failure on sda, disabling device. kernel: md/raid:md125: Operation continuing on 3 devices. kernel: md: recovery of RAID array md124 kernel: md: delaying recovery of md125 until md124 has finished (they share one or more physical units) mdadm[628]: mdadm: Fail event detected on md device /dev/md125, component device /dev/sda mdadm[628]: mdadm: RebuildStarted event detected on md device /dev/md124 Intel mdadm[628]: mdadm: Fail event detected on md device /dev/md124, component device /dev/sda --------------------------------------- ps. Belated thanks too for your solution to my previous problem here on 2021/08/02. That fix showed no sign it had succeeded until reboot but after that all was fine. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Can't replace drive in imsm RAID 5 array, spare not shown 2024-10-10 8:25 ` 19 Devices @ 2024-10-10 10:42 ` Mariusz Tkaczyk 0 siblings, 0 replies; 4+ messages in thread From: Mariusz Tkaczyk @ 2024-10-10 10:42 UTC (permalink / raw) To: 19 Devices; +Cc: linux-raid On Thu, 10 Oct 2024 09:25:58 +0100 19 Devices <19devices@gmail.com> wrote: > ps. Belated thanks too for your solution to my previous problem here on > 2021/08/02. That fix showed no sign it had succeeded until reboot but after > that all was fine. Cool. I'm glad to hear that. Mariusz ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-10-10 10:42 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-10-06 6:00 Can't replace drive in imsm RAID 5 array, spare not shown 19 Devices 2024-10-09 10:09 ` Mariusz Tkaczyk 2024-10-10 8:25 ` 19 Devices 2024-10-10 10:42 ` Mariusz Tkaczyk
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).