is "replaceable" in 3.2 considered stable

is "replaceable" in 3.2 considered stable

[Mikael Abrahamsson]

Hello.

I have an ubuntu 12.04 system, (running ubuntu kernel 3.2), and I have a 
drive I want to swap out that has a lot of offline uncorrectable block on 
my raid6 mdadm-raid with superblock v1.2, the drive is failing.

I remembered Neil announcing the "replaceable" option, and it seems this 
came in 3.2 according to <https://lwn.net/Articles/465048/>.

Has this function as implemented in 3.2 received a lot of testing and can 
be considered safe for production use, or is it still as said in the 
email, for testing only?

My kernel:

3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: is "replaceable" in 3.2 considered stable

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

On Mon, 5 Nov 2012 05:51:18 +0100 (CET) Mikael Abrahamsson <swmike@swm.pp.se>
wrote:

> 
> Hello.
> 
> I have an ubuntu 12.04 system, (running ubuntu kernel 3.2), and I have a 
> drive I want to swap out that has a lot of offline uncorrectable block on 
> my raid6 mdadm-raid with superblock v1.2, the drive is failing.
> 
> I remembered Neil announcing the "replaceable" option, and it seems this 
> came in 3.2 according to <https://lwn.net/Articles/465048/>.
> 
> Has this function as implemented in 3.2 received a lot of testing and can 
> be considered safe for production use, or is it still as said in the 
> email, for testing only?
> 
> My kernel:
> 
> 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> 

Try it.  If it works for you, we can be more confident that it is stable :-)

Seriously I think it is safe to try but I won't promise that nothing will go
wrong.  I do promise that if something goes wrong (which is fairly unlikely),
I'll do my best to help you fix it, and think it is extremely unlikely that
the experiment will cause data loss.

One thing to be aware of:  if you need to reboot during the replacement
process, the mdadm you have installed will not be able to put the array back
together properly (it will leave out either the failing device or the
replacement).  The mdadm in the 'master' branch of my git tree can do this so
if it comes to that, there is a way forward.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: is "replaceable" in 3.2 considered stable

[Mikael Abrahamsson]

On Mon, 5 Nov 2012, NeilBrown wrote:

> Try it.  If it works for you, we can be more confident that it is stable 
> :-)

root:~# cat /sys/block/md0/md/dev-sdh/state
in_sync
root:~# echo replaceable > /sys/block/md0/md/dev-sdh/state
-bash: echo: write error: Invalid argument
root:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid6 sdi[10](S) sdg[11] sdj[15] sdh[14] sdk[13] sde[12] sdd[9] sdb[6] sdf[7] sdc[2]
       13674601024 blocks super 1.2 level 6, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
       bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>

Does this mean Ubuntu decided to not support this option in their kernel, 
or am I misunderstanding something?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: is "replaceable" in 3.2 considered stable

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

On Mon, 5 Nov 2012 09:11:54 +0100 (CET) Mikael Abrahamsson <swmike@swm.pp.se>
wrote:

> On Mon, 5 Nov 2012, NeilBrown wrote:
> 
> > Try it.  If it works for you, we can be more confident that it is stable 
> > :-)
> 
> root:~# cat /sys/block/md0/md/dev-sdh/state
> in_sync
> root:~# echo replaceable > /sys/block/md0/md/dev-sdh/state
> -bash: echo: write error: Invalid argument
> root:~# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
> md0 : active raid6 sdi[10](S) sdg[11] sdj[15] sdh[14] sdk[13] sde[12] sdd[9] sdb[6] sdf[7] sdc[2]
>        13674601024 blocks super 1.2 level 6, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
>        bitmap: 0/15 pages [0KB], 65536KB chunk
> 
> unused devices: <none>
> 
> Does this mean Ubuntu decided to not support this option in their kernel, 
> or am I misunderstanding something?
> 

You want
   echo want_replacement > ..../state

I changed that from replaceable at some point, so I guess you are looking at
old code :-(

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: is "replaceable" in 3.2 considered stable

[Mikael Abrahamsson]

On Mon, 5 Nov 2012, NeilBrown wrote:

> You want
>   echo want_replacement > ..../state
>
> I changed that from replaceable at some point, so I guess you are looking at
> old code :-(

I was looking at the LWN article...

Still no go though:

root:~# echo want_replacement > /sys/block/md0/md/dev-sdh/state
-bash: echo: write error: Invalid argument
root:~# cat /sys/block/md0/md/dev-sdh/state
in_sync
root:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid6 sdi[10](S) sdg[11] sdj[15] sdh[14] sdk[13] sde[12] sdd[9] sdb[6] sdf[7] sdc[2]
       13674601024 blocks super 1.2 level 6, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
       bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: is "replaceable" in 3.2 considered stable

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]

On Mon, 5 Nov 2012 09:46:47 +0100 (CET) Mikael Abrahamsson <swmike@swm.pp.se>
wrote:

> On Mon, 5 Nov 2012, NeilBrown wrote:
> 
> > You want
> >   echo want_replacement > ..../state
> >
> > I changed that from replaceable at some point, so I guess you are looking at
> > old code :-(
> 
> I was looking at the LWN article...
> 
> Still no go though:
> 
> root:~# echo want_replacement > /sys/block/md0/md/dev-sdh/state
> -bash: echo: write error: Invalid argument
> root:~# cat /sys/block/md0/md/dev-sdh/state
> in_sync
> root:~# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
> md0 : active raid6 sdi[10](S) sdg[11] sdj[15] sdh[14] sdk[13] sde[12] sdd[9] sdb[6] sdf[7] sdc[2]
>        13674601024 blocks super 1.2 level 6, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
>        bitmap: 0/15 pages [0KB], 65536KB chunk
> 
> unused devices: <none>
> 

The code actually appears in 3.3, not 3.2.  Sorry.

The email in the lwn.net article does say:

> I hope to submit this together with support for RAID10 (and maybe some
> minimal support for RAID1) for Linux-3.3. 

It also says:


> The following series - on top of my for-linus branch which should appear in
> 3.2-rc1 eventually - implements hot-replace for RAID4/5/6.

which is probably what made you think it was in 3.2.
However is was the "for-linus branch; that would appear in 3.2-rc1, not the
series of patches :-(


So: you need 3.3 or later.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: is "replaceable" in 3.2 considered stable

[Mikael Abrahamsson]

On Mon, 5 Nov 2012, NeilBrown wrote:

> So: you need 3.3 or later.

Thanks for the clarification! I have now failed the drive instead and it's 
correctly rebuilding to what was before a spare.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: is "replaceable" in 3.2 considered stable

[Brad Campbell]

On 05/11/12 13:22, NeilBrown wrote:

>
> Try it.  If it works for you, we can be more confident that it is stable :-)
>
> Seriously I think it is safe to try but I won't promise that nothing will go
> wrong.  I do promise that if something goes wrong (which is fairly unlikely),
> I'll do my best to help you fix it, and think it is extremely unlikely that
> the experiment will cause data loss.

I have a test system set up at the moment with a RAID10 n,2 across 6 1TB 
drives + 2 spares.

I've run several hot-replace tests using 3.6.2 and found it works as 
advertised.

I also have two drives with hard write errors (excellent for failure 
testing), and trying to replace a drive with one that contains a write 
error fails the replace as you would expect.

The test machine is on UPS, so I have not done any testing that involves 
reboots during a re-sync.

Re: is "replaceable" in 3.2 considered stable

[joystick]

On 11/07/12 02:42, Brad Campbell wrote:
> On 05/11/12 13:22, NeilBrown wrote:
>
> I have a test system set up at the moment with a RAID10 n,2 across 6 
> 1TB drives + 2 spares.
>
> I've run several hot-replace tests using 3.6.2 and found it works as 
> advertised.
>
> I also have two drives with hard write errors (excellent for failure 
> testing), and trying to replace a drive with one that contains a write 
> error fails the replace as you would expect.

Thanks for your testing Brad

we still need someone to test the other case, a more common scenario I'd 
say: the disk to be replaced fails during hot-replace

> The test machine is on UPS, so I have not done any testing that 
> involves reboots during a re-sync.

And also this one...
Best simulation would be unpulling the plug so that disks do not flush
if you have still time for us, of course :-)

Thank you

Re: is "replaceable" in 3.2 considered stable

[Brad Campbell]

On 07/11/12 17:55, joystick wrote:

>
> we still need someone to test the other case, a more common scenario I'd
> say: the disk to be replaced fails during hot-replace

I suspect I can do that by creating some "media errors" using hdparm 
while the replacement is in progress.

I'd have *thought* that the procedure should be to re-direct the read to 
another mirror (it is a raid-10 after all).

>> The test machine is on UPS, so I have not done any testing that
>> involves reboots during a re-sync.
>
> And also this one...
> Best simulation would be unpulling the plug so that disks do not flush
> if you have still time for us, of course :-)

I assume to test that adequately (as in re-assemble an array with a 
replacement in progress) I'd need to upgrade mdadm to the latest git code?

No biggie, just requires a bit more tweaking the initramfs stuff.

The tests thus far have been conducted in spare drive slots in my 
production server. This afternoon I picked up a new motherboard for my 
test box so tomorrow I'll have a dedicated machine to play with.

Debian on SSD, and 10 x 1TB 7200 RPM drives that have been retired from 
active hard service (they have about 22,000 hours on them) and ready as 
a test mule.

If you put together a set of tests you'd like performed I'll be happy to 
run them and see what happens. The machine is on a managed APC PDU (yay 
Gumtree!), so remote power cycling is a lot easier than it used to be 
and I really don't mind hammering the disks with emergency parks or 
excessive cycles.

Re: is "replaceable" in 3.2 considered stable

[joystick]

On 11/07/12 14:44, Brad Campbell wrote:
> On 07/11/12 17:55, joystick wrote:
>
>>
>> we still need someone to test the other case, a more common scenario I'd
>> say: the disk to be replaced fails during hot-replace
>
> I suspect I can do that by creating some "media errors" using hdparm 
> while the replacement is in progress.

Hi Brad

That seems to be one way, but only for read errors, another would be 
with dm-flakey, which AFAIU would also allow to fail writes, and the 
faulty raid type (man mdadm)
the last two should allow to also test with write failures and hence 
complete failure of the source disk which should interrupt hotreplace 
and fallback to normal rebuild (I don't know if restarting from first 
byte or more intelligently going ahead with rebuild from the point of 
failure of hot-replace).

Then, the presence of the bad-blocks-list also changes the behaviour of 
hot-replace, so that one also would have to be tested if you are 
interested, so that makes already a lot of cases:

- read error only or readwrite error in source device
- presence of bad-block-list yes/no
(4 cases)

- write error of destination device
- power loss in the middle of hot-replace
- bad-blocks set for many devices (more than parity or mirror level) all 
on the same strip (4k wide) including the disk to be replaced, which 
AFAIR should cause a corresponding bad-block hole in the device 
destination of hotreplace, which in turn would cause read-error to be 
returned immediately by MD if that area is read.

- simultaneous hot-replace of 2+ drives; I don't know if this is 
supposed to work...

> If you put together a set of tests you'd like performed I'll be happy 
> to run them and see what happens. The machine is on a managed APC PDU 
> (yay Gumtree!), so remote power cycling is a lot easier than it used 
> to be and I really don't mind hammering the disks with emergency parks 
> or excessive cycles. 

Did that :-P
I'll tell you if some other test comes to my mind

Thanks for your work