Online Drive Replacement: BTRFS with RAID 6

All of lore.kernel.org
 help / color / mirror / Atom feed

* Online Drive Replacement: BTRFS with RAID 6
@ 2014-12-01 14:47 Oliver
  2014-12-01 19:33 ` Robert White
  0 siblings, 1 reply; 3+ messages in thread
From: Oliver @ 2014-12-01 14:47 UTC (permalink / raw)
  To: linux-btrfs

Hi All,

on a testing machine I installed four HDDs and they are configured as 
RAID6. For a test I removed one of the drives (/dev/sdk) while the 
volume was mounted and data was written to it. This worked well, as far 
as I can see. Some I/O errors were written to /var/log/syslog, but the 
volume kept working. Unfortunately the command "btrfs fi sh" did not 
show any missing drives. So I remounted the volume in degraded mode: 
"mount -t btrfs /dev/sdx1 -o remount,rw,degraded,noatime /mnt". This way 
the drive in question was reported as missing. Then I plugged in the HDD 
again (it is of course /dev/sdk again) and started a balancing in hope 
that this will restore RAID6: "btrfs filesystem balance start /mnt". Now 
the volume looks like this:

$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
     Total devices 4 FS bytes used 257.05GiB
     devid    1 size 465.76GiB used 262.03GiB path /dev/sdi1
     devid    2 size 465.76GiB used 262.00GiB path /dev/sdj1
     devid    3 size 465.76GiB used 261.03GiB path /dev/sdh1
     devid    4 size 465.76GiB used 0.00 path /dev/sdk1

How do I reinitiate /dev/sdk1? As running "btrfs fi ba start /mnt" does 
not help, I tried to remove the hdd, but

$ btrfs de de /dev/sdk1 /mnt/
ERROR: error removing the device '/dev/sdk1' - unable to go below four 
devices on raid6

A replacement does not work this way either:

$ btrfs replace start -f -r /dev/sdk1 /dev/sdk1 /mnt
/dev/sdk1 is mounted

Are there other ways to replace/reinitiate the hdd then converting to 
RAID 5?

Here are some more information about my configuration:

$   uname -a
Linux hostname 3.13.0-30-generic #55-Ubuntu SMP Fri Jul 4 21:40:53 UTC 
2014 x86_64 x86_64 x86_64 GNU/Linux
$   btrfs --version
Btrfs v3.12
$ btrfs fi df /mnt
Data, RAID6: total=263.00GiB, used=256.82GiB
System, RAID1: total=32.00MiB, used=36.00KiB
Metadata, RAID1: total=1.00GiB, used=271.13MiB

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Online Drive Replacement: BTRFS with RAID 6
  2014-12-01 14:47 Online Drive Replacement: BTRFS with RAID 6 Oliver
@ 2014-12-01 19:33 ` Robert White
  2014-12-02  7:44   ` Oliver
  0 siblings, 1 reply; 3+ messages in thread
From: Robert White @ 2014-12-01 19:33 UTC (permalink / raw)
  To: Oliver, linux-btrfs

On 12/01/2014 06:47 AM, Oliver wrote:
> Hi All,
>
> on a testing machine I installed four HDDs and they are configured as
> RAID6. For a test I removed one of the drives (/dev/sdk) while the
> volume was mounted and data was written to it. This worked well, as far
> as I can see. Some I/O errors were written to /var/log/syslog, but the
> volume kept working. Unfortunately the command "btrfs fi sh" did not
> show any missing drives. So I remounted the volume in degraded mode:
> "mount -t btrfs /dev/sdx1 -o remount,rw,degraded,noatime /mnt". This way
> the drive in question was reported as missing. Then I plugged in the HDD
> again (it is of course /dev/sdk again) and started a balancing in hope
> that this will restore RAID6: "btrfs filesystem balance start /mnt". Now
> the volume looks like this:

Since it was already running and such, remounting it as degraded was 
probably not a good thing (or even vaguely necessary).

The WIKI, in discussing add/remove and failed drives goes to great 
lengths (big red box) to discuss the current instability of RAID5/6 format.

I am guessing here but I _think_ you should do the following...

(0) Backup your data. [okay this is a test system that you deliberately 
purturbed but still... 8-) ]

Option 1:

(reasonable, but undocumented .. Either blance or scrub _ought_ to look 
at the disk sectors and trigger some re-copying from the good parts.)

The disk is in the array (again), you may just need to re-balance or 
scrub the array to get the data on the drive back in harmony with the 
state of the array overall.

Option 2:

(unlikely :: add and remove are about making the geometry smaller/larger 
and, as stated, a RAID 6 cannot be less than 4 drives by definition, so 
there is no three-drive geometry for a RAID 6.)

re-unplug the device, then use btrfs remove /dev/sdk /mnt
then re-plug-in the device and use btrfs add /dev/sdk /mnt

Option 3:

(reasonable, but undocumented :: replace by device id -- 4 in your 
example case -- instead of system path. This would, I should think, skip 
the check of /dev/sdk1's separate status)

btrfs replace start -f 4 /dev/sdk1 /mnt

Option 3a:

(got to get /dev/sdk1 back out of the list of active devices for /mnt so 
the system wont see /dev/sdk1 as "mounted" (e.g. held by a subsystem))

unplug device.
mount -o remount,degraded etc...
plug in device.
btrfs replace start -f 4 /dev/sdk1 /mnt

Option 4:

(most likely, most time consuming)

Unplug /dev/sdk. Plug it into another computer and zero a decent chunk 
of partition 1.
Plug it back into the original computer
do the replace operation as in Option 3.

This is the most-likely correct option if a simple rebalance or scrub 
doesn't work, as you will be presenting the system with three attached 
drives, one "missing" drive that will not match any necessary 
signatures, and a "new, blank" drive in its place.

===

In all cases, you may need to unmount or remount or remount degraded  in 
there somewhere, particularly because you have already done so at least 
once.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Online Drive Replacement: BTRFS with RAID 6
  2014-12-01 19:33 ` Robert White
@ 2014-12-02  7:44   ` Oliver
  0 siblings, 0 replies; 3+ messages in thread
From: Oliver @ 2014-12-02  7:44 UTC (permalink / raw)
  To: Robert White, linux-btrfs

Dear Robert,

thank you for all the possibilities! I think option 4 + 3 would be my 
prefered ones.

Until I received your answer I allready played around with that volume 
and converting it to Raid 5 seems to work too, I'll attach the steps I 
took. Soon  I'll try your solution and do some checksuming to verify the 
integrity of the files that were written while I pulled out the drive.

Please check in the steps below that I can violate Raid6 geometry by 
removing sdk again.
--------------------
$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 4 FS bytes used 257.05GiB
	devid    1 size 465.76GiB used 262.03GiB path /dev/sdi1
	devid    2 size 465.76GiB used 262.00GiB path /dev/sdj1
	devid    3 size 465.76GiB used 261.03GiB path /dev/sdh1
	devid    4 size 465.76GiB used 0.00 path /dev/sdk1


$ btrfs fi balance start -dconvert=raid5 /mnt
Done, had to relocate 261 out of 263 chunks


$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 4 FS bytes used 257.09GiB
	devid    1 size 465.76GiB used 262.03GiB path /dev/sdi1
	devid    2 size 465.76GiB used 262.00GiB path /dev/sdj1
	devid    3 size 465.76GiB used 261.03GiB path /dev/sdh1
	devid    4 size 465.76GiB used 0.00 path /dev/sdk1


$ btrfs fi df /mnt
Data, RAID5: total=522.00GiB, used=256.82GiB
System, RAID1: total=32.00MiB, used=56.00KiB
Metadata, RAID1: total=1.00GiB, used=271.49MiB


$ btrfs de de /dev/sdk1 /mnt


$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 3 FS bytes used 257.09GiB
	devid    1 size 465.76GiB used 262.03GiB path /dev/sdi1
	devid    2 size 465.76GiB used 262.00GiB path /dev/sdj1
	devid    3 size 465.76GiB used 261.03GiB path /dev/sdh1


$ btrfs de ad /dev/sdk1 /mnt
/dev/sdk1 appears to contain an existing filesystem (btrfs).
Use the -f option to force overwrite.


$ btrfs de ad -f /dev/sdk1 /mnt
/dev/sdk1 is mounted


$ mount
...
/dev/sdi1 on /mnt type btrfs (rw,degraded)


$ mount -t btrfs -o remount,rw /dev/sdi1 /mnt


$ btrfs de ad -f /dev/sdk1 /mnt
/dev/sdk1 is mounted


$ umount /dev/sdk1
umount: /dev/sdk1 ist nicht eingehängt


$ dd if=/dev/zero of=/dev/sdk bs=1M count=100
100+0 Datensätze ein
100+0 Datensätze aus
104857600 Bytes (105 MB) kopiert, 1,68492 s, 62,2 MB/s

$ fdisk /dev/sdk # o+n+w

$ btrfs de ad -f /dev/sdk1 /mnt
/dev/sdk1 is mounted
# still not able to mount sdk, but this is most likely the cause:

$ btrfs fi sh /dev/sdk1
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 3 FS bytes used 257.09GiB
	devid    1 size 465.76GiB used 262.03GiB path /dev/sdi1
	devid    2 size 465.76GiB used 262.00GiB path /dev/sdj1
	devid    3 size 465.76GiB used 261.03GiB path /dev/sdh1


$ wipefs /dev/sdk1
# this did not help, wipefs does not see any signatures


# I don't know what I did yesterday, but it suddently worked:
$ btrfs de ad -f /dev/sdk1 /mnt

$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 4 FS bytes used 257.05GiB
	devid    1 size 465.76GiB used 131.00GiB path /dev/sdi1
	devid    2 size 465.76GiB used 130.03GiB path /dev/sdj1
	devid    3 size 465.76GiB used 131.03GiB path /dev/sdh1
	devid    4 size 465.76GiB used 0.00 path /dev/sdk1

$ btrfs fi balance start -dconvert=raid6 /mnt
Done, had to relocate 130 out of 132 chunks

$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 4 FS bytes used 257.09GiB
	devid    1 size 465.76GiB used 131.00GiB path /dev/sdi1
	devid    2 size 465.76GiB used 130.03GiB path /dev/sdj1
	devid    3 size 465.76GiB used 131.03GiB path /dev/sdh1
	devid    4 size 465.76GiB used 130.00GiB path /dev/sdk1

$ btrfs de de /dev/sdk1 /mnt
# now I can remove sdk and violate Raid6 geometry:

$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 3 FS bytes used 257.03GiB
	devid    1 size 465.76GiB used 267.00GiB path /dev/sdi1
	devid    2 size 465.76GiB used 266.03GiB path /dev/sdj1
	devid    3 size 465.76GiB used 267.03GiB path /dev/sdh1

$ btrfs fi df /mnt
Data, RAID6: total=266.00GiB, used=256.77GiB
System, RAID1: total=32.00MiB, used=52.00KiB
Metadata, RAID1: total=1.00GiB, used=270.72MiB
$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 4 FS bytes used 257.05GiB
	devid    1 size 465.76GiB used 262.03GiB path /dev/sdi1
	devid    2 size 465.76GiB used 262.00GiB path /dev/sdj1
	devid    3 size 465.76GiB used 261.03GiB path /dev/sdh1
	devid    4 size 465.76GiB used 0.00 path /dev/sdk1


$ btrfs fi balance start -dconvert=raid5 /mnt
Done, had to relocate 261 out of 263 chunks


$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 4 FS bytes used 257.09GiB
	devid    1 size 465.76GiB used 262.03GiB path /dev/sdi1
	devid    2 size 465.76GiB used 262.00GiB path /dev/sdj1
	devid    3 size 465.76GiB used 261.03GiB path /dev/sdh1
	devid    4 size 465.76GiB used 0.00 path /dev/sdk1


$ btrfs fi df /mnt
Data, RAID5: total=522.00GiB, used=256.82GiB
System, RAID1: total=32.00MiB, used=56.00KiB
Metadata, RAID1: total=1.00GiB, used=271.49MiB


$ btrfs de de /dev/sdk1 /mnt


$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 3 FS bytes used 257.09GiB
	devid    1 size 465.76GiB used 262.03GiB path /dev/sdi1
	devid    2 size 465.76GiB used 262.00GiB path /dev/sdj1
	devid    3 size 465.76GiB used 261.03GiB path /dev/sdh1


$ btrfs de ad /dev/sdk1 /mnt
/dev/sdk1 appears to contain an existing filesystem (btrfs).
Use the -f option to force overwrite.


$ btrfs de ad -f /dev/sdk1 /mnt
/dev/sdk1 is mounted


$ mount
...
/dev/sdi1 on /mnt type btrfs (rw,degraded)


$ mount -t btrfs -o remount,rw /dev/sdi1 /mnt


$ btrfs de ad -f /dev/sdk1 /mnt
/dev/sdk1 is mounted


$ umount /dev/sdk1
umount: /dev/sdk1 ist nicht eingehängt


$ dd if=/dev/zero of=/dev/sdk bs=1M count=100
100+0 Datensätze ein
100+0 Datensätze aus
104857600 Bytes (105 MB) kopiert, 1,68492 s, 62,2 MB/s

$ fdisk /dev/sdk # o+n+w

$ btrfs de ad -f /dev/sdk1 /mnt
/dev/sdk1 is mounted
# still not able to mount sdk, but this is most likely the cause:

$ btrfs fi sh /dev/sdk1
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 3 FS bytes used 257.09GiB
	devid    1 size 465.76GiB used 262.03GiB path /dev/sdi1
	devid    2 size 465.76GiB used 262.00GiB path /dev/sdj1
	devid    3 size 465.76GiB used 261.03GiB path /dev/sdh1


$ wipefs /dev/sdk1
# this did not help, wipefs does not see any signatures


# I don't know what I did yesterday, but it suddently worked:
$ btrfs de ad -f /dev/sdk1 /mnt

$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 4 FS bytes used 257.05GiB
	devid    1 size 465.76GiB used 131.00GiB path /dev/sdi1
	devid    2 size 465.76GiB used 130.03GiB path /dev/sdj1
	devid    3 size 465.76GiB used 131.03GiB path /dev/sdh1
	devid    4 size 465.76GiB used 0.00 path /dev/sdk1

$ btrfs fi balance start -dconvert=raid6 /mnt
Done, had to relocate 130 out of 132 chunks

$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 4 FS bytes used 257.09GiB
	devid    1 size 465.76GiB used 131.00GiB path /dev/sdi1
	devid    2 size 465.76GiB used 130.03GiB path /dev/sdj1
	devid    3 size 465.76GiB used 131.03GiB path /dev/sdh1
	devid    4 size 465.76GiB used 130.00GiB path /dev/sdk1

$ btrfs de de /dev/sdk1 /mnt
# now I can remove sdk and violate Raid6 geometry:

$ btrfs fi sh
Label: none  uuid: 28410e37-77c1-4c01-8075-0d5068d9ffc2
	Total devices 3 FS bytes used 257.03GiB
	devid    1 size 465.76GiB used 267.00GiB path /dev/sdi1
	devid    2 size 465.76GiB used 266.03GiB path /dev/sdj1
	devid    3 size 465.76GiB used 267.03GiB path /dev/sdh1

$ btrfs fi df /mnt
Data, RAID6: total=266.00GiB, used=256.77GiB
System, RAID1: total=32.00MiB, used=52.00KiB
Metadata, RAID1: total=1.00GiB, used=270.72MiB



Am 01.12.2014 um 20:33 schrieb Robert White:
> On 12/01/2014 06:47 AM, Oliver wrote:
>> Hi All,
>>
>> on a testing machine I installed four HDDs and they are configured as
>> RAID6. For a test I removed one of the drives (/dev/sdk) while the
>> volume was mounted and data was written to it. This worked well, as far
>> as I can see. Some I/O errors were written to /var/log/syslog, but the
>> volume kept working. Unfortunately the command "btrfs fi sh" did not
>> show any missing drives. So I remounted the volume in degraded mode:
>> "mount -t btrfs /dev/sdx1 -o remount,rw,degraded,noatime /mnt". This way
>> the drive in question was reported as missing. Then I plugged in the HDD
>> again (it is of course /dev/sdk again) and started a balancing in hope
>> that this will restore RAID6: "btrfs filesystem balance start /mnt". Now
>> the volume looks like this:
>
> Since it was already running and such, remounting it as degraded was
> probably not a good thing (or even vaguely necessary).
>
> The WIKI, in discussing add/remove and failed drives goes to great
> lengths (big red box) to discuss the current instability of RAID5/6 format.
>
> I am guessing here but I _think_ you should do the following...
>
> (0) Backup your data. [okay this is a test system that you deliberately
> purturbed but still... 8-) ]
>
> Option 1:
>
> (reasonable, but undocumented .. Either blance or scrub _ought_ to look
> at the disk sectors and trigger some re-copying from the good parts.)
>
> The disk is in the array (again), you may just need to re-balance or
> scrub the array to get the data on the drive back in harmony with the
> state of the array overall.
>
> Option 2:
>
> (unlikely :: add and remove are about making the geometry smaller/larger
> and, as stated, a RAID 6 cannot be less than 4 drives by definition, so
> there is no three-drive geometry for a RAID 6.)
>
> re-unplug the device, then use btrfs remove /dev/sdk /mnt
> then re-plug-in the device and use btrfs add /dev/sdk /mnt
>
>
> Option 3:
>
> (reasonable, but undocumented :: replace by device id -- 4 in your
> example case -- instead of system path. This would, I should think, skip
> the check of /dev/sdk1's separate status)
>
> btrfs replace start -f 4 /dev/sdk1 /mnt
>
> Option 3a:
>
> (got to get /dev/sdk1 back out of the list of active devices for /mnt so
> the system wont see /dev/sdk1 as "mounted" (e.g. held by a subsystem))
>
> unplug device.
> mount -o remount,degraded etc...
> plug in device.
> btrfs replace start -f 4 /dev/sdk1 /mnt
>
> Option 4:
>
> (most likely, most time consuming)
>
> Unplug /dev/sdk. Plug it into another computer and zero a decent chunk
> of partition 1.
> Plug it back into the original computer
> do the replace operation as in Option 3.
>
> This is the most-likely correct option if a simple rebalance or scrub
> doesn't work, as you will be presenting the system with three attached
> drives, one "missing" drive that will not match any necessary
> signatures, and a "new, blank" drive in its place.
>
> ===
>
> In all cases, you may need to unmount or remount or remount degraded  in
> there somewhere, particularly because you have already done so at least
> once.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2014-12-02  7:45 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-12-01 14:47 Online Drive Replacement: BTRFS with RAID 6 Oliver
2014-12-01 19:33 ` Robert White
2014-12-02  7:44   ` Oliver

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.