Best practices for raid 1

Best practices for raid 1

[Vinko Magecic]

Hello,

I set up a raid 1 with two btrfs devices and came across some situations in my testing that I can't get a straight answer on.

1) When replacing a volume, do I still need to `umount /path` and then `mount -o degraded ...` the good volume before doing the `btrfs replace start ...` ? I didn't see anything that said I had to and when I tested it without mounting the volume it was able to replace the device without any issue. Is that considered bad and could risk damage or has `replace` made it possible to replace devices without umounting the filesystem?

2) Everything I see about replacing a drive says to use `/old/device /new/device` but what if the old device can't be read or no longer exists? Would that be a `btrfs device add /new/device; btrfs balance start /new/device` ?

3) When I have the RAID1 with two devices and I want to grow it out, which is the better practice? Create a larger volume, replace the old device with the new device and then do it a second time for the other device, or attaching the new volumes to the label/uuid one at a time and with each one use `btrfs filesystem resize devid:max /mountpoint`.

Thanks


     

    

Re: Best practices for raid 1

[Chris Murphy]

On Tue, Jan 10, 2017 at 2:07 PM, Vinko Magecic
<vinko.magecic@construction.com> wrote:
> Hello,
>
> I set up a raid 1 with two btrfs devices and came across some situations in my testing that I can't get a straight answer on.
>
> 1) When replacing a volume, do I still need to `umount /path` and then `mount -o degraded ...` the good volume before doing the `btrfs replace start ...` ?

No. If the device being replaced is unreliable, use -r to limit the
reads from the device being replaced.



> I didn't see anything that said I had to and when I tested it without mounting the volume it was able to replace the device without any issue. Is that considered bad and could risk damage or has `replace` made it possible to replace devices without umounting the filesystem?

It's always been possible even before 'replace'.
btrfs dev add <dev3>
btrfs dev rem <dev1>

But there are some bugs in dev replace that Qu is working on; I think
they mainly negatively impact raid56 though.

The one limitation of 'replace' is that the new block device must be
equal to or larger than the block device being replaced; where dev add
> dev rem doesn't require this.


> 2) Everything I see about replacing a drive says to use `/old/device /new/device` but what if the old device can't be read or no longer exists?

The command works whether the device is present or not; but if it's
present and working then any errors on one device can be corrected by
the other, whereas if the device is missing, then any errors on the
remaining device can't be corrected. Off hand I'm not sure if the
replace continues and an error just logged...I think that's what
should happen.


>Would that be a `btrfs device add /new/device; btrfs balance start /new/device` ?

dev add then dev rem; the balance isn't necessary.

>
> 3) When I have the RAID1 with two devices and I want to grow it out, which is the better practice? Create a larger volume, replace the old device with the new device and then do it a second time for the other device, or attaching the new volumes to the label/uuid one at a time and with each one use `btrfs filesystem resize devid:max /mountpoint`.

If you're replacing a 2x raid1 with two bigger replacements, you'd use
'btrfs replace' twice. Maybe it'd work concurrently, I've never tried
it, but useful for someone to test and see if it explodes because if
it's allowed, it should work or fail gracefully.

There's no need to do filesystem resizes when doing either 'replace'
or 'dev add' followed by 'dev rem' because the fs resize is implied.
First it's resized/grown with add; and then it's resized/shrink with
remove. For replace there's a consolidation of steps, it's been a
while since I've looked at the code so I can't tell you what steps it
skips, what the state of the devices are in during the replace, which
one active writes go to.


-- 
Chris Murphy

Re: Best practices for raid 1

[Austin S. Hemmelgarn]

On 2017-01-10 16:49, Chris Murphy wrote:
> On Tue, Jan 10, 2017 at 2:07 PM, Vinko Magecic
> <vinko.magecic@construction.com> wrote:
>> Hello,
>>
>> I set up a raid 1 with two btrfs devices and came across some situations in my testing that I can't get a straight answer on.
>>
>> 1) When replacing a volume, do I still need to `umount /path` and then `mount -o degraded ...` the good volume before doing the `btrfs replace start ...` ?
>
> No. If the device being replaced is unreliable, use -r to limit the
> reads from the device being replaced.
>
>
>
>> I didn't see anything that said I had to and when I tested it without mounting the volume it was able to replace the device without any issue. Is that considered bad and could risk damage or has `replace` made it possible to replace devices without umounting the filesystem?
>
> It's always been possible even before 'replace'.
> btrfs dev add <dev3>
> btrfs dev rem <dev1>
>
> But there are some bugs in dev replace that Qu is working on; I think
> they mainly negatively impact raid56 though.
>
> The one limitation of 'replace' is that the new block device must be
> equal to or larger than the block device being replaced; where dev add
>> dev rem doesn't require this.
The other thing to remember is that you can resize the FS on the device 
being replaced so that it will fit on the new device.  I actually 
regularly do this when re-partitioning or moving filesystems between 
devices as a safety precaution so that I can be sure it will fit in the 
new location.  I would only suggest doing this though if that device is 
still reliable, as it may move data around on that device, and it 
obviously doesn't work if the device being replaced is missing.
>
>
>> 2) Everything I see about replacing a drive says to use `/old/device /new/device` but what if the old device can't be read or no longer exists?
>
> The command works whether the device is present or not; but if it's
> present and working then any errors on one device can be corrected by
> the other, whereas if the device is missing, then any errors on the
> remaining device can't be corrected. Off hand I'm not sure if the
> replace continues and an error just logged...I think that's what
> should happen.
IIRC, that's what happens up to some (arbitrary) threshold, at which 
point the replace fails.
>
>
>> Would that be a `btrfs device add /new/device; btrfs balance start /new/device` ?
>
> dev add then dev rem; the balance isn't necessary.
A better way to put it is that the balance is implicit in the removal of 
the device.  The data that was on that device has to go somewhere, and 
the easiest way to do that is just to run a balance that's not allowed 
to allocate anything on the device being removed.
>
>>
>> 3) When I have the RAID1 with two devices and I want to grow it out, which is the better practice? Create a larger volume, replace the old device with the new device and then do it a second time for the other device, or attaching the new volumes to the label/uuid one at a time and with each one use `btrfs filesystem resize devid:max /mountpoint`.
>
> If you're replacing a 2x raid1 with two bigger replacements, you'd use
> 'btrfs replace' twice. Maybe it'd work concurrently, I've never tried
> it, but useful for someone to test and see if it explodes because if
> it's allowed, it should work or fail gracefully.
In theory, it _might_ be possible to get dev replace to work 
concurrently.  As of right now, I know that the current implementation 
does not work with more than one instance running per FS (because it 
uses devid 0 for the new device during the replace, and devids have to 
be unique), but I don't know for certain what it does if you try to run 
another (it _should_ refuse to start, I'm not certain if that's what it 
actually does, and I don't have the time to check right now).

That said, there are many reasons to just serialize replaces most of the 
time, the most notable being that replace does not just read from the 
device being replaced (although most of the reads go to that device), 
and that serializing the replace operations has less impact on the rest 
of the system (it is designed to be used on live systems).
>
> There's no need to do filesystem resizes when doing either 'replace'
> or 'dev add' followed by 'dev rem' because the fs resize is implied.
> First it's resized/grown with add; and then it's resized/shrink with
> remove. For replace there's a consolidation of steps, it's been a
> while since I've looked at the code so I can't tell you what steps it
> skips, what the state of the devices are in during the replace, which
> one active writes go to.
Last time I checked, this was not the case for replace, and a resize to 
max size was still necessary.  That was almost 3 months ago though (I've 
been lucky and not needed to replace anything since then), so I may be 
incorrect about the current state of things.

Re: Best practices for raid 1

[Tomasz Kusmierz]

I would like to use this thread to ask few questions: 

If we have 2 devices dying on us and we run RAID6 - this theoretically will still run (despite our current problems). Now let’s say that we booted up raid6 of 10 disk and 2 of them dies but operator does NOT know what are dev ID of disk that died, How does one removes those devices other than using “-missing” ??? I ask because it’s in multiple places stated to use “replace” when your device dies but nobody ever states how to find out which /dev/ node is actually missing  …. so when I want to use a replace, I don’t know what to use within command :/ … This whole thing might have an additional complication - if FS is fool, than one would need to add disks than remove missing. 


> On 10 Jan 2017, at 21:49, Chris Murphy <lists@colorremedies.com> wrote:
> 
> On Tue, Jan 10, 2017 at 2:07 PM, Vinko Magecic
> <vinko.magecic@construction.com> wrote:
>> Hello,
>> 
>> I set up a raid 1 with two btrfs devices and came across some situations in my testing that I can't get a straight answer on.
>> 
>> 1) When replacing a volume, do I still need to `umount /path` and then `mount -o degraded ...` the good volume before doing the `btrfs replace start ...` ?
> 
> No. If the device being replaced is unreliable, use -r to limit the
> reads from the device being replaced.
> 
> 
> 
>> I didn't see anything that said I had to and when I tested it without mounting the volume it was able to replace the device without any issue. Is that considered bad and could risk damage or has `replace` made it possible to replace devices without umounting the filesystem?
> 
> It's always been possible even before 'replace'.
> btrfs dev add <dev3>
> btrfs dev rem <dev1>
> 
> But there are some bugs in dev replace that Qu is working on; I think
> they mainly negatively impact raid56 though.
> 
> The one limitation of 'replace' is that the new block device must be
> equal to or larger than the block device being replaced; where dev add
>> dev rem doesn't require this.
> 
> 
>> 2) Everything I see about replacing a drive says to use `/old/device /new/device` but what if the old device can't be read or no longer exists?
> 
> The command works whether the device is present or not; but if it's
> present and working then any errors on one device can be corrected by
> the other, whereas if the device is missing, then any errors on the
> remaining device can't be corrected. Off hand I'm not sure if the
> replace continues and an error just logged...I think that's what
> should happen.
> 
> 
>> Would that be a `btrfs device add /new/device; btrfs balance start /new/device` ?
> 
> dev add then dev rem; the balance isn't necessary.
> 
>> 
>> 3) When I have the RAID1 with two devices and I want to grow it out, which is the better practice? Create a larger volume, replace the old device with the new device and then do it a second time for the other device, or attaching the new volumes to the label/uuid one at a time and with each one use `btrfs filesystem resize devid:max /mountpoint`.
> 
> If you're replacing a 2x raid1 with two bigger replacements, you'd use
> 'btrfs replace' twice. Maybe it'd work concurrently, I've never tried
> it, but useful for someone to test and see if it explodes because if
> it's allowed, it should work or fail gracefully.
> 
> There's no need to do filesystem resizes when doing either 'replace'
> or 'dev add' followed by 'dev rem' because the fs resize is implied.
> First it's resized/grown with add; and then it's resized/shrink with
> remove. For replace there's a consolidation of steps, it's been a
> while since I've looked at the code so I can't tell you what steps it
> skips, what the state of the devices are in during the replace, which
> one active writes go to.
> 
> 
> -- 
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best practices for raid 1

[Austin S. Hemmelgarn]

On 2017-01-11 15:37, Tomasz Kusmierz wrote:
> I would like to use this thread to ask few questions:
>
> If we have 2 devices dying on us and we run RAID6 - this theoretically will still run (despite our current problems). Now let’s say that we booted up raid6 of 10 disk and 2 of them dies but operator does NOT know what are dev ID of disk that died, How does one removes those devices other than using “-missing” ??? I ask because it’s in multiple places stated to use “replace” when your device dies but nobody ever states how to find out which /dev/ node is actually missing  …. so when I want to use a replace, I don’t know what to use within command :/ … This whole thing might have an additional complication - if FS is fool, than one would need to add disks than remove missing.
raid6 is a special case right now (aside from the fact that it's not 
safe for general usage) because it's the only profile on BTRFS that can 
sustain more than one failed disk.  In the case that the devices aren't 
actually listed as missing (most disks won't disappear unless the 
cabling, storage controller, or disk electronics are bad), you can use 
btrfs fi show to see what the mapping is.  If the disks are missing 
(again, not likely unless there's a pretty severe electrical failure 
somewhere), it's safer in that case to add enough devices to satisfy 
replication and storage constraints, then just run 'btrfs device delete 
missing' to get rid of the other disks.
>
>
>> On 10 Jan 2017, at 21:49, Chris Murphy <lists@colorremedies.com> wrote:
>>
>> On Tue, Jan 10, 2017 at 2:07 PM, Vinko Magecic
>> <vinko.magecic@construction.com> wrote:
>>> Hello,
>>>
>>> I set up a raid 1 with two btrfs devices and came across some situations in my testing that I can't get a straight answer on.
>>>
>>> 1) When replacing a volume, do I still need to `umount /path` and then `mount -o degraded ...` the good volume before doing the `btrfs replace start ...` ?
>>
>> No. If the device being replaced is unreliable, use -r to limit the
>> reads from the device being replaced.
>>
>>
>>
>>> I didn't see anything that said I had to and when I tested it without mounting the volume it was able to replace the device without any issue. Is that considered bad and could risk damage or has `replace` made it possible to replace devices without umounting the filesystem?
>>
>> It's always been possible even before 'replace'.
>> btrfs dev add <dev3>
>> btrfs dev rem <dev1>
>>
>> But there are some bugs in dev replace that Qu is working on; I think
>> they mainly negatively impact raid56 though.
>>
>> The one limitation of 'replace' is that the new block device must be
>> equal to or larger than the block device being replaced; where dev add
>>> dev rem doesn't require this.
>>
>>
>>> 2) Everything I see about replacing a drive says to use `/old/device /new/device` but what if the old device can't be read or no longer exists?
>>
>> The command works whether the device is present or not; but if it's
>> present and working then any errors on one device can be corrected by
>> the other, whereas if the device is missing, then any errors on the
>> remaining device can't be corrected. Off hand I'm not sure if the
>> replace continues and an error just logged...I think that's what
>> should happen.
>>
>>
>>> Would that be a `btrfs device add /new/device; btrfs balance start /new/device` ?
>>
>> dev add then dev rem; the balance isn't necessary.
>>
>>>
>>> 3) When I have the RAID1 with two devices and I want to grow it out, which is the better practice? Create a larger volume, replace the old device with the new device and then do it a second time for the other device, or attaching the new volumes to the label/uuid one at a time and with each one use `btrfs filesystem resize devid:max /mountpoint`.
>>
>> If you're replacing a 2x raid1 with two bigger replacements, you'd use
>> 'btrfs replace' twice. Maybe it'd work concurrently, I've never tried
>> it, but useful for someone to test and see if it explodes because if
>> it's allowed, it should work or fail gracefully.
>>
>> There's no need to do filesystem resizes when doing either 'replace'
>> or 'dev add' followed by 'dev rem' because the fs resize is implied.
>> First it's resized/grown with add; and then it's resized/shrink with
>> remove. For replace there's a consolidation of steps, it's been a
>> while since I've looked at the code so I can't tell you what steps it
>> skips, what the state of the devices are in during the replace, which
>> one active writes go to.
>>
>>
>> --
>> Chris Murphy
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Best practices for raid 1

[Tomasz Kusmierz]

That was long winded way of saying “there is no mechanism in btrfs to tell you exactly which device is missing” but thanks anyway.

> On 12 Jan 2017, at 12:47, Austin S. Hemmelgarn <ahferroin7@gmail.com> wrote:
> 
> On 2017-01-11 15:37, Tomasz Kusmierz wrote:
>> I would like to use this thread to ask few questions:
>> 
>> If we have 2 devices dying on us and we run RAID6 - this theoretically will still run (despite our current problems). Now let’s say that we booted up raid6 of 10 disk and 2 of them dies but operator does NOT know what are dev ID of disk that died, How does one removes those devices other than using “-missing” ??? I ask because it’s in multiple places stated to use “replace” when your device dies but nobody ever states how to find out which /dev/ node is actually missing  …. so when I want to use a replace, I don’t know what to use within command :/ … This whole thing might have an additional complication - if FS is fool, than one would need to add disks than remove missing.
> raid6 is a special case right now (aside from the fact that it's not safe for general usage) because it's the only profile on BTRFS that can sustain more than one failed disk.  In the case that the devices aren't actually listed as missing (most disks won't disappear unless the cabling, storage controller, or disk electronics are bad), you can use btrfs fi show to see what the mapping is.  If the disks are missing (again, not likely unless there's a pretty severe electrical failure somewhere), it's safer in that case to add enough devices to satisfy replication and storage constraints, then just run 'btrfs device delete missing' to get rid of the other disks.
>> 
>> 
>>> On 10 Jan 2017, at 21:49, Chris Murphy <lists@colorremedies.com> wrote:
>>> 
>>> On Tue, Jan 10, 2017 at 2:07 PM, Vinko Magecic
>>> <vinko.magecic@construction.com> wrote:
>>>> Hello,
>>>> 
>>>> I set up a raid 1 with two btrfs devices and came across some situations in my testing that I can't get a straight answer on.
>>>> 
>>>> 1) When replacing a volume, do I still need to `umount /path` and then `mount -o degraded ...` the good volume before doing the `btrfs replace start ...` ?
>>> 
>>> No. If the device being replaced is unreliable, use -r to limit the
>>> reads from the device being replaced.
>>> 
>>> 
>>> 
>>>> I didn't see anything that said I had to and when I tested it without mounting the volume it was able to replace the device without any issue. Is that considered bad and could risk damage or has `replace` made it possible to replace devices without umounting the filesystem?
>>> 
>>> It's always been possible even before 'replace'.
>>> btrfs dev add <dev3>
>>> btrfs dev rem <dev1>
>>> 
>>> But there are some bugs in dev replace that Qu is working on; I think
>>> they mainly negatively impact raid56 though.
>>> 
>>> The one limitation of 'replace' is that the new block device must be
>>> equal to or larger than the block device being replaced; where dev add
>>>> dev rem doesn't require this.
>>> 
>>> 
>>>> 2) Everything I see about replacing a drive says to use `/old/device /new/device` but what if the old device can't be read or no longer exists?
>>> 
>>> The command works whether the device is present or not; but if it's
>>> present and working then any errors on one device can be corrected by
>>> the other, whereas if the device is missing, then any errors on the
>>> remaining device can't be corrected. Off hand I'm not sure if the
>>> replace continues and an error just logged...I think that's what
>>> should happen.
>>> 
>>> 
>>>> Would that be a `btrfs device add /new/device; btrfs balance start /new/device` ?
>>> 
>>> dev add then dev rem; the balance isn't necessary.
>>> 
>>>> 
>>>> 3) When I have the RAID1 with two devices and I want to grow it out, which is the better practice? Create a larger volume, replace the old device with the new device and then do it a second time for the other device, or attaching the new volumes to the label/uuid one at a time and with each one use `btrfs filesystem resize devid:max /mountpoint`.
>>> 
>>> If you're replacing a 2x raid1 with two bigger replacements, you'd use
>>> 'btrfs replace' twice. Maybe it'd work concurrently, I've never tried
>>> it, but useful for someone to test and see if it explodes because if
>>> it's allowed, it should work or fail gracefully.
>>> 
>>> There's no need to do filesystem resizes when doing either 'replace'
>>> or 'dev add' followed by 'dev rem' because the fs resize is implied.
>>> First it's resized/grown with add; and then it's resized/shrink with
>>> remove. For replace there's a consolidation of steps, it's been a
>>> while since I've looked at the code so I can't tell you what steps it
>>> skips, what the state of the devices are in during the replace, which
>>> one active writes go to.
>>> 
>>> 
>>> --
>>> Chris Murphy
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
> 

Re: Best practices for raid 1

[Tomasz Kusmierz]

> On 10 Jan 2017, at 21:07, Vinko Magecic <vinko.magecic@construction.com> wrote:
> 
> Hello,
> 
> I set up a raid 1 with two btrfs devices and came across some situations in my testing that I can't get a straight answer on.
> 1) When replacing a volume, do I still need to `umount /path` and then `mount -o degraded ...` the good volume before doing the `btrfs replace start ...` ? I didn't see anything that said I had to and when I tested it without mounting the volume it was able to replace the device without any issue. Is that considered bad and could risk damage or has `replace` made it possible to replace devices without umounting the filesystem?

No need to unmount, just replace old with new. Your scenario seems very convoluted and it’s pointless

> 2) Everything I see about replacing a drive says to use `/old/device /new/device` but what if the old device can't be read or no longer exists? Would that be a `btrfs device add /new/device; btrfs balance start /new/device` ?
In case where old device is missing you’ve got few options:
- if you have enough space to fit the data and enough of disks to comply with redundancy - just remove the drive, So for example is you have 3 x 1TB drives with raid 1 And use less than 1TB of data total - juste remove one drive and you will have 2 x 1TB drives in raid 1 and btrfs fill just rebalance stuff for you !
- if you have not enough space to fi the data / not enough disks left to comply with raid lever - your only option is to add disk first then remove missing (btrfs dev delete missing /mount_point_of_your_fs)

> 3) When I have the RAID1 with two devices and I want to grow it out, which is the better practice? Create a larger volume, replace the old device with the new device and then do it a second time for the other device, or attaching the new volumes to the label/uuid one at a time and with each one use `btrfs filesystem resize devid:max /mountpoint`.

You kinda misunderstand the principal of btrfs. Btrfs will span across ALL the available space you’ve got. If you have multiple devices in this setup (remember that partition IS A DEVICE), it will span across multiple devices and you can’t change this. Now btrfs resize is mean for resizing a file system occupying a device (or partition). So work flow is that is you want to shrink a device (partition) you first shrink fs on this device than size down the device (partition) … if you want to increase the size of device (partition) you increase size of device (partition) than you grow filesystem within this device (partition). This is 100% irrespective of total cumulative size of file system. 

Let’s say you’ve got a btrfs file system that is spanning across 3 x 1TB devices … and those devices are partitions. You have raid 1 setup - your complete amount of available space is 1.5 TB. Let’s say you want to shrink of of partitions to 0.5TB -> first you shrink FS on this partition (balance will runn automatically) -> you shrink partition down to 0.5TB -> from now on your total available space is 1.25TB. 

Simples right ? :)

> Thanks
> 
> 
> 
> 
>    --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html