Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Torsten Bronger]

Hallöchen!

I want to replace a device in a RAID1 and converted it temporarily
to “single”:

btrfs balance start -f -mconvert=single,soft -dconvert=single,soft /

This takes very long. I don’t see why, and wonder whether this is
the right approach in the first place. After all, no (significant)
amount of data should be needed to be transferred in the process,
should it?

A side question: Is -sconvert also necessary?

Regards,
Torsten.

-- 
Torsten Bronger

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Andrei Borzenkov]

On 18.02.2023 11:10, Torsten Bronger wrote:
> Hallöchen!
> 
> I want to replace a device in a RAID1 and converted it temporarily
> to “single”:
> 
> btrfs balance start -f -mconvert=single,soft -dconvert=single,soft /
> 
> This takes very long. I don’t see why, and wonder whether this is
> the right approach in the first place. After all, no (significant)
> amount of data should be needed to be transferred in the process,
> should it?
> 

Converting between RAID profiles copies data from chunks with the old 
profile to the chunks with the new profile. Old chunk is not modified in 
place, they are removed after all data is moved to the new chunk.

> A side question: Is -sconvert also necessary?
> 
> Regards,
> Torsten.
> 

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Torsten Bronger]

Hallöchen!

Andrei Borzenkov writes:

> On 18.02.2023 11:10, Torsten Bronger wrote:
>
>> I want to replace a device in a RAID1 and converted it
>> temporarily to “single”: [...]  This takes very long. I don’t see
>> why, [...]
>
> Converting between RAID profiles copies data from chunks with the
> old profile to the chunks with the new profile. Old chunk is not
> modified in place, they are removed after all data is moved to the
> new chunk.

Thank you!  I see the reason now (while it is unfortunate
nevertheless that this means that the bulk data has to be moved as
well).

Regards,
Torsten.

-- 
Torsten Bronger

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Goffredo Baroncelli]

On 18/02/2023 09.10, Torsten Bronger wrote:
> Hallöchen!
> 
> I want to replace a device in a RAID1 and converted it temporarily
> to “single”:

I suggest you to evaluate
- remove a disk when the FS is offline
- mount the FS in 'degraded' mode
- attach a new disk

This would minimize the cost of the reshaping. But wait for some other
feedback because I am not sure how robust is working in a 'degraded'
mode for "I/O-intensive operation".

BR


> btrfs balance start -f -mconvert=single,soft -dconvert=single,soft /
> 
> This takes very long. I don’t see why, and wonder whether this is
> the right approach in the first place. After all, no (significant)
> amount of data should be needed to be transferred in the process,
> should it?
> 
> A side question: Is -sconvert also necessary?
> 
> Regards,
> Torsten.
> 

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Torsten Bronger]

Hallöchen!

Goffredo Baroncelli writes:

> On 18/02/2023 09.10, Torsten Bronger wrote:
>
>
>> I want to replace a device in a RAID1 and converted it
>> temporarily to “single”:
>
> I suggest you to evaluate
> - remove a disk when the FS is offline
> - mount the FS in 'degraded' mode
> - attach a new disk

I agree that converting to single is not the fastest way to replace
a disk, but the safest AFAICS.  It is the boot partition in a simple
home server without monitor or keyboard, which makes mounting as
degraded difficult.  Besides, you can only mount in degraded mode
once.  If anything goes wrong, I have to rebuild from scratch.

Mileage may vary in a more professional environment.

Regards,
Torsten.

-- 
Torsten Bronger

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[me]

On Sat, Feb 18, 2023 at 7:45 AM Torsten Bronger
<bronger@physik.rwth-aachen.de> wrote:
>
> Hallöchen!
>
> Goffredo Baroncelli writes:
>
> > On 18/02/2023 09.10, Torsten Bronger wrote:
> >
> >
> >> I want to replace a device in a RAID1 and converted it
> >> temporarily to “single”:
> >
> > I suggest you to evaluate
> > - remove a disk when the FS is offline
> > - mount the FS in 'degraded' mode
> > - attach a new disk
>
> I agree that converting to single is not the fastest way to replace
> a disk, but the safest AFAICS.  It is the boot partition in a simple
> home server without monitor or keyboard, which makes mounting as
> degraded difficult.  Besides, you can only mount in degraded mode
> once.  If anything goes wrong, I have to rebuild from scratch.
I just want to make clear, this bug was fixed long ago. You would need
to update your bootloader conf to use degraded at boot time in this
case, but in theory, it should work identical to have a raid1 degraded
as converting it all back to single.

I agree though, it would be a great feature addition to reduce IO load
to replicating single chunks to RAID1, or converting them back to
single but not rewriting them out. On SSDs it would reduce wear, and
on HDDs would reduce seeking/head thrashing.
>
> Mileage may vary in a more professional environment.
>
> Regards,
> Torsten.
>
> --
> Torsten Bronger
>

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Torsten Bronger]

Hallöchen!

me@jse.io writes:

> On Sat, Feb 18, 2023 at 7:45 AM Torsten Bronger
> <bronger@physik.rwth-aachen.de> wrote:
>>
>> [...]
>>
>> I agree that converting to single is not the fastest way to
>> replace a disk, but the safest AFAICS.  It is the boot partition
>> in a simple home server without monitor or keyboard, which makes
>> mounting as degraded difficult.  Besides, you can only mount in
>> degraded mode once.  If anything goes wrong, I have to rebuild
>> from scratch.
>
> I just want to make clear, this bug was fixed long ago. You would
> need to update your bootloader conf to use degraded at boot time
> in this case, but in theory, it should work identical to have a
> raid1 degraded as converting it all back to single.

What was fixed?  Can you now mount in in degraded mode read-write as
often as you wish?

Regards,
Torsten.
-- 
Torsten Bronger

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[me]

On Sat, Feb 18, 2023 at 4:11 PM Torsten Bronger
<bronger@physik.rwth-aachen.de> wrote:
>
> Hallöchen!
>
> me@jse.io writes:
>
> > On Sat, Feb 18, 2023 at 7:45 AM Torsten Bronger
> > <bronger@physik.rwth-aachen.de> wrote:
> >>
> >> [...]
> >>
> >> I agree that converting to single is not the fastest way to
> >> replace a disk, but the safest AFAICS.  It is the boot partition
> >> in a simple home server without monitor or keyboard, which makes
> >> mounting as degraded difficult.  Besides, you can only mount in
> >> degraded mode once.  If anything goes wrong, I have to rebuild
> >> from scratch.
> >
> > I just want to make clear, this bug was fixed long ago. You would
> > need to update your bootloader conf to use degraded at boot time
> > in this case, but in theory, it should work identical to have a
> > raid1 degraded as converting it all back to single.
>
> What was fixed?  Can you now mount in in degraded mode read-write as
> often as you wish?
Yes, and has been for some time. Since kernel 4.14 iirc.
> Regards,
> Torsten.
> --
> Torsten Bronger
>

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Goffredo Baroncelli]

On 18/02/2023 12.38, Torsten Bronger wrote:
> Hallöchen!
> 
> Goffredo Baroncelli writes:
> 
>> On 18/02/2023 09.10, Torsten Bronger wrote:
>>
>>
>>> I want to replace a device in a RAID1 and converted it
>>> temporarily to “single”:
>>
>> I suggest you to evaluate
>> - remove a disk when the FS is offline
>> - mount the FS in 'degraded' mode
>> - attach a new disk
> 
> I agree that converting to single is not the fastest way to replace
> a disk, but the safest AFAICS.  It is the boot partition in a simple
> home server without monitor or keyboard, which makes mounting as
> degraded difficult.  Besides, you can only mount in degraded mode
> once.  If anything goes wrong, I have to rebuild from scratch.
> 

Just for curiosity, I know that the BTRFS RAID1 is not the fastest implementation,
but how slower is in your "I/O-intensive operation".

In theory the differences should not be so huge

> Mileage may vary in a more professional environment.
> 
> Regards,
> Torsten.
> 

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Torsten Bronger]

Hallöchen!

Goffredo Baroncelli writes:

> [...] 
>
> Just for curiosity, I know that the BTRFS RAID1 is not the fastest
> implementation, but how slower is in your "I/O-intensive
> operation".

What exactly do you want to compare?  Be that as it may, I have no
benchmarks.  Originally, I was just wondering why RAID1 --> single
may take hours, given that all data is on both disks anyway.

Regards,
Torsten.

-- 
Torsten Bronger

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[waxhead]

> Hallöchen!
> 
> Goffredo Baroncelli writes:
> 
>> [...]
>>
>> Just for curiosity, I know that the BTRFS RAID1 is not the fastest
>> implementation, but how slower is in your "I/O-intensive
>> operation".
> 
> What exactly do you want to compare?  Be that as it may, I have no
> benchmarks.  Originally, I was just wondering why RAID1 --> single
> may take hours, given that all data is on both disks anyway.
> 
> Regards,
> Torsten.
> 
So what I think you would like to have answered is : Since RAID1 has two 
duplicate chunks (A1 and A2) why could not BTRFS simply change the chunk 
type of A1 to single and discard chunk A2.

Personally I don't have the slightest idea why , but I imagine that this 
perhaps might be possible with the extent tree V2 changes sometime in 
the future. Meanwhile perhaps someone else would like to explain why...

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Torsten Bronger]

Hallöchen!

waxhead writes:

> [...]
>
> So what I think you would like to have answered is : Since RAID1
> has two duplicate chunks (A1 and A2) why could not BTRFS simply
> change the chunk type of A1 to single and discard chunk A2.

Exactly.

Regards,
Torsten.

-- 
Torsten Bronger

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Patrik Lundquist]

Replace the disk offline, mount degraded, use btrfs replace with the
devid of the missing disk.

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Forza]

On 2023-02-20 00:39, waxhead wrote:
>> Hallöchen!
>>
>> Goffredo Baroncelli writes:
>>
>>> [...]
>>>
>>> Just for curiosity, I know that the BTRFS RAID1 is not the fastest
>>> implementation, but how slower is in your "I/O-intensive
>>> operation".
>>
>> What exactly do you want to compare?  Be that as it may, I have no
>> benchmarks.  Originally, I was just wondering why RAID1 --> single
>> may take hours, given that all data is on both disks anyway.
>>
>> Regards,
>> Torsten.
>>
> So what I think you would like to have answered is : Since RAID1 has two 
> duplicate chunks (A1 and A2) why could not BTRFS simply change the chunk 
> type of A1 to single and discard chunk A2.
>  > Personally I don't have the slightest idea why , but I imagine that this
> perhaps might be possible with the extent tree V2 changes sometime in 
> the future. Meanwhile perhaps someone else would like to explain why...

I believe this would violate the "CoW" nature of Btrfs. In other words, 
it would introduce a change in-place, which is not possible with current 
current way of working.

There probably are ways to make it safe and atomic, but maybe not a high 
priority from the current devs?

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Torsten Bronger]

Hallöchen!

Forza writes:

> [...]
>
> I believe this would violate the "CoW" nature of Btrfs. In other
> words, it would introduce a change in-place, which is not possible
> with current current way of working.
>
> There probably are ways to make it safe and atomic, but maybe not
> a high priority from the current devs?

My naive assumption was: There is the bulk data, and there is
metadata that also contains the profile of the data.  If the profile
is changed, only the metadata has to be re-written atomically (in
particular, the CoW way).

But apparently, the profile is part of the bulk data chunk itself.
This may well be necessary, but I was surprised by that.

Regards,
Torsten.

-- 
Torsten Bronger

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[me]

On Tue, Feb 21, 2023 at 3:21 AM Torsten Bronger
<bronger@physik.rwth-aachen.de> wrote:
>
> Hallöchen!
>
> Forza writes:
>
> > [...]
> >
> > I believe this would violate the "CoW" nature of Btrfs. In other
> > words, it would introduce a change in-place, which is not possible
> > with current current way of working.
> >
> > There probably are ways to make it safe and atomic, but maybe not
> > a high priority from the current devs?
>
> My naive assumption was: There is the bulk data, and there is
> metadata that also contains the profile of the data.  If the profile
> is changed, only the metadata has to be re-written atomically (in
> particular, the CoW way).
>
> But apparently, the profile is part of the bulk data chunk itself.
> This may well be necessary, but I was surprised by that.
If you want to go back to single on a 2 device RAID1, and avoid the
wear, especially if you plan to return to RAID1 at a later date, why
not just use it degraded? A degraded RAID1 chunk is basically just
single anyway. You can convert the metadata to dup to reduce write amp
without making the fs itself less resilient. New chunks will become
single while degraded anyway. You can convert those back later as
well.

When you do want to return using the old disk, use btrfs replace
instead. It will simply copy the existing data rather than balance it
to replicate it.
>
> Regards,
> Torsten.
>
> --
> Torsten Bronger
>

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Torsten Bronger]

Hallöchen!

me@jse.io writes:

> [...]
>
> If you want to go back to single on a 2 device RAID1, and avoid
> the wear, especially if you plan to return to RAID1 at a later
> date, why not just use it degraded?

Now that I know that the mount-degraded-rw-only-once thing has long
been fixed, I will do that of course.

Bye,
Torsten.

-- 
Torsten Bronger

Re: Why is converting from RAID1 to single in Btrfs an I/O-intensive operation?

[Ferry Toth]

Op 18-02-2023 om 12:38 schreef Torsten Bronger:
> Hallöchen!
> 
> Goffredo Baroncelli writes:
> 
>> On 18/02/2023 09.10, Torsten Bronger wrote:
>>
>>
>>> I want to replace a device in a RAID1 and converted it
>>> temporarily to “single”:
>>
>> I suggest you to evaluate
>> - remove a disk when the FS is offline
>> - mount the FS in 'degraded' mode
>> - attach a new disk
Better (if you have the hardware for it, i.e. sufficient connectors) add 
the extra disk to the pool. Then remove the old disk from the pool.
Then physically remove the (now unused) disk.
> I agree that converting to single is not the fastest way to replace
> a disk, but the safest AFAICS.  It is the boot partition in a simple
> home server without monitor or keyboard, which makes mounting as
> degraded difficult.  Besides, you can only mount in degraded mode
> once.  If anything goes wrong, I have to rebuild from scratch.
> 
> Mileage may vary in a more professional environment.
> 
> Regards,
> Torsten.
>