Data single *and* raid?

Data single *and* raid?

[Hendrik Friedel]

Hello,

I converted an array to raid5 by
btrfs device add /dev/sdd /mnt/new_storage
btrfs device add /dev/sdc /mnt/new_storage
btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/

The Balance went through. But now:
Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
         Total devices 3 FS bytes used 5.28TiB
         devid    1 size 2.73TiB used 2.57TiB path /dev/sde
         devid    2 size 2.73TiB used 2.73TiB path /dev/sdc
         devid    3 size 2.73TiB used 2.73TiB path /dev/sdd
btrfs-progs v4.1.1

Already the 2.57TiB is a bit surprising:
root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
Data, single: total=2.55TiB, used=2.55TiB
Data, RAID5: total=2.73TiB, used=2.72TiB
System, RAID5: total=32.00MiB, used=736.00KiB
Metadata, RAID1: total=6.00GiB, used=5.33GiB
Metadata, RAID5: total=3.00GiB, used=2.99GiB

Why is there Data single and Raid?
Why is Metadata RAID1 and Raid5?

A scrub is currently running and showed no errors yet.

Greetings,
Hendrik

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Re: Data single *and* raid?

[Chris Murphy]

On Sat, Aug 1, 2015 at 2:09 PM, Hendrik Friedel <hendrik@friedels.name> wrote:
> Hello,
>
> I converted an array to raid5 by
> btrfs device add /dev/sdd /mnt/new_storage
> btrfs device add /dev/sdc /mnt/new_storage
> btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
>
> The Balance went through. But now:
> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>         Total devices 3 FS bytes used 5.28TiB
>         devid    1 size 2.73TiB used 2.57TiB path /dev/sde
>         devid    2 size 2.73TiB used 2.73TiB path /dev/sdc
>         devid    3 size 2.73TiB used 2.73TiB path /dev/sdd
> btrfs-progs v4.1.1
>
> Already the 2.57TiB is a bit surprising:
> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
> Data, single: total=2.55TiB, used=2.55TiB
> Data, RAID5: total=2.73TiB, used=2.72TiB
> System, RAID5: total=32.00MiB, used=736.00KiB
> Metadata, RAID1: total=6.00GiB, used=5.33GiB
> Metadata, RAID5: total=3.00GiB, used=2.99GiB
>
> Why is there Data single and Raid?
> Why is Metadata RAID1 and Raid5?


What kernel version?

-- 
Chris Murphy

Re: Data single *and* raid?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1832 bytes --]

On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
> Hello,
> 
> I converted an array to raid5 by
> btrfs device add /dev/sdd /mnt/new_storage
> btrfs device add /dev/sdc /mnt/new_storage
> btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
> 
> The Balance went through. But now:
> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>         Total devices 3 FS bytes used 5.28TiB
>         devid    1 size 2.73TiB used 2.57TiB path /dev/sde
>         devid    2 size 2.73TiB used 2.73TiB path /dev/sdc
>         devid    3 size 2.73TiB used 2.73TiB path /dev/sdd
> btrfs-progs v4.1.1
> 
> Already the 2.57TiB is a bit surprising:
> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
> Data, single: total=2.55TiB, used=2.55TiB
> Data, RAID5: total=2.73TiB, used=2.72TiB
> System, RAID5: total=32.00MiB, used=736.00KiB
> Metadata, RAID1: total=6.00GiB, used=5.33GiB
> Metadata, RAID5: total=3.00GiB, used=2.99GiB

   Looking at the btrfs fi show output, you've probably run out of
space during the conversion, probably due to an uneven distribution of
the original "single" chunks.

   I think I would suggest balancing the single chunks, and trying the
conversion (of the unconverted parts) again:

# btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/
# btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/

   You may have to do this more than once.

   Hugo.

> Why is there Data single and Raid?
> Why is Metadata RAID1 and Raid5?
> 
> A scrub is currently running and showed no errors yet.

-- 
Hugo Mills             | You've read the project plan. Forget that. We're
hugo@... carfax.org.uk | going to Do Stuff and Have Fun doing it.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                           Jeremy Frey

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Data single *and* raid?

[Chris Murphy]

On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
>> Hello,
>>
>> I converted an array to raid5 by
>> btrfs device add /dev/sdd /mnt/new_storage
>> btrfs device add /dev/sdc /mnt/new_storage
>> btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
>>
>> The Balance went through. But now:
>> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>>         Total devices 3 FS bytes used 5.28TiB
>>         devid    1 size 2.73TiB used 2.57TiB path /dev/sde
>>         devid    2 size 2.73TiB used 2.73TiB path /dev/sdc
>>         devid    3 size 2.73TiB used 2.73TiB path /dev/sdd
>> btrfs-progs v4.1.1
>>
>> Already the 2.57TiB is a bit surprising:
>> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
>> Data, single: total=2.55TiB, used=2.55TiB
>> Data, RAID5: total=2.73TiB, used=2.72TiB
>> System, RAID5: total=32.00MiB, used=736.00KiB
>> Metadata, RAID1: total=6.00GiB, used=5.33GiB
>> Metadata, RAID5: total=3.00GiB, used=2.99GiB
>
>    Looking at the btrfs fi show output, you've probably run out of
> space during the conversion, probably due to an uneven distribution of
> the original "single" chunks.
>
>    I think I would suggest balancing the single chunks, and trying the
> conversion (of the unconverted parts) again:
>
> # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/
> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/
>

Yep I bet that's it also. btrfs fi usage might be better at exposing this case.


-- 
Chris Murphy

Re: Data single *and* raid?

[Duncan]

Chris Murphy posted on Sat, 01 Aug 2015 14:44:52 -0600 as excerpted:

> On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>> On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
>>> Hello,
>>>
>>> I converted an array to raid5 by
>>> btrfs device add /dev/sdd /mnt/new_storage
>>> btrfs device add /dev/sdc /mnt/new_storage btrfs
>>> balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
>>>
>>> The Balance went through. But now:
>>> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>>>         Total devices 3 FS bytes used 5.28TiB
>>>	    devid    1 size 2.73TiB  used 2.57TiB path /dev/sde
>>> 	    devid    2 size 2.73TiB used  2.73TiB path /dev/sdc
>>>	    devid    3 size 2.73TiB used 2.73TiB  path /dev/sdd
>>> btrfs-progs v4.1.1
>>>
>>> Already the 2.57TiB is a bit surprising:
>>> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
>>> Data, single: total=2.55TiB, used=2.55TiB
>>> Data, RAID5: total=2.73TiB, used=2.72TiB
>>> System, RAID5: total=32.00MiB, used=736.00KiB
>>> Metadata, RAID1: total=6.00GiB, used=5.33GiB
>>> Metadata, RAID5: total=3.00GiB, used=2.99GiB
>>
>>    Looking at the btrfs fi show output, you've probably run out of
>> space during the conversion, probably due to an uneven distribution of
>> the original "single" chunks.
>>
>>    I think I would suggest balancing the single chunks, and trying the
>> conversion (of the unconverted parts) again:
>>
>> # btrfs balance start -dprofiles=single -mprofile=raid1
>> /mnt/new_storage/
>> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
>> /mnt/new_storage/
>>
>>
> Yep I bet that's it also. btrfs fi usage might be better at exposing
> this case.

Does fi usage deal with raid5 yet?

Last I knew it didn't deal with raid56 (which I'm not using) or with 
mixed-bg (which I am using on one btrfs).  On the mixed-bg btrfs it still 
warns it doesn't handle that, reporting unallocated of 16 EiB on a 256 
MiB filesystem, so clearly the warning is valid.  16 EiB unallocated on a 
256 MiB btrfs, I wish!  progs v4.1.2, latest as of yesterday, I believe.


Meanwhile, three devices the same size.  He just added two of them and 
didn't do a rebalance after that until the raid5 conversion, so except 
for anything added since, all data/metadata should have started on a 
single device, with single data and dup metadata.  The conversion was 
specifically to raid5 for both data/metadata, so that's what he should 
have ended up with.

But, somewhere along the line he got 6 GiB of raid1 metadata.  Either he 
added/rewrote a *BUNCH* of files after adding at least one device before 
the conversion that he didn't tell us about, or the conversion screwed 
up, because that's a lot of raid1 metadata coming out of nowhere!


But I'm *strongly* suspecting a pre-full-raid56-support kernel, because 
btrfs-progs is certainly reasonably new (v4.1.1, as of yesterday 4.1.2 is 
the newest as I mention above), but the fi df doesn't report global 
reserve.  The only way I know of that it wouldn't report that with a new 
userspace is if the kernelspace is too old.  And AFAIK, the kernel was 
reporting global reserve (with fi df listing it as unknown if it was too 
old) _well_ before full raid56 support.  So it's gotta be an old kernel, 
with only partial raid56 support, which might explain the weird to-raid56 
conversion results.


Finally, that's btrfs-progs 4.1.1, the one that's blacklisted due to a 
buggy mkfs.btrfs.  If he created the filesystem with that mkfs.btrfs... 
maybe that explains the funky results, as well.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Data single *and* raid?

[Chris Murphy]

On Sat, Aug 1, 2015 at 3:45 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Does fi usage deal with raid5 yet?

Now that you mention it, I think it doesn't. But if it did, it would
show this problem better than df I think.

> But, somewhere along the line he got 6 GiB of raid1 metadata.  Either he
> added/rewrote a *BUNCH* of files after adding at least one device before
> the conversion that he didn't tell us about, or the conversion screwed
> up, because that's a lot of raid1 metadata coming out of nowhere!

Yeah it's a little confusing, that's why I asked about the kernel version.
>
>
> But I'm *strongly* suspecting a pre-full-raid56-support kernel, because
> btrfs-progs is certainly reasonably new (v4.1.1, as of yesterday 4.1.2 is
> the newest as I mention above), but the fi df doesn't report global
> reserve.  The only way I know of that it wouldn't report that with a new
> userspace is if the kernelspace is too old.

Yes and that also puts it in the realm of kernels that weren't
releasing/deallocating empty chunks; although I don't know if that's a
factor, if dconvert forcibly deals with this..


> Finally, that's btrfs-progs 4.1.1, the one that's blacklisted due to a
> buggy mkfs.btrfs.  If he created the filesystem with that mkfs.btrfs...
> maybe that explains the funky results, as well.

Good catch. That really ought to be filed as a bug with that distro to
flat out remove 4.1.1 from the repos.



-- 
Chris Murphy

Re: Data single *and* raid?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2238 bytes --]

On Sat, Aug 01, 2015 at 04:26:25PM -0600, Chris Murphy wrote:
> On Sat, Aug 1, 2015 at 3:45 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> > Does fi usage deal with raid5 yet?
> 
> Now that you mention it, I think it doesn't. But if it did, it would
> show this problem better than df I think.
> 
> > But, somewhere along the line he got 6 GiB of raid1 metadata.  Either he
> > added/rewrote a *BUNCH* of files after adding at least one device before
> > the conversion that he didn't tell us about, or the conversion screwed
> > up, because that's a lot of raid1 metadata coming out of nowhere!
> 
> Yeah it's a little confusing, that's why I asked about the kernel version.

   It's 6 GiB of metadata for 5.7 GIB of data, or thereabouts. 0.1% is
about the expected size.

   Now, given the original description, it's not clear at all why the
data has suddenly doubled in size -- unless there's some snapshots,
and the OP did a defrag as well.

> > But I'm *strongly* suspecting a pre-full-raid56-support kernel, because
> > btrfs-progs is certainly reasonably new (v4.1.1, as of yesterday 4.1.2 is
> > the newest as I mention above), but the fi df doesn't report global
> > reserve.  The only way I know of that it wouldn't report that with a new
> > userspace is if the kernelspace is too old.
> 
> Yes and that also puts it in the realm of kernels that weren't
> releasing/deallocating empty chunks; although I don't know if that's a
> factor, if dconvert forcibly deals with this..

   It does -- you only have to look at the btrfs fi df output to see
that there's no empty block groups (to within 0.1% or so)

> > Finally, that's btrfs-progs 4.1.1, the one that's blacklisted due to a
> > buggy mkfs.btrfs.  If he created the filesystem with that mkfs.btrfs...
> > maybe that explains the funky results, as well.
> 
> Good catch. That really ought to be filed as a bug with that distro to
> flat out remove 4.1.1 from the repos.

   If they picked up 4.1.1 fast enough, they should pick up 4.1.2 just
as quickly...

   Hugo.

-- 
Hugo Mills             | "What are we going to do tonight?"
hugo@... carfax.org.uk | "The same thing we do every night, Pinky. Try to
http://carfax.org.uk/  | take over the world!"
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Data single *and* raid?

[Duncan]

Hugo Mills posted on Sat, 01 Aug 2015 22:34:13 +0000 as excerpted:

>> Yes and that also puts it in the realm of kernels that weren't
>> releasing/deallocating empty chunks; although I don't know if that's a
>> factor, if dconvert forcibly deals with this..
> 
>    It does -- you only have to look at the btrfs fi df output to see
> that there's no empty block groups (to within 0.1% or so)

Exactly.  The allocations are all full.

And the fi show says there's little to no room to allocate more, as 
well.  There's room on one device, but that's not going to help except 
with single, which shouldn't be allocated any more.

I'd say...

1) If this fs was created with btrfs-progs v4.1.1, get what you need to 
retrieve off of it immediately, then blow it away and start over, as the 
thing isn't stable and all data is at risk.

2) If it wasn't created with progs v4.1.1, the next issue is that kernel, 
since it's obviously from before raid56 was fully functional (well either 
that or there's a more serious bug going on).  Existing data should at 
least be reasonably stable, but with raid56 mode being so new, the newer 
the kernel you're using to work with it, the better.  4.1.x at LEAST, if 
not 4.2-rc, as we're nearing the end of the 4.2 development cycle.  And 
plan on keeping even better than normal backups and on current on kernels 
for at least another several kernel cycles, if you're going to use raid56 
mode, as while it's complete now, it's going to take a bit to stabilize 
to the level of the rest of btrfs itself, which of course is stabilizing 
now, but not really fully stable and mature yet, so the sysadmin's rule 
that data with any value is backed up, or by definition it's throw-away 
data, despite any claims to the contrary, continues to apply double on 
btrfs, compared to more mature and stable filesystems.

So definitely upgrade the kernel.  Then see where things stand.

3) Meanwhile, based on raid56 mode's newness, I've been recommending that 
people stay off it until 4.5-ish or so, basically a year after initial 
nominal full support, unless of course their intent is to be a leading/
bleeding edge testing and report deployment.  Otherwise, use raid1 or 
raid10 mode until then, and evaluate raid56 mode stability around 4.5, 
before deploying.

And if you're one of the brave doing current raid56 deployment, testing 
and bug reporting in full knowledge of its newness and lack of current 
stability and maturity, THANKS, it's your work that's helping to 
stabilize it for others, when they do switch to it. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Data single *and* raid?

[Chris Murphy]

On Sat, Aug 1, 2015 at 6:27 PM, Duncan <1i5t5.duncan@cox.net> wrote:

> 1) If this fs was created with btrfs-progs v4.1.1, get what you need to
> retrieve off of it immediately, then blow it away and start over, as the
> thing isn't stable and all data is at risk.

Agreed. But I'd go so far as to say at this point it looks like it's
wedged itself into a kind of self-induced faux-ENOSPC state because
there isn't room to allocate more raid5 chunks. So, I think it's stuck
in any case.

It'd be great to reproduce this with a current kernel and see if it's
still happening.


-- 
Chris Murphy

Re: Data single *and* raid?

[Duncan]

Chris Murphy posted on Sat, 01 Aug 2015 19:14:13 -0600 as excerpted:

> On Sat, Aug 1, 2015 at 6:27 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> 
>> 1) If this fs was created with btrfs-progs v4.1.1, get what you need to
>> retrieve off of it immediately, then blow it away and start over, as
>> the thing isn't stable and all data is at risk.
> 
> Agreed. But I'd go so far as to say at this point it looks like it's
> wedged itself into a kind of self-induced faux-ENOSPC state because
> there isn't room to allocate more raid5 chunks. So, I think it's stuck
> in any case.

Well, yes and no.  If it was setup with progs v4.1.1, save what you can 
and blow it away as it's not stable enough to try anything else.

If it was setup with something earlier (not sure about 4.1.0, was it 
affected? but 4.0.x and earlier should be fine for setup), however, once 
on a new kernel the usual ENOSPC workarounds can be given a try.  That 
would include a first balance start -dusage=0 -musage=0, and if that 
didn't free up at least a gig on a second device, I'd try the old add-a-
device trick and see what happens.  (A few GiB thumb drive should work in 
a pinch, or even a ramdrive if you're willing to risk loss in the event 
of a crash vaporizing everything in memory including the ramdrive.)

After all, even if he didn't know the risk of the still very new raid56 
mode before, he does after reading my earlier message, and anything of 
value should be backed up before he attempts anything, so at that point, 
there's nothing to lose and it's upto him whether he wants to simply blow 
away the current setup and start over with either raid1 or (with another 
device) raid10, avoiding the current risk of raid56, or blow away and 
start over with raid56 again, knowing the risks now, or try to recover 
what's there, viewing it not as the easiest way but as practice in 
disaster recovery, again, with anything of value backed up so there's 
nothing to lose besides the time of fiddling with it and ending up having 
to blow away and restart from backups anyway, regardless of how it goes.

> It'd be great to reproduce this with a current kernel and see if it's
> still happening.

Absolutely.

Tho at this point I believe the chances are pretty good it's simply 
either that bad 4.1.1 mkfs.btrfs, or an older pre-full-raid56-support 
kernel that didn't handle balance to raid56 so well, yet, and that on the 
latest userspace and kernel the problem shouldn't reoccur.

But it'd be nice to *KNOW* that, by trying to reproduce, absolutely.  He 
may well have stumbled upon yet another confirmation of my recommendation 
to wait on raid56 unless you're deliberately testing it, and confirmation 
thereof would be halfway to getting it fixed, so those who /do/ choose to 
wait won't be dealing with it. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Data single *and* raid?

[Hendrik Friedel]

Hello,

>>     Looking at the btrfs fi show output, you've probably run out of
>> space during the conversion, probably due to an uneven distribution of
>> the original "single" chunks.
>>
>>     I think I would suggest balancing the single chunks, and trying the
>> conversion (of the unconverted parts) again:
>>
>> # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/
>> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/
>>
>
> Yep I bet that's it also. btrfs fi usage might be better at exposing this case.


Thanks for your hints.
The balance is running now for about 11h:
The status is a bit surprising to me:
0 out of about 2619 chunks balanced (4165 considered), 100% left

btrfs fi usage is also surprising:
Overall:
     Device size:                   8.19TiB
     Device allocated:              2.56TiB
     Device unallocated:            5.62TiB
     Device missing:                  0.00B
     Used:                          2.56TiB
     Free (estimated):             11.65TiB      (min: 2.81TiB)
     Data ratio:                       0.48
     Metadata ratio:                   1.33
     Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:2.55TiB, Used:2.55TiB
    /dev/sdc        1.60TiB
    /dev/sde      975.44GiB

Data,RAID5: Size:2.73TiB, Used:2.72TiB
    /dev/sdc        1.12TiB
    /dev/sdd        2.73TiB
    /dev/sde        1.61TiB

Metadata,RAID1: Size:6.00GiB, Used:5.33GiB
    /dev/sdc        5.00GiB
    /dev/sdd        1.00GiB
    /dev/sde        6.00GiB

Metadata,RAID5: Size:3.00GiB, Used:2.99GiB
    /dev/sdc        2.00GiB
    /dev/sdd        1.00GiB
    /dev/sde        3.00GiB

System,RAID5: Size:32.00MiB, Used:736.00KiB
    /dev/sdd       32.00MiB
    /dev/sde       32.00MiB

Unallocated:
    /dev/sdc        1.02MiB
    /dev/sdd        1.02MiB
    /dev/sde      164.59GiB


I hope, that is because Raid5 is not implemented yet?

Regards,
Hendrik

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Re: Data single *and* raid?

[Chris Murphy]

On Sat, Aug 1, 2015 at 9:46 PM, Duncan <1i5t5.duncan@cox.net> wrote:

> If it was setup with something earlier (not sure about 4.1.0, was it
> affected?

No.

> but 4.0.x and earlier should be fine for setup), however, once
> on a new kernel the usual ENOSPC workarounds can be given a try.  That
> would include a first balance start -dusage=0 -musage=0, and if that
> didn't free up at least a gig on a second device,

If I'm following this correctly, the reproduce steps would be to
create a single device Btrfs that's ~94% full, add two devices, then
convert to raid5. I think what's going on here is empty single profile
data chunks aren't being deallocated, and it's effectively a 2 device
raid5.

So actually, you're right, either -dusage=0 might fix it, or better,
newer kernel that automatically deallocated empty/converted single
profile data chunks. But right now it will take another balance in the
end because it looks like this is effectively a 2 device raid5, with
the 3rd drive full of single only chunks (which might be empty?).


-- 
Chris Murphy

Re: Data single *and* raid?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1498 bytes --]

On Sun, Aug 02, 2015 at 12:31:13PM -0600, Chris Murphy wrote:
> On Sat, Aug 1, 2015 at 9:46 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> 
> > If it was setup with something earlier (not sure about 4.1.0, was it
> > affected?
> 
> No.
> 
> > but 4.0.x and earlier should be fine for setup), however, once
> > on a new kernel the usual ENOSPC workarounds can be given a try.  That
> > would include a first balance start -dusage=0 -musage=0, and if that
> > didn't free up at least a gig on a second device,
> 
> If I'm following this correctly, the reproduce steps would be to
> create a single device Btrfs that's ~94% full, add two devices, then
> convert to raid5. I think what's going on here is empty single profile
> data chunks aren't being deallocated, and it's effectively a 2 device
> raid5.

   This isn't supported by the btrfs fi df output: all of the
allocated space is used.

   Hugo.

> So actually, you're right, either -dusage=0 might fix it, or better,
> newer kernel that automatically deallocated empty/converted single
> profile data chunks. But right now it will take another balance in the
> end because it looks like this is effectively a 2 device raid5, with
> the 3rd drive full of single only chunks (which might be empty?).

-- 
Hugo Mills             | I'll take your bet, but make it ten thousand francs.
hugo@... carfax.org.uk | I'm only a _poor_ corrupt official.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                              Capt. Renaud, Casablanca

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Data single *and* raid?

[Hendrik Friedel]

Hello Hugo,
hello Chris,

thanks for your advice. Now I am here:
btrfs balance start -dprofiles=single -mprofiles=raid1 /mnt/__Complete_Disk/
Done, had to relocate 0 out of 3939 chunks


root@homeserver:/mnt/__Complete_Disk# btrfs fi show
Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
         Total devices 3 FS bytes used 3.78TiB
         devid    1 size 2.73TiB used 2.72TiB path /dev/sde
         devid    2 size 2.73TiB used 2.23TiB path /dev/sdc
         devid    3 size 2.73TiB used 2.73TiB path /dev/sdd

btrfs-progs v4.1.1


So, that looks good.

But then:
root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/
Data, RAID5: total=3.83TiB, used=3.78TiB
System, RAID5: total=32.00MiB, used=576.00KiB
Metadata, RAID5: total=6.46GiB, used=4.84GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Is the RAID5 expected here?
I did not yet run:
btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft 
/mnt/new_storage/

Regards,
Hendrik


On 01.08.2015 22:44, Chris Murphy wrote:
> On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>> On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
>>> Hello,
>>>
>>> I converted an array to raid5 by
>>> btrfs device add /dev/sdd /mnt/new_storage
>>> btrfs device add /dev/sdc /mnt/new_storage
>>> btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
>>>
>>> The Balance went through. But now:
>>> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>>>          Total devices 3 FS bytes used 5.28TiB
>>>          devid    1 size 2.73TiB used 2.57TiB path /dev/sde
>>>          devid    2 size 2.73TiB used 2.73TiB path /dev/sdc
>>>          devid    3 size 2.73TiB used 2.73TiB path /dev/sdd
>>> btrfs-progs v4.1.1
>>>
>>> Already the 2.57TiB is a bit surprising:
>>> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
>>> Data, single: total=2.55TiB, used=2.55TiB
>>> Data, RAID5: total=2.73TiB, used=2.72TiB
>>> System, RAID5: total=32.00MiB, used=736.00KiB
>>> Metadata, RAID1: total=6.00GiB, used=5.33GiB
>>> Metadata, RAID5: total=3.00GiB, used=2.99GiB
>>
>>     Looking at the btrfs fi show output, you've probably run out of
>> space during the conversion, probably due to an uneven distribution of
>> the original "single" chunks.
>>
>>     I think I would suggest balancing the single chunks, and trying the
>> conversion (of the unconverted parts) again:
>>
>> # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/
>> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/
>>
>
> Yep I bet that's it also. btrfs fi usage might be better at exposing this case.
>
>


-- 
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Tel. 04203 8394854
Mobil 0178 1874363

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Re: Data single *and* raid?

[Qu Wenruo]

Hendrik Friedel wrote on 2015/08/06 20:57 +0200:
> Hello Hugo,
> hello Chris,
>
> thanks for your advice. Now I am here:
> btrfs balance start -dprofiles=single -mprofiles=raid1
> /mnt/__Complete_Disk/
> Done, had to relocate 0 out of 3939 chunks
>
>
> root@homeserver:/mnt/__Complete_Disk# btrfs fi show
> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>          Total devices 3 FS bytes used 3.78TiB
>          devid    1 size 2.73TiB used 2.72TiB path /dev/sde
>          devid    2 size 2.73TiB used 2.23TiB path /dev/sdc
>          devid    3 size 2.73TiB used 2.73TiB path /dev/sdd
>
> btrfs-progs v4.1.1
>
>
> So, that looks good.
>
> But then:
> root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/
> Data, RAID5: total=3.83TiB, used=3.78TiB
> System, RAID5: total=32.00MiB, used=576.00KiB
> Metadata, RAID5: total=6.46GiB, used=4.84GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
GlobalReserve is not a chunk type, it just means a range of metadata 
reserved for overcommiting.
And it's always single.

Personally, I don't think it should be output in "fi df" command, as 
it's in a higher level than chunk.

At least for your case, nothing is needed to worry about.

Thanks,
Qu

>
> Is the RAID5 expected here?
> I did not yet run:
> btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
> /mnt/new_storage/
>
> Regards,
> Hendrik
>
>
> On 01.08.2015 22:44, Chris Murphy wrote:
>> On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>>> On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
>>>> Hello,
>>>>
>>>> I converted an array to raid5 by
>>>> btrfs device add /dev/sdd /mnt/new_storage
>>>> btrfs device add /dev/sdc /mnt/new_storage
>>>> btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
>>>>
>>>> The Balance went through. But now:
>>>> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>>>>          Total devices 3 FS bytes used 5.28TiB
>>>>          devid    1 size 2.73TiB used 2.57TiB path /dev/sde
>>>>          devid    2 size 2.73TiB used 2.73TiB path /dev/sdc
>>>>          devid    3 size 2.73TiB used 2.73TiB path /dev/sdd
>>>> btrfs-progs v4.1.1
>>>>
>>>> Already the 2.57TiB is a bit surprising:
>>>> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
>>>> Data, single: total=2.55TiB, used=2.55TiB
>>>> Data, RAID5: total=2.73TiB, used=2.72TiB
>>>> System, RAID5: total=32.00MiB, used=736.00KiB
>>>> Metadata, RAID1: total=6.00GiB, used=5.33GiB
>>>> Metadata, RAID5: total=3.00GiB, used=2.99GiB
>>>
>>>     Looking at the btrfs fi show output, you've probably run out of
>>> space during the conversion, probably due to an uneven distribution of
>>> the original "single" chunks.
>>>
>>>     I think I would suggest balancing the single chunks, and trying the
>>> conversion (of the unconverted parts) again:
>>>
>>> # btrfs balance start -dprofiles=single -mprofile=raid1
>>> /mnt/new_storage/
>>> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
>>> /mnt/new_storage/
>>>
>>
>> Yep I bet that's it also. btrfs fi usage might be better at exposing
>> this case.
>>
>>
>
>

Re: Data single *and* raid?

[Hendrik Friedel]

Hello Quo,

thanks for your reply.

 >> But then:
>> root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/
>> Data, RAID5: total=3.83TiB, used=3.78TiB
>> System, RAID5: total=32.00MiB, used=576.00KiB
>> Metadata, RAID5: total=6.46GiB, used=4.84GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
> GlobalReserve is not a chunk type, it just means a range of metadata
> reserved for overcommiting.
> And it's always single.
>
> Personally, I don't think it should be output in "fi df" command, as
> it's in a higher level than chunk.
>
> At least for your case, nothing is needed to worry about.


But this seems to be a RAID5 now, right?
Well, that's what I want, but the command was:
btrfs balance start -dprofiles=single -mprofiles=raid1
/mnt/__Complete_Disk/

So, we would expect raid1 here, no?

Greetings,
Hendrik


>>
>>
>> On 01.08.2015 22:44, Chris Murphy wrote:
>>> On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>>>> On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
>>>>> Hello,
>>>>>
>>>>> I converted an array to raid5 by
>>>>> btrfs device add /dev/sdd /mnt/new_storage
>>>>> btrfs device add /dev/sdc /mnt/new_storage
>>>>> btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
>>>>>
>>>>> The Balance went through. But now:
>>>>> Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>>>>>          Total devices 3 FS bytes used 5.28TiB
>>>>>          devid    1 size 2.73TiB used 2.57TiB path /dev/sde
>>>>>          devid    2 size 2.73TiB used 2.73TiB path /dev/sdc
>>>>>          devid    3 size 2.73TiB used 2.73TiB path /dev/sdd
>>>>> btrfs-progs v4.1.1
>>>>>
>>>>> Already the 2.57TiB is a bit surprising:
>>>>> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
>>>>> Data, single: total=2.55TiB, used=2.55TiB
>>>>> Data, RAID5: total=2.73TiB, used=2.72TiB
>>>>> System, RAID5: total=32.00MiB, used=736.00KiB
>>>>> Metadata, RAID1: total=6.00GiB, used=5.33GiB
>>>>> Metadata, RAID5: total=3.00GiB, used=2.99GiB
>>>>
>>>>     Looking at the btrfs fi show output, you've probably run out of
>>>> space during the conversion, probably due to an uneven distribution of
>>>> the original "single" chunks.
>>>>
>>>>     I think I would suggest balancing the single chunks, and trying the
>>>> conversion (of the unconverted parts) again:
>>>>
>>>> # btrfs balance start -dprofiles=single -mprofile=raid1
>>>> /mnt/new_storage/
>>>> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
>>>> /mnt/new_storage/
>>>>
>>>
>>> Yep I bet that's it also. btrfs fi usage might be better at exposing
>>> this case.
>>>
>>>
>>
>>


-- 
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Tel. 04203 8394854
Mobil 0178 1874363

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Re: Data single *and* raid?

[Duncan]

Hendrik Friedel posted on Fri, 07 Aug 2015 07:16:04 +0200 as excerpted:

>>> But then:
>>> # btrfs fi df /mnt/__Complete_Disk/
>>> Data, RAID5: total=3.83TiB, used=3.78TiB
>>> System, RAID5: total=32.00MiB, used=576.00KiB
>>> Metadata, RAID5: total=6.46GiB, used=4.84GiB
>>> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> [T]his seems to be a RAID5 now, right?
> Well, that's what I want, but the command was:
> btrfs balance start -dprofiles=single -mprofiles=raid1
> /mnt/__Complete_Disk/
> 
> So, we would expect raid1 here, no?

No.  The behavior might be a bit counterintuitive on first glance, but 
once the logic is understood, it makes sense.

1) You had tried the initial raid5 convert using an earlier kernel that 
had incomplete raid5 support, as evidenced by the lack of the global-
reserve line in btrfs fi df, on a new enough userspace that it should 
have had it.

2) That initial attempt ran out of space, possibly because it was keeping 
the single and raid1 chunks around due to fragmentation (Hugo's guess), 
or due to a now fixed raid5 conversion bugs in the old kernel[1] (my 
guess), or possibly due to some other bug that's apparently fixed in 
newer kernels, thus the successful completion of the conversion below.

3) But that initial attempt still did one critical thing -- set the 
default new-chunk type to raid5, for both data and metadata.

4) So when the second btrfs balance attempt came along, this one 
primarily intended to clean up that fragmentation that Hugo expected, and 
thus targeted at those old single data and raid1 metadata chunks, when it 
rewrote those chunks it used the new chunk default, rewriting them into 
raid5.

That was a result that Hugo obviously didn't predict as his instructions 
suggested following up with another balance command to complete the 
conversion.  And neither Chris (apparently) nor I (definitely!) foresaw 
it either.  But the behavior does make sense, once you take into account 
the default chunk type, and that a balance-convert does normally change 
it.

And FWIW, the precise behavior of this default chunk type selector and 
when it falls back to single data and raid1 or dup metadata (as it will 
in some instances with a degraded filesystem), has both been problematic 
before, and is being debated in a current thread, due to the implications 
for writable mounts of degraded single-device raid1s, for instance.  It's 
behavior in corner-cases like these that is much of the reason most 
regulars on this list don't consider btrfs fully stable and mature, just 
yet, because sometimes that corner-case behavior can mean the filesystem 
doing the wrong thing, going read-only, without any way to correct the 
problem even tho things are generally still fine, because correcting the 
problem would require a writable filesystem, thus creating a chicken and 
egg situation where correcting the problem requires a writable 
filesystem, but a writable filesystem isn't allowed until the problem is 
corrected, for instance.  (As of now, in that situation a user has little 
choice but to copy the data on that read-only filesystem elsewhere, do a 
mkfs to wipe away the problem, and restore to the fresh filesystem.  
Technically, that shouldn't be required.)

---
[1] FWIW, for "online" tasks like btrfs balance, the btrfs-progs 
userspace simply issues the commands to the kernel, which does the real 
work.  For "offline" tasks such as btrfs check or btrfs restore, 
userspace is the real brains and the kernel simply relays the commands at 
the device level, without much involvement by the kernel's btrfs code at 
all.  So while you had a current userspace, the old kernel was the 
critical part since btrfs balance is an online command in which it's the 
kernel's btrfs code that does the real work.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Data single *and* raid?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3173 bytes --]

On Fri, Aug 07, 2015 at 07:16:04AM +0200, Hendrik Friedel wrote:
> Hello Quo,
> 
> thanks for your reply.
> 
> >> But then:
> >>root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/
> >>Data, RAID5: total=3.83TiB, used=3.78TiB
> >>System, RAID5: total=32.00MiB, used=576.00KiB
> >>Metadata, RAID5: total=6.46GiB, used=4.84GiB
> >>GlobalReserve, single: total=512.00MiB, used=0.00B
> >GlobalReserve is not a chunk type, it just means a range of metadata
> >reserved for overcommiting.
> >And it's always single.
> >
> >Personally, I don't think it should be output in "fi df" command, as
> >it's in a higher level than chunk.
> >
> >At least for your case, nothing is needed to worry about.
> 
> 
> But this seems to be a RAID5 now, right?
> Well, that's what I want, but the command was:
> btrfs balance start -dprofiles=single -mprofiles=raid1
> /mnt/__Complete_Disk/
> 
> So, we would expect raid1 here, no?

   No, profiles= selects chunks of that profile. If you wanted to
change the RAID level, you should use convert=.

   Hugo.

> Greetings,
> Hendrik
> 
> 
> >>
> >>
> >>On 01.08.2015 22:44, Chris Murphy wrote:
> >>>On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
> >>>>On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
> >>>>>Hello,
> >>>>>
> >>>>>I converted an array to raid5 by
> >>>>>btrfs device add /dev/sdd /mnt/new_storage
> >>>>>btrfs device add /dev/sdc /mnt/new_storage
> >>>>>btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
> >>>>>
> >>>>>The Balance went through. But now:
> >>>>>Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
> >>>>>         Total devices 3 FS bytes used 5.28TiB
> >>>>>         devid    1 size 2.73TiB used 2.57TiB path /dev/sde
> >>>>>         devid    2 size 2.73TiB used 2.73TiB path /dev/sdc
> >>>>>         devid    3 size 2.73TiB used 2.73TiB path /dev/sdd
> >>>>>btrfs-progs v4.1.1
> >>>>>
> >>>>>Already the 2.57TiB is a bit surprising:
> >>>>>root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
> >>>>>Data, single: total=2.55TiB, used=2.55TiB
> >>>>>Data, RAID5: total=2.73TiB, used=2.72TiB
> >>>>>System, RAID5: total=32.00MiB, used=736.00KiB
> >>>>>Metadata, RAID1: total=6.00GiB, used=5.33GiB
> >>>>>Metadata, RAID5: total=3.00GiB, used=2.99GiB
> >>>>
> >>>>    Looking at the btrfs fi show output, you've probably run out of
> >>>>space during the conversion, probably due to an uneven distribution of
> >>>>the original "single" chunks.
> >>>>
> >>>>    I think I would suggest balancing the single chunks, and trying the
> >>>>conversion (of the unconverted parts) again:
> >>>>
> >>>># btrfs balance start -dprofiles=single -mprofile=raid1
> >>>>/mnt/new_storage/
> >>>># btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
> >>>>/mnt/new_storage/
> >>>>
> >>>
> >>>Yep I bet that's it also. btrfs fi usage might be better at exposing
> >>>this case.
> >>>
> >>>
> >>
> >>
> 
> 

-- 
Hugo Mills             | There is no dark side to the Moon, really. As a
hugo@... carfax.org.uk | matter of fact, it's all dark.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                            Pink Floyd

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]