* Re: Data single *and* raid?
2015-08-01 20:44 ` Chris Murphy
@ 2015-08-01 21:45 ` Duncan
2015-08-01 22:26 ` Chris Murphy
2015-08-02 12:54 ` Hendrik Friedel
2015-08-06 18:57 ` Hendrik Friedel
2 siblings, 1 reply; 18+ messages in thread
From: Duncan @ 2015-08-01 21:45 UTC (permalink / raw)
To: linux-btrfs
Chris Murphy posted on Sat, 01 Aug 2015 14:44:52 -0600 as excerpted:
> On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>> On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
>>> Hello,
>>>
>>> I converted an array to raid5 by
>>> btrfs device add /dev/sdd /mnt/new_storage
>>> btrfs device add /dev/sdc /mnt/new_storage btrfs
>>> balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
>>>
>>> The Balance went through. But now:
>>> Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>>> Total devices 3 FS bytes used 5.28TiB
>>> devid 1 size 2.73TiB used 2.57TiB path /dev/sde
>>> devid 2 size 2.73TiB used 2.73TiB path /dev/sdc
>>> devid 3 size 2.73TiB used 2.73TiB path /dev/sdd
>>> btrfs-progs v4.1.1
>>>
>>> Already the 2.57TiB is a bit surprising:
>>> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
>>> Data, single: total=2.55TiB, used=2.55TiB
>>> Data, RAID5: total=2.73TiB, used=2.72TiB
>>> System, RAID5: total=32.00MiB, used=736.00KiB
>>> Metadata, RAID1: total=6.00GiB, used=5.33GiB
>>> Metadata, RAID5: total=3.00GiB, used=2.99GiB
>>
>> Looking at the btrfs fi show output, you've probably run out of
>> space during the conversion, probably due to an uneven distribution of
>> the original "single" chunks.
>>
>> I think I would suggest balancing the single chunks, and trying the
>> conversion (of the unconverted parts) again:
>>
>> # btrfs balance start -dprofiles=single -mprofile=raid1
>> /mnt/new_storage/
>> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
>> /mnt/new_storage/
>>
>>
> Yep I bet that's it also. btrfs fi usage might be better at exposing
> this case.
Does fi usage deal with raid5 yet?
Last I knew it didn't deal with raid56 (which I'm not using) or with
mixed-bg (which I am using on one btrfs). On the mixed-bg btrfs it still
warns it doesn't handle that, reporting unallocated of 16 EiB on a 256
MiB filesystem, so clearly the warning is valid. 16 EiB unallocated on a
256 MiB btrfs, I wish! progs v4.1.2, latest as of yesterday, I believe.
Meanwhile, three devices the same size. He just added two of them and
didn't do a rebalance after that until the raid5 conversion, so except
for anything added since, all data/metadata should have started on a
single device, with single data and dup metadata. The conversion was
specifically to raid5 for both data/metadata, so that's what he should
have ended up with.
But, somewhere along the line he got 6 GiB of raid1 metadata. Either he
added/rewrote a *BUNCH* of files after adding at least one device before
the conversion that he didn't tell us about, or the conversion screwed
up, because that's a lot of raid1 metadata coming out of nowhere!
But I'm *strongly* suspecting a pre-full-raid56-support kernel, because
btrfs-progs is certainly reasonably new (v4.1.1, as of yesterday 4.1.2 is
the newest as I mention above), but the fi df doesn't report global
reserve. The only way I know of that it wouldn't report that with a new
userspace is if the kernelspace is too old. And AFAIK, the kernel was
reporting global reserve (with fi df listing it as unknown if it was too
old) _well_ before full raid56 support. So it's gotta be an old kernel,
with only partial raid56 support, which might explain the weird to-raid56
conversion results.
Finally, that's btrfs-progs 4.1.1, the one that's blacklisted due to a
buggy mkfs.btrfs. If he created the filesystem with that mkfs.btrfs...
maybe that explains the funky results, as well.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-01 21:45 ` Duncan
@ 2015-08-01 22:26 ` Chris Murphy
2015-08-01 22:34 ` Hugo Mills
0 siblings, 1 reply; 18+ messages in thread
From: Chris Murphy @ 2015-08-01 22:26 UTC (permalink / raw)
To: Btrfs BTRFS
On Sat, Aug 1, 2015 at 3:45 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Does fi usage deal with raid5 yet?
Now that you mention it, I think it doesn't. But if it did, it would
show this problem better than df I think.
> But, somewhere along the line he got 6 GiB of raid1 metadata. Either he
> added/rewrote a *BUNCH* of files after adding at least one device before
> the conversion that he didn't tell us about, or the conversion screwed
> up, because that's a lot of raid1 metadata coming out of nowhere!
Yeah it's a little confusing, that's why I asked about the kernel version.
>
>
> But I'm *strongly* suspecting a pre-full-raid56-support kernel, because
> btrfs-progs is certainly reasonably new (v4.1.1, as of yesterday 4.1.2 is
> the newest as I mention above), but the fi df doesn't report global
> reserve. The only way I know of that it wouldn't report that with a new
> userspace is if the kernelspace is too old.
Yes and that also puts it in the realm of kernels that weren't
releasing/deallocating empty chunks; although I don't know if that's a
factor, if dconvert forcibly deals with this..
> Finally, that's btrfs-progs 4.1.1, the one that's blacklisted due to a
> buggy mkfs.btrfs. If he created the filesystem with that mkfs.btrfs...
> maybe that explains the funky results, as well.
Good catch. That really ought to be filed as a bug with that distro to
flat out remove 4.1.1 from the repos.
--
Chris Murphy
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-01 22:26 ` Chris Murphy
@ 2015-08-01 22:34 ` Hugo Mills
2015-08-02 0:27 ` Duncan
0 siblings, 1 reply; 18+ messages in thread
From: Hugo Mills @ 2015-08-01 22:34 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
[-- Attachment #1: Type: text/plain, Size: 2238 bytes --]
On Sat, Aug 01, 2015 at 04:26:25PM -0600, Chris Murphy wrote:
> On Sat, Aug 1, 2015 at 3:45 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> > Does fi usage deal with raid5 yet?
>
> Now that you mention it, I think it doesn't. But if it did, it would
> show this problem better than df I think.
>
> > But, somewhere along the line he got 6 GiB of raid1 metadata. Either he
> > added/rewrote a *BUNCH* of files after adding at least one device before
> > the conversion that he didn't tell us about, or the conversion screwed
> > up, because that's a lot of raid1 metadata coming out of nowhere!
>
> Yeah it's a little confusing, that's why I asked about the kernel version.
It's 6 GiB of metadata for 5.7 GIB of data, or thereabouts. 0.1% is
about the expected size.
Now, given the original description, it's not clear at all why the
data has suddenly doubled in size -- unless there's some snapshots,
and the OP did a defrag as well.
> > But I'm *strongly* suspecting a pre-full-raid56-support kernel, because
> > btrfs-progs is certainly reasonably new (v4.1.1, as of yesterday 4.1.2 is
> > the newest as I mention above), but the fi df doesn't report global
> > reserve. The only way I know of that it wouldn't report that with a new
> > userspace is if the kernelspace is too old.
>
> Yes and that also puts it in the realm of kernels that weren't
> releasing/deallocating empty chunks; although I don't know if that's a
> factor, if dconvert forcibly deals with this..
It does -- you only have to look at the btrfs fi df output to see
that there's no empty block groups (to within 0.1% or so)
> > Finally, that's btrfs-progs 4.1.1, the one that's blacklisted due to a
> > buggy mkfs.btrfs. If he created the filesystem with that mkfs.btrfs...
> > maybe that explains the funky results, as well.
>
> Good catch. That really ought to be filed as a bug with that distro to
> flat out remove 4.1.1 from the repos.
If they picked up 4.1.1 fast enough, they should pick up 4.1.2 just
as quickly...
Hugo.
--
Hugo Mills | "What are we going to do tonight?"
hugo@... carfax.org.uk | "The same thing we do every night, Pinky. Try to
http://carfax.org.uk/ | take over the world!"
PGP: E2AB1DE4 |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-01 22:34 ` Hugo Mills
@ 2015-08-02 0:27 ` Duncan
2015-08-02 1:14 ` Chris Murphy
0 siblings, 1 reply; 18+ messages in thread
From: Duncan @ 2015-08-02 0:27 UTC (permalink / raw)
To: linux-btrfs
Hugo Mills posted on Sat, 01 Aug 2015 22:34:13 +0000 as excerpted:
>> Yes and that also puts it in the realm of kernels that weren't
>> releasing/deallocating empty chunks; although I don't know if that's a
>> factor, if dconvert forcibly deals with this..
>
> It does -- you only have to look at the btrfs fi df output to see
> that there's no empty block groups (to within 0.1% or so)
Exactly. The allocations are all full.
And the fi show says there's little to no room to allocate more, as
well. There's room on one device, but that's not going to help except
with single, which shouldn't be allocated any more.
I'd say...
1) If this fs was created with btrfs-progs v4.1.1, get what you need to
retrieve off of it immediately, then blow it away and start over, as the
thing isn't stable and all data is at risk.
2) If it wasn't created with progs v4.1.1, the next issue is that kernel,
since it's obviously from before raid56 was fully functional (well either
that or there's a more serious bug going on). Existing data should at
least be reasonably stable, but with raid56 mode being so new, the newer
the kernel you're using to work with it, the better. 4.1.x at LEAST, if
not 4.2-rc, as we're nearing the end of the 4.2 development cycle. And
plan on keeping even better than normal backups and on current on kernels
for at least another several kernel cycles, if you're going to use raid56
mode, as while it's complete now, it's going to take a bit to stabilize
to the level of the rest of btrfs itself, which of course is stabilizing
now, but not really fully stable and mature yet, so the sysadmin's rule
that data with any value is backed up, or by definition it's throw-away
data, despite any claims to the contrary, continues to apply double on
btrfs, compared to more mature and stable filesystems.
So definitely upgrade the kernel. Then see where things stand.
3) Meanwhile, based on raid56 mode's newness, I've been recommending that
people stay off it until 4.5-ish or so, basically a year after initial
nominal full support, unless of course their intent is to be a leading/
bleeding edge testing and report deployment. Otherwise, use raid1 or
raid10 mode until then, and evaluate raid56 mode stability around 4.5,
before deploying.
And if you're one of the brave doing current raid56 deployment, testing
and bug reporting in full knowledge of its newness and lack of current
stability and maturity, THANKS, it's your work that's helping to
stabilize it for others, when they do switch to it. =:^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-02 0:27 ` Duncan
@ 2015-08-02 1:14 ` Chris Murphy
2015-08-02 3:46 ` Duncan
0 siblings, 1 reply; 18+ messages in thread
From: Chris Murphy @ 2015-08-02 1:14 UTC (permalink / raw)
To: Btrfs BTRFS
On Sat, Aug 1, 2015 at 6:27 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> 1) If this fs was created with btrfs-progs v4.1.1, get what you need to
> retrieve off of it immediately, then blow it away and start over, as the
> thing isn't stable and all data is at risk.
Agreed. But I'd go so far as to say at this point it looks like it's
wedged itself into a kind of self-induced faux-ENOSPC state because
there isn't room to allocate more raid5 chunks. So, I think it's stuck
in any case.
It'd be great to reproduce this with a current kernel and see if it's
still happening.
--
Chris Murphy
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-02 1:14 ` Chris Murphy
@ 2015-08-02 3:46 ` Duncan
2015-08-02 18:31 ` Chris Murphy
0 siblings, 1 reply; 18+ messages in thread
From: Duncan @ 2015-08-02 3:46 UTC (permalink / raw)
To: linux-btrfs
Chris Murphy posted on Sat, 01 Aug 2015 19:14:13 -0600 as excerpted:
> On Sat, Aug 1, 2015 at 6:27 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>
>> 1) If this fs was created with btrfs-progs v4.1.1, get what you need to
>> retrieve off of it immediately, then blow it away and start over, as
>> the thing isn't stable and all data is at risk.
>
> Agreed. But I'd go so far as to say at this point it looks like it's
> wedged itself into a kind of self-induced faux-ENOSPC state because
> there isn't room to allocate more raid5 chunks. So, I think it's stuck
> in any case.
Well, yes and no. If it was setup with progs v4.1.1, save what you can
and blow it away as it's not stable enough to try anything else.
If it was setup with something earlier (not sure about 4.1.0, was it
affected? but 4.0.x and earlier should be fine for setup), however, once
on a new kernel the usual ENOSPC workarounds can be given a try. That
would include a first balance start -dusage=0 -musage=0, and if that
didn't free up at least a gig on a second device, I'd try the old add-a-
device trick and see what happens. (A few GiB thumb drive should work in
a pinch, or even a ramdrive if you're willing to risk loss in the event
of a crash vaporizing everything in memory including the ramdrive.)
After all, even if he didn't know the risk of the still very new raid56
mode before, he does after reading my earlier message, and anything of
value should be backed up before he attempts anything, so at that point,
there's nothing to lose and it's upto him whether he wants to simply blow
away the current setup and start over with either raid1 or (with another
device) raid10, avoiding the current risk of raid56, or blow away and
start over with raid56 again, knowing the risks now, or try to recover
what's there, viewing it not as the easiest way but as practice in
disaster recovery, again, with anything of value backed up so there's
nothing to lose besides the time of fiddling with it and ending up having
to blow away and restart from backups anyway, regardless of how it goes.
> It'd be great to reproduce this with a current kernel and see if it's
> still happening.
Absolutely.
Tho at this point I believe the chances are pretty good it's simply
either that bad 4.1.1 mkfs.btrfs, or an older pre-full-raid56-support
kernel that didn't handle balance to raid56 so well, yet, and that on the
latest userspace and kernel the problem shouldn't reoccur.
But it'd be nice to *KNOW* that, by trying to reproduce, absolutely. He
may well have stumbled upon yet another confirmation of my recommendation
to wait on raid56 unless you're deliberately testing it, and confirmation
thereof would be halfway to getting it fixed, so those who /do/ choose to
wait won't be dealing with it. =:^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-02 3:46 ` Duncan
@ 2015-08-02 18:31 ` Chris Murphy
2015-08-02 19:06 ` Hugo Mills
0 siblings, 1 reply; 18+ messages in thread
From: Chris Murphy @ 2015-08-02 18:31 UTC (permalink / raw)
Cc: Btrfs BTRFS
On Sat, Aug 1, 2015 at 9:46 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> If it was setup with something earlier (not sure about 4.1.0, was it
> affected?
No.
> but 4.0.x and earlier should be fine for setup), however, once
> on a new kernel the usual ENOSPC workarounds can be given a try. That
> would include a first balance start -dusage=0 -musage=0, and if that
> didn't free up at least a gig on a second device,
If I'm following this correctly, the reproduce steps would be to
create a single device Btrfs that's ~94% full, add two devices, then
convert to raid5. I think what's going on here is empty single profile
data chunks aren't being deallocated, and it's effectively a 2 device
raid5.
So actually, you're right, either -dusage=0 might fix it, or better,
newer kernel that automatically deallocated empty/converted single
profile data chunks. But right now it will take another balance in the
end because it looks like this is effectively a 2 device raid5, with
the 3rd drive full of single only chunks (which might be empty?).
--
Chris Murphy
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-02 18:31 ` Chris Murphy
@ 2015-08-02 19:06 ` Hugo Mills
0 siblings, 0 replies; 18+ messages in thread
From: Hugo Mills @ 2015-08-02 19:06 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
[-- Attachment #1: Type: text/plain, Size: 1498 bytes --]
On Sun, Aug 02, 2015 at 12:31:13PM -0600, Chris Murphy wrote:
> On Sat, Aug 1, 2015 at 9:46 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>
> > If it was setup with something earlier (not sure about 4.1.0, was it
> > affected?
>
> No.
>
> > but 4.0.x and earlier should be fine for setup), however, once
> > on a new kernel the usual ENOSPC workarounds can be given a try. That
> > would include a first balance start -dusage=0 -musage=0, and if that
> > didn't free up at least a gig on a second device,
>
> If I'm following this correctly, the reproduce steps would be to
> create a single device Btrfs that's ~94% full, add two devices, then
> convert to raid5. I think what's going on here is empty single profile
> data chunks aren't being deallocated, and it's effectively a 2 device
> raid5.
This isn't supported by the btrfs fi df output: all of the
allocated space is used.
Hugo.
> So actually, you're right, either -dusage=0 might fix it, or better,
> newer kernel that automatically deallocated empty/converted single
> profile data chunks. But right now it will take another balance in the
> end because it looks like this is effectively a 2 device raid5, with
> the 3rd drive full of single only chunks (which might be empty?).
--
Hugo Mills | I'll take your bet, but make it ten thousand francs.
hugo@... carfax.org.uk | I'm only a _poor_ corrupt official.
http://carfax.org.uk/ |
PGP: E2AB1DE4 | Capt. Renaud, Casablanca
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-01 20:44 ` Chris Murphy
2015-08-01 21:45 ` Duncan
@ 2015-08-02 12:54 ` Hendrik Friedel
2015-08-06 18:57 ` Hendrik Friedel
2 siblings, 0 replies; 18+ messages in thread
From: Hendrik Friedel @ 2015-08-02 12:54 UTC (permalink / raw)
To: Chris Murphy, Hugo Mills, Btrfs BTRFS
Hello,
>> Looking at the btrfs fi show output, you've probably run out of
>> space during the conversion, probably due to an uneven distribution of
>> the original "single" chunks.
>>
>> I think I would suggest balancing the single chunks, and trying the
>> conversion (of the unconverted parts) again:
>>
>> # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/
>> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/
>>
>
> Yep I bet that's it also. btrfs fi usage might be better at exposing this case.
Thanks for your hints.
The balance is running now for about 11h:
The status is a bit surprising to me:
0 out of about 2619 chunks balanced (4165 considered), 100% left
btrfs fi usage is also surprising:
Overall:
Device size: 8.19TiB
Device allocated: 2.56TiB
Device unallocated: 5.62TiB
Device missing: 0.00B
Used: 2.56TiB
Free (estimated): 11.65TiB (min: 2.81TiB)
Data ratio: 0.48
Metadata ratio: 1.33
Global reserve: 512.00MiB (used: 0.00B)
Data,single: Size:2.55TiB, Used:2.55TiB
/dev/sdc 1.60TiB
/dev/sde 975.44GiB
Data,RAID5: Size:2.73TiB, Used:2.72TiB
/dev/sdc 1.12TiB
/dev/sdd 2.73TiB
/dev/sde 1.61TiB
Metadata,RAID1: Size:6.00GiB, Used:5.33GiB
/dev/sdc 5.00GiB
/dev/sdd 1.00GiB
/dev/sde 6.00GiB
Metadata,RAID5: Size:3.00GiB, Used:2.99GiB
/dev/sdc 2.00GiB
/dev/sdd 1.00GiB
/dev/sde 3.00GiB
System,RAID5: Size:32.00MiB, Used:736.00KiB
/dev/sdd 32.00MiB
/dev/sde 32.00MiB
Unallocated:
/dev/sdc 1.02MiB
/dev/sdd 1.02MiB
/dev/sde 164.59GiB
I hope, that is because Raid5 is not implemented yet?
Regards,
Hendrik
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: Data single *and* raid?
2015-08-01 20:44 ` Chris Murphy
2015-08-01 21:45 ` Duncan
2015-08-02 12:54 ` Hendrik Friedel
@ 2015-08-06 18:57 ` Hendrik Friedel
2015-08-07 1:26 ` Qu Wenruo
2 siblings, 1 reply; 18+ messages in thread
From: Hendrik Friedel @ 2015-08-06 18:57 UTC (permalink / raw)
To: Chris Murphy, Hugo Mills, Btrfs BTRFS
Hello Hugo,
hello Chris,
thanks for your advice. Now I am here:
btrfs balance start -dprofiles=single -mprofiles=raid1 /mnt/__Complete_Disk/
Done, had to relocate 0 out of 3939 chunks
root@homeserver:/mnt/__Complete_Disk# btrfs fi show
Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
Total devices 3 FS bytes used 3.78TiB
devid 1 size 2.73TiB used 2.72TiB path /dev/sde
devid 2 size 2.73TiB used 2.23TiB path /dev/sdc
devid 3 size 2.73TiB used 2.73TiB path /dev/sdd
btrfs-progs v4.1.1
So, that looks good.
But then:
root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/
Data, RAID5: total=3.83TiB, used=3.78TiB
System, RAID5: total=32.00MiB, used=576.00KiB
Metadata, RAID5: total=6.46GiB, used=4.84GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
Is the RAID5 expected here?
I did not yet run:
btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
/mnt/new_storage/
Regards,
Hendrik
On 01.08.2015 22:44, Chris Murphy wrote:
> On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>> On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
>>> Hello,
>>>
>>> I converted an array to raid5 by
>>> btrfs device add /dev/sdd /mnt/new_storage
>>> btrfs device add /dev/sdc /mnt/new_storage
>>> btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
>>>
>>> The Balance went through. But now:
>>> Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>>> Total devices 3 FS bytes used 5.28TiB
>>> devid 1 size 2.73TiB used 2.57TiB path /dev/sde
>>> devid 2 size 2.73TiB used 2.73TiB path /dev/sdc
>>> devid 3 size 2.73TiB used 2.73TiB path /dev/sdd
>>> btrfs-progs v4.1.1
>>>
>>> Already the 2.57TiB is a bit surprising:
>>> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
>>> Data, single: total=2.55TiB, used=2.55TiB
>>> Data, RAID5: total=2.73TiB, used=2.72TiB
>>> System, RAID5: total=32.00MiB, used=736.00KiB
>>> Metadata, RAID1: total=6.00GiB, used=5.33GiB
>>> Metadata, RAID5: total=3.00GiB, used=2.99GiB
>>
>> Looking at the btrfs fi show output, you've probably run out of
>> space during the conversion, probably due to an uneven distribution of
>> the original "single" chunks.
>>
>> I think I would suggest balancing the single chunks, and trying the
>> conversion (of the unconverted parts) again:
>>
>> # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/
>> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/
>>
>
> Yep I bet that's it also. btrfs fi usage might be better at exposing this case.
>
>
--
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Tel. 04203 8394854
Mobil 0178 1874363
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: Data single *and* raid?
2015-08-06 18:57 ` Hendrik Friedel
@ 2015-08-07 1:26 ` Qu Wenruo
2015-08-07 5:16 ` Hendrik Friedel
0 siblings, 1 reply; 18+ messages in thread
From: Qu Wenruo @ 2015-08-07 1:26 UTC (permalink / raw)
To: Hendrik Friedel, Chris Murphy, Hugo Mills, Btrfs BTRFS
Hendrik Friedel wrote on 2015/08/06 20:57 +0200:
> Hello Hugo,
> hello Chris,
>
> thanks for your advice. Now I am here:
> btrfs balance start -dprofiles=single -mprofiles=raid1
> /mnt/__Complete_Disk/
> Done, had to relocate 0 out of 3939 chunks
>
>
> root@homeserver:/mnt/__Complete_Disk# btrfs fi show
> Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
> Total devices 3 FS bytes used 3.78TiB
> devid 1 size 2.73TiB used 2.72TiB path /dev/sde
> devid 2 size 2.73TiB used 2.23TiB path /dev/sdc
> devid 3 size 2.73TiB used 2.73TiB path /dev/sdd
>
> btrfs-progs v4.1.1
>
>
> So, that looks good.
>
> But then:
> root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/
> Data, RAID5: total=3.83TiB, used=3.78TiB
> System, RAID5: total=32.00MiB, used=576.00KiB
> Metadata, RAID5: total=6.46GiB, used=4.84GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
GlobalReserve is not a chunk type, it just means a range of metadata
reserved for overcommiting.
And it's always single.
Personally, I don't think it should be output in "fi df" command, as
it's in a higher level than chunk.
At least for your case, nothing is needed to worry about.
Thanks,
Qu
>
> Is the RAID5 expected here?
> I did not yet run:
> btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
> /mnt/new_storage/
>
> Regards,
> Hendrik
>
>
> On 01.08.2015 22:44, Chris Murphy wrote:
>> On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>>> On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
>>>> Hello,
>>>>
>>>> I converted an array to raid5 by
>>>> btrfs device add /dev/sdd /mnt/new_storage
>>>> btrfs device add /dev/sdc /mnt/new_storage
>>>> btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
>>>>
>>>> The Balance went through. But now:
>>>> Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>>>> Total devices 3 FS bytes used 5.28TiB
>>>> devid 1 size 2.73TiB used 2.57TiB path /dev/sde
>>>> devid 2 size 2.73TiB used 2.73TiB path /dev/sdc
>>>> devid 3 size 2.73TiB used 2.73TiB path /dev/sdd
>>>> btrfs-progs v4.1.1
>>>>
>>>> Already the 2.57TiB is a bit surprising:
>>>> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
>>>> Data, single: total=2.55TiB, used=2.55TiB
>>>> Data, RAID5: total=2.73TiB, used=2.72TiB
>>>> System, RAID5: total=32.00MiB, used=736.00KiB
>>>> Metadata, RAID1: total=6.00GiB, used=5.33GiB
>>>> Metadata, RAID5: total=3.00GiB, used=2.99GiB
>>>
>>> Looking at the btrfs fi show output, you've probably run out of
>>> space during the conversion, probably due to an uneven distribution of
>>> the original "single" chunks.
>>>
>>> I think I would suggest balancing the single chunks, and trying the
>>> conversion (of the unconverted parts) again:
>>>
>>> # btrfs balance start -dprofiles=single -mprofile=raid1
>>> /mnt/new_storage/
>>> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
>>> /mnt/new_storage/
>>>
>>
>> Yep I bet that's it also. btrfs fi usage might be better at exposing
>> this case.
>>
>>
>
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-07 1:26 ` Qu Wenruo
@ 2015-08-07 5:16 ` Hendrik Friedel
2015-08-07 6:25 ` Duncan
2015-08-07 8:11 ` Hugo Mills
0 siblings, 2 replies; 18+ messages in thread
From: Hendrik Friedel @ 2015-08-07 5:16 UTC (permalink / raw)
To: Qu Wenruo, Chris Murphy, Hugo Mills, Btrfs BTRFS
Hello Quo,
thanks for your reply.
>> But then:
>> root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/
>> Data, RAID5: total=3.83TiB, used=3.78TiB
>> System, RAID5: total=32.00MiB, used=576.00KiB
>> Metadata, RAID5: total=6.46GiB, used=4.84GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
> GlobalReserve is not a chunk type, it just means a range of metadata
> reserved for overcommiting.
> And it's always single.
>
> Personally, I don't think it should be output in "fi df" command, as
> it's in a higher level than chunk.
>
> At least for your case, nothing is needed to worry about.
But this seems to be a RAID5 now, right?
Well, that's what I want, but the command was:
btrfs balance start -dprofiles=single -mprofiles=raid1
/mnt/__Complete_Disk/
So, we would expect raid1 here, no?
Greetings,
Hendrik
>>
>>
>> On 01.08.2015 22:44, Chris Murphy wrote:
>>> On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>>>> On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
>>>>> Hello,
>>>>>
>>>>> I converted an array to raid5 by
>>>>> btrfs device add /dev/sdd /mnt/new_storage
>>>>> btrfs device add /dev/sdc /mnt/new_storage
>>>>> btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
>>>>>
>>>>> The Balance went through. But now:
>>>>> Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
>>>>> Total devices 3 FS bytes used 5.28TiB
>>>>> devid 1 size 2.73TiB used 2.57TiB path /dev/sde
>>>>> devid 2 size 2.73TiB used 2.73TiB path /dev/sdc
>>>>> devid 3 size 2.73TiB used 2.73TiB path /dev/sdd
>>>>> btrfs-progs v4.1.1
>>>>>
>>>>> Already the 2.57TiB is a bit surprising:
>>>>> root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
>>>>> Data, single: total=2.55TiB, used=2.55TiB
>>>>> Data, RAID5: total=2.73TiB, used=2.72TiB
>>>>> System, RAID5: total=32.00MiB, used=736.00KiB
>>>>> Metadata, RAID1: total=6.00GiB, used=5.33GiB
>>>>> Metadata, RAID5: total=3.00GiB, used=2.99GiB
>>>>
>>>> Looking at the btrfs fi show output, you've probably run out of
>>>> space during the conversion, probably due to an uneven distribution of
>>>> the original "single" chunks.
>>>>
>>>> I think I would suggest balancing the single chunks, and trying the
>>>> conversion (of the unconverted parts) again:
>>>>
>>>> # btrfs balance start -dprofiles=single -mprofile=raid1
>>>> /mnt/new_storage/
>>>> # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
>>>> /mnt/new_storage/
>>>>
>>>
>>> Yep I bet that's it also. btrfs fi usage might be better at exposing
>>> this case.
>>>
>>>
>>
>>
--
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Tel. 04203 8394854
Mobil 0178 1874363
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-07 5:16 ` Hendrik Friedel
@ 2015-08-07 6:25 ` Duncan
2015-08-07 8:11 ` Hugo Mills
1 sibling, 0 replies; 18+ messages in thread
From: Duncan @ 2015-08-07 6:25 UTC (permalink / raw)
To: linux-btrfs
Hendrik Friedel posted on Fri, 07 Aug 2015 07:16:04 +0200 as excerpted:
>>> But then:
>>> # btrfs fi df /mnt/__Complete_Disk/
>>> Data, RAID5: total=3.83TiB, used=3.78TiB
>>> System, RAID5: total=32.00MiB, used=576.00KiB
>>> Metadata, RAID5: total=6.46GiB, used=4.84GiB
>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> [T]his seems to be a RAID5 now, right?
> Well, that's what I want, but the command was:
> btrfs balance start -dprofiles=single -mprofiles=raid1
> /mnt/__Complete_Disk/
>
> So, we would expect raid1 here, no?
No. The behavior might be a bit counterintuitive on first glance, but
once the logic is understood, it makes sense.
1) You had tried the initial raid5 convert using an earlier kernel that
had incomplete raid5 support, as evidenced by the lack of the global-
reserve line in btrfs fi df, on a new enough userspace that it should
have had it.
2) That initial attempt ran out of space, possibly because it was keeping
the single and raid1 chunks around due to fragmentation (Hugo's guess),
or due to a now fixed raid5 conversion bugs in the old kernel[1] (my
guess), or possibly due to some other bug that's apparently fixed in
newer kernels, thus the successful completion of the conversion below.
3) But that initial attempt still did one critical thing -- set the
default new-chunk type to raid5, for both data and metadata.
4) So when the second btrfs balance attempt came along, this one
primarily intended to clean up that fragmentation that Hugo expected, and
thus targeted at those old single data and raid1 metadata chunks, when it
rewrote those chunks it used the new chunk default, rewriting them into
raid5.
That was a result that Hugo obviously didn't predict as his instructions
suggested following up with another balance command to complete the
conversion. And neither Chris (apparently) nor I (definitely!) foresaw
it either. But the behavior does make sense, once you take into account
the default chunk type, and that a balance-convert does normally change
it.
And FWIW, the precise behavior of this default chunk type selector and
when it falls back to single data and raid1 or dup metadata (as it will
in some instances with a degraded filesystem), has both been problematic
before, and is being debated in a current thread, due to the implications
for writable mounts of degraded single-device raid1s, for instance. It's
behavior in corner-cases like these that is much of the reason most
regulars on this list don't consider btrfs fully stable and mature, just
yet, because sometimes that corner-case behavior can mean the filesystem
doing the wrong thing, going read-only, without any way to correct the
problem even tho things are generally still fine, because correcting the
problem would require a writable filesystem, thus creating a chicken and
egg situation where correcting the problem requires a writable
filesystem, but a writable filesystem isn't allowed until the problem is
corrected, for instance. (As of now, in that situation a user has little
choice but to copy the data on that read-only filesystem elsewhere, do a
mkfs to wipe away the problem, and restore to the fresh filesystem.
Technically, that shouldn't be required.)
---
[1] FWIW, for "online" tasks like btrfs balance, the btrfs-progs
userspace simply issues the commands to the kernel, which does the real
work. For "offline" tasks such as btrfs check or btrfs restore,
userspace is the real brains and the kernel simply relays the commands at
the device level, without much involvement by the kernel's btrfs code at
all. So while you had a current userspace, the old kernel was the
critical part since btrfs balance is an online command in which it's the
kernel's btrfs code that does the real work.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Data single *and* raid?
2015-08-07 5:16 ` Hendrik Friedel
2015-08-07 6:25 ` Duncan
@ 2015-08-07 8:11 ` Hugo Mills
1 sibling, 0 replies; 18+ messages in thread
From: Hugo Mills @ 2015-08-07 8:11 UTC (permalink / raw)
To: Hendrik Friedel; +Cc: Qu Wenruo, Chris Murphy, Btrfs BTRFS
[-- Attachment #1: Type: text/plain, Size: 3173 bytes --]
On Fri, Aug 07, 2015 at 07:16:04AM +0200, Hendrik Friedel wrote:
> Hello Quo,
>
> thanks for your reply.
>
> >> But then:
> >>root@homeserver:/mnt/__Complete_Disk# btrfs fi df /mnt/__Complete_Disk/
> >>Data, RAID5: total=3.83TiB, used=3.78TiB
> >>System, RAID5: total=32.00MiB, used=576.00KiB
> >>Metadata, RAID5: total=6.46GiB, used=4.84GiB
> >>GlobalReserve, single: total=512.00MiB, used=0.00B
> >GlobalReserve is not a chunk type, it just means a range of metadata
> >reserved for overcommiting.
> >And it's always single.
> >
> >Personally, I don't think it should be output in "fi df" command, as
> >it's in a higher level than chunk.
> >
> >At least for your case, nothing is needed to worry about.
>
>
> But this seems to be a RAID5 now, right?
> Well, that's what I want, but the command was:
> btrfs balance start -dprofiles=single -mprofiles=raid1
> /mnt/__Complete_Disk/
>
> So, we would expect raid1 here, no?
No, profiles= selects chunks of that profile. If you wanted to
change the RAID level, you should use convert=.
Hugo.
> Greetings,
> Hendrik
>
>
> >>
> >>
> >>On 01.08.2015 22:44, Chris Murphy wrote:
> >>>On Sat, Aug 1, 2015 at 2:32 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
> >>>>On Sat, Aug 01, 2015 at 10:09:35PM +0200, Hendrik Friedel wrote:
> >>>>>Hello,
> >>>>>
> >>>>>I converted an array to raid5 by
> >>>>>btrfs device add /dev/sdd /mnt/new_storage
> >>>>>btrfs device add /dev/sdc /mnt/new_storage
> >>>>>btrfs balance start -dconvert=raid5 -mconvert=raid5 /mnt/new_storage/
> >>>>>
> >>>>>The Balance went through. But now:
> >>>>>Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
> >>>>> Total devices 3 FS bytes used 5.28TiB
> >>>>> devid 1 size 2.73TiB used 2.57TiB path /dev/sde
> >>>>> devid 2 size 2.73TiB used 2.73TiB path /dev/sdc
> >>>>> devid 3 size 2.73TiB used 2.73TiB path /dev/sdd
> >>>>>btrfs-progs v4.1.1
> >>>>>
> >>>>>Already the 2.57TiB is a bit surprising:
> >>>>>root@homeserver:/mnt# btrfs fi df /mnt/new_storage/
> >>>>>Data, single: total=2.55TiB, used=2.55TiB
> >>>>>Data, RAID5: total=2.73TiB, used=2.72TiB
> >>>>>System, RAID5: total=32.00MiB, used=736.00KiB
> >>>>>Metadata, RAID1: total=6.00GiB, used=5.33GiB
> >>>>>Metadata, RAID5: total=3.00GiB, used=2.99GiB
> >>>>
> >>>> Looking at the btrfs fi show output, you've probably run out of
> >>>>space during the conversion, probably due to an uneven distribution of
> >>>>the original "single" chunks.
> >>>>
> >>>> I think I would suggest balancing the single chunks, and trying the
> >>>>conversion (of the unconverted parts) again:
> >>>>
> >>>># btrfs balance start -dprofiles=single -mprofile=raid1
> >>>>/mnt/new_storage/
> >>>># btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft
> >>>>/mnt/new_storage/
> >>>>
> >>>
> >>>Yep I bet that's it also. btrfs fi usage might be better at exposing
> >>>this case.
> >>>
> >>>
> >>
> >>
>
>
--
Hugo Mills | There is no dark side to the Moon, really. As a
hugo@... carfax.org.uk | matter of fact, it's all dark.
http://carfax.org.uk/ |
PGP: E2AB1DE4 | Pink Floyd
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 18+ messages in thread