All of lore.kernel.org
 help / color / mirror / Atom feed
* btrfs check --repair now runs in minutes instead of hours? aborting
@ 2017-09-05  1:05 Marc MERLIN
  2017-09-05  1:21 ` Qu Wenruo
  0 siblings, 1 reply; 12+ messages in thread
From: Marc MERLIN @ 2017-09-05  1:05 UTC (permalink / raw)
  To: quwenruo, Lu Fengqi, Chris Mason; +Cc: Btrfs BTRFS, David Sterba

Ok, I don't want to sound like I'm complaining :) but I updated
btrfs-progs to top of tree in git, installed it, and ran it on an 8TiB
filesystem that used to take 12H or so to check.

It finished in maybe 10mn, just 10mn! :)
gargamel:/var/local/src/btrfs-progs# btrfs check --repair /dev/mapper/dshelf1
enabling repair mode
Checking filesystem on /dev/mapper/dshelf1
UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 11674263347200 bytes used, no error found
total csum bytes: 11384482936
total tree bytes: 13704495104
total fs tree bytes: 724729856
total extent tree bytes: 482639872
btree space waste bytes: 1167025205
file data blocks allocated: 12041456693248
 referenced 12063146434560

This is great news, but can I trust that the program worked properly and
indeed that my filesystem is fully clean?
Or at this point if I'm not running --mode=lowmem, the regular mode is
really doesn't check much and only lowmem can do a proper check? (even
though it can't fix problems once it finds them)

Thanks
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-05  1:05 btrfs check --repair now runs in minutes instead of hours? aborting Marc MERLIN
@ 2017-09-05  1:21 ` Qu Wenruo
  2017-09-05  2:55   ` Marc MERLIN
  0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2017-09-05  1:21 UTC (permalink / raw)
  To: Marc MERLIN, quwenruo, Lu Fengqi, Chris Mason; +Cc: Btrfs BTRFS, David Sterba



On 2017年09月05日 09:05, Marc MERLIN wrote:
> Ok, I don't want to sound like I'm complaining :) but I updated
> btrfs-progs to top of tree in git, installed it, and ran it on an 8TiB
> filesystem that used to take 12H or so to check.

How much space allocated for that 8T fs?
If metadata is not that large, 10min is valid.

Here fi df output could help.

And, without --repair, how much time it takes to run?

> 
> It finished in maybe 10mn, just 10mn! :)
> gargamel:/var/local/src/btrfs-progs# btrfs check --repair /dev/mapper/dshelf1
> enabling repair mode
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
> checking extents
> Fixed 0 roots.
> checking free space cache
> cache and super generation don't match, space cache will be invalidated
> checking fs roots
> checking csums
> checking root refs
> found 11674263347200 bytes used, no error found
> total csum bytes: 11384482936
> total tree bytes: 13704495104
> total fs tree bytes: 724729856
> total extent tree bytes: 482639872
> btree space waste bytes: 1167025205
> file data blocks allocated: 12041456693248
>   referenced 12063146434560
> 
> This is great news, but can I trust that the program worked properly and
> indeed that my filesystem is fully clean?

Normally we rely on btrfs-progs selftest to make sure fsck works.
But maybe we could cross check original mode and lowmem mode.

> Or at this point if I'm not running --mode=lowmem, the regular mode is
> really doesn't check much and only lowmem can do a proper check? (even
> though it can't fix problems once it finds them)

No, original mode is still the core (although code is not that elegant), 
please consider lowmem mode as a corner case for guys with small memory 
(without swap) but want to check super large fs.

So if it turns out to be a bug, we must fix it.

Thanks,
Qu
> 
> Thanks
> Marc
> 

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-05  1:21 ` Qu Wenruo
@ 2017-09-05  2:55   ` Marc MERLIN
  2017-09-05  4:06     ` Marc MERLIN
  2017-09-05  8:05     ` Qu Wenruo
  0 siblings, 2 replies; 12+ messages in thread
From: Marc MERLIN @ 2017-09-05  2:55 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: quwenruo, Lu Fengqi, Chris Mason, Btrfs BTRFS, David Sterba

On Tue, Sep 05, 2017 at 09:21:55AM +0800, Qu Wenruo wrote:
> 
> 
> On 2017年09月05日 09:05, Marc MERLIN wrote:
> >Ok, I don't want to sound like I'm complaining :) but I updated
> >btrfs-progs to top of tree in git, installed it, and ran it on an 8TiB
> >filesystem that used to take 12H or so to check.
> 
> How much space allocated for that 8T fs?
> If metadata is not that large, 10min is valid.
> 
> Here fi df output could help.

gargamel:~# btrfs fi df /mnt/btrfs_pool1
Data, single: total=10.60TiB, used=10.54TiB
System, DUP: total=32.00MiB, used=1.19MiB
Metadata, DUP: total=58.00GiB, used=12.69GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

> And, without --repair, how much time it takes to run?

Well, funny that you ask, it's now been running for hours, still waiting...
 
Just before, I ran lowmem, and it was pretty quick too (didn't time it,
but less than 1h):
gargamel:/var/local/src/btrfs-progs# btrfs check --mode=lowmem
/dev/mapper/dshelf1
Checking filesystem on /dev/mapper/dshelf1
UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 11674263330816 bytes used, no error found
total csum bytes: 11384482936
total tree bytes: 13738737664
total fs tree bytes: 758988800
total extent tree bytes: 482623488
btree space waste bytes: 1171475737
file data blocks allocated: 12888981110784
 referenced 12930453286912

Now, this is good news for my filesystem being probably clean (previous
versions of lowmem before my git update found issues that were unclear, but
apparently errors in the code, and this version finds nothing)

But I'm not sure why --repair would be fast, and not --repair would be slow?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-05  2:55   ` Marc MERLIN
@ 2017-09-05  4:06     ` Marc MERLIN
  2017-09-05  8:05     ` Qu Wenruo
  1 sibling, 0 replies; 12+ messages in thread
From: Marc MERLIN @ 2017-09-05  4:06 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: quwenruo, Lu Fengqi, Chris Mason, Btrfs BTRFS, David Sterba

Ok, not quite hours, but check takes 88mn, check --repair takes 11mn

gargamel:/var/local/src/btrfs-progs# time btrfs check /dev/mapper/dshelf1
Checking filesystem on /dev/mapper/dshelf1
UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 11674263330816 bytes used, no error found
total csum bytes: 11384482936
total tree bytes: 13704478720
total fs tree bytes: 724729856
total extent tree bytes: 482623488
btree space waste bytes: 1167009013
file data blocks allocated: 12041456693248
referenced 12063146434560

real    88m56.597s
user    2m13.985s
sys     2m7.880s

gargamel:/var/local/src/btrfs-progs# time btrfs check --repair
/dev/mapper/dshelf1
enabling repair mode
Checking filesystem on /dev/mapper/dshelf1
UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 11674263330816 bytes used, no error found
total csum bytes: 11384482936
total tree bytes: 13704478720
total fs tree bytes: 724729856
total extent tree bytes: 482623488
btree space waste bytes: 1167009013
file data blocks allocated: 12041456693248
referenced 12063146434560

real    11m10.499s
user    1m55.067s
sys     1m31.666s

And lowmem is 24mn:
gargamel:/var/local/src/btrfs-progs# time btrfs check --mode=lowmem
/dev/mapper/dshelf1
Checking filesystem on /dev/mapper/dshelf1
UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 11674263363584 bytes used, no error found
total csum bytes: 11384482936
total tree bytes: 13738770432
total fs tree bytes: 758988800
total extent tree bytes: 482656256
btree space waste bytes: 1171508121
file data blocks allocated: 12888981110784
referenced 12930453286912

real    24m20.493s
user    5m45.749s
sys     1m40.204s


Does this make any sense at all that check without --repair is so much
slower than with --repair or lowmem?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-05  2:55   ` Marc MERLIN
  2017-09-05  4:06     ` Marc MERLIN
@ 2017-09-05  8:05     ` Qu Wenruo
  2017-09-05  8:54       ` Duncan
  2017-09-05 14:45       ` Marc MERLIN
  1 sibling, 2 replies; 12+ messages in thread
From: Qu Wenruo @ 2017-09-05  8:05 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: quwenruo, Lu Fengqi, Chris Mason, Btrfs BTRFS, David Sterba



On 2017年09月05日 10:55, Marc MERLIN wrote:
> On Tue, Sep 05, 2017 at 09:21:55AM +0800, Qu Wenruo wrote:
>>
>>
>> On 2017年09月05日 09:05, Marc MERLIN wrote:
>>> Ok, I don't want to sound like I'm complaining :) but I updated
>>> btrfs-progs to top of tree in git, installed it, and ran it on an 8TiB
>>> filesystem that used to take 12H or so to check.
>>
>> How much space allocated for that 8T fs?
>> If metadata is not that large, 10min is valid.
>>
>> Here fi df output could help.
> 
> gargamel:~# btrfs fi df /mnt/btrfs_pool1
> Data, single: total\x10.60TiB, used\x10.54TiB
> System, DUP: total2.00MiB, used=1.19MiB
> Metadata, DUP: totalX.00GiB, used\x12.69GiB

Wait for a minute.

Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the 
format (again)?
This output format must be changed, at least to 0.69 GiB, or 706 MiB.

I'll fix this first.

> GlobalReserve, single: totalQ2.00MiB, used=0.00B
> 
>> And, without --repair, how much time it takes to run?
> 
> Well, funny that you ask, it's now been running for hours, still waiting...
>   
> Just before, I ran lowmem, and it was pretty quick too (didn't time it,
> but less than 1h):

You mean lowmem is actually FASTER than original mode?
That's very surprising.

Is there any special operation done for that btrfs?
Like offline dedupe or tons of reflinks?

IIRC original mode did a quite slow check for tons of reflink, which may 
be related.

BTW, how many subvolumes do you have in the fs?

> gargamel:/var/local/src/btrfs-progs# btrfs check --mode=wmem
> /dev/mapper/dshelf1
> Checking filesystem on /dev/mapper/dshelf1
> UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
> checking extents
> checking free space cache
> cache and super generation don't match, space cache will be invalidated
> checking fs roots
> checking csums
> checking root refs
> found 11674263330816 bytes used, no error found
> total csum bytes: 11384482936
> total tree bytes: 13738737664
> total fs tree bytes: 758988800
> total extent tree bytes: 482623488
> btree space waste bytes: 1171475737
> file data blocks allocated: 12888981110784
>   referenced 12930453286912
> 
> Now, this is good news for my filesystem being probably clean (previous
> versions of lowmem before my git update found issues that were unclear, but
> apparently errors in the code, and this version finds nothing)
> 
> But I'm not sure why --repair would be fast, and not --repair would be slow?

This looks like a bug. My first guess is related to number of 
subvolumes/reflinks, but I'm not sure since I don't have many real-world 
btrfs.

I'll take sometime to look into it.

Thanks for the very interesting report,
Qu

> 
> Marc
> 

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-05  8:05     ` Qu Wenruo
@ 2017-09-05  8:54       ` Duncan
  2017-09-05  9:06         ` Qu Wenruo
  2017-09-05 14:45       ` Marc MERLIN
  1 sibling, 1 reply; 12+ messages in thread
From: Duncan @ 2017-09-05  8:54 UTC (permalink / raw)
  To: linux-btrfs

Qu Wenruo posted on Tue, 05 Sep 2017 16:05:04 +0800 as excerpted:

> On 2017年09月05日 10:55, Marc MERLIN wrote:
>> 
>> gargamel:~# btrfs fi df /mnt/btrfs_pool1
>> Data, single: total\x10.60TiB, used\x10.54TiB
>> System, DUP: total2.00MiB, used=1.19MiB
>> Metadata, DUP: totalX.00GiB, used\x12.69GiB
> 
> Wait for a minute.
> 
> Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the
> format (again)?

It appears to be your end.  Based on the fact that I'm seeing a a bunch 
of weird characters in your quote of the message that I didn't see in the 
original, I'm guessing it's charset related, very possibly due to the 
"equal" sign being an escape character for mime/quoted-printable (tho his 
post was text/plain; charset equals utf-8, full 8-bit, so not quoted-
printable encoded at all) and I believe various i18n escapes as well, 
with the latter being an issue if the client assumes local charset 
despite the utf8 specified in the header.

See if these numbers, copied and reformatted from his post with spaces 
inserted either side of the numbers and the equals signs deleted, arrive 
any less garbled:

Data, single: total 10.60 TiB, used 10.54 TiB
System, DUP: total 32.00 MiB, used 1.19 MiB
Metadata, DUP: total 58.00 GiB, used 12.69 GiB
GlobalReserve, single: total 512.00 MiB, used 0.00 B

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-05  8:54       ` Duncan
@ 2017-09-05  9:06         ` Qu Wenruo
  2017-09-05  9:35           ` Duncan
  0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2017-09-05  9:06 UTC (permalink / raw)
  To: Duncan, linux-btrfs



On 2017年09月05日 16:54, Duncan wrote:
> Qu Wenruo posted on Tue, 05 Sep 2017 16:05:04 +0800 as excerpted:
> 
>> On 2017年09月05日 10:55, Marc MERLIN wrote:
>>>
>>> gargamel:~# btrfs fi df /mnt/btrfs_pool1
>>> Data, single: total\x10.60TiB, used\x10.54TiB
>>> System, DUP: total2.00MiB, used=1.19MiB
>>> Metadata, DUP: totalX.00GiB, used\x12.69GiB
>>
>> Wait for a minute.
>>
>> Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the
>> format (again)?
> 
> It appears to be your end.  Based on the fact that I'm seeing a a bunch
> of weird characters in your quote of the message that I didn't see in the
> original, I'm guessing it's charset related, very possibly due to the
> "equal" sign being an escape character for mime/quoted-printable (tho his
> post was text/plain; charset equals utf-8, full 8-bit, so not quoted-
> printable encoded at all) and I believe various i18n escapes as well,
> with the latter being an issue if the client assumes local charset
> despite the utf8 specified in the header.
> 
> See if these numbers, copied and reformatted from his post with spaces
> inserted either side of the numbers and the equals signs deleted, arrive
> any less garbled:
> 
> Data, single: total 10.60 TiB, used 10.54 TiB
> System, DUP: total 32.00 MiB, used 1.19 MiB
> Metadata, DUP: total 58.00 GiB, used 12.69 GiB
> GlobalReserve, single: total 512.00 MiB, used 0.00 B
> 
Thanks a lot for this.

I'd better double check my client setup to avoid such embrassment.

Thanks,
Qu

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-05  9:06         ` Qu Wenruo
@ 2017-09-05  9:35           ` Duncan
  0 siblings, 0 replies; 12+ messages in thread
From: Duncan @ 2017-09-05  9:35 UTC (permalink / raw)
  To: linux-btrfs

Qu Wenruo posted on Tue, 05 Sep 2017 17:06:35 +0800 as excerpted:

>> See if these numbers, copied and reformatted from his post with spaces
>> inserted either side of the numbers and the equals signs deleted,
>> arrive any less garbled:
>> 
>> Data, single: total 10.60 TiB, used 10.54 TiB System, DUP: total 32.00
>> MiB, used 1.19 MiB Metadata, DUP: total 58.00 GiB, used 12.69 GiB
>> GlobalReserve, single: total 512.00 MiB, used 0.00 B
>> 
> Thanks a lot for this.

It worked. =:^)  (But thinking about it now, that smiley with an equals 
sign probably won't!)


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-05  8:05     ` Qu Wenruo
  2017-09-05  8:54       ` Duncan
@ 2017-09-05 14:45       ` Marc MERLIN
  2017-09-09 17:44         ` Marc MERLIN
  1 sibling, 1 reply; 12+ messages in thread
From: Marc MERLIN @ 2017-09-05 14:45 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: quwenruo, Lu Fengqi, Chris Mason, Btrfs BTRFS, David Sterba

On Tue, Sep 05, 2017 at 04:05:04PM +0800, Qu Wenruo wrote:
> > gargamel:~# btrfs fi df /mnt/btrfs_pool1
> > Data, single: total\x10.60TiB, used\x10.54TiB
> > System, DUP: total2.00MiB, used=1.19MiB
> > Metadata, DUP: totalX.00GiB, used\x12.69GiB
> 
> Wait for a minute.
> 
> Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the format
> (again)?
> This output format must be changed, at least to 0.69 GiB, or 706 MiB.
 
Email client problem. I see control characters in what you quoted.

Let's try again
gargamel:~# btrfs fi df /mnt/btrfs_pool1
Data, single: total=10.66TiB, used=10.60TiB      => 10TB
System, DUP: total=64.00MiB, used=1.20MiB        => 1.2MB
Metadata, DUP: total=57.50GiB, used=12.76GiB     => 13GB
GlobalReserve, single: total=512.00MiB, used=0.00B  => 0

> You mean lowmem is actually FASTER than original mode?
> That's very surprising.
 
Correct, unless I add --repair and then original mode is 2x faster than
lowmem.

> Is there any special operation done for that btrfs?
> Like offline dedupe or tons of reflinks?

In this case, no.
Note that btrfs check used to take many hours overnight until I did a
git pull of btrfs progs and built the latest from TOT.

> BTW, how many subvolumes do you have in the fs?
 
gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | wc -l
91

If I remove snapshots for btrfs send and historical 'backups':
gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | grep -Ev '(hourly|daily|weekly|rw|ro)' | wc -l
5

> This looks like a bug. My first guess is related to number of
> subvolumes/reflinks, but I'm not sure since I don't have many real-world
> btrfs.
> 
> I'll take sometime to look into it.
> 
> Thanks for the very interesting report,

Thanks for having a look :)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-05 14:45       ` Marc MERLIN
@ 2017-09-09 17:44         ` Marc MERLIN
  2017-09-10  6:01           ` Qu Wenruo
  0 siblings, 1 reply; 12+ messages in thread
From: Marc MERLIN @ 2017-09-09 17:44 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: quwenruo, Lu Fengqi, Chris Mason, Btrfs BTRFS, David Sterba

So, should I assume that btrfs progs git has some issue since there is
no plausible way that a check --repair should be faster than a regular
check?

Thanks,
Marc

On Tue, Sep 05, 2017 at 07:45:25AM -0700, Marc MERLIN wrote:
> On Tue, Sep 05, 2017 at 04:05:04PM +0800, Qu Wenruo wrote:
> > > gargamel:~# btrfs fi df /mnt/btrfs_pool1
> > > Data, single: total\x10.60TiB, used\x10.54TiB
> > > System, DUP: total2.00MiB, used=1.19MiB
> > > Metadata, DUP: totalX.00GiB, used\x12.69GiB
> > 
> > Wait for a minute.
> > 
> > Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the format
> > (again)?
> > This output format must be changed, at least to 0.69 GiB, or 706 MiB.
>  
> Email client problem. I see control characters in what you quoted.
> 
> Let's try again
> gargamel:~# btrfs fi df /mnt/btrfs_pool1
> Data, single: total=10.66TiB, used=10.60TiB      => 10TB
> System, DUP: total=64.00MiB, used=1.20MiB        => 1.2MB
> Metadata, DUP: total=57.50GiB, used=12.76GiB     => 13GB
> GlobalReserve, single: total=512.00MiB, used=0.00B  => 0
> 
> > You mean lowmem is actually FASTER than original mode?
> > That's very surprising.
>  
> Correct, unless I add --repair and then original mode is 2x faster than
> lowmem.
> 
> > Is there any special operation done for that btrfs?
> > Like offline dedupe or tons of reflinks?
> 
> In this case, no.
> Note that btrfs check used to take many hours overnight until I did a
> git pull of btrfs progs and built the latest from TOT.
> 
> > BTW, how many subvolumes do you have in the fs?
>  
> gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | wc -l
> 91
> 
> If I remove snapshots for btrfs send and historical 'backups':
> gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | grep -Ev '(hourly|daily|weekly|rw|ro)' | wc -l
> 5
> 
> > This looks like a bug. My first guess is related to number of
> > subvolumes/reflinks, but I'm not sure since I don't have many real-world
> > btrfs.
> > 
> > I'll take sometime to look into it.
> > 
> > Thanks for the very interesting report,
> 
> Thanks for having a look :)
> 
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-09 17:44         ` Marc MERLIN
@ 2017-09-10  6:01           ` Qu Wenruo
  2017-09-10 13:18             ` Marc MERLIN
  0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2017-09-10  6:01 UTC (permalink / raw)
  To: Marc MERLIN; +Cc: quwenruo, Lu Fengqi, Chris Mason, Btrfs BTRFS, David Sterba



On 2017年09月10日 01:44, Marc MERLIN wrote:
> So, should I assume that btrfs progs git has some issue since there is
> no plausible way that a check --repair should be faster than a regular
> check?

Yes, the assumption that repair should be no faster than RO check is 
correct.
Especially for clean fs, repair should just behave the same as RO check.

And I'll first submit a patch (or patches) to output the consumed time 
for each tree, so we could have a clue what is going wrong.
(Digging the code is just a little too boring for me)

Thanks,
Qu

> 
> Thanks,
> Marc
> 
> On Tue, Sep 05, 2017 at 07:45:25AM -0700, Marc MERLIN wrote:
>> On Tue, Sep 05, 2017 at 04:05:04PM +0800, Qu Wenruo wrote:
>>>> gargamel:~# btrfs fi df /mnt/btrfs_pool1
>>>> Data, single: total\x10.60TiB, used\x10.54TiB
>>>> System, DUP: total2.00MiB, used=1.19MiB
>>>> Metadata, DUP: totalX.00GiB, used\x12.69GiB
>>>
>>> Wait for a minute.
>>>
>>> Is that .69GiB means 706 MiB? Or my email client/GMX screwed up the format
>>> (again)?
>>> This output format must be changed, at least to 0.69 GiB, or 706 MiB.
>>   
>> Email client problem. I see control characters in what you quoted.
>>
>> Let's try again
>> gargamel:~# btrfs fi df /mnt/btrfs_pool1
>> Data, single: total=10.66TiB, used=10.60TiB      => 10TB
>> System, DUP: total=64.00MiB, used=1.20MiB        => 1.2MB
>> Metadata, DUP: total=57.50GiB, used=12.76GiB     => 13GB
>> GlobalReserve, single: total=512.00MiB, used=0.00B  => 0
>>
>>> You mean lowmem is actually FASTER than original mode?
>>> That's very surprising.
>>   
>> Correct, unless I add --repair and then original mode is 2x faster than
>> lowmem.
>>
>>> Is there any special operation done for that btrfs?
>>> Like offline dedupe or tons of reflinks?
>>
>> In this case, no.
>> Note that btrfs check used to take many hours overnight until I did a
>> git pull of btrfs progs and built the latest from TOT.
>>
>>> BTW, how many subvolumes do you have in the fs?
>>   
>> gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | wc -l
>> 91
>>
>> If I remove snapshots for btrfs send and historical 'backups':
>> gargamel:/mnt/btrfs_pool1# btrfs subvolume list . | grep -Ev '(hourly|daily|weekly|rw|ro)' | wc -l
>> 5
>>
>>> This looks like a bug. My first guess is related to number of
>>> subvolumes/reflinks, but I'm not sure since I don't have many real-world
>>> btrfs.
>>>
>>> I'll take sometime to look into it.
>>>
>>> Thanks for the very interesting report,
>>
>> Thanks for having a look :)
>>
>> Marc
>> -- 
>> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>> Microsoft is to operating systems ....
>>                                        .... what McDonalds is to gourmet cooking
>> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: btrfs check --repair now runs in minutes instead of hours? aborting
  2017-09-10  6:01           ` Qu Wenruo
@ 2017-09-10 13:18             ` Marc MERLIN
  0 siblings, 0 replies; 12+ messages in thread
From: Marc MERLIN @ 2017-09-10 13:18 UTC (permalink / raw)
  To: Qu Wenruo; +Cc: quwenruo, Lu Fengqi, Chris Mason, Btrfs BTRFS, David Sterba

On Sun, Sep 10, 2017 at 02:01:58PM +0800, Qu Wenruo wrote:
> 
> 
> On 2017年09月10日 01:44, Marc MERLIN wrote:
> > So, should I assume that btrfs progs git has some issue since there is
> > no plausible way that a check --repair should be faster than a regular
> > check?
> 
> Yes, the assumption that repair should be no faster than RO check is
> correct.
> Especially for clean fs, repair should just behave the same as RO check.
> 
> And I'll first submit a patch (or patches) to output the consumed time for
> each tree, so we could have a clue what is going wrong.
> (Digging the code is just a little too boring for me)

Cool. Let me know when I should sync and re-try.
In the meantime, though, my check --repair went back to 170mn after
triggering an FS bug for Josef, so it seems back to normal.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2017-09-10 13:18 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-05  1:05 btrfs check --repair now runs in minutes instead of hours? aborting Marc MERLIN
2017-09-05  1:21 ` Qu Wenruo
2017-09-05  2:55   ` Marc MERLIN
2017-09-05  4:06     ` Marc MERLIN
2017-09-05  8:05     ` Qu Wenruo
2017-09-05  8:54       ` Duncan
2017-09-05  9:06         ` Qu Wenruo
2017-09-05  9:35           ` Duncan
2017-09-05 14:45       ` Marc MERLIN
2017-09-09 17:44         ` Marc MERLIN
2017-09-10  6:01           ` Qu Wenruo
2017-09-10 13:18             ` Marc MERLIN

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.