'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[i.r.e.c.c.a.k.u.n+kernel.org]

(Originally reported on Kernel.org Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=219033)

There is a very weird quirk I found with 'btrfs filesystem defragment' command. And no, it's not about reflinks removal, I'm aware of that.

It is kinda hard to replicate, but I found a somewhat reliable way. It reaches extremes with fallocated files specifically.

1. Create a file on a Btrfs filesystem using 'fallocate' and fill it. The easy way to do that is just to copy some files with 'rsync --preallocate'.

2. Check compsize info:

# compsize foo
Processed 71 files, 71 regular extents (71 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      630M         630M         630M
none       100%      630M         630M         630M

All is fine here for now. 1 extent per 1 file, "Disk Usage" = "Referenced".

3. Run defragment:

# btrfs filesystem defragment -r foo

4. Check compsize again:

# compsize foo
Processed 71 files, 76 regular extents (76 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      638M         638M         630M
none       100%      638M         638M         630M

Oops, besides the fact that the amount of extents is actually increased, which means 'btrfs filesystem defragment' actually made fragmentation worse, physical disk usage increased for no reason. And I didn't find any way to shrink it back.

---

The end result seems to be random though. But I managed to achieve some truly horrifying results.

# compsize foo
Processed 45 files, 45 regular extents (45 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      360M         360M         360M
none       100%      360M         360M         360M

# btrfs filesystem defragment -r -t 1G foo

# compsize foo
Processed 45 files, 144 regular extents (144 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      716M         716M         360M
none       100%      716M         716M         360M

Yikes! Triple the extents! Double increase in size!

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Qu Wenruo]

在 2024/8/4 18:50, i.r.e.c.c.a.k.u.n+kernel.org@gmail.com 写道:
> (Originally reported on Kernel.org Bugzilla: 
> https://bugzilla.kernel.org/show_bug.cgi?id=219033)
> 
> There is a very weird quirk I found with 'btrfs filesystem defragment' 
> command. And no, it's not about reflinks removal, I'm aware of that.
> 
> It is kinda hard to replicate, but I found a somewhat reliable way. It 
> reaches extremes with fallocated files specifically.
> 
> 1. Create a file on a Btrfs filesystem using 'fallocate' and fill it. 
> The easy way to do that is just to copy some files with 'rsync 
> --preallocate'.
> 
> 2. Check compsize info:

Mind to dump the filemap output (xfs_io -c "fiemap -v") before and after 
the defrag?

Thanks,
Qu
> 
> # compsize foo
> Processed 71 files, 71 regular extents (71 refs), 0 inline.
> Type       Perc     Disk Usage   Uncompressed Referenced
> TOTAL      100%      630M         630M         630M
> none       100%      630M         630M         630M
> 
> All is fine here for now. 1 extent per 1 file, "Disk Usage" = "Referenced".
> 
> 3. Run defragment:
> 
> # btrfs filesystem defragment -r foo
> 
> 4. Check compsize again:
> 
> # compsize foo
> Processed 71 files, 76 regular extents (76 refs), 0 inline.
> Type       Perc     Disk Usage   Uncompressed Referenced
> TOTAL      100%      638M         638M         630M
> none       100%      638M         638M         630M
> 
> Oops, besides the fact that the amount of extents is actually increased, 
> which means 'btrfs filesystem defragment' actually made fragmentation 
> worse, physical disk usage increased for no reason. And I didn't find 
> any way to shrink it back.
> 
> ---
> 
> The end result seems to be random though. But I managed to achieve some 
> truly horrifying results.
> 
> # compsize foo
> Processed 45 files, 45 regular extents (45 refs), 0 inline.
> Type       Perc     Disk Usage   Uncompressed Referenced
> TOTAL      100%      360M         360M         360M
> none       100%      360M         360M         360M
> 
> # btrfs filesystem defragment -r -t 1G foo
> 
> # compsize foo
> Processed 45 files, 144 regular extents (144 refs), 0 inline.
> Type       Perc     Disk Usage   Uncompressed Referenced
> TOTAL      100%      716M         716M         360M
> none       100%      716M         716M         360M
> 
> Yikes! Triple the extents! Double increase in size!
> 

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Hanabishi]

On 8/4/24 22:19, Qu Wenruo wrote:
> Mind to dump the filemap output (xfs_io -c "fiemap -v") before and after the defrag?
> 
> Thanks,
> Qu

Sure.

# compsize mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
Processed 1 file, 1 regular extents (1 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      224M         224M         224M
none       100%      224M         224M         224M

# xfs_io -c "fiemap -v" mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
  EXT: FILE-OFFSET      BLOCK-RANGE           TOTAL FLAGS
    0: [0..460303]:     545974648..546434951 460304   0x1

# btrfs filesystem defragment -t 1G mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst

# compsize mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
Processed 1 file, 8 regular extents (8 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      420M         420M         224M
none       100%      420M         420M         224M

# xfs_io -c "fiemap -v" mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
  EXT: FILE-OFFSET      BLOCK-RANGE           TOTAL FLAGS
    0: [0..511]:        15754800..15755311      512   0x0
    1: [512..2559]:     22070192..22072239     2048   0x0
    2: [2560..6655]:    22632216..22636311     4096   0x0
    3: [6656..14335]:   22072240..22079919     7680   0x0
    4: [14336..385023]: 546434952..546805639 370688   0x0
    5: [385024..400383]: 44592672..44608031    15360   0x0
    6: [400384..460303]: 546375032..546434951  59920   0x1

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Qu Wenruo]

在 2024/8/6 03:46, Hanabishi 写道:
> On 8/4/24 22:19, Qu Wenruo wrote:
>> Mind to dump the filemap output (xfs_io -c "fiemap -v") before and
>> after the defrag?
>>
>> Thanks,
>> Qu
>
> Sure.
>
> # compsize mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
> Processed 1 file, 1 regular extents (1 refs), 0 inline.
> Type       Perc     Disk Usage   Uncompressed Referenced
> TOTAL      100%      224M         224M         224M
> none       100%      224M         224M         224M
>
> # xfs_io -c "fiemap -v" mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
> mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
>   EXT: FILE-OFFSET      BLOCK-RANGE           TOTAL FLAGS
>     0: [0..460303]:     545974648..546434951 460304   0x1

Weird, there is no fallocated space involved at all.

>
> # btrfs filesystem defragment -t 1G

Oh you're using non-default threshold.
Unfortunately 1G makes no sense, as btrfs's largest extent size is only
128M.

(Although the above output shows an extent with 224M size, it's because
btrfs merges the fiemap result internally when possible).

It's recommended to go the default values anyway.

> mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
> mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
>
> # compsize mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
> Processed 1 file, 8 regular extents (8 refs), 0 inline.
> Type       Perc     Disk Usage   Uncompressed Referenced
> TOTAL      100%      420M         420M         224M
> none       100%      420M         420M         224M
>
> # xfs_io -c "fiemap -v" mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
> mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
>   EXT: FILE-OFFSET      BLOCK-RANGE           TOTAL FLAGS
>     0: [0..511]:        15754800..15755311      512   0x0
>     1: [512..2559]:     22070192..22072239     2048   0x0
>     2: [2560..6655]:    22632216..22636311     4096   0x0
>     3: [6656..14335]:   22072240..22079919     7680   0x0
>     4: [14336..385023]: 546434952..546805639 370688   0x0
>     5: [385024..400383]: 44592672..44608031    15360   0x0

All the above extents are new extents.

>     6: [400384..460303]: 546375032..546434951  59920   0x1
>

While this one is the old one.

This looks like a recent bug fix e42b9d8b9ea2 ("btrfs: defrag: avoid
unnecessary defrag caused by incorrect extent size"), which is merged in
v6.8 kernel.

Mind to provide the kernel version?


Furthermore, there is another problem, according to your fiemap result,
the fs seems to cause fragmented new extents by somehow.

Is there any memory pressure or the fs itself is fragmented?
Btrfs defrag is only re-dirty the data, then write them back.
This expects them to be written in a continuous extent, but both memory
pressure and fragmented fs can all break such assumption.

Thanks,
Qu

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Hanabishi]

On 8/5/24 22:47, Qu Wenruo wrote:

> It's recommended to go the default values anyway.

It's for testing purposes. As you can see in original message, it happens regardless.
I simply noticed that increasing the threshold makes the problem worse.

> Mind to provide the kernel version?

Originally reported at 6.10-rc7. Current tests with 6.11-rc1 and 6.11-rc2. Still the same results.

> Is there any memory pressure or the fs itself is fragmented?

No. I tested it on multiple machines with lots of free RAM, also tested with like 99% empty disks.

Could you please try it yourself? It is fairly easy to follow the steps.
I use 'rsync --preallocate' to copy the files over (and maybe call 'sync' after to be sure).
Then run defragment on them and see if the problem reproduces.

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Qu Wenruo]

在 2024/8/6 16:49, Hanabishi 写道:
> On 8/5/24 22:47, Qu Wenruo wrote:
>
>> It's recommended to go the default values anyway.
>
> It's for testing purposes. As you can see in original message, it
> happens regardless.
> I simply noticed that increasing the threshold makes the problem worse.
>
>> Mind to provide the kernel version?
>
> Originally reported at 6.10-rc7. Current tests with 6.11-rc1 and
> 6.11-rc2. Still the same results.
>
>> Is there any memory pressure or the fs itself is fragmented?
>
> No. I tested it on multiple machines with lots of free RAM, also tested
> with like 99% empty disks.
>
> Could you please try it yourself? It is fairly easy to follow the steps.
> I use 'rsync --preallocate' to copy the files over (and maybe call
> 'sync' after to be sure).
> Then run defragment on them and see if the problem reproduces.
>

The problem is, I can not reproduce the problem here.
Or I'm already submitting patch to fix it.

# xfs_io  -c "fiemap -v"
/mnt/btrfs/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
/mnt/btrfs/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
  EXT: FILE-OFFSET      BLOCK-RANGE       TOTAL FLAGS
    0: [0..460303]:     583680..1043983  460304   0x1

# btrfs fi defrag /mnt/btrfs/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
# sync

# xfs_io  -c "fiemap -v"
/mnt/btrfs/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
/mnt/btrfs/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
  EXT: FILE-OFFSET      BLOCK-RANGE       TOTAL FLAGS
    0: [0..460303]:     583680..1043983  460304   0x1

In fact, with your initial fiemap layout, btrfs won't even try to defrag
it, due to the extent size is already larger than the default threshold.

I also tried "rsync --preallocate" as request, the same:

# rsync --preallocate
/home/adam/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst  /mnt/btrfs/

# xfs_io  -c "fiemap -v"
/mnt/btrfs/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
/mnt/btrfs/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
  EXT: FILE-OFFSET      BLOCK-RANGE       TOTAL FLAGS
    0: [0..460303]:     1043984..1504287 460304   0x1

# btrfs fi defrag /mnt/btrfs/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
# sync
# xfs_io  -c "fiemap -v"
/mnt/btrfs/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
/mnt/btrfs/mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
  EXT: FILE-OFFSET      BLOCK-RANGE       TOTAL FLAGS
    0: [0..460303]:     1043984..1504287 460304   0x1

The same, btrfs detects the extent is large enough and refuse to do
defrag. (Although tried "-t 1G" for defrag, no difference)

That's why I'm asking you all the information, because:

- The kernel code should skip large enough extents
   At least if using the default parameter.

- Even for preallocated cases, as long as the file occupy
   the whole length, it's no different.

- Even btrfs chose to do defrag, and you have no memory pressure
   there should be new continuous data extents.
   Not the smaller ones you shown.

So either there is something like cgroup involved (which can limits the
dirty page cache and trigger write backs), or some other weird
behavior/bugs.

Thanks,
Qu

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Hanabishi]

On 8/6/24 09:55, Qu Wenruo wrote:

> So either there is something like cgroup involved (which can limits the
> dirty page cache and trigger write backs), or some other weird
> behavior/bugs.

Yes, this line reveals something. I do have modified dirty page cache values. I tend to keep it on low values.

Now playing around with it - yes, it is seems to be the cause. When I tune 'vm.dirty_ratio' and 'vm.dirty_background_ratio' up to higher values, the problem becames less prevalent.

Which means lowering them cranks up the problem to extremes. E.g. try

# sysctl -w vm.dirty_bytes=8192
# sysctl -w vm.dirty_background_ratio=0

With that setup defrag completely obliterates files even with default threshold value.

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Qu Wenruo]

在 2024/8/6 19:53, Hanabishi 写道:
> On 8/6/24 09:55, Qu Wenruo wrote:
>
>> So either there is something like cgroup involved (which can limits the
>> dirty page cache and trigger write backs), or some other weird
>> behavior/bugs.
>
> Yes, this line reveals something. I do have modified dirty page cache
> values. I tend to keep it on low values.
>
> Now playing around with it - yes, it is seems to be the cause. When I
> tune 'vm.dirty_ratio' and 'vm.dirty_background_ratio' up to higher
> values, the problem becames less prevalent.
>
> Which means lowering them cranks up the problem to extremes. E.g. try
>
> # sysctl -w vm.dirty_bytes=8192
> # sysctl -w vm.dirty_background_ratio=0
>
> With that setup defrag completely obliterates files even with default
> threshold value.

At least I no longer need to live under the fear of new defrag bugs.

This also explains why defrag (even with default values) would trigger
rewrites of extents, because although fiemap is only showing a single
extent, it will be a lot of small extents on the larger pre-allocated range.

Thus btrfs believe it can merge all of them into a larger extent, but VM
settings forces btrfs to write them early, causing extra data COW, and
cause worse fragmentation.

Too low values means kernel will trigger dirty writeback aggressively, I
believe for all extent based file systems (ext4/xfs/btrfs etc), it would
cause a huge waste of metadata, due to the huge amount of small extents.

So yes, that setting is the cause, although it will reduce the memory
used by page cache (it still counts as memory pressure), but the cost is
more fragmented extents and overall worse fs performance and possibly
more wear on NAND based storage.

Thanks,
Qu

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Hanabishi]

On 8/6/24 10:42, Qu Wenruo wrote:

> Too low values means kernel will trigger dirty writeback aggressively, I
> believe for all extent based file systems (ext4/xfs/btrfs etc), it would
> cause a huge waste of metadata, due to the huge amount of small extents.
> 
> So yes, that setting is the cause, although it will reduce the memory
> used by page cache (it still counts as memory pressure), but the cost is
> more fragmented extents and overall worse fs performance and possibly
> more wear on NAND based storage.

Thanks for explanation. I'm aware of low dirty page cache performance tradeoffs, I prefer more reliability in case of system failure / power outage.
But that rises questions anyway.

1. Why are files ok initially regardless of page cache size? It only blows up with explicit run of the defragment command. And I didn't face anything similar with other filesystems either.

2. How I get my space back without deleting the files? Even if I crank up the page cache amount and then defragment "properly", it doesn't reclaim the actual space back.

# btrfs filesystem defragment mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst

# compsize mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
Processed 1 file, 3 regular extents (3 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      449M         449M         224M
none       100%      449M         449M         224M

There are 3 extents, it's defenitely not a metadata overhead.

3. Regardless of settings, what if users do end up in low memory conditions for some reason? It's not an uncommon scenario.
You end up with Btrfs borking your disk space. In my opinion it looks like a bug and should not happen.

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Qu Wenruo]

在 2024/8/6 20:35, Hanabishi 写道:
> On 8/6/24 10:42, Qu Wenruo wrote:
>
>> Too low values means kernel will trigger dirty writeback aggressively, I
>> believe for all extent based file systems (ext4/xfs/btrfs etc), it would
>> cause a huge waste of metadata, due to the huge amount of small extents.
>>
>> So yes, that setting is the cause, although it will reduce the memory
>> used by page cache (it still counts as memory pressure), but the cost is
>> more fragmented extents and overall worse fs performance and possibly
>> more wear on NAND based storage.
>
> Thanks for explanation. I'm aware of low dirty page cache performance
> tradeoffs, I prefer more reliability in case of system failure / power
> outage.
> But that rises questions anyway.
>
> 1. Why are files ok initially regardless of page cache size? It only
> blows up with explicit run of the defragment command. And I didn't face
> anything similar with other filesystems either.

Because btrfs merges extents that are physically adjacent at fiemap time.

Especially if you go fallocate, then the initial write are ensured to
land in that preallocated range.
Although they may be split into many small extents, they are still
physically adjacent.

When defrag happens, it triggers data COW, and screw up everything.

>
> 2. How I get my space back without deleting the files? Even if I crank
> up the page cache amount and then defragment "properly", it doesn't
> reclaim the actual space back.
>
> # btrfs filesystem defragment mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
>
> # compsize mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
> Processed 1 file, 3 regular extents (3 refs), 0 inline.
> Type       Perc     Disk Usage   Uncompressed Referenced
> TOTAL      100%      449M         449M         224M
> none       100%      449M         449M         224M
>
> There are 3 extents, it's defenitely not a metadata overhead.

I'm not sure how high the value you set, but at least please do
everything with default kernel config, not just crank the settings up.

And have you tried sync before compsize/fiemap?

If you still have problems reclaiming the space, please provide the
fiemap output (before defrag, and after defrag and sync)

>
> 3. Regardless of settings, what if users do end up in low memory
> conditions for some reason? It's not an uncommon scenario.
> You end up with Btrfs borking your disk space. In my opinion it looks
> like a bug and should not happen.
>

If we try to lock the defrag range, to ensure them to land in a larger
extent, I'm 100% sure MM guys won't be happy, it's blocking the most
common way to reclaim memory.

By that method we're only going to exhaust the system memory at the
worst timing.


IIRC it's already in the document, although not that clear:

   The value is only advisory and the final size of the extents may
   differ, depending on the state of the free space and fragmentation or
   other internal logic.

To be honest, defrag is not recommended for modern extent based file
systems already, thus there is no longer a common and good example to
follow.

And for COW file systems, along with btrfs' specific bookend behavior,
it brings a new level of complexity.

So overall, if you're not sure what the defrag internal logic is, nor
have a clear problem you want to solve, do not defrag.

Thanks,
Qu

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Hanabishi]

On 8/6/24 11:23, Qu Wenruo wrote:

> When defrag happens, it triggers data COW, and screw up everything.

Yeah, making test files in NOCOW mode seems to prevent the issue.

> I'm not sure how high the value you set, but at least please do
> everything with default kernel config, not just crank the settings up.

Up to 50% of 32G RAM machine. More than enough for a 224M file.

Kernel defaults are meaningless anyway, as they are relative to RAM size. Even Linus admitted that: https://lwn.net/Articles/572921/

> And have you tried sync before compsize/fiemap?

Of course. I do sync on every step.

> If we try to lock the defrag range, to ensure them to land in a larger
> extent, I'm 100% sure MM guys won't be happy, it's blocking the most
> common way to reclaim memory.

Hmm, but couldn't Btrfs simply preallocate that space? I copied files much larger in size than the page cache and even entire RAM, they are totally fine as you could guess.
Is moving extents under the hood that different from copying files around?

> IIRC it's already in the document, although not that clear:
> 
>    The value is only advisory and the final size of the extents may
>    differ, depending on the state of the free space and fragmentation or
>    other internal logic.
> 
> To be honest, defrag is not recommended for modern extent based file
> systems already, thus there is no longer a common and good example to
> follow.
> 
> And for COW file systems, along with btrfs' specific bookend behavior,
> it brings a new level of complexity.
> 
> So overall, if you're not sure what the defrag internal logic is, nor
> have a clear problem you want to solve, do not defrag.

Well, I went into this hole for a reason.
I worked with some software piece which writes files sequentally, but in a very primitive POSIX-compliant way. For reference, ~17G file it produced was split into more than 1 million(!) extents. Basically shredding entire file into 16K pieces. Producing a no-joke access performance penalty even on SSD. In fact I only noticed the problem because of horrible disk performance with the file.

And I even tried to write it in NOCOW mode, but it didn't help, fragmentation level was the same. So it has nothing to do with CoW, it's Btrfs itself not really getting intentions of the software.
I'm not sure how it would behave with other filesystems, but for me it doesn't really look as a FS fault anyway.

So I ended up falling back to the old good defragmentation. Discovering the reported issue along the way, becaming a double-trouble for me.

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Qu Wenruo]

在 2024/8/6 21:38, Hanabishi 写道:
> On 8/6/24 11:23, Qu Wenruo wrote:
[...]
>
>> If we try to lock the defrag range, to ensure them to land in a larger
>> extent, I'm 100% sure MM guys won't be happy, it's blocking the most
>> common way to reclaim memory.
>
> Hmm, but couldn't Btrfs simply preallocate that space? I copied files
> much larger in size than the page cache and even entire RAM, they are
> totally fine as you could guess.

For preallocation, welcome into the rabbit hole.

TL;DR, preallocation of btrfs is never reliable, it doesn't even ensure
the next write will success.

The biggest reason here is snapshot.

Even if we preallocate the range, the end user is still fully allowed to
do any snapshot.

And a preallocated range shared by multiple subvolumes will never be
overwritten, causing the same problem.


As you have already experienced, if set to NOCOW, everything will be
(mostly) fine, just as all the other non-COW filesystems.

But you're using btrfs for its super fast snapshot, and that will force
data COW, causing all the complexity.

> Is moving extents under the hood that different from copying files around?

Ext4 uses move_extent to do defrag, but unfortunately we do not go that
path, as mentioned even preallocation is not reliable.

>
>> IIRC it's already in the document, although not that clear:
>>
>>    The value is only advisory and the final size of the extents may
>>    differ, depending on the state of the free space and fragmentation or
>>    other internal logic.
>>
>> To be honest, defrag is not recommended for modern extent based file
>> systems already, thus there is no longer a common and good example to
>> follow.
>>
>> And for COW file systems, along with btrfs' specific bookend behavior,
>> it brings a new level of complexity.
>>
>> So overall, if you're not sure what the defrag internal logic is, nor
>> have a clear problem you want to solve, do not defrag.
>
> Well, I went into this hole for a reason.
> I worked with some software piece which writes files sequentally, but in
> a very primitive POSIX-compliant way. For reference, ~17G file it
> produced was split into more than 1 million(!) extents. Basically
> shredding entire file into 16K pieces. Producing a no-joke access
> performance penalty even on SSD. In fact I only noticed the problem
> because of horrible disk performance with the file.
>
> And I even tried to write it in NOCOW mode, but it didn't help,
> fragmentation level was the same. So it has nothing to do with CoW, it's
> Btrfs itself not really getting intentions of the software.
> I'm not sure how it would behave with other filesystems, but for me it
> doesn't really look as a FS fault anyway.

To me, any fs will follow the sync/fsync request from the user space
process, so if the tool wants fragmentation, it will get fragmentation.

>
> So I ended up falling back to the old good defragmentation. Discovering
> the reported issue along the way, becaming a double-trouble for me.
>

In that case, if you do not use snapshot for that subvolume, it's
recommended to go with NOCOW first, then preallocate space for the log
file. By this, the log file is always using continous range on disk.

And finally go with defrag (with high enough writeback threshold), to
reduce the number of extents (fully internal, won't even be reported by
fiemap).


BTW, forgot to mention in your previous reply mentioning the powerloss
of a fs, it doesn't help btrfs at least.

Even we aggressively writeback the dirty data, our metadata is purely
protected by COW, and a transactional system.

It means, even you have written 10GiB new data, as long as our
transaction is not committed, you will only get all the old data after a
power loss (unless it's explicitly fsynced).
That's another point very different from old non-COW filesystems.

Instead "commit=" with a lower value is more helpful for btrfs, but that
would cause more metadata writes though.

Thanks,
Qu

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Hanabishi]

On 8/6/24 22:10, Qu Wenruo wrote:

> But you're using btrfs for its super fast snapshot, and that will force
> data COW, causing all the complexity.

For me the data checksumming is more of a selling point. I.e. yes, using Btrfs in a NOCOW mode kinda defies the point.

> It means, even you have written 10GiB new data, as long as our
> transaction is not committed, you will only get all the old data after a
> power loss (unless it's explicitly fsynced).
> That's another point very different from old non-COW filesystems.
> 
> Instead "commit=" with a lower value is more helpful for btrfs, but that
> would cause more metadata writes though.

What about "flushoncommit" mount option? Does it make data view more resilient?

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Qu Wenruo]

在 2024/8/7 08:12, Hanabishi 写道:
> On 8/6/24 22:10, Qu Wenruo wrote:
>
>> But you're using btrfs for its super fast snapshot, and that will force
>> data COW, causing all the complexity.
>
> For me the data checksumming is more of a selling point. I.e. yes, using
> Btrfs in a NOCOW mode kinda defies the point.

In that data csum (and COW) case, then I guess one has to choose if
preallocation is really wanted very carefully.

Or it's super easy to cause unexpected on-disk space waste.
(COW is already going to cause space waste, but preallocation amplifies
that much faster)

>
>> It means, even you have written 10GiB new data, as long as our
>> transaction is not committed, you will only get all the old data after a
>> power loss (unless it's explicitly fsynced).
>> That's another point very different from old non-COW filesystems.
>>
>> Instead "commit=" with a lower value is more helpful for btrfs, but that
>> would cause more metadata writes though.
>
> What about "flushoncommit" mount option? Does it make data view more
> resilient?
>

If combined with lower commit= value, yes, it will be data view more
consistent with transactions.

But as usual, it amplifies the metadata writes, which is already pretty
bad for btrfs.

And may worse the extra space usage problem too, if using data COW (as
it forces dirty pages writeback at every transaction commit, causing
smaller writes)

(I guess a UPS would be better for everyone except the budget?)

Thanks,
Qu

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Hanabishi]

On 8/6/24 22:51, Qu Wenruo wrote:

> (I guess a UPS would be better for everyone except the budget?)

Of course, I have it. But you underestimate level of my paranoia :)

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Hanabishi]

On 8/6/24 11:23, Qu Wenruo wrote:

> If you still have problems reclaiming the space, please provide the
> fiemap output (before defrag, and after defrag and sync)

Before defrag:

# compsize mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
Processed 1 file, 883 regular extents (883 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      449M         449M         224M
none       100%      449M         449M         224M

# xfs_io -c "fiemap -v" mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
  EXT: FILE-OFFSET      BLOCK-RANGE        TOTAL FLAGS
    0: [0..511]:        15754800..15755311   512   0x0
    1: [512..1023]:     15063136..15063647   512   0x0
    2: [1024..1535]:    15751920..15752431   512   0x0
    3: [1536..2047]:    15145832..15146343   512   0x0
    4: [2048..3071]:    22070192..22071215  1024   0x0
    5: [3072..3583]:    22632216..22632727   512   0x0
    6: [3584..4095]:    22071216..22071727   512   0x0
    7: [4096..4607]:    22632728..22633239   512   0x0
    8: [4608..5119]:    22071728..22072239   512   0x0
    9: [5120..5631]:    22633240..22633751   512   0x0
   10: [5632..6143]:    22072240..22072751   512   0x0
   11: [6144..6655]:    22633752..22634263   512   0x0
   12: [6656..7167]:    22072752..22073263   512   0x0
   13: [7168..7679]:    22634264..22634775   512   0x0
   14: [7680..8191]:    22073264..22073775   512   0x0
   15: [8192..8703]:    21106888..21107399   512   0x0
   16: [8704..9215]:    22863440..22863951   512   0x0
   17: [9216..9727]:    22634776..22635287   512   0x0
   18: [9728..10239]:   22073776..22074287   512   0x0
   19: [10240..10751]:  21107400..21107911   512   0x0
   20: [10752..11263]:  22863952..22864463   512   0x0
   21: [11264..11775]:  22635288..22635799   512   0x0
   22: [11776..12287]:  22074288..22074799   512   0x0
   23: [12288..12799]:  21107912..21108423   512   0x0
   24: [12800..13311]:  22864464..22864975   512   0x0
   25: [13312..13823]:  22635800..22636311   512   0x0
   26: [13824..14335]:  22074800..22075311   512   0x0
   27: [14336..14847]:  21108424..21108935   512   0x0
   28: [14848..15359]:  22864976..22865487   512   0x0
   29: [15360..15871]:  22106224..22106735   512   0x0
   30: [15872..16383]:  22636312..22636823   512   0x0
   31: [16384..16895]:  22075312..22075823   512   0x0
   32: [16896..17407]:  21108936..21109447   512   0x0
   33: [17408..17919]:  22865488..22865999   512   0x0
   34: [17920..18431]:  22106736..22107247   512   0x0
   35: [18432..18943]:  21587608..21588119   512   0x0
   36: [18944..19455]:  21821096..21821607   512   0x0
   37: [19456..19967]:  22636824..22637335   512   0x0
   38: [19968..20479]:  22612032..22612543   512   0x0
   39: [20480..20991]:  22075824..22076335   512   0x0
   40: [20992..21503]:  22886512..22887023   512   0x0
   41: [21504..22015]:  21109448..21109959   512   0x0
   42: [22016..22527]:  22866000..22866511   512   0x0
   43: [22528..23039]:  22107248..22107759   512   0x0
   44: [23040..23551]:  21588120..21588631   512   0x0
   45: [23552..24063]:  21821608..21822119   512   0x0
   46: [24064..24575]:  22637336..22637847   512   0x0
   47: [24576..25087]:  22612544..22613055   512   0x0
   48: [25088..25599]:  22076336..22076847   512   0x0
   49: [25600..26111]:  22887024..22887535   512   0x0
   50: [26112..26623]:  21109960..21110471   512   0x0
   51: [26624..27135]:  22866512..22867023   512   0x0
   52: [27136..27647]:  22107760..22108271   512   0x0
   53: [27648..28159]:  21588632..21589143   512   0x0
   54: [28160..28671]:  21560104..21560615   512   0x0
   55: [28672..29183]:  21822120..21822631   512   0x0
   56: [29184..29695]:  22637848..22638359   512   0x0
   57: [29696..30207]:  22613056..22613567   512   0x0
   58: [30208..30719]:  22076848..22077359   512   0x0
   59: [30720..31231]:  22887536..22888047   512   0x0
   60: [31232..31743]:  21110472..21110983   512   0x0
   61: [31744..32255]:  22867024..22867535   512   0x0
   62: [32256..32767]:  22108272..22108783   512   0x0
   63: [32768..33279]:  21589144..21589655   512   0x0
   64: [33280..33791]:  21067632..21068143   512   0x0
   65: [33792..34303]:  22320824..22321335   512   0x0
   66: [34304..34815]:  21560616..21561127   512   0x0
   67: [34816..35327]:  21822632..21823143   512   0x0
   68: [35328..35839]:  22902232..22902743   512   0x0
   69: [35840..36351]:  22638360..22638871   512   0x0
   70: [36352..36863]:  22613568..22614079   512   0x0
   71: [36864..37375]:  22077360..22077871   512   0x0
   72: [37376..37887]:  21165368..21165879   512   0x0
   73: [37888..38399]:  22888048..22888559   512   0x0
   74: [38400..38911]:  21110984..21111495   512   0x0
   75: [38912..39423]:  22867536..22868047   512   0x0
   76: [39424..39935]:  22108784..22109295   512   0x0
   77: [39936..40447]:  21589656..21590167   512   0x0
   78: [40448..40959]:  21063264..21063775   512   0x0
   79: [40960..41471]:  22389400..22389911   512   0x0
   80: [41472..41983]:  21068144..21068655   512   0x0
   81: [41984..42495]:  22321336..22321847   512   0x0
   82: [42496..43007]:  21324480..21324991   512   0x0
   83: [43008..43519]:  21561128..21561639   512   0x0
   84: [43520..44031]:  21823144..21823655   512   0x0
   85: [44032..44543]:  22902744..22903255   512   0x0
   86: [44544..45055]:  22638872..22639383   512   0x0
   87: [45056..45567]:  22614080..22614591   512   0x0
   88: [45568..46079]:  22216920..22217431   512   0x0
   89: [46080..46591]:  22077872..22078383   512   0x0
   90: [46592..47103]:  21165880..21166391   512   0x0
   91: [47104..47615]:  22888560..22889071   512   0x0
   92: [47616..48127]:  21111496..21112007   512   0x0
   93: [48128..48639]:  21664808..21665319   512   0x0
   94: [48640..49151]:  22868048..22868559   512   0x0
   95: [49152..49663]:  22109296..22109807   512   0x0
   96: [49664..50175]:  21590168..21590679   512   0x0
   97: [50176..50687]:  21063776..21064287   512   0x0
   98: [50688..51199]:  22389912..22390423   512   0x0
   99: [51200..51711]:  21068656..21069167   512   0x0
  100: [51712..52223]:  22321848..22322359   512   0x0
  101: [52224..52735]:  21324992..21325503   512   0x0
  102: [52736..53247]:  21561640..21562151   512   0x0
  103: [53248..53759]:  21823656..21824167   512   0x0
  104: [53760..54271]:  22903256..22903767   512   0x0
  105: [54272..54783]:  22639384..22639895   512   0x0
  106: [54784..55295]:  21076216..21076727   512   0x0
  107: [55296..55807]:  22614592..22615103   512   0x0
  108: [55808..56319]:  22217432..22217943   512   0x0
  109: [56320..56831]:  22078384..22078895   512   0x0
  110: [56832..57343]:  21166392..21166903   512   0x0
  111: [57344..57855]:  22889072..22889583   512   0x0
  112: [57856..58367]:  21112008..21112519   512   0x0
  113: [58368..58879]:  21665320..21665831   512   0x0
  114: [58880..59391]:  22868560..22869071   512   0x0
  115: [59392..59903]:  22109808..22110319   512   0x0
  116: [59904..60415]:  21590680..21591191   512   0x0
  117: [60416..60927]:  21064288..21064799   512   0x0
  118: [60928..61439]:  22341784..22342295   512   0x0
  119: [61440..61951]:  22390424..22390935   512   0x0
  120: [61952..62463]:  21069168..21069679   512   0x0
  121: [62464..62975]:  22322360..22322871   512   0x0
  122: [62976..63487]:  21325504..21326015   512   0x0
  123: [63488..63999]:  21562152..21562663   512   0x0
  124: [64000..64511]:  21824168..21824679   512   0x0
  125: [64512..65023]:  22903768..22904279   512   0x0
  126: [65024..65535]:  22639896..22640407   512   0x0
  127: [65536..66047]:  21076728..21077239   512   0x0
  128: [66048..66559]:  22615104..22615615   512   0x0
  129: [66560..67071]:  22217944..22218455   512   0x0
  130: [67072..67583]:  22078896..22079407   512   0x0
  131: [67584..68095]:  21166904..21167415   512   0x0
  132: [68096..68607]:  22889584..22890095   512   0x0
  133: [68608..69119]:  21112520..21113031   512   0x0
  134: [69120..69631]:  22609256..22609767   512   0x0
  135: [69632..70143]:  21665832..21666343   512   0x0
  136: [70144..70655]:  22869072..22869583   512   0x0
  137: [70656..71167]:  22110320..22110831   512   0x0
  138: [71168..71679]:  21617320..21617831   512   0x0
  139: [71680..72191]:  21591192..21591703   512   0x0
  140: [72192..72703]:  21282824..21283335   512   0x0
  141: [72704..73215]:  21064800..21065311   512   0x0
  142: [73216..73727]:  22342296..22342807   512   0x0
  143: [73728..74239]:  22390936..22391447   512   0x0
  144: [74240..74751]:  21069680..21070191   512   0x0
  145: [74752..75263]:  22322872..22323383   512   0x0
  146: [75264..75775]:  21326016..21326527   512   0x0
  147: [75776..76287]:  21562664..21563175   512   0x0
  148: [76288..76799]:  21824680..21825191   512   0x0
  149: [76800..77311]:  22904280..22904791   512   0x0
  150: [77312..77823]:  22640408..22640919   512   0x0
  151: [77824..78335]:  21077240..21077751   512   0x0
  152: [78336..78847]:  22615616..22616127   512   0x0
  153: [78848..79359]:  22218456..22218967   512   0x0
  154: [79360..79871]:  22079408..22079919   512   0x0
  155: [79872..80383]:  21167416..21167927   512   0x0
  156: [80384..80895]:  22686944..22687455   512   0x0
  157: [80896..81407]:  22890096..22890607   512   0x0
  158: [81408..81919]:  22310672..22311183   512   0x0
  159: [81920..82431]:  21113032..21113543   512   0x0
  160: [82432..82943]:  22609768..22610279   512   0x0
  161: [82944..83455]:  21666344..21666855   512   0x0
  162: [83456..83967]:  22869584..22870095   512   0x0
  163: [83968..84479]:  22110832..22111343   512   0x0
  164: [84480..84991]:  21617832..21618343   512   0x0
  165: [84992..85503]:  21591704..21592215   512   0x0
  166: [85504..86015]:  21283336..21283847   512   0x0
  167: [86016..86527]:  21065312..21065823   512   0x0
  168: [86528..87039]:  22342808..22343319   512   0x0
  169: [87040..87551]:  22391448..22391959   512   0x0
  170: [87552..88063]:  21070192..21070703   512   0x0
  171: [88064..88575]:  22323384..22323895   512   0x0
  172: [88576..89087]:  21326528..21327039   512   0x0
  173: [89088..89599]:  21563176..21563687   512   0x0
  174: [89600..90111]:  21825192..21825703   512   0x0
  175: [90112..90623]:  22904792..22905303   512   0x0
  176: [90624..91135]:  22585336..22585847   512   0x0
  177: [91136..91647]:  22640920..22641431   512   0x0
  178: [91648..92159]:  21077752..21078263   512   0x0
  179: [92160..92671]:  22616128..22616639   512   0x0
  180: [92672..93183]:  21293176..21293687   512   0x0
  181: [93184..93695]:  22218968..22219479   512   0x0
  182: [93696..94207]:  22079920..22080431   512   0x0
  183: [94208..94719]:  21167928..21168439   512   0x0
  184: [94720..95231]:  22687456..22687967   512   0x0
  185: [95232..95743]:  22890608..22891119   512   0x0
  186: [95744..96255]:  22311184..22311695   512   0x0
  187: [96256..96767]:  21113544..21114055   512   0x0
  188: [96768..97279]:  22610280..22610791   512   0x0
  189: [97280..97791]:  21666856..21667367   512   0x0
  190: [97792..98303]:  22870096..22870607   512   0x0
  191: [98304..98815]:  22111344..22111855   512   0x0
  192: [98816..99327]:  21618344..21618855   512   0x0
  193: [99328..99839]:  21592216..21592727   512   0x0
  194: [99840..100351]: 21283848..21284359   512   0x0
  195: [100352..100863]: 21065824..21066335   512   0x0
  196: [100864..101375]: 22343320..22343831   512   0x0
  197: [101376..101887]: 22391960..22392471   512   0x0
  198: [101888..102399]: 21070704..21071215   512   0x0
  199: [102400..102911]: 22323896..22324407   512   0x0
  200: [102912..103423]: 21327040..21327551   512   0x0
  201: [103424..103935]: 21563688..21564199   512   0x0
  202: [103936..104447]: 21825704..21826215   512   0x0
  203: [104448..104959]: 22905304..22905815   512   0x0
  204: [104960..105471]: 22585848..22586359   512   0x0
  205: [105472..105983]: 22641432..22641943   512   0x0
  206: [105984..106495]: 21078264..21078775   512   0x0
  207: [106496..107007]: 22616640..22617151   512   0x0
  208: [107008..107519]: 21234104..21234615   512   0x0
  209: [107520..108031]: 22427128..22427639   512   0x0
  210: [108032..108543]: 21293688..21294199   512   0x0
  211: [108544..109055]: 22219480..22219991   512   0x0
  212: [109056..109567]: 22080432..22080943   512   0x0
  213: [109568..110079]: 21168440..21168951   512   0x0
  214: [110080..110591]: 21409320..21409831   512   0x0
  215: [110592..111103]: 22687968..22688479   512   0x0
  216: [111104..111615]: 22891120..22891631   512   0x0
  217: [111616..112127]: 21684712..21685223   512   0x0
  218: [112128..112639]: 22311696..22312207   512   0x0
  219: [112640..113151]: 21719392..21719903   512   0x0
  220: [113152..113663]: 21114056..21114567   512   0x0
  221: [113664..114175]: 23113504..23114015   512   0x0
  222: [114176..114687]: 22610792..22611303   512   0x0
  223: [114688..115199]: 22807696..22808207   512   0x0
  224: [115200..115711]: 21667368..21667879   512   0x0
  225: [115712..116223]: 22870608..22871119   512   0x0
  226: [116224..116735]: 22937760..22938271   512   0x0
  227: [116736..117247]: 22111856..22112367   512   0x0
  228: [117248..118271]: 21618856..21619879  1024   0x0
  229: [118272..118783]: 21592728..21593239   512   0x0
  230: [118784..119295]: 23056312..23056823   512   0x0
  231: [119296..119807]: 21284360..21284871   512   0x0
  232: [119808..120319]: 22465704..22466215   512   0x0
  233: [120320..120831]: 21066336..21066847   512   0x0
  234: [120832..121343]: 22343832..22344343   512   0x0
  235: [121344..121855]: 22392472..22392983   512   0x0
  236: [121856..122367]: 21071216..21071727   512   0x0
  237: [122368..122879]: 22324408..22324919   512   0x0
  238: [122880..123391]: 21327552..21328063   512   0x0
  239: [123392..123903]: 21564200..21564711   512   0x0
  240: [123904..124415]: 21826216..21826727   512   0x0
  241: [124416..124927]: 22905816..22906327   512   0x0
  242: [124928..125439]: 22586360..22586871   512   0x0
  243: [125440..125951]: 21237560..21238071   512   0x0
  244: [125952..126463]: 22641944..22642455   512   0x0
  245: [126464..126975]: 21078776..21079287   512   0x0
  246: [126976..127487]: 22617152..22617663   512   0x0
  247: [127488..127999]: 21234616..21235127   512   0x0
  248: [128000..128511]: 21606672..21607183   512   0x0
  249: [128512..129023]: 21722976..21723487   512   0x0
  250: [129024..129535]: 22427640..22428151   512   0x0
  251: [129536..130047]: 21294200..21294711   512   0x0
  252: [130048..130559]: 22219992..22220503   512   0x0
  253: [130560..131071]: 22811288..22811799   512   0x0
  254: [131072..132095]: 23927696..23928719  1024   0x0
  255: [132096..132607]: 21084104..21084615   512   0x0
  256: [132608..133119]: 22080944..22081455   512   0x0
  257: [133120..133631]: 21412952..21413463   512   0x0
  258: [133632..134143]: 21567528..21568039   512   0x0
  259: [134144..134655]: 21059024..21059535   512   0x0
  260: [134656..135167]: 22872112..22872623   512   0x0
  261: [135168..135679]: 21168952..21169463   512   0x0
  262: [135680..136191]: 22315000..22315511   512   0x0
  263: [136192..136703]: 23116776..23117287   512   0x0
  264: [136704..137215]: 21409832..21410343   512   0x0
  265: [137216..137727]: 22688480..22688991   512   0x0
  266: [137728..138239]: 22349944..22350455   512   0x0
  267: [138240..138751]: 22891632..22892143   512   0x0
  268: [138752..139263]: 21996600..21997111   512   0x0
  269: [139264..139775]: 21603192..21603703   512   0x0
  270: [139776..140799]: 23928720..23929743  1024   0x0
  271: [140800..141311]: 21050072..21050583   512   0x0
  272: [141312..141823]: 22684136..22684647   512   0x0
  273: [141824..142335]: 21446816..21447327   512   0x0
  274: [142336..142847]: 21685224..21685735   512   0x0
  275: [142848..143359]: 22312208..22312719   512   0x0
  276: [143360..143871]: 21287480..21287991   512   0x0
  277: [143872..144383]: 22430560..22431071   512   0x0
  278: [144384..144895]: 21719904..21720415   512   0x0
  279: [144896..145407]: 21114568..21115079   512   0x0
  280: [145408..145919]: 23114016..23114527   512   0x0
  281: [145920..146431]: 22441448..22441959   512   0x0
  282: [146432..146943]: 22611304..22611815   512   0x0
  283: [146944..147455]: 22808208..22808719   512   0x0
  284: [147456..147967]: 21667880..21668391   512   0x0
  285: [147968..148479]: 22871120..22871631   512   0x0
  286: [148480..148991]: 21179632..21180143   512   0x0
  287: [148992..149503]: 22938272..22938783   512   0x0
  288: [149504..150015]: 22231104..22231615   512   0x0
  289: [150016..150527]: 21080456..21080967   512   0x0
  290: [150528..151039]: 22112368..22112879   512   0x0
  291: [151040..151551]: 21593240..21593751   512   0x0
  292: [151552..152063]: 21495184..21495695   512   0x0
  293: [152064..152575]: 22952056..22952567   512   0x0
  294: [152576..153087]: 23056824..23057335   512   0x0
  295: [153088..153599]: 21531352..21531863   512   0x0
  296: [153600..154111]: 21678384..21678895   512   0x0
  297: [154112..154623]: 21762912..21763423   512   0x0
  298: [154624..155135]: 23033240..23033751   512   0x0
  299: [155136..155647]: 22699112..22699623   512   0x0
  300: [155648..156159]: 21248448..21248959   512   0x0
  301: [156160..156671]: 21953552..21954063   512   0x0
  302: [156672..157183]: 21284872..21285383   512   0x0
  303: [157184..157695]: 22403776..22404287   512   0x0
  304: [157696..158207]: 21616512..21617023   512   0x0
  305: [158208..158719]: 22466216..22466727   512   0x0
  306: [158720..159231]: 21083216..21083727   512   0x0
  307: [159232..159743]: 21066848..21067359   512   0x0
  308: [159744..160255]: 22344344..22344855   512   0x0
  309: [160256..163327]: 23929744..23932815  3072   0x0
  310: [163328..163839]: 24648112..24648623   512   0x0
  311: [163840..164351]: 23892640..23893151   512   0x0
  312: [164352..164863]: 23932816..23933327   512   0x0
  313: [164864..165375]: 24648624..24649135   512   0x0
  314: [165376..165887]: 23893152..23893663   512   0x0
  315: [165888..166399]: 23933328..23933839   512   0x0
  316: [166400..166911]: 24649136..24649647   512   0x0
  317: [166912..167423]: 23893664..23894175   512   0x0
  318: [167424..167935]: 24657336..24657847   512   0x0
  319: [167936..168447]: 24638272..24638783   512   0x0
  320: [168448..168959]: 23933840..23934351   512   0x0
  321: [168960..169471]: 24649648..24650159   512   0x0
  322: [169472..169983]: 23894176..23894687   512   0x0
  323: [169984..170495]: 24657848..24658359   512   0x0
  324: [170496..171007]: 24638784..24639295   512   0x0
  325: [171008..171519]: 23934352..23934863   512   0x0
  326: [171520..172031]: 24650160..24650671   512   0x0
  327: [172032..172543]: 23894688..23895199   512   0x0
  328: [172544..173055]: 24658360..24658871   512   0x0
  329: [173056..173567]: 24639296..24639807   512   0x0
  330: [173568..174079]: 23934864..23935375   512   0x0
  331: [174080..174591]: 24650672..24651183   512   0x0
  332: [174592..175103]: 23895200..23895711   512   0x0
  333: [175104..175615]: 24658872..24659383   512   0x0
  334: [175616..176127]: 24639808..24640319   512   0x0
  335: [176128..176639]: 23935376..23935887   512   0x0
  336: [176640..177151]: 24651184..24651695   512   0x0
  337: [177152..177663]: 23895712..23896223   512   0x0
  338: [177664..178175]: 24659384..24659895   512   0x0
  339: [178176..178687]: 24640320..24640831   512   0x0
  340: [178688..179199]: 23935888..23936399   512   0x0
  341: [179200..179711]: 23950248..23950759   512   0x0
  342: [179712..180223]: 24651696..24652207   512   0x0
  343: [180224..180735]: 23896224..23896735   512   0x0
  344: [180736..181247]: 23916888..23917399   512   0x0
  345: [181248..181759]: 24659896..24660407   512   0x0
  346: [181760..182271]: 23902144..23902655   512   0x0
  347: [182272..182783]: 24640832..24641343   512   0x0
  348: [182784..183295]: 23936400..23936911   512   0x0
  349: [183296..183807]: 23950760..23951271   512   0x0
  350: [183808..184319]: 24652208..24652719   512   0x0
  351: [184320..184831]: 23896736..23897247   512   0x0
  352: [184832..185343]: 23917400..23917911   512   0x0
  353: [185344..185855]: 24660408..24660919   512   0x0
  354: [185856..186367]: 23902656..23903167   512   0x0
  355: [186368..186879]: 24641344..24641855   512   0x0
  356: [186880..187391]: 23936912..23937423   512   0x0
  357: [187392..187903]: 23951272..23951783   512   0x0
  358: [187904..188415]: 24652720..24653231   512   0x0
  359: [188416..188927]: 23897248..23897759   512   0x0
  360: [188928..189439]: 23917912..23918423   512   0x0
  361: [189440..189951]: 24660920..24661431   512   0x0
  362: [189952..190463]: 23903168..23903679   512   0x0
  363: [190464..190975]: 24641856..24642367   512   0x0
  364: [190976..191487]: 24038696..24039207   512   0x0
  365: [191488..191999]: 23937424..23937935   512   0x0
  366: [192000..192511]: 23886312..23886823   512   0x0
  367: [192512..193023]: 23951784..23952295   512   0x0
  368: [193024..193535]: 24624208..24624719   512   0x0
  369: [193536..194047]: 24653232..24653743   512   0x0
  370: [194048..194559]: 23897760..23898271   512   0x0
  371: [194560..195071]: 23918424..23918935   512   0x0
  372: [195072..195583]: 23914568..23915079   512   0x0
  373: [195584..196095]: 24661432..24661943   512   0x0
  374: [196096..196607]: 23903680..23904191   512   0x0
  375: [196608..197119]: 24642368..24642879   512   0x0
  376: [197120..197631]: 24039208..24039719   512   0x0
  377: [197632..198143]: 23937936..23938447   512   0x0
  378: [198144..198655]: 23886824..23887335   512   0x0
  379: [198656..199167]: 23952296..23952807   512   0x0
  380: [199168..199679]: 24624720..24625231   512   0x0
  381: [199680..200191]: 24653744..24654255   512   0x0
  382: [200192..200703]: 23898272..23898783   512   0x0
  383: [200704..201215]: 23918936..23919447   512   0x0
  384: [201216..201727]: 23915080..23915591   512   0x0
  385: [201728..202239]: 24661944..24662455   512   0x0
  386: [202240..202751]: 23904192..23904703   512   0x0
  387: [202752..203263]: 24642880..24643391   512   0x0
  388: [203264..203775]: 24039720..24040231   512   0x0
  389: [203776..204287]: 24892160..24892671   512   0x0
  390: [204288..204799]: 23938448..23938959   512   0x0
  391: [204800..205311]: 23635360..23635871   512   0x0
  392: [205312..205823]: 23887336..23887847   512   0x0
  393: [205824..206335]: 23952808..23953319   512   0x0
  394: [206336..206847]: 24625232..24625743   512   0x0
  395: [206848..207359]: 23922512..23923023   512   0x0
  396: [207360..207871]: 24654256..24654767   512   0x0
  397: [207872..208383]: 23898784..23899295   512   0x0
  398: [208384..208895]: 23919448..23919959   512   0x0
  399: [208896..209407]: 24666688..24667199   512   0x0
  400: [209408..209919]: 23915592..23916103   512   0x0
  401: [209920..210431]: 24662456..24662967   512   0x0
  402: [210432..210943]: 23904704..23905215   512   0x0
  403: [210944..211455]: 24643392..24643903   512   0x0
  404: [211456..211967]: 24040232..24040743   512   0x0
  405: [211968..212479]: 23477400..23477911   512   0x0
  406: [212480..212991]: 24892672..24893183   512   0x0
  407: [212992..213503]: 23938960..23939471   512   0x0
  408: [213504..214015]: 23635872..23636383   512   0x0
  409: [214016..214527]: 23887848..23888359   512   0x0
  410: [214528..215039]: 23953320..23953831   512   0x0
  411: [215040..215551]: 24625744..24626255   512   0x0
  412: [215552..216063]: 23923024..23923535   512   0x0
  413: [216064..216575]: 24654768..24655279   512   0x0
  414: [216576..217087]: 23899296..23899807   512   0x0
  415: [217088..217599]: 25475768..25476279   512   0x0
  416: [217600..218111]: 25567712..25568223   512   0x0
  417: [218112..218623]: 25409728..25410239   512   0x0
  418: [218624..219135]: 25309808..25310319   512   0x0
  419: [219136..219647]: 25476280..25476791   512   0x0
  420: [219648..220159]: 25568224..25568735   512   0x0
  421: [220160..220671]: 25410240..25410751   512   0x0
  422: [220672..221183]: 26502880..26503391   512   0x0
  423: [221184..221695]: 25310320..25310831   512   0x0
  424: [221696..222207]: 25476792..25477303   512   0x0
  425: [222208..222719]: 26124504..26125015   512   0x0
  426: [222720..223231]: 26872056..26872567   512   0x0
  427: [223232..223743]: 25568736..25569247   512   0x0
  428: [223744..224255]: 25410752..25411263   512   0x0
  429: [224256..224767]: 26399624..26400135   512   0x0
  430: [224768..225279]: 26104440..26104951   512   0x0
  431: [225280..225791]: 26503392..26503903   512   0x0
  432: [225792..226303]: 25310832..25311343   512   0x0
  433: [226304..226815]: 25477304..25477815   512   0x0
  434: [226816..227327]: 26188568..26189079   512   0x0
  435: [227328..227839]: 25717352..25717863   512   0x0
  436: [227840..228351]: 26986488..26986999   512   0x0
  437: [228352..228863]: 26125016..26125527   512   0x0
  438: [228864..229375]: 26872568..26873079   512   0x0
  439: [229376..229887]: 25689296..25689807   512   0x0
  440: [229888..230399]: 26684400..26684911   512   0x0
  441: [230400..230911]: 27235296..27235807   512   0x0
  442: [230912..231423]: 25569248..25569759   512   0x0
  443: [231424..231935]: 25411264..25411775   512   0x0
  444: [231936..232447]: 25681648..25682159   512   0x0
  445: [232448..232959]: 26400136..26400647   512   0x0
  446: [232960..233471]: 26104952..26105463   512   0x0
  447: [233472..233983]: 26376072..26376583   512   0x0
  448: [233984..234495]: 26501088..26501599   512   0x0
  449: [234496..235007]: 26428392..26428903   512   0x0
  450: [235008..235519]: 25638672..25639183   512   0x0
  451: [235520..236031]: 26841936..26842447   512   0x0
  452: [236032..236543]: 26844728..26845239   512   0x0
  453: [236544..237055]: 26215208..26215719   512   0x0
  454: [237056..237567]: 26413440..26413951   512   0x0
  455: [237568..238079]: 26874568..26875079   512   0x0
  456: [238080..238591]: 26503904..26504415   512   0x0
  457: [238592..239103]: 25623704..25624215   512   0x0
  458: [239104..239615]: 26992208..26992719   512   0x0
  459: [239616..240127]: 25993696..25994207   512   0x0
  460: [240128..240639]: 27005016..27005527   512   0x0
  461: [240640..241151]: 25311344..25311855   512   0x0
  462: [241152..241663]: 25477816..25478327   512   0x0
  463: [241664..242175]: 26331512..26332023   512   0x0
  464: [242176..242687]: 26189080..26189591   512   0x0
  465: [242688..243199]: 25407936..25408447   512   0x0
  466: [243200..251903]: 29021776..29030479  8704   0x0
  467: [251904..252415]: 28872544..28873055   512   0x0
  468: [252416..252927]: 29297240..29297751   512   0x0
  469: [252928..253439]: 29030480..29030991   512   0x0
  470: [253440..253951]: 28873056..28873567   512   0x0
  471: [253952..254463]: 29297752..29298263   512   0x0
  472: [254464..254975]: 29030992..29031503   512   0x0
  473: [254976..255487]: 28873568..28874079   512   0x0
  474: [255488..255999]: 29298264..29298775   512   0x0
  475: [256000..256511]: 29031504..29032015   512   0x0
  476: [256512..257023]: 28874080..28874591   512   0x0
  477: [257024..257535]: 28612088..28612599   512   0x0
  478: [257536..258047]: 29298776..29299287   512   0x0
  479: [258048..258559]: 29032016..29032527   512   0x0
  480: [258560..259071]: 28874592..28875103   512   0x0
  481: [259072..259583]: 28612600..28613111   512   0x0
  482: [259584..260095]: 29299288..29299799   512   0x0
  483: [260096..260607]: 29032528..29033039   512   0x0
  484: [260608..261119]: 28875104..28875615   512   0x0
  485: [261120..261631]: 28613112..28613623   512   0x0
  486: [261632..262143]: 29299800..29300311   512   0x0
  487: [262144..262655]: 29033040..29033551   512   0x0
  488: [262656..263167]: 28875616..28876127   512   0x0
  489: [263168..263679]: 28613624..28614135   512   0x0
  490: [263680..264191]: 29300312..29300823   512   0x0
  491: [264192..264703]: 29033552..29034063   512   0x0
  492: [264704..265215]: 28876128..28876639   512   0x0
  493: [265216..265727]: 28614136..28614647   512   0x0
  494: [265728..266239]: 27729832..27730343   512   0x0
  495: [266240..266751]: 28700920..28701431   512   0x0
  496: [266752..267263]: 29300824..29301335   512   0x0
  497: [267264..267775]: 27679200..27679711   512   0x0
  498: [267776..268287]: 27389096..27389607   512   0x0
  499: [268288..268799]: 29147000..29147511   512   0x0
  500: [268800..269311]: 29034064..29034575   512   0x0
  501: [269312..269823]: 28876640..28877151   512   0x0
  502: [269824..270335]: 29019776..29020287   512   0x0
  503: [270336..270847]: 28614648..28615159   512   0x0
  504: [270848..271359]: 29239832..29240343   512   0x0
  505: [271360..271871]: 27730344..27730855   512   0x0
  506: [271872..272383]: 28701432..28701943   512   0x0
  507: [272384..272895]: 29301336..29301847   512   0x0
  508: [272896..273407]: 27679712..27680223   512   0x0
  509: [273408..273919]: 27389608..27390119   512   0x0
  510: [273920..274431]: 29147512..29148023   512   0x0
  511: [274432..274943]: 29034576..29035087   512   0x0
  512: [274944..275455]: 27489120..27489631   512   0x0
  513: [275456..275967]: 28877152..28877663   512   0x0
  514: [275968..276479]: 29020288..29020799   512   0x0
  515: [276480..276991]: 28615160..28615671   512   0x0
  516: [276992..277503]: 29240344..29240855   512   0x0
  517: [277504..278015]: 27730856..27731367   512   0x0
  518: [278016..278527]: 28701944..28702455   512   0x0
  519: [278528..279039]: 29301848..29302359   512   0x0
  520: [279040..280063]: 30184504..30185527  1024   0x0
  521: [280064..280575]: 30379504..30380015   512   0x0
  522: [280576..281087]: 30447392..30447903   512   0x0
  523: [281088..281599]: 30185528..30186039   512   0x0
  524: [281600..282111]: 30380016..30380527   512   0x0
  525: [282112..282623]: 30447904..30448415   512   0x0
  526: [282624..283135]: 29611968..29612479   512   0x0
  527: [283136..283647]: 30186040..30186551   512   0x0
  528: [283648..284159]: 30380528..30381039   512   0x0
  529: [284160..284671]: 30448416..30448927   512   0x0
  530: [284672..285183]: 29612480..29612991   512   0x0
  531: [285184..285695]: 30186552..30187063   512   0x0
  532: [285696..286207]: 30381040..30381551   512   0x0
  533: [286208..286719]: 30448928..30449439   512   0x0
  534: [286720..287231]: 29612992..29613503   512   0x0
  535: [287232..287743]: 30187064..30187575   512   0x0
  536: [287744..288255]: 30381552..30382063   512   0x0
  537: [288256..288767]: 30449440..30449951   512   0x0
  538: [288768..289279]: 29613504..29614015   512   0x0
  539: [289280..289791]: 30187576..30188087   512   0x0
  540: [289792..290303]: 30382064..30382575   512   0x0
  541: [290304..290815]: 30449952..30450463   512   0x0
  542: [290816..291327]: 29614016..29614527   512   0x0
  543: [291328..291839]: 30188088..30188599   512   0x0
  544: [291840..292351]: 30382576..30383087   512   0x0
  545: [292352..292863]: 30450464..30450975   512   0x0
  546: [292864..293375]: 29614528..29615039   512   0x0
  547: [293376..293887]: 30315896..30316407   512   0x0
  548: [293888..294399]: 30188600..30189111   512   0x0
  549: [294400..294911]: 30383088..30383599   512   0x0
  550: [294912..295423]: 30450976..30451487   512   0x0
  551: [295424..295935]: 29615040..29615551   512   0x0
  552: [295936..296447]: 30316408..30316919   512   0x0
  553: [296448..296959]: 30189112..30189623   512   0x0
  554: [296960..297471]: 30383600..30384111   512   0x0
  555: [297472..297983]: 30451488..30451999   512   0x0
  556: [297984..300543]: 32891744..32894303  2560   0x0
  557: [300544..301055]: 33570880..33571391   512   0x0
  558: [301056..301567]: 32894304..32894815   512   0x0
  559: [301568..302079]: 33571392..33571903   512   0x0
  560: [302080..302591]: 32894816..32895327   512   0x0
  561: [302592..303103]: 33571904..33572415   512   0x0
  562: [303104..303615]: 32895328..32895839   512   0x0
  563: [303616..304127]: 33572416..33572927   512   0x0
  564: [304128..304639]: 32895840..32896351   512   0x0
  565: [304640..305151]: 33572928..33573439   512   0x0
  566: [305152..305663]: 32896352..32896863   512   0x0
  567: [305664..306175]: 33573440..33573951   512   0x0
  568: [306176..306687]: 32896864..32897375   512   0x0
  569: [306688..307199]: 33573952..33574463   512   0x0
  570: [307200..307711]: 32897376..32897887   512   0x0
  571: [307712..308223]: 31963960..31964471   512   0x0
  572: [308224..308735]: 33574464..33574975   512   0x0
  573: [308736..309247]: 41524800..41525311   512   0x0
  574: [309248..309759]: 41960416..41960927   512   0x0
  575: [309760..310271]: 41589776..41590287   512   0x0
  576: [310272..310783]: 40839608..40840119   512   0x0
  577: [310784..311295]: 41525312..41525823   512   0x0
  578: [311296..311807]: 41960928..41961439   512   0x0
  579: [311808..312319]: 41590288..41590799   512   0x0
  580: [312320..312831]: 40840120..40840631   512   0x0
  581: [312832..313343]: 41525824..41526335   512   0x0
  582: [313344..313855]: 41961440..41961951   512   0x0
  583: [313856..314367]: 41590800..41591311   512   0x0
  584: [314368..314879]: 40840632..40841143   512   0x0
  585: [314880..315391]: 41784272..41784783   512   0x0
  586: [315392..315903]: 41526336..41526847   512   0x0
  587: [315904..316415]: 41595760..41596271   512   0x0
  588: [316416..316927]: 41961952..41962463   512   0x0
  589: [316928..317439]: 41591312..41591823   512   0x0
  590: [317440..317951]: 40841144..40841655   512   0x0
  591: [317952..318463]: 41784784..41785295   512   0x0
  592: [318464..318975]: 41526848..41527359   512   0x0
  593: [318976..319487]: 41596272..41596783   512   0x0
  594: [319488..319999]: 41962464..41962975   512   0x0
  595: [320000..320511]: 41591824..41592335   512   0x0
  596: [320512..321023]: 40841656..40842167   512   0x0
  597: [321024..321535]: 41785296..41785807   512   0x0
  598: [321536..322047]: 41451544..41452055   512   0x0
  599: [322048..322559]: 41527360..41527871   512   0x0
  600: [322560..323071]: 41596784..41597295   512   0x0
  601: [323072..323583]: 41962976..41963487   512   0x0
  602: [323584..324095]: 41600424..41600935   512   0x0
  603: [324096..324607]: 41592336..41592847   512   0x0
  604: [324608..325119]: 40842168..40842679   512   0x0
  605: [325120..325631]: 41785808..41786319   512   0x0
  606: [325632..326143]: 41452056..41452567   512   0x0
  607: [326144..326655]: 41527872..41528383   512   0x0
  608: [326656..327167]: 41597296..41597807   512   0x0
  609: [327168..327679]: 41963488..41963999   512   0x0
  610: [327680..328191]: 41600936..41601447   512   0x0
  611: [328192..328703]: 41592848..41593359   512   0x0
  612: [328704..329215]: 40990472..40990983   512   0x0
  613: [329216..329727]: 41271880..41272391   512   0x0
  614: [329728..330239]: 41438976..41439487   512   0x0
  615: [330240..330751]: 40842680..40843191   512   0x0
  616: [330752..331263]: 41873688..41874199   512   0x0
  617: [331264..331775]: 41786320..41786831   512   0x0
  618: [331776..332287]: 40027960..40028471   512   0x0
  619: [332288..332799]: 41651928..41652439   512   0x0
  620: [332800..333311]: 41452568..41453079   512   0x0
  621: [333312..333823]: 41528384..41528895   512   0x0
  622: [333824..334335]: 41898280..41898791   512   0x0
  623: [334336..334847]: 40035688..40036199   512   0x0
  624: [334848..335359]: 41534936..41535447   512   0x0
  625: [335360..335871]: 41597808..41598319   512   0x0
  626: [335872..336383]: 41964000..41964511   512   0x0
  627: [336384..336895]: 41097552..41098063   512   0x0
  628: [336896..337407]: 40746288..40746799   512   0x0
  629: [337408..337919]: 41601448..41601959   512   0x0
  630: [337920..338431]: 41378544..41379055   512   0x0
  631: [338432..338943]: 41593360..41593871   512   0x0
  632: [338944..339455]: 40990984..40991495   512   0x0
  633: [339456..339967]: 41968536..41969047   512   0x0
  634: [339968..340479]: 41272392..41272903   512   0x0
  635: [340480..340991]: 41439488..41439999   512   0x0
  636: [340992..341503]: 40277504..40278015   512   0x0
  637: [341504..342015]: 40843192..40843703   512   0x0
  638: [342016..342527]: 41915264..41915775   512   0x0
  639: [342528..343039]: 41874200..41874711   512   0x0
  640: [343040..343551]: 40448704..40449215   512   0x0
  641: [343552..344063]: 40758976..40759487   512   0x0
  642: [344064..344575]: 41786832..41787343   512   0x0
  643: [344576..345087]: 40028472..40028983   512   0x0
  644: [345088..345599]: 41652440..41652951   512   0x0
  645: [345600..346111]: 41453080..41453591   512   0x0
  646: [346112..346623]: 41528896..41529407   512   0x0
  647: [346624..347135]: 40217400..40217911   512   0x0
  648: [347136..347647]: 41443864..41444375   512   0x0
  649: [347648..348159]: 41898792..41899303   512   0x0
  650: [348160..348671]: 40036200..40036711   512   0x0
  651: [348672..349183]: 41535448..41535959   512   0x0
  652: [349184..349695]: 41598320..41598831   512   0x0
  653: [349696..350207]: 41964512..41965023   512   0x0
  654: [350208..350719]: 41098064..41098575   512   0x0
  655: [350720..351231]: 40470248..40470759   512   0x0
  656: [351232..351743]: 40746800..40747311   512   0x0
  657: [351744..352255]: 40872232..40872743   512   0x0
  658: [352256..352767]: 40764248..40764759   512   0x0
  659: [352768..353279]: 41601960..41602471   512   0x0
  660: [353280..353791]: 41379056..41379567   512   0x0
  661: [353792..354303]: 41593872..41594383   512   0x0
  662: [354304..354815]: 40991496..40992007   512   0x0
  663: [354816..355327]: 41969048..41969559   512   0x0
  664: [355328..355839]: 41272904..41273415   512   0x0
  665: [355840..356351]: 41440000..41440511   512   0x0
  666: [356352..356863]: 40278016..40278527   512   0x0
  667: [356864..357375]: 40282848..40283359   512   0x0
  668: [357376..357887]: 40843704..40844215   512   0x0
  669: [357888..358399]: 41915776..41916287   512   0x0
  670: [358400..358911]: 40863568..40864079   512   0x0
  671: [358912..359423]: 41823336..41823847   512   0x0
  672: [359424..359935]: 41874712..41875223   512   0x0
  673: [359936..360447]: 40449216..40449727   512   0x0
  674: [360448..360959]: 40597816..40598327   512   0x0
  675: [360960..361471]: 40759488..40759999   512   0x0
  676: [361472..361983]: 41316432..41316943   512   0x0
  677: [361984..362495]: 40106288..40106799   512   0x0
  678: [362496..363007]: 41787344..41787855   512   0x0
  679: [363008..363519]: 40028984..40029495   512   0x0
  680: [363520..364031]: 41652952..41653463   512   0x0
  681: [364032..364543]: 41453592..41454103   512   0x0
  682: [364544..365055]: 41529408..41529919   512   0x0
  683: [365056..365567]: 40217912..40218423   512   0x0
  684: [365568..366079]: 41444376..41444887   512   0x0
  685: [366080..366591]: 40238248..40238759   512   0x0
  686: [366592..367103]: 41899304..41899815   512   0x0
  687: [367104..367615]: 40977312..40977823   512   0x0
  688: [367616..368127]: 40036712..40037223   512   0x0
  689: [368128..368639]: 41535960..41536471   512   0x0
  690: [368640..369151]: 40232424..40232935   512   0x0
  691: [369152..369663]: 41428608..41429119   512   0x0
  692: [369664..370175]: 41598832..41599343   512   0x0
  693: [370176..370687]: 41965024..41965535   512   0x0
  694: [370688..371199]: 41098576..41099087   512   0x0
  695: [371200..371711]: 40470760..40471271   512   0x0
  696: [371712..372223]: 40747312..40747823   512   0x0
  697: [372224..372735]: 40929888..40930399   512   0x0
  698: [372736..373247]: 40872744..40873255   512   0x0
  699: [373248..373759]: 40764760..40765271   512   0x0
  700: [373760..374271]: 41602472..41602983   512   0x0
  701: [374272..374783]: 41379568..41380079   512   0x0
  702: [374784..375295]: 41594384..41594895   512   0x0
  703: [375296..375807]: 40643960..40644471   512   0x0
  704: [375808..376319]: 40478632..40479143   512   0x0
  705: [376320..376831]: 40992008..40992519   512   0x0
  706: [376832..377343]: 40964216..40964727   512   0x0
  707: [377344..377855]: 41450224..41450735   512   0x0
  708: [377856..378367]: 41969560..41970071   512   0x0
  709: [378368..378879]: 41273416..41273927   512   0x0
  710: [378880..379391]: 40837280..40837791   512   0x0
  711: [379392..379903]: 41440512..41441023   512   0x0
  712: [379904..380415]: 40415224..40415735   512   0x0
  713: [380416..380927]: 40278528..40279039   512   0x0
  714: [380928..381439]: 40283360..40283871   512   0x0
  715: [381440..381951]: 40844216..40844727   512   0x0
  716: [381952..382463]: 41220200..41220711   512   0x0
  717: [382464..382975]: 41916288..41916799   512   0x0
  718: [382976..383487]: 40860256..40860767   512   0x0
  719: [383488..383999]: 39948008..39948519   512   0x0
  720: [384000..384511]: 40864080..40864591   512   0x0
  721: [384512..385023]: 41823848..41824359   512   0x0
  722: [385024..385535]: 41875224..41875735   512   0x0
  723: [385536..386047]: 40905368..40905879   512   0x0
  724: [386048..386559]: 41268600..41269111   512   0x0
  725: [386560..387071]: 41799352..41799863   512   0x0
  726: [387072..387583]: 40575928..40576439   512   0x0
  727: [387584..388095]: 41197520..41198031   512   0x0
  728: [388096..388607]: 40449728..40450239   512   0x0
  729: [388608..389119]: 41949000..41949511   512   0x0
  730: [389120..389631]: 40598328..40598839   512   0x0
  731: [389632..390143]: 40760000..40760511   512   0x0
  732: [390144..390655]: 41316944..41317455   512   0x0
  733: [390656..391167]: 41092320..41092831   512   0x0
  734: [391168..391679]: 40106800..40107311   512   0x0
  735: [391680..392191]: 41787856..41788367   512   0x0
  736: [392192..392703]: 40029496..40030007   512   0x0
  737: [392704..393215]: 41610488..41610999   512   0x0
  738: [393216..393727]: 41736560..41737071   512   0x0
  739: [393728..394239]: 41653464..41653975   512   0x0
  740: [394240..394751]: 41454104..41454615   512   0x0
  741: [394752..395263]: 40612216..40612727   512   0x0
  742: [395264..395775]: 41095496..41096007   512   0x0
  743: [395776..396287]: 41529920..41530431   512   0x0
  744: [396288..396799]: 41475544..41476055   512   0x0
  745: [396800..397311]: 40218424..40218935   512   0x0
  746: [397312..397823]: 41444888..41445399   512   0x0
  747: [397824..398335]: 40393408..40393919   512   0x0
  748: [398336..398847]: 40238760..40239271   512   0x0
  749: [398848..399359]: 41899816..41900327   512   0x0
  750: [399360..399871]: 40076552..40077063   512   0x0
  751: [399872..400383]: 40977824..40978335   512   0x0
  752: [400384..400895]: 40037224..40037735   512   0x0
  753: [400896..401407]: 41536472..41536983   512   0x0
  754: [401408..401919]: 40042312..40042823   512   0x0
  755: [401920..402431]: 40280848..40281359   512   0x0
  756: [402432..402943]: 40693776..40694287   512   0x0
  757: [402944..403455]: 41324256..41324767   512   0x0
  758: [403456..403967]: 40812752..40813263   512   0x0
  759: [403968..404479]: 41214544..41215055   512   0x0
  760: [404480..404991]: 39937024..39937535   512   0x0
  761: [404992..405503]: 40139880..40140391   512   0x0
  762: [405504..406015]: 41015120..41015631   512   0x0
  763: [406016..406527]: 40232936..40233447   512   0x0
  764: [406528..407039]: 41429120..41429631   512   0x0
  765: [407040..407551]: 41599344..41599855   512   0x0
  766: [407552..408063]: 41965536..41966047   512   0x0
  767: [408064..408575]: 40944608..40945119   512   0x0
  768: [408576..409087]: 40830200..40830711   512   0x0
  769: [409088..409599]: 41099088..41099599   512   0x0
  770: [409600..410111]: 40589432..40589943   512   0x0
  771: [410112..410623]: 41531976..41532487   512   0x0
  772: [410624..411135]: 40471272..40471783   512   0x0
  773: [411136..411647]: 40747824..40748335   512   0x0
  774: [411648..412159]: 40465256..40465767   512   0x0
  775: [412160..412671]: 40561720..40562231   512   0x0
  776: [412672..413183]: 41802664..41803175   512   0x0
  777: [413184..413695]: 40930400..40930911   512   0x0
  778: [413696..414207]: 40873256..40873767   512   0x0
  779: [414208..414719]: 39961272..39961783   512   0x0
  780: [414720..415743]: 44592672..44593695  1024   0x0
  781: [415744..416255]: 39981600..39982111   512   0x0
  782: [416256..416767]: 40765272..40765783   512   0x0
  783: [416768..417279]: 41588640..41589151   512   0x0
  784: [417280..417791]: 40724408..40724919   512   0x0
  785: [417792..418303]: 41602984..41603495   512   0x0
  786: [418304..418815]: 41380080..41380591   512   0x0
  787: [418816..419327]: 41594896..41595407   512   0x0
  788: [419328..419839]: 40242832..40243343   512   0x0
  789: [419840..420351]: 41184512..41185023   512   0x0
  790: [420352..420863]: 41985208..41985719   512   0x0
  791: [420864..421375]: 40644472..40644983   512   0x0
  792: [421376..421887]: 40479144..40479655   512   0x0
  793: [421888..422399]: 40992520..40993031   512   0x0
  794: [422400..422911]: 40407232..40407743   512   0x0
  795: [422912..423423]: 40964728..40965239   512   0x0
  796: [423424..423935]: 39942072..39942583   512   0x0
  797: [423936..424447]: 41466536..41467047   512   0x0
  798: [424448..424959]: 40225592..40226103   512   0x0
  799: [424960..425471]: 40551696..40552207   512   0x0
  800: [425472..425983]: 40128472..40128983   512   0x0
  801: [425984..427007]: 44593696..44594719  1024   0x0
  802: [427008..427519]: 41450736..41451247   512   0x0
  803: [427520..428543]: 44594720..44595743  1024   0x0
  804: [428544..429055]: 41970072..41970583   512   0x0
  805: [429056..429567]: 41273928..41274439   512   0x0
  806: [429568..430079]: 41361216..41361727   512   0x0
  807: [430080..430591]: 41689184..41689695   512   0x0
  808: [430592..431103]: 40837792..40838303   512   0x0
  809: [431104..431615]: 41441024..41441535   512   0x0
  810: [431616..432127]: 40519160..40519671   512   0x0
  811: [432128..432639]: 40415736..40416247   512   0x0
  812: [432640..433151]: 40279040..40279551   512   0x0
  813: [433152..433663]: 40283872..40284383   512   0x0
  814: [433664..434175]: 40844728..40845239   512   0x0
  815: [434176..434687]: 41164672..41165183   512   0x0
  816: [434688..435199]: 41220712..41221223   512   0x0
  817: [435200..435711]: 41916800..41917311   512   0x0
  818: [435712..436223]: 39957440..39957951   512   0x0
  819: [436224..437247]: 45496648..45497671  1024   0x0
  820: [437248..437759]: 40927080..40927591   512   0x0
  821: [437760..438783]: 44595744..44596767  1024   0x0
  822: [438784..440319]: 45497672..45499207  1536   0x0
  823: [440320..440831]: 41090536..41091047   512   0x0
  824: [440832..441343]: 40336616..40337127   512   0x0
  825: [441344..441855]: 40860768..40861279   512   0x0
  826: [441856..442879]: 44596768..44597791  1024   0x0
  827: [442880..443391]: 40289768..40290279   512   0x0
  828: [443392..444415]: 44275392..44276415  1024   0x0
  829: [444416..444927]: 41137328..41137839   512   0x0
  830: [444928..445951]: 44597792..44598815  1024   0x0
  831: [445952..446463]: 41918592..41919103   512   0x0
  832: [446464..447487]: 45499208..45500231  1024   0x0
  833: [447488..447999]: 40787120..40787631   512   0x0
  834: [448000..449023]: 44577400..44578423  1024   0x0
  835: [449024..449535]: 39948520..39949031   512   0x0
  836: [449536..450047]: 40864592..40865103   512   0x0
  837: [450048..450559]: 41824360..41824871   512   0x0
  838: [450560..451071]: 41875736..41876247   512   0x0
  839: [451072..451583]: 40905880..40906391   512   0x0
  840: [451584..452095]: 41269112..41269623   512   0x0
  841: [452096..452607]: 41435192..41435703   512   0x0
  842: [452608..453119]: 42001720..42002231   512   0x0
  843: [453120..453631]: 41799864..41800375   512   0x0
  844: [453632..454143]: 40637832..40638343   512   0x0
  845: [454144..454655]: 41929600..41930111   512   0x0
  846: [454656..455167]: 40576440..40576951   512   0x0
  847: [455168..455679]: 41198032..41198543   512   0x0
  848: [455680..456191]: 40431640..40432151   512   0x0
  849: [456192..456703]: 40450240..40450751   512   0x0
  850: [456704..457215]: 40539664..40540175   512   0x0
  851: [457216..457727]: 41821640..41822151   512   0x0
  852: [457728..458239]: 41889416..41889927   512   0x0
  853: [458240..458751]: 40254464..40254975   512   0x0
  854: [458752..459263]: 41616880..41617391   512   0x0
  855: [459264..459775]: 41641960..41642471   512   0x0
  856: [459776..460287]: 41949512..41950023   512   0x0
  857: [460288..460303]: 546895240..546895255    16   0x1

After defrag:

# compsize mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
Processed 1 file, 3 regular extents (3 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      449M         449M         224M
none       100%      449M         449M         224M

# xfs_io -c "fiemap -v" mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
  EXT: FILE-OFFSET      BLOCK-RANGE           TOTAL FLAGS
    0: [0..262143]:     406841344..407103487 262144   0x0
    1: [262144..460287]: 436002368..436200511 198144   0x0
    2: [460288..460303]: 546895240..546895255     16   0x1

In fact fiemap "TOTAL" adds up correctly to the actual file size here.
So maybe it is actually compsize lying with "Disk Usage" or something else weird happening.

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Hanabishi]

On 8/6/24 12:17, Hanabishi wrote:

> In fact fiemap "TOTAL" adds up correctly to the actual file size here.
> So maybe it is actually compsize lying with "Disk Usage" or something else weird happening.

I reproduced the results on a dedicated disk.
No, compsize is not lying. Confirmed by looking at total fs usage.

# compsize mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
Processed 1 file, 3 regular extents (3 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL      100%      449M         449M         224M
none       100%      449M         449M         224M

# btrfs filesystem usage /mnt
Overall:
     Device size:		 29.88GiB
     Device allocated:		  1.52GiB
     Device unallocated:		 28.36GiB
     Device missing:		    0.00B
     Device slack:		    0.00B
     Used:			450.82MiB
     Free (estimated):		 28.92GiB	(min: 14.74GiB)
     Free (statfs, df):		 28.92GiB
     Data ratio:			     1.00
     Metadata ratio:		     2.00
     Global reserve:		  5.50MiB	(used: 16.00KiB)
     Multiple profiles:		       no

Data,single: Size:1.00GiB, Used:449.51MiB (43.90%)
    /dev/sdc1	  1.00GiB

Metadata,DUP: Size:256.00MiB, Used:656.00KiB (0.25%)
    /dev/sdc1	512.00MiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
    /dev/sdc1	 16.00MiB

Unallocated:
    /dev/sdc1	 28.36GiB

Notice that the space overhead does *not* belong to metadata. It is the actual data space wasted. So the problem is real.
Which also means that fiemap is the one who lies here.

# xfs_io -c "fiemap -v" mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
  EXT: FILE-OFFSET      BLOCK-RANGE       TOTAL FLAGS
    0: [0..460287]:     7335440..7795727 460288   0x0
    1: [460288..460303]: 7335424..7335439     16   0x1

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Qu Wenruo]

在 2024/8/6 22:52, Hanabishi 写道:
> On 8/6/24 12:17, Hanabishi wrote:
> 
>> In fact fiemap "TOTAL" adds up correctly to the actual file size here.
>> So maybe it is actually compsize lying with "Disk Usage" or something 
>> else weird happening.
> 
> I reproduced the results on a dedicated disk.
> No, compsize is not lying. Confirmed by looking at total fs usage.
> 
> # compsize mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
> Processed 1 file, 3 regular extents (3 refs), 0 inline.
> Type       Perc     Disk Usage   Uncompressed Referenced
> TOTAL      100%      449M         449M         224M
> none       100%      449M         449M         224M
> 
> # btrfs filesystem usage /mnt
> Overall:
>      Device size:         29.88GiB
>      Device allocated:          1.52GiB
>      Device unallocated:         28.36GiB
>      Device missing:            0.00B
>      Device slack:            0.00B
>      Used:            450.82MiB
>      Free (estimated):         28.92GiB    (min: 14.74GiB)
>      Free (statfs, df):         28.92GiB
>      Data ratio:                 1.00
>      Metadata ratio:             2.00
>      Global reserve:          5.50MiB    (used: 16.00KiB)
>      Multiple profiles:               no
> 
> Data,single: Size:1.00GiB, Used:449.51MiB (43.90%)
>     /dev/sdc1      1.00GiB
> 
> Metadata,DUP: Size:256.00MiB, Used:656.00KiB (0.25%)
>     /dev/sdc1    512.00MiB
> 
> System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
>     /dev/sdc1     16.00MiB
> 
> Unallocated:
>     /dev/sdc1     28.36GiB
> 
> Notice that the space overhead does *not* belong to metadata. It is the 
> actual data space wasted. So the problem is real.
> Which also means that fiemap is the one who lies here.
> 
> # xfs_io -c "fiemap -v" mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst
> mingw-w64-gcc-13.1.0-1-x86_64.pkg.tar.zst:
>   EXT: FILE-OFFSET      BLOCK-RANGE       TOTAL FLAGS
>     0: [0..460287]:     7335440..7795727 460288   0x0
>     1: [460288..460303]: 7335424..7335439     16   0x1
> 
> 

I'm pretty sure it's the last extent causing the problem.
It's still referring to the old large preallocated extent, as btrfs can 
only free the whole extent when every part of it has no one referring to.

But since it's the last extent, it doesn't get fully defragged because 
we believe it can not be merged with other extents.

In that case, preallocation with COW is causing the problem.
If you sync the file without preallocation (but with COW), defrag should 
work fine.

Or if you sync the file with preallocation but without COW, defrag 
should also work fine (well, in that case you won't even need to defrag).


There is an attempt to enforce defragging for such preallocated extents, 
but not yet merged upstream due to interface change:

https://lore.kernel.org/linux-btrfs/cover.1710213625.git.wqu@suse.com/T/

Thanks,
Qu

Re: 'btrfs filesystem defragment' makes files explode in size, especially fallocated ones

[Hanabishi]

On 8/6/24 22:18, Qu Wenruo wrote:

> There is an attempt to enforce defragging for such preallocated extents, but not yet merged upstream due to interface change:
> 
> https://lore.kernel.org/linux-btrfs/cover.1710213625.git.wqu@suse.com/T/

So, can we count that you are aware of the problem at this point? Because the main goal of my report is to make devs aware.

For me personally "Don't use defragment. Ever." advice is enough. In real need of defragmentation I can manually copy files without reflinking (rsync never reflinks anyway) or from another drive. Not a big deal really.