btrfs fi du is unreliable

btrfs fi du is unreliable

[Chris Murphy]

Hi,

[chris@f25s ~]$ uname -r
4.8.11-300.fc25.x86_64
[chris@f25s ~]$ rpm -q btrfs-progs
btrfs-progs-4.8.5-1.fc26.x86_64


I'm not finding any pattern to this so far, but it's definitely not
always reliable. Here is today's example.

[chris@f25s ~]$ sudo btrfs fi du -s /mnt/second/jackson.2015/
     Total   Exclusive  Set shared  Filename
 111.67GiB   111.67GiB       0.00B  /mnt/second/jackson.2015/

Super. This is correct. The jackson.2015 subvolume contains only
exclusively used data, there are no snapshots, and there are no
reflinks involved for any files.


[chris@f25s ~]$ sudo btrfs send  /mnt/second/jackson.2015/ | sudo
btrfs receive /mnt/int/

Send the subvolume to a different btrfs volume. This is not an
incremental send. After completion however:


[chris@f25s ~]$ sudo btrfs fi du -s /mnt/int/jackson.2015/
     Total   Exclusive  Set shared  Filename
 111.67GiB    37.20GiB    29.93GiB  /mnt/int/jackson.2015/

That makes zero sense. What's going on here?


Chris Murphy

Re: btrfs fi du is unreliable

[Chris Murphy]

Another example:

[chris@f25s ~]$ sudo btrfs fi du -s /mnt/first/everything-new
     Total   Exclusive  Set shared  Filename
 367.54GiB    14.21GiB   350.95GiB  /mnt/first/everything-new

The problem here is that Exclusive + Shared ≠ Total. Rather those two
add up to 365.16GiB, which suggests 2.38GiB is neither shared nor
exclusive. What would that be? The total not equalling shared +
exclusive is the most common bug I see with fi du.



---
Chris Murphy

Re: btrfs fi du is unreliable

[Henk Slager]

On Sun, Dec 4, 2016 at 7:30 PM, Chris Murphy <lists@colorremedies.com> wrote:
> Hi,
>
> [chris@f25s ~]$ uname -r
> 4.8.11-300.fc25.x86_64
> [chris@f25s ~]$ rpm -q btrfs-progs
> btrfs-progs-4.8.5-1.fc26.x86_64
>
>
> I'm not finding any pattern to this so far, but it's definitely not
> always reliable. Here is today's example.
>
> [chris@f25s ~]$ sudo btrfs fi du -s /mnt/second/jackson.2015/
>      Total   Exclusive  Set shared  Filename
>  111.67GiB   111.67GiB       0.00B  /mnt/second/jackson.2015/
>
> Super. This is correct. The jackson.2015 subvolume contains only
> exclusively used data, there are no snapshots, and there are no
> reflinks involved for any files.
>
>
> [chris@f25s ~]$ sudo btrfs send  /mnt/second/jackson.2015/ | sudo
> btrfs receive /mnt/int/
>
> Send the subvolume to a different btrfs volume. This is not an
> incremental send. After completion however:
>
>
> [chris@f25s ~]$ sudo btrfs fi du -s /mnt/int/jackson.2015/
>      Total   Exclusive  Set shared  Filename
>  111.67GiB    37.20GiB    29.93GiB  /mnt/int/jackson.2015/
>
> That makes zero sense. What's going on here?

I tried to reproduce, with progs from git v4.8.5 and kernel
4.8.10-1-default (tumbleweed):

# /net/src/btrfs-progs/btrfs send testfidu |
/net/src/btrfs-progs/btrfs receive /local/omedia/

# /net/src/btrfs-progs/btrfs fi du -s ./testfidu/
     Total   Exclusive  Set shared  Filename
  42.56GiB    42.56GiB       0.00B  ./testfidu/

# /net/src/btrfs-progs/btrfs fi du -s /local/omedia/testfidu/
     Total   Exclusive  Set shared  Filename
  42.56GiB    42.56GiB       0.00B  /local/omedia/testfidu/

There are no btrfs changes between kernels 4.8.10 and 4.8.11. There is
no compress mount option in my case, that is the only thing I
currently can think of that could make your Set shared number
non-zero.

Re: btrfs fi du is unreliable

[Chris Murphy]

On Sun, Dec 4, 2016 at 3:17 PM, Henk Slager <eye1tm@gmail.com> wrote:
> On Sun, Dec 4, 2016 at 7:30 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> Hi,
>>
>> [chris@f25s ~]$ uname -r
>> 4.8.11-300.fc25.x86_64
>> [chris@f25s ~]$ rpm -q btrfs-progs
>> btrfs-progs-4.8.5-1.fc26.x86_64
>>
>>
>> I'm not finding any pattern to this so far, but it's definitely not
>> always reliable. Here is today's example.
>>
>> [chris@f25s ~]$ sudo btrfs fi du -s /mnt/second/jackson.2015/
>>      Total   Exclusive  Set shared  Filename
>>  111.67GiB   111.67GiB       0.00B  /mnt/second/jackson.2015/
>>
>> Super. This is correct. The jackson.2015 subvolume contains only
>> exclusively used data, there are no snapshots, and there are no
>> reflinks involved for any files.
>>
>>
>> [chris@f25s ~]$ sudo btrfs send  /mnt/second/jackson.2015/ | sudo
>> btrfs receive /mnt/int/
>>
>> Send the subvolume to a different btrfs volume. This is not an
>> incremental send. After completion however:
>>
>>
>> [chris@f25s ~]$ sudo btrfs fi du -s /mnt/int/jackson.2015/
>>      Total   Exclusive  Set shared  Filename
>>  111.67GiB    37.20GiB    29.93GiB  /mnt/int/jackson.2015/
>>
>> That makes zero sense. What's going on here?
>
> I tried to reproduce, with progs from git v4.8.5 and kernel
> 4.8.10-1-default (tumbleweed):
>
> # /net/src/btrfs-progs/btrfs send testfidu |
> /net/src/btrfs-progs/btrfs receive /local/omedia/
>
> # /net/src/btrfs-progs/btrfs fi du -s ./testfidu/
>      Total   Exclusive  Set shared  Filename
>   42.56GiB    42.56GiB       0.00B  ./testfidu/
>
> # /net/src/btrfs-progs/btrfs fi du -s /local/omedia/testfidu/
>      Total   Exclusive  Set shared  Filename
>   42.56GiB    42.56GiB       0.00B  /local/omedia/testfidu/
>
> There are no btrfs changes between kernels 4.8.10 and 4.8.11. There is
> no compress mount option in my case, that is the only thing I
> currently can think of that could make your Set shared number
> non-zero.

No compression on either end; default mount options for both volumes.


-- 
Chris Murphy

Re: btrfs fi du is unreliable

[Duncan]

Henk Slager posted on Sun, 04 Dec 2016 23:17:23 +0100 as excerpted:

> There are no btrfs changes between kernels 4.8.10 and 4.8.11. There is
> no compress mount option in my case, that is the only thing I currently
> can think of that could make your Set shared number non-zero.

Compression... might actually explain what I'm seeing here, even if it 
doesn't explain Chris Murphy's issue.

I don't do subvolumes or snapshots here, and I've not, knowingly at 
least, setup any reflinked files, so I expected my btrfs fi du's to all 
show 0 bytes shared.

And / shows exactly that, 0 bytes shared, but /home (/h here) shows a 
shared count in the /summary/ while a grep -v ' -  /h' of the full output 
only outputs the summary line, no actual file listing as shared.

(As my admin user so I can sudo w/o password, $$ indicates a command 
line.  Again, /home is a symlink to /h here, /h is thus the actual 
mountpoint for /home, an entirely separate btrfs, not a subvolume.)

$$ sudo btrfs fi du -s /
     Total   Exclusive  Set shared  Filename
   2.94GiB     2.94GiB       0.00B  /

$$ sudo btrfs fi du -s /h
     Total   Exclusive  Set shared  Filename
  16.16GiB    16.16GiB    32.00KiB  /h

$$ sudo btrfs fi du /h | grep -v ' -  /h' 
     Total   Exclusive  Set shared  Filename
  16.16GiB    16.16GiB    32.00KiB  /h

But I do use compress=lzo on both filesystems.  If compression accounts 
for the shared, however, I wonder why / isn't showing any shared, since 
it's mounted with compress=lzo as well.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman