Where is the disk space?

Where is the disk space?

[Marc MERLIN]

root@polgara:/mnt/btrfs_root# du -sh *
28G     @
28G     @_hourly.20151113_08:04:01
4.0K    @_last
4.0K    @_last_rw
28G     @_rw.20151113_00:02:01
root@polgara:/mnt/btrfs_root# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb5        56G   40G  5.4G  89% /mnt/btrfs_root

root@polgara:/mnt/btrfs_root# btrfs fi df .
Data, single: total=39.85GiB, used=38.52GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=6.00GiB, used=579.17MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=208.00MiB, used=0.00B

root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root'  uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
        Total devices 1 FS bytes used 39.09GiB
        devid    1 size 55.88GiB used 51.88GiB path /dev/sdb5

btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root# 

root@polgara:/mnt/btrfs_root# btrfs balance start -dusage=80 -v /mnt/btrfs_root
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=80
Done, had to relocate 1 out of 55 chunks

Sadly, it's only running 3.17.8 because of complicated reasons, but still, 

1) I have 28GB used (modulo a few files between the btrfs send snapshots and
current status)

2) fi show shows I'm using 39GB, not sure where the extra 11GB came from

3) fi df agrees with fi show

4) regular df agrees on used too, but shows 5GB free instead of 15GB despite
the filesystem being balanced.

I did have a bunch of snapshots that I did delete a while ago now, but it
looks like their blocks aren't being reclaimed.

Any ideas?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Where is the disk space?

[Duncan]

Marc MERLIN posted on Fri, 13 Nov 2015 09:41:01 -0800 as excerpted:

> Any ideas?

Without addressing the main question, a couple targets of opportunity:

1) A quick balance with -mprofiles=single should kill those unused single 
metadata and system mkfs.btrfs legacies so you don't have to see them in 
fi df.

2) While fi df reports the data total vs. used spread being close enough 
that it's unsurprising a balance -dusage=80 didn't give you anything 
back, the spread on metadata is rather higher, dup, 6.0 gig total, less 
than 1 gig used, so a balance -musage=50 or 80 should return a few gigs, 
4-5 I'd guess, leaving you 1-2 gig metadata total.

Maybe that's part of the missing snapshot overhead?

(The -musage run should kill the unused single profile metadata/system as 
well, but for a different reason, so I made it a different point and used 
-mprofiles=single there to avoid conflating the two.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Where is the disk space?

[Marc MERLIN]

On Fri, Nov 13, 2015 at 07:45:21PM +0000, Duncan wrote:
> Marc MERLIN posted on Fri, 13 Nov 2015 09:41:01 -0800 as excerpted:
> 
> > Any ideas?
> 
> Without addressing the main question, a couple targets of opportunity:
> 
> 1) A quick balance with -mprofiles=single should kill those unused single 
> metadata and system mkfs.btrfs legacies so you don't have to see them in 
> fi df.
> 
> 2) While fi df reports the data total vs. used spread being close enough 
> that it's unsurprising a balance -dusage=80 didn't give you anything 
> back, the spread on metadata is rather higher, dup, 6.0 gig total, less 
> than 1 gig used, so a balance -musage=50 or 80 should return a few gigs, 
> 4-5 I'd guess, leaving you 1-2 gig metadata total.

Well, -musage helped, I didn't know about that one, thanks:
root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root'  uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
        Total devices 1 FS bytes used 39.09GiB
        devid    1 size 55.88GiB used 51.88GiB path /dev/sdb5

btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root# btrfs balance start -musage=80 -v /mnt/btrfs_root
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=80
  SYSTEM (flags 0x2): balancing, usage=80
Done, had to relocate 14 out of 55 chunks
root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root'  uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
        Total devices 1 FS bytes used 39.75GiB
        devid    1 size 55.88GiB used 41.91GiB path /dev/sdb5

btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb5        56G   41G   15G  74% /mnt/btrfs_root

I'm still seeing 39GB used for 28GB of actual data, but I definitely
fixed one bit already thanks to you.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Where is the disk space?

[Duncan]

Marc MERLIN posted on Fri, 13 Nov 2015 12:01:58 -0800 as excerpted:

> I'm still seeing 39GB used for 28GB of actual data, but I definitely
> fixed one bit already thanks to you.

For the data side, I think I understand what's going on with the space, 
but am not in sufficient mastery of the concept to feel confident that I 
can explain it well.  Never-the-less, here's a go at it.  One of the devs 
did a post complete with nice ascii diagrams if you're interested in 
trying to look it up.

What happens is that with larger files (particularly VM images and the 
like if they're not set nocow or if they're nocow but cow1-ed due to 
snapshotting, which AFAIK fits your use-case directly) originally setup 
in a few reasonably large extents, as rewrites occur, they cow and thus 
unmap random smaller extents from within the larger extents.  But, btrfs 
doesn't do extent splitting, so those larger extents remain pinned as 
long as at least some of the data within them remains referenced.  The 
result can eventually be rather cavernous mostly empty original extents 
still pinned in place by the few 4K blocks that haven't ever been 
rewritten.

If you believe you know what files are likely to be the culprit, and if 
you're doing VMs that's exactly where I'd look first, try temporarily 
moving them (and any old snapshots of them) out of the filesystem, doing 
a quick fi df to see if it has indeed emptied out some data chunks, and 
if so, a balance -dusage=80 or whatever to try to reclaim them, then move 
the file(s) back.

Of course if you still have other snapshots of the file pinning down 
those extents it won't free them, so you have to either delete the 
snapshots or at least delete the culprit file(s) from within them 
(obviously only with writable snapshots) in ordered for this to work.

In theory, defrag can do much the same thing, assuming of course there 
aren't snapshots containing the same file and still locking down its old 
extents, but that doesn't get the file off the filesystem for the 
cleanup, so it's likely to be somewhat less effective, perhaps 
considerably less effective if the remaining free space inside existing 
data chunks is itself highly fragmented.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Where is the disk space?

[Liu Bo]

Hi,

On Fri, Nov 13, 2015 at 09:41:01AM -0800, Marc MERLIN wrote:
> root@polgara:/mnt/btrfs_root# du -sh *
> 28G     @
> 28G     @_hourly.20151113_08:04:01
> 4.0K    @_last
> 4.0K    @_last_rw
> 28G     @_rw.20151113_00:02:01
> root@polgara:/mnt/btrfs_root# df -h .
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sdb5        56G   40G  5.4G  89% /mnt/btrfs_root
> 
> root@polgara:/mnt/btrfs_root# btrfs fi df .
> Data, single: total=39.85GiB, used=38.52GiB
> System, DUP: total=8.00MiB, used=16.00KiB
> System, single: total=4.00MiB, used=0.00B
> Metadata, DUP: total=6.00GiB, used=579.17MiB
> Metadata, single: total=8.00MiB, used=0.00B
> GlobalReserve, single: total=208.00MiB, used=0.00B
> 
> root@polgara:/mnt/btrfs_root# btrfs fi show .
> Label: 'btrfs_root'  uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
>         Total devices 1 FS bytes used 39.09GiB
>         devid    1 size 55.88GiB used 51.88GiB path /dev/sdb5
> 
> btrfs-progs v4.0-dirty
> root@polgara:/mnt/btrfs_root# 
> 
> root@polgara:/mnt/btrfs_root# btrfs balance start -dusage=80 -v /mnt/btrfs_root
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x2): balancing, usage=80
> Done, had to relocate 1 out of 55 chunks
> 
> Sadly, it's only running 3.17.8 because of complicated reasons, but still, 
> 
> 1) I have 28GB used (modulo a few files between the btrfs send snapshots and
> current status)
> 
> 2) fi show shows I'm using 39GB, not sure where the extra 11GB came from
> 
> 3) fi df agrees with fi show
> 
> 4) regular df agrees on used too, but shows 5GB free instead of 15GB despite
> the filesystem being balanced.
> 
> I did have a bunch of snapshots that I did delete a while ago now, but it
> looks like their blocks aren't being reclaimed.
> 
> Any ideas?
> 

Since you said you have some snapshots in between...I can think of one
case to prove where the space goes,

Say, you have a file with size=10M on a freshly created partition(the total used data space is 10M), and you have a snapshot which owns this file, then you modify the original file by overwrite the range [3M, 5M], and right now you can find that the total used data space increases to 15M or maybe more (because of unaliged write and extent pads to 4K length).

This comes from our COW and extent references implementation, so you get
the benefit of COW, meanwhile have to live with the un-reclaimed space.

It's sort of something I was trying to fix, but I found that my approach
led to other problems so I decided to give it up.

Thanks,

-liubo

> Thanks,
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/  
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Where is the disk space?

[Marc MERLIN]

On Sat, Nov 14, 2015 at 10:35:39PM -0800, Liu Bo wrote:
> Since you said you have some snapshots in between...I can think of one
> case to prove where the space goes,
> 
> Say, you have a file with size=10M on a freshly created partition(the total used data space is 10M), and you have a snapshot which owns this file, then you modify the original file by overwrite the range [3M, 5M], and right now you can find that the total used data space increases to 15M or maybe more (because of unaliged write and extent pads to 4K length).
> 
> This comes from our COW and extent references implementation, so you get
> the benefit of COW, meanwhile have to live with the un-reclaimed space.
> 
> It's sort of something I was trying to fix, but I found that my approach
> led to other problems so I decided to give it up.

That's an interesting scenario. Thanks for explaining this.

MArc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Where is the disk space?

[Qu Wenruo]

Liu Bo wrote on 2015/11/14 22:35 -0800:
> Hi,
>
> On Fri, Nov 13, 2015 at 09:41:01AM -0800, Marc MERLIN wrote:
>> root@polgara:/mnt/btrfs_root# du -sh *
>> 28G     @
>> 28G     @_hourly.20151113_08:04:01
>> 4.0K    @_last
>> 4.0K    @_last_rw
>> 28G     @_rw.20151113_00:02:01
>> root@polgara:/mnt/btrfs_root# df -h .
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/sdb5        56G   40G  5.4G  89% /mnt/btrfs_root
>>
>> root@polgara:/mnt/btrfs_root# btrfs fi df .
>> Data, single: total=39.85GiB, used=38.52GiB
>> System, DUP: total=8.00MiB, used=16.00KiB
>> System, single: total=4.00MiB, used=0.00B
>> Metadata, DUP: total=6.00GiB, used=579.17MiB
>> Metadata, single: total=8.00MiB, used=0.00B
>> GlobalReserve, single: total=208.00MiB, used=0.00B
>>
>> root@polgara:/mnt/btrfs_root# btrfs fi show .
>> Label: 'btrfs_root'  uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
>>          Total devices 1 FS bytes used 39.09GiB
>>          devid    1 size 55.88GiB used 51.88GiB path /dev/sdb5
>>
>> btrfs-progs v4.0-dirty
>> root@polgara:/mnt/btrfs_root#
>>
>> root@polgara:/mnt/btrfs_root# btrfs balance start -dusage=80 -v /mnt/btrfs_root
>> Dumping filters: flags 0x1, state 0x0, force is off
>>    DATA (flags 0x2): balancing, usage=80
>> Done, had to relocate 1 out of 55 chunks
>>
>> Sadly, it's only running 3.17.8 because of complicated reasons, but still,
>>
>> 1) I have 28GB used (modulo a few files between the btrfs send snapshots and
>> current status)
>>
>> 2) fi show shows I'm using 39GB, not sure where the extra 11GB came from
>>
>> 3) fi df agrees with fi show
>>
>> 4) regular df agrees on used too, but shows 5GB free instead of 15GB despite
>> the filesystem being balanced.
>>
>> I did have a bunch of snapshots that I did delete a while ago now, but it
>> looks like their blocks aren't being reclaimed.
>>
>> Any ideas?
>>
>
> Since you said you have some snapshots in between...I can think of one
> case to prove where the space goes,
>
> Say, you have a file with size=10M on a freshly created partition(the total used data space is 10M), and you have a snapshot which owns this file, then you modify the original file by overwrite the range [3M, 5M], and right now you can find that the total used data space increases to 15M or maybe more (because of unaliged write and extent pads to 4K length).
>
> This comes from our COW and extent references implementation, so you get
> the benefit of COW, meanwhile have to live with the un-reclaimed space.
>
> It's sort of something I was trying to fix, but I found that my approach
> led to other problems so I decided to give it up.
>
> Thanks,
>
> -liubo

The case is quite right, but the example maybe a little incorrect BTW.

For 10M file in one subvolume and rewrite [3M, 5M](including the last 
byte) in the snapshot will only increase used space to 12M + 4K, as 
above numbers except the last byte are already sectorsize aligned.


The root cause is, btrfs' lazy extent freeing behavior.
Use a new 12M case to describe:

======
Step 0:
Subv A:
[0,12M): Shared between A and B (Extent 1)

on disk:
Extent 1: record the above [0,12M) data

Subv B:
[0,4M): Shared between A and B (Extent 1)
[4M,8M): Exclusive in B (Extent 2)
[8M,12M): Shared between A and B (Extent 1)

on disk:
Extent 1: is reused in [0,4M) and [8M,12M) ranges
Extent 2: record data of [4M,8M)

------
Step 1:
Write [4M,8M) of subv A:
[0,4M): Shared between A and B (Extent 1)
[4M,8M): Exclusive in A (Extent 3)
[8M,12M): Shared between A and B (Extent 1)

Extent 3: record the above [4M,8M) data
And Extent 1 is not changed.

Subv B:
[0,4M): Shared between A and B (Extent 1)
[4M,8M): Exclusive in B (Extent 2)
[8M,12M): Shared between A and B (Extent 1)
======

After step 2, the used space will be 12M (Extent 1) + 4M (Extent 2) + 4M 
(Extent 3) = 20M.

Even only 8M of extent 1 is really referred by.
So the middle 4M of extent 1 is totally wasted and Btrfs won't free it, 
until *ALL* of the extent is not referred by any one.


One solution would be defrag, but IIRC defrag under multi-subvolume case 
is not supported yet...


BTW, personally speaking, to find out how much space a subvolume takes, 
btrfs qgroup would be quite handy (after 4.2 kernel).
Shows how much space a subvolume takes exclusively.

Thanks,
Qu
>
>> Thanks,
>> Marc
>> --
>> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>> Microsoft is to operating systems ....
>>                                        .... what McDonalds is to gourmet cooking
>> Home page: http://marc.merlins.org/
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>