Unable to allocate for space usage in particular btrfs volume

Unable to allocate for space usage in particular btrfs volume

[OmegaPhil]

[-- Attachment #1: Type: text/plain, Size: 3337 bytes --]

Back in September I noticed that 'sudo du -chs /mnt/storage-1' reported
887GB used and 'df -h' 920GB for this particular volume - I went on
#btrfs for any suggestions, and balancing + defraging made no
difference. It had no subvolumes/snapshots etc, I basically used it like
a checksumed ext4fs.

Since the volume was converted from ext4, I redid it from scratch (so
made with kernel v4.1.3 or v4.1.6 on this Debian Testing machine), and
the problem went away.

After a couple of months, df reports 907GB used, whereas du says 884GB -
I currently have 8 large (1-5.5TB volumes) btrfs volumes in use,
storage-1 is the only SSD volume and the only one with this problem.

No balancing or defraging this time, it didn't make a difference before
and this is a relatively new volume.

Are there any sysadmin-level ways I can account for the ~23GB lost space?

Thanks for any help.

=========================================================

$ uname -a

Linux omega1 4.2.0-1-amd64 #1 SMP Debian 4.2.5-1 (2015-10-27) x86_64
GNU/Linux

$ btrfs --version

btrfs-progs v4.2.2

$ sudo btrfs fi usage /mnt/storage-1

Overall:
    Device size:		 953.87GiB
    Device allocated:		 932.04GiB
    Device unallocated:		  21.83GiB
    Device missing:		     0.00B
    Used:			 906.10GiB
    Free (estimated):		  45.35GiB	(min: 34.43GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:925.01GiB, Used:901.50GiB
   /dev/sdb	 925.01GiB

Metadata,single: Size:8.00MiB, Used:0.00B
   /dev/sdb	   8.00MiB

Metadata,DUP: Size:3.50GiB, Used:2.30GiB
   /dev/sdb	   7.00GiB

System,single: Size:4.00MiB, Used:0.00B
   /dev/sdb	   4.00MiB

System,DUP: Size:8.00MiB, Used:128.00KiB
   /dev/sdb	  16.00MiB

Unallocated:
   /dev/sdb	  21.83GiB

$ sudo btrfs-show-super /dev/sdb

superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum			0x7f6b70be [match]
bytenr			65536
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			27430475-c49a-4e3f-8f8d-be5c14be59db
label			storage-1
generation		114344
root			683413471232
sys_array_size		226
chunk_root_generation	114251
root_level		1
chunk_root		21004288
chunk_root_level	1
log_root		683413979136
log_root_transid	0
log_root_level		0
total_bytes		1024209543168
bytes_used		971565568000
sectorsize		4096
nodesize		16384
leafsize		16384
stripesize		4096
root_dir		6
num_devices		1
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x161
			( MIXED_BACKREF |
			  BIG_METADATA |
			  EXTENDED_IREF |
			  SKINNY_METADATA )
csum_type		0
csum_size		4
cache_generation	114344
uuid_tree_generation	114344
dev_item.uuid		c6b32341-6300-4f21-8c3b-3d7d458c3668
dev_item.fsid		27430475-c49a-4e3f-8f8d-be5c14be59db [match]
dev_item.type		0
dev_item.total_bytes	1024209543168
dev_item.bytes_used	1000765128704
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		1
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0

=========================================================

dmesg contains a lot of information which is superfluous to btrfs and
personal, I can filter on a regex and report if necessary.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Unable to allocate for space usage in particular btrfs volume

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 4379 bytes --]

On Wed, Nov 04, 2015 at 09:10:42PM +0000, OmegaPhil wrote:
> Back in September I noticed that 'sudo du -chs /mnt/storage-1' reported
> 887GB used and 'df -h' 920GB for this particular volume - I went on
> #btrfs for any suggestions, and balancing + defraging made no
> difference. It had no subvolumes/snapshots etc, I basically used it like
> a checksumed ext4fs.
> 
> Since the volume was converted from ext4, I redid it from scratch (so
> made with kernel v4.1.3 or v4.1.6 on this Debian Testing machine), and
> the problem went away.
> 
> After a couple of months, df reports 907GB used, whereas du says 884GB -
> I currently have 8 large (1-5.5TB volumes) btrfs volumes in use,
> storage-1 is the only SSD volume and the only one with this problem.
> 
> No balancing or defraging this time, it didn't make a difference before
> and this is a relatively new volume.
> 
> Are there any sysadmin-level ways I can account for the ~23GB lost space?

   There's an issue where replacing blocks in the middle of an
existing extent won't split the extent, and thus the "old" blocks
aren't freed up, because they're held by the original extent (even
though not actually referenced by any existing file). This might be
what you're seeing.

   I'm not sure how to confirm this theory, or what to do about it if
it's true. (Defrag the file? Copy it elsewhere? Other?)

   Two other cases for df > du are orphaned files, although 23 GiB of
orphans is large; and missing out the dot-files in the directory that
du is run from (if doing, say, "du *" rather than "du ."). I've been
bitten by both of those in the past.

   Hugo.

> Thanks for any help.
> 
> =========================================================
> 
> $ uname -a
> 
> Linux omega1 4.2.0-1-amd64 #1 SMP Debian 4.2.5-1 (2015-10-27) x86_64
> GNU/Linux
> 
> $ btrfs --version
> 
> btrfs-progs v4.2.2
> 
> $ sudo btrfs fi usage /mnt/storage-1
> 
> Overall:
>     Device size:		 953.87GiB
>     Device allocated:		 932.04GiB
>     Device unallocated:		  21.83GiB
>     Device missing:		     0.00B
>     Used:			 906.10GiB
>     Free (estimated):		  45.35GiB	(min: 34.43GiB)
>     Data ratio:			      1.00
>     Metadata ratio:		      2.00
>     Global reserve:		 512.00MiB	(used: 0.00B)
> 
> Data,single: Size:925.01GiB, Used:901.50GiB
>    /dev/sdb	 925.01GiB
> 
> Metadata,single: Size:8.00MiB, Used:0.00B
>    /dev/sdb	   8.00MiB
> 
> Metadata,DUP: Size:3.50GiB, Used:2.30GiB
>    /dev/sdb	   7.00GiB
> 
> System,single: Size:4.00MiB, Used:0.00B
>    /dev/sdb	   4.00MiB
> 
> System,DUP: Size:8.00MiB, Used:128.00KiB
>    /dev/sdb	  16.00MiB
> 
> Unallocated:
>    /dev/sdb	  21.83GiB
> 
> $ sudo btrfs-show-super /dev/sdb
> 
> superblock: bytenr=65536, device=/dev/sdb
> ---------------------------------------------------------
> csum			0x7f6b70be [match]
> bytenr			65536
> flags			0x1
> 			( WRITTEN )
> magic			_BHRfS_M [match]
> fsid			27430475-c49a-4e3f-8f8d-be5c14be59db
> label			storage-1
> generation		114344
> root			683413471232
> sys_array_size		226
> chunk_root_generation	114251
> root_level		1
> chunk_root		21004288
> chunk_root_level	1
> log_root		683413979136
> log_root_transid	0
> log_root_level		0
> total_bytes		1024209543168
> bytes_used		971565568000
> sectorsize		4096
> nodesize		16384
> leafsize		16384
> stripesize		4096
> root_dir		6
> num_devices		1
> compat_flags		0x0
> compat_ro_flags		0x0
> incompat_flags		0x161
> 			( MIXED_BACKREF |
> 			  BIG_METADATA |
> 			  EXTENDED_IREF |
> 			  SKINNY_METADATA )
> csum_type		0
> csum_size		4
> cache_generation	114344
> uuid_tree_generation	114344
> dev_item.uuid		c6b32341-6300-4f21-8c3b-3d7d458c3668
> dev_item.fsid		27430475-c49a-4e3f-8f8d-be5c14be59db [match]
> dev_item.type		0
> dev_item.total_bytes	1024209543168
> dev_item.bytes_used	1000765128704
> dev_item.io_align	4096
> dev_item.io_width	4096
> dev_item.sector_size	4096
> dev_item.devid		1
> dev_item.dev_group	0
> dev_item.seek_speed	0
> dev_item.bandwidth	0
> dev_item.generation	0
> 
> =========================================================
> 
> dmesg contains a lot of information which is superfluous to btrfs and
> personal, I can filter on a regex and report if necessary.
> 



-- 
Hugo Mills             | There are three things you should never see being
hugo@... carfax.org.uk | made: laws, standards, and sausages.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Unable to allocate for space usage in particular btrfs volume

[OmegaPhil]

[-- Attachment #1: Type: text/plain, Size: 2236 bytes --]

On 04/11/15 21:30, Hugo Mills wrote:
> On Wed, Nov 04, 2015 at 09:10:42PM +0000, OmegaPhil wrote:
>> Back in September I noticed that 'sudo du -chs /mnt/storage-1' reported
>> 887GB used and 'df -h' 920GB for this particular volume - I went on
>> #btrfs for any suggestions, and balancing + defraging made no
>> difference. It had no subvolumes/snapshots etc, I basically used it like
>> a checksumed ext4fs.
>>
>> Since the volume was converted from ext4, I redid it from scratch (so
>> made with kernel v4.1.3 or v4.1.6 on this Debian Testing machine), and
>> the problem went away.
>>
>> After a couple of months, df reports 907GB used, whereas du says 884GB -
>> I currently have 8 large (1-5.5TB volumes) btrfs volumes in use,
>> storage-1 is the only SSD volume and the only one with this problem.
>>
>> No balancing or defraging this time, it didn't make a difference before
>> and this is a relatively new volume.
>>
>> Are there any sysadmin-level ways I can account for the ~23GB lost space?
> 
>    There's an issue where replacing blocks in the middle of an
> existing extent won't split the extent, and thus the "old" blocks
> aren't freed up, because they're held by the original extent (even
> though not actually referenced by any existing file). This might be
> what you're seeing.
> 
>    I'm not sure how to confirm this theory, or what to do about it if
> it's true. (Defrag the file? Copy it elsewhere? Other?)
> 
>    Two other cases for df > du are orphaned files, although 23 GiB of
> orphans is large; and missing out the dot-files in the directory that
> du is run from (if doing, say, "du *" rather than "du ."). I've been
> bitten by both of those in the past.
> 
>    Hugo.


The volume doesn't change hugely over time, so it really ought not to
have broken so quickly - a quick rundown of the storage usage:

36% general (small files, some smallish videos)
24% music
23% pr0n
17% VMs

But in terms of 'large files changing', it could be the VM disks perhaps
- I'll move them out, balance, and then back in again, hopefully that'd
be a meaningful test.

du-wise was direct on the root directory, any idea how I could audit
orphan files?

Thanks





[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Unable to allocate for space usage in particular btrfs volume

[Duncan]

OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:

> The volume doesn't change hugely over time, so it really ought not to
> have broken so quickly - a quick rundown of the storage usage:
> 
> 36% general (small files, some smallish videos)
> 24% music 23% pr0n 17% VMs
> 
> But in terms of 'large files changing', it could be the VM disks perhaps
> -
> I'll move them out, balance, and then back in again, hopefully that'd be
> a meaningful test.

VM image files (and large database files, for the same reason) are a bit 
of a problem on btrfs, and indeed, any COW-based filesystem, since the 
random rewrite pattern matching that use-case is pretty much the absolute 
worst-case match for a COW-based filesystem there is.

And that would be the worst-case in terms of the unsplit extents issue 
Hugo was talking about as well.  So they may well be the problem, indeed.

Since you're not doing snapshotting (which conflicts with this option, 
with an imperfect workaround), setting nocow on those files may well 
eliminate the problem, but be aware if you aren't already that (1) nocow 
does turn off checksumming as well, in ordered to avoid a race that could 
easily lead to data corruption, and (2) you can't just activate nocow on 
the existing file and expect it to work, the procedure is a bit more 
complicated than that, since nocow is only guaranteed to work if it's set 
at file creation.  Detailed instructions for #2 skipped as they've been 
posted many times, but if you are interested and haven't seen them, ask.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Unable to allocate for space usage in particular btrfs volume

[OmegaPhil]

[-- Attachment #1: Type: text/plain, Size: 2989 bytes --]

On 05/11/15 04:18, Duncan wrote:
> OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:
> 
>> The volume doesn't change hugely over time, so it really ought not to
>> have broken so quickly - a quick rundown of the storage usage:
>>
>> 36% general (small files, some smallish videos)
>> 24% music 23% pr0n 17% VMs
>>
>> But in terms of 'large files changing', it could be the VM disks perhaps
>> -
>> I'll move them out, balance, and then back in again, hopefully that'd be
>> a meaningful test.
> 
> VM image files (and large database files, for the same reason) are a bit 
> of a problem on btrfs, and indeed, any COW-based filesystem, since the 
> random rewrite pattern matching that use-case is pretty much the absolute 
> worst-case match for a COW-based filesystem there is.
> 
> And that would be the worst-case in terms of the unsplit extents issue 
> Hugo was talking about as well.  So they may well be the problem, indeed.
> 
> Since you're not doing snapshotting (which conflicts with this option, 
> with an imperfect workaround), setting nocow on those files may well 
> eliminate the problem, but be aware if you aren't already that (1) nocow 
> does turn off checksumming as well, in ordered to avoid a race that could 
> easily lead to data corruption, and (2) you can't just activate nocow on 
> the existing file and expect it to work, the procedure is a bit more 
> complicated than that, since nocow is only guaranteed to work if it's set 
> at file creation.  Detailed instructions for #2 skipped as they've been 
> posted many times, but if you are interested and haven't seen them, ask.


Thankyou Hugo, Duncan - moving VMs out, then:

=====================================================

sudo chattr +C '/mnt/storage-1/Virtual Machines'
sudo btrfs balance start -mconvert=dup /mnt/storage-1
sudo btrfs fi defrag -rv /mnt/storage-1

=====================================================

And after moving VMs back, du reports 884GB used,

=====================================================

$ sudo btrfs fi usage /mnt/storage-1

Overall:
    Device size:		 953.87GiB
    Device allocated:		 924.07GiB
    Device unallocated:		  29.80GiB
    Device missing:		     0.00B
    Used:			 886.11GiB
    Free (estimated):		  65.72GiB	(min: 50.82GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:918.01GiB, Used:882.09GiB
   /dev/sdb	 918.01GiB

Metadata,DUP: Size:3.00GiB, Used:2.01GiB
   /dev/sdb	   6.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB
   /dev/sdb	  64.00MiB

Unallocated:
   /dev/sdb	  29.80GiB

=====================================================

So a couple of gig still unaccountable but irrelevant. Thanks, problem
solved! Although hopefully checksumming will be allowed on nocow files
in the future as thats currently 17% of all data unprotected and will
get worse...


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Unable to allocate for space usage in particular btrfs volume

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

On Thu, Nov 05, 2015 at 10:44:51AM +0000, OmegaPhil wrote:
> So a couple of gig still unaccountable but irrelevant. Thanks, problem
> solved! Although hopefully checksumming will be allowed on nocow files
> in the future as thats currently 17% of all data unprotected and will
> get worse...

   It's not likely to happen. If you're modifying data in-place (as is
the case on nodatacow files), then there's a window of vulnerability
between modifying the data and modifying the checksum records during
which the checksums are incorrect. This breaks the consistency
guarantees of the FS.

   Hugo.

-- 
Hugo Mills             | Nostalgia isn't what it used to be.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Unable to allocate for space usage in particular btrfs volume

[Calvin Walton]

On Thu, 2015-11-05 at 10:44 +0000, OmegaPhil wrote:
> On 05/11/15 04:18, Duncan wrote:
> > OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:
> > VM image files (and large database files, for the same reason) are
> > a bit 
> > of a problem on btrfs, and indeed, any COW-based filesystem, since
> > the 
> > random rewrite pattern matching that use-case is pretty much the
> > absolute 
> > worst-case match for a COW-based filesystem there is.
> > Since you're not doing snapshotting (which conflicts with this
> > option, 
> > with an imperfect workaround), setting nocow on those files may
> > well 
> > eliminate the problem, but be aware if you aren't already that (1)
> > nocow 
> > does turn off checksumming as well, in ordered to avoid a race that
> > could 
> > easily lead to data corruption, and (2) you can't just activate

> So a couple of gig still unaccountable but irrelevant. Thanks,
> problem
> solved! Although hopefully checksumming will be allowed on nocow
> files
> in the future as thats currently 17% of all data unprotected and will
> get worse...

There's actually an interesting workaround to this: Although the VM
disk images aren't checksummed on the host filesystem, you can use
btrfs *inside* the VMs and enable checksumming there. The downside is
that you can only verify the VM data by booting the VM and running a
scrub from inside.

This of course doesn't help if your VMs are Windows or legacy versions
of Linux without btrfs support. On BSD you could try ZFS.

-- 
Calvin Walton <calvin.walton@kepstin.ca>

Re: Unable to allocate for space usage in particular btrfs volume

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2591 bytes --]

On 2015-11-06 15:15, Calvin Walton wrote:
> On Thu, 2015-11-05 at 10:44 +0000, OmegaPhil wrote:
>> On 05/11/15 04:18, Duncan wrote:
>>> OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:
>>> VM image files (and large database files, for the same reason) are
>>> a bit
>>> of a problem on btrfs, and indeed, any COW-based filesystem, since
>>> the
>>> random rewrite pattern matching that use-case is pretty much the
>>> absolute
>>> worst-case match for a COW-based filesystem there is.
>>> Since you're not doing snapshotting (which conflicts with this
>>> option,
>>> with an imperfect workaround), setting nocow on those files may
>>> well
>>> eliminate the problem, but be aware if you aren't already that (1)
>>> nocow
>>> does turn off checksumming as well, in ordered to avoid a race that
>>> could
>>> easily lead to data corruption, and (2) you can't just activate
>
>> So a couple of gig still unaccountable but irrelevant. Thanks,
>> problem
>> solved! Although hopefully checksumming will be allowed on nocow
>> files
>> in the future as thats currently 17% of all data unprotected and will
>> get worse...
>
> There's actually an interesting workaround to this: Although the VM
> disk images aren't checksummed on the host filesystem, you can use
> btrfs *inside* the VMs and enable checksumming there. The downside is
> that you can only verify the VM data by booting the VM and running a
> scrub from inside.
Actually, by using a combination of loop devices and kpartx, it's fully 
possible to mount the FS and verify it without booting the VM.  Of 
course, doing this usually requires root access on the host system, but 
for most people I know, that's usually not an issue.  I do this on 
occasion when I need to pull a file off of one of my VM disks on my 
laptop and don't have the time to spin up the VM itself.

Another option if you're doing a direct boot of the kernel (for example, 
when using a fully paravirtualized domain on Xen, or using some of the 
QEMU ARM systems) is to just do the volume management (partitioning and 
such) on the host, and expose each filesystem to the guest as a separate 
disk.  I do this with most of my Linux VM's on my Xen system where I use 
LVM as the back-end storage for the virtual disk images, as it allows me 
to easily directly mount the VM's filesystems on the host if need be 
(and let's you do all kinds of cool things like using a cluster-aware 
filesystem for the VM's root so that you can mount it from the host 
safely while the VM is still online).



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]