problems with free space cache

problems with free space cache

[Nicholas D Steeves]

The volume was created with btrfs-progs-4.4.41; I upgraded to
linux-4.4.10 today, and here is what I grepped from my dmesg:

[  +0.002613] Btrfs loaded
[  +0.080621] BTRFS: device label Red devid 1 transid 108234 /dev/sda3
[  +0.000106] BTRFS: device label Red devid 2 transid 108234 /dev/sdb2
[  +0.011888] BTRFS info (device sdb2): disk space caching is enabled
[  +0.000001] BTRFS: has skinny extents
[  +0.063532] BTRFS info (device sdb2): disk space caching is enabled
[  +3.615389] BTRFS warning (device sdb2): block group 183639212032
has wrong amount of free space
[  +0.000002] BTRFS warning (device sdb2): failed to load free space
cache for block group 183639212032, rebuilding it now
[  +1.718348] BTRFS warning (device sdb2): block group 859022819328
has wrong amount of free space
[  +0.000003] BTRFS warning (device sdb2): failed to load free space
cache for block group 859022819328, rebuilding it now
[  +4.357965] BTRFS info (device sdb2): The free space cache file
(1082361118720) is invalid. skip it
[  +0.026489] BTRFS info (device sdb2): The free space cache file
(1094172278784) is invalid. skip it
[  +0.740340] BTRFS info (device sdb2): The free space cache file
(1183292850176) is invalid. skip it
[  +0.610161] BTRFS info (device sdb2): The free space cache file
(1248791101440) is invalid. skip it
[  +0.353670] BTRFS info (device sdb2): The free space cache file
(1284224581632) is invalid. skip it

What is the standard procedure for fixing the cache?  Rootfs is a
subvolume and the first entry in fstab.  Second entry is /btrfs-admin,
which is where I mount the whole volume.  Is it sufficient to add the
clear_cache option to the rootfs entry, or does the /btrfs-admin entry
also need it?

>From what I've read in the documentation one modifies fstab, reboots,
removes modification from fstab, and it's fixed.

Cheers,
Nicholas

Re: problems with free space cache

[Qu Wenruo]

Nicholas D Steeves wrote on 2016/05/15 22:11 -0400:
> The volume was created with btrfs-progs-4.4.41; I upgraded to
> linux-4.4.10 today, and here is what I grepped from my dmesg:
>
> [  +0.002613] Btrfs loaded
> [  +0.080621] BTRFS: device label Red devid 1 transid 108234 /dev/sda3
> [  +0.000106] BTRFS: device label Red devid 2 transid 108234 /dev/sdb2
> [  +0.011888] BTRFS info (device sdb2): disk space caching is enabled
> [  +0.000001] BTRFS: has skinny extents
> [  +0.063532] BTRFS info (device sdb2): disk space caching is enabled
> [  +3.615389] BTRFS warning (device sdb2): block group 183639212032
> has wrong amount of free space
> [  +0.000002] BTRFS warning (device sdb2): failed to load free space
> cache for block group 183639212032, rebuilding it now
> [  +1.718348] BTRFS warning (device sdb2): block group 859022819328
> has wrong amount of free space
> [  +0.000003] BTRFS warning (device sdb2): failed to load free space
> cache for block group 859022819328, rebuilding it now
> [  +4.357965] BTRFS info (device sdb2): The free space cache file
> (1082361118720) is invalid. skip it
> [  +0.026489] BTRFS info (device sdb2): The free space cache file
> (1094172278784) is invalid. skip it
> [  +0.740340] BTRFS info (device sdb2): The free space cache file
> (1183292850176) is invalid. skip it
> [  +0.610161] BTRFS info (device sdb2): The free space cache file
> (1248791101440) is invalid. skip it
> [  +0.353670] BTRFS info (device sdb2): The free space cache file
> (1284224581632) is invalid. skip it
>
> What is the standard procedure for fixing the cache?  Rootfs is a
> subvolume and the first entry in fstab.  Second entry is /btrfs-admin,
> which is where I mount the whole volume.  Is it sufficient to add the
> clear_cache option to the rootfs entry, or does the /btrfs-admin entry
> also need it?

IIRC clear_cache option is fs level option.
So the first mount with clear_cache, then all subvolume will have 
clear_cache.

$ sudo mount /dev/sda6 -o subvol=sub1,clear_cache /mnt/data/
$ sudo mount /dev/sda6 -o subvol=sub2 /mnt/test/
$ mount | grep btrfs
/dev/sda6 on /mnt/data type btrfs 
(rw,relatime,space_cache,clear_cache,subvolid=257,subvol=/sub1)
/dev/sda6 on /mnt/test type btrfs 
(rw,relatime,space_cache,clear_cache,subvolid=258,subvol=/sub2)

Thanks,
Qu
>
> From what I've read in the documentation one modifies fstab, reboots,
> removes modification from fstab, and it's fixed.
>
> Cheers,
> Nicholas
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: problems with free space cache

[Nicholas D Steeves]

On 15 May 2016 at 22:24, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> Nicholas D Steeves wrote on 2016/05/15 22:11 -0400:
>> What is the standard procedure for fixing the cache?  Rootfs is a
>> subvolume and the first entry in fstab.  Second entry is /btrfs-admin,
>> which is where I mount the whole volume.  Is it sufficient to add the
>> clear_cache option to the rootfs entry, or does the /btrfs-admin entry
>> also need it?
>
>
> IIRC clear_cache option is fs level option.
> So the first mount with clear_cache, then all subvolume will have
> clear_cache.
>

So that means that in this case mounting subvol=rootfs with
clear_cache will clear the cache for all subsequently mounted
subvolumes, and for the whole volume, correct?

Thanks,
Nicholas

Re: problems with free space cache

[Qu Wenruo]

Nicholas D Steeves wrote on 2016/05/15 23:02 -0400:
> On 15 May 2016 at 22:24, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>>
>>
>> Nicholas D Steeves wrote on 2016/05/15 22:11 -0400:
>>> What is the standard procedure for fixing the cache?  Rootfs is a
>>> subvolume and the first entry in fstab.  Second entry is /btrfs-admin,
>>> which is where I mount the whole volume.  Is it sufficient to add the
>>> clear_cache option to the rootfs entry, or does the /btrfs-admin entry
>>> also need it?
>>
>>
>> IIRC clear_cache option is fs level option.
>> So the first mount with clear_cache, then all subvolume will have
>> clear_cache.
>>
>
> So that means that in this case mounting subvol=rootfs with
> clear_cache will clear the cache for all subsequently mounted
> subvolumes, and for the whole volume, correct?
>

Yes, although I'm not sure whether clear_cache will fix all the problems.
I see other guy reporting such problem but clear_cache only fixed some 
of them.

Thanks,
Qu


> Thanks,
> Nicholas
>
>

Re: problems with free space cache

[Duncan]

Qu Wenruo posted on Mon, 16 May 2016 10:24:23 +0800 as excerpted:

> IIRC clear_cache option is fs level option.
> So the first mount with clear_cache, then all subvolume will have
> clear_cache.

Question:  Does clear_cache work with a read-only mount?

I could see it being like the log replay and being done even on ro 
mounts, particularly since unlike the log replay, clear_cache is a 
specific mount option so could be seen as specifically requested even if 
otherwise ro.  Or not.  So I don't know.

He mentioned root, which is normally mounted read-only first.  That's 
what got me thinking about it.  Does that complicate things?

If root is mounted ro without clear_cache by the initr*, will a 
subsequent remount,rw,clear_cache do it?

And for those like me who keep root mounted ro by default, would 
clear_cache work at all, or would it not work until such time as I 
mounted root, or some other subvolume (which I don't have here to worry 
about, but for those who do...) writable?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: problems with free space cache

[Qu Wenruo]

Duncan wrote on 2016/05/16 05:59 +0000:
> Qu Wenruo posted on Mon, 16 May 2016 10:24:23 +0800 as excerpted:
>
>> IIRC clear_cache option is fs level option.
>> So the first mount with clear_cache, then all subvolume will have
>> clear_cache.
>
> Question:  Does clear_cache work with a read-only mount?
Good question.

But easy to check.
I just checked it and found even that's possible, it doesn't work.

Free space cache inode bytenr doesn't change and no generation change.
While without ro, it did rebuild free space cache for *SOME* chunks, not 
*ALL* chunks.

And that's the problem I'm just chasing today.


Short conclude: clear_cache mount option will only rebuild free space 
cache for chunks which we allocated space from, during the mount time of 
clear_cache.
(Maybe I'm just out-of-date and some other devs may already know that)

And kernel fix for that is a little tricky, as we don't really read out 
all free space cache, but only when we are going to allocate space from 
them.
For any free space cache we didn't read out, they won't be rebuilt.

You can find my reproducer in my just submitted 
patch(https://patchwork.kernel.org/patch/9098431/).

That behavior makes things a little confusing, which users may continue 
hitting annoying free space cache warning from kernel, even they try to 
use clear_cache mount option.


Anyway, I'll add ability for manually wipe out all/given free space 
cache to btrfsck, at least creating a solution to really rebuild all v1 
free space cache.

Thanks,
Qu
>
> I could see it being like the log replay and being done even on ro
> mounts, particularly since unlike the log replay, clear_cache is a
> specific mount option so could be seen as specifically requested even if
> otherwise ro.  Or not.  So I don't know.
>
> He mentioned root, which is normally mounted read-only first.  That's
> what got me thinking about it.  Does that complicate things?
>
> If root is mounted ro without clear_cache by the initr*, will a
> subsequent remount,rw,clear_cache do it?
>
> And for those like me who keep root mounted ro by default, would
> clear_cache work at all, or would it not work until such time as I
> mounted root, or some other subvolume (which I don't have here to worry
> about, but for those who do...) writable?
>

Re: problems with free space cache

[Austin S. Hemmelgarn]

On 2016-05-16 02:20, Qu Wenruo wrote:
>
>
> Duncan wrote on 2016/05/16 05:59 +0000:
>> Qu Wenruo posted on Mon, 16 May 2016 10:24:23 +0800 as excerpted:
>>
>>> IIRC clear_cache option is fs level option.
>>> So the first mount with clear_cache, then all subvolume will have
>>> clear_cache.
>>
>> Question:  Does clear_cache work with a read-only mount?
> Good question.
>
> But easy to check.
> I just checked it and found even that's possible, it doesn't work.
>
> Free space cache inode bytenr doesn't change and no generation change.
> While without ro, it did rebuild free space cache for *SOME* chunks, not
> *ALL* chunks.
>
> And that's the problem I'm just chasing today.
>
>
> Short conclude: clear_cache mount option will only rebuild free space
> cache for chunks which we allocated space from, during the mount time of
> clear_cache.
> (Maybe I'm just out-of-date and some other devs may already know that)
>
> And kernel fix for that is a little tricky, as we don't really read out
> all free space cache, but only when we are going to allocate space from
> them.
> For any free space cache we didn't read out, they won't be rebuilt.
>
> You can find my reproducer in my just submitted
> patch(https://patchwork.kernel.org/patch/9098431/).
>
> That behavior makes things a little confusing, which users may continue
> hitting annoying free space cache warning from kernel, even they try to
> use clear_cache mount option.
>
>
> Anyway, I'll add ability for manually wipe out all/given free space
> cache to btrfsck, at least creating a solution to really rebuild all v1
> free space cache.
>
FWIW, I think it's possible to do this by mounting with clear_cache and 
then running a full balance on the filesystem.  Having an option to do 
this on an unmounted FS would be preferred of course, as that would 
almost certainly be more efficient on any reasonably sized filesystem.

Re: problems with free space cache

[Nicholas D Steeves]

On 16 May 2016 at 07:36, Austin S. Hemmelgarn <ahferroin7@gmail.com> wrote:
> On 2016-05-16 02:20, Qu Wenruo wrote:
>>
>>
>>
>> Duncan wrote on 2016/05/16 05:59 +0000:
>>>
>>> Qu Wenruo posted on Mon, 16 May 2016 10:24:23 +0800 as excerpted:
>>>
>>>> IIRC clear_cache option is fs level option.
>>>> So the first mount with clear_cache, then all subvolume will have
>>>> clear_cache.
>>>
>>>
>>> Question:  Does clear_cache work with a read-only mount?
>>
>> Good question.
>>
>> But easy to check.
>> I just checked it and found even that's possible, it doesn't work.

+1  I had to use my USB flash rescue disk to mount with clear_cache.

>> Free space cache inode bytenr doesn't change and no generation change.
>> While without ro, it did rebuild free space cache for *SOME* chunks, not
>> *ALL* chunks.
>>
>> And that's the problem I'm just chasing today.

+1  Unfortunately, it didn't fix the affected free space cache files.

>> Short conclude: clear_cache mount option will only rebuild free space
>> cache for chunks which we allocated space from, during the mount time of
>> clear_cache.
>> (Maybe I'm just out-of-date and some other devs may already know that)

Does this mean creating a huge file filled with zeros, while mounted
with clear_cache would solve this?  I think that would be faster than
a full rebalance, but I'm not convinced it would work, because of #1
(see below).

One of the following might have caused this situation:

1) I created a new subvolume, and tested a full restore from
(non-btrfs aware) backups to this subvolume; some time later,
verification (rsync -c with other options) completed, and the backup
was confirmed to be usable; I deleted the subvolume and watched the
cleaner get to work and reduce the allocated space, remembering a time
a balance was necessary.  If this is what caused it, cloud there be a
bug in the cleaner and/or cleaner<->space_cache interaction?

2) The same week I reorganised several hundred gigabytes of short term
backups, moving them from one subvolume to another.  I used cp -ar
--reflink=always from within /btrfs-admin, which is where I mount the
whole volume, because / is subvol=rootfs.  After the copy I removed
the source, then used the checksums I keep in my short-term backup
directory to verify everything was ok (I could have restored from a
fresh long-term backup if this failed).  As expected, everything was
ok.  If this is what caused the free space cache bug, then could there
be a bug in the intersubvolume reflink code?

>> That behavior makes things a little confusing, which users may continue
>> hitting annoying free space cache warning from kernel, even they try to
>> use clear_cache mount option.

So this warning is totally harmless and will not cause future problems?

>> Anyway, I'll add ability for manually wipe out all/given free space
>> cache to btrfsck, at least creating a solution to really rebuild all v1
>> free space cache.

Nice!  Am I correct in understanding that this is as safe as a full
balance, and not dangerous like btrfs check --repair?  Is it likely
that the v2 free space cache will be default by linux-4.10?

> FWIW, I think it's possible to do this by mounting with clear_cache and then
> running a full balance on the filesystem.  Having an option to do this on an
> unmounted FS would be preferred of course, as that would almost certainly
> be more efficient on any reasonably sized filesystem.

I'm running a balance now, like it's 2015! ;-)

Thank you,
Nicholas

Re: problems with free space cache

[Qu Wenruo]

Nicholas D Steeves wrote on 2016/05/17 14:17 -0400:
> On 16 May 2016 at 07:36, Austin S. Hemmelgarn <ahferroin7@gmail.com> wrote:
>> On 2016-05-16 02:20, Qu Wenruo wrote:
>>>
>>>
>>>
>>> Duncan wrote on 2016/05/16 05:59 +0000:
>>>>
>>>> Qu Wenruo posted on Mon, 16 May 2016 10:24:23 +0800 as excerpted:
>>>>
>>>>> IIRC clear_cache option is fs level option.
>>>>> So the first mount with clear_cache, then all subvolume will have
>>>>> clear_cache.
>>>>
>>>>
>>>> Question:  Does clear_cache work with a read-only mount?
>>>
>>> Good question.
>>>
>>> But easy to check.
>>> I just checked it and found even that's possible, it doesn't work.
>
> +1  I had to use my USB flash rescue disk to mount with clear_cache.
>
>>> Free space cache inode bytenr doesn't change and no generation change.
>>> While without ro, it did rebuild free space cache for *SOME* chunks, not
>>> *ALL* chunks.
>>>
>>> And that's the problem I'm just chasing today.
>
> +1  Unfortunately, it didn't fix the affected free space cache files.
>
>>> Short conclude: clear_cache mount option will only rebuild free space
>>> cache for chunks which we allocated space from, during the mount time of
>>> clear_cache.
>>> (Maybe I'm just out-of-date and some other devs may already know that)
>
> Does this mean creating a huge file filled with zeros, while mounted
> with clear_cache would solve this?  I think that would be faster than
> a full rebalance, but I'm not convinced it would work, because of #1
> (see below).

No, it won't work, or at least won't work for all chunks.

Even you create a large file which would take up, for example, 50% of 
your total btrfs space.

It's still very possible that a lot of existing data chunk won't be 
modified, and their space cache is not not updated.

Not to mention that the large file won't cause too much metadata 
pressure to modify many metadata block group.

So without the new fsck method to offline clean space cache, either full 
balance with clear_cache mount option, or change kernel to do super slow 
mount to clean all free space cache.

>
> One of the following might have caused this situation:
>
> 1) I created a new subvolume, and tested a full restore from
> (non-btrfs aware) backups to this subvolume; some time later,
> verification (rsync -c with other options) completed, and the backup
> was confirmed to be usable; I deleted the subvolume and watched the
> cleaner get to work and reduce the allocated space, remembering a time
> a balance was necessary.  If this is what caused it, cloud there be a
> bug in the cleaner and/or cleaner<->space_cache interaction?

Unforunately, not sure, as I'm not familiar with btrfs cleaner thread.
But as long as fsck doesn't complain anything other than free space 
cache, cleaner thread is at least not causing big problem.

>
> 2) The same week I reorganised several hundred gigabytes of short term
> backups, moving them from one subvolume to another.  I used cp -ar
> --reflink=always from within /btrfs-admin, which is where I mount the
> whole volume, because / is subvol=rootfs.  After the copy I removed
> the source, then used the checksums I keep in my short-term backup
> directory to verify everything was ok (I could have restored from a
> fresh long-term backup if this failed).  As expected, everything was
> ok.  If this is what caused the free space cache bug, then could there
> be a bug in the intersubvolume reflink code?
>
>>> That behavior makes things a little confusing, which users may continue
>>> hitting annoying free space cache warning from kernel, even they try to
>>> use clear_cache mount option.
>
> So this warning is totally harmless and will not cause future problems?

Yes, at least for end-user. (despite the possible allocation performance 
impact)

For developer, this may means there is something wrong preventing us 
from commit free space cache to disk.

>
>>> Anyway, I'll add ability for manually wipe out all/given free space
>>> cache to btrfsck, at least creating a solution to really rebuild all v1
>>> free space cache.
>
> Nice!  Am I correct in understanding that this is as safe as a full
> balance, and not dangerous like btrfs check --repair?  Is it likely
> that the v2 free space cache will be default by linux-4.10?

I'm not certain about how safe full balance is.
(Common problem like ENOSPC will still prevent you from real *full* balance)
However, as balance is long tested and reviewed function, I assume it's 
still safer than current "btrfsck --clear-space-cache".

But at least, a little less dangerous than --repair.
If specifying to clear space cache, it will override the --repair 
option, and only to clear space cache.
So even carelessly typed "--clear-space-cache --repair", it will only 
clear cache.

And we designed the behavior to use transaction to protect most operation.
Every space cache inode deletion is in one transaction.
If bad things happened half way, only operation successfully done will 
be write to disk.


BUT! as the patchset has not been fully reviewed, it can still cause 
problem, and further more, for some extreme case, like the whole tree 
root already contains corrupted data, it may still cause problems.


So currently, it's still dangerous than read-only fsck, but IMHO less 
dangerous than --repair/--init-extent-tree.

Thans,
Qu
>
>> FWIW, I think it's possible to do this by mounting with clear_cache and then
>> running a full balance on the filesystem.  Having an option to do this on an
>> unmounted FS would be preferred of course, as that would almost certainly
>> be more efficient on any reasonably sized filesystem.
>
> I'm running a balance now, like it's 2015! ;-)
>
> Thank you,
> Nicholas
>
>