Fwd: [suggestion] Add verbose notification about inode-cache rebuild to kernel log

Fwd: [suggestion] Add verbose notification about inode-cache rebuild to kernel log

[Tomáš Pružina]

Good day,
I ran into some troubles with inode-cache rebuilding on root fs after
filesystem was mounted without inode_cache, which stalls boot of my
box by several minutes.

I boot from commandline like:
root=/dev/sda4 rootfstype=btrfs
rootflags=inode_cache,space_cache,autodefrag rw ...

However when I manually remount root fs without inode_cache (for
example via kexec), it triggers cache rebuild at next mount which
takes several tens of minutes and mimick 'freeze' on boot.

Would it be possible to put some printk saying that this is happening
so users don't get confused about it??

Thanks.

Re: Fwd: [suggestion] Add verbose notification about inode-cache rebuild to kernel log

[David Sterba]

On Thu, May 08, 2014 at 04:22:13PM +0200, Tomáš Pružina wrote:
> I ran into some troubles with inode-cache rebuilding on root fs after
> filesystem was mounted without inode_cache, which stalls boot of my
> box by several minutes.
> 
> I boot from commandline like:
> root=/dev/sda4 rootfstype=btrfs
> rootflags=inode_cache,space_cache,autodefrag rw ...
> 
> However when I manually remount root fs without inode_cache (for
> example via kexec), it triggers cache rebuild at next mount which
> takes several tens of minutes and mimick 'freeze' on boot.
> 
> Would it be possible to put some printk saying that this is happening
> so users don't get confused about it??

Adding the printk is probably a good thing, but I'd rather reconsider
using inode_cache at all. IMO it's supposed to fix problems with inode
numbers that we don't have.

https://btrfs.wiki.kernel.org/index.php/Project_ideas#Free_inode_number_cache

"As the filesystem fills up, finding a free inode number will become
expensive. This should be cached the same way we do free blocks."

Without inode_cache the new inode number is a simple increment from the
last number, in the 64bit space, per subvolume. We're not going to
reach the limits anytime soon.

The documentation does not say anything about that, I'll update the
wiki.

Re: Fwd: [suggestion] Add verbose notification about inode-cache rebuild to kernel log

[Goffredo Baroncelli]

On 05/09/2014 04:51 PM, David Sterba wrote:
> On Thu, May 08, 2014 at 04:22:13PM +0200, Tomáš Pružina wrote:
>> I ran into some troubles with inode-cache rebuilding on root fs after
>> filesystem was mounted without inode_cache, which stalls boot of my
>> box by several minutes.
>>
>> I boot from commandline like:
>> root=/dev/sda4 rootfstype=btrfs
>> rootflags=inode_cache,space_cache,autodefrag rw ...
>>
>> However when I manually remount root fs without inode_cache (for
>> example via kexec), it triggers cache rebuild at next mount which
>> takes several tens of minutes and mimick 'freeze' on boot.
>>
>> Would it be possible to put some printk saying that this is happening
>> so users don't get confused about it??
> 
> Adding the printk is probably a good thing, but I'd rather reconsider
> using inode_cache at all. IMO it's supposed to fix problems with inode
> numbers that we don't have.

IIRC, the problem is for the 32 bit system, were the API wants the inode number to be 32 bit..

In the past the inode_cache was introduced because the inode often reached 2^32 [1]

Because most modern hardware is 64 bit (with the exception of ARM ?), could be make sense to allow btrfs to work without inode_cache only on 64bit, loosing the possibility to be used on 32 bit system.

Instead when the inode_cache is used, btrfs still be compatible with both 32bit and 64bit systems.

How often a filesystem is moved between a 64bit and 32bit systems ?


[1] http://article.gmane.org/gmane.comp.file-systems.btrfs/9573
-- 
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Fwd: [suggestion] Add verbose notification about inode-cache rebuild to kernel log

[David Sterba]

On Fri, May 09, 2014 at 08:12:54PM +0200, Goffredo Baroncelli wrote:
> > Adding the printk is probably a good thing, but I'd rather reconsider
> > using inode_cache at all. IMO it's supposed to fix problems with inode
> > numbers that we don't have.
> 
> IIRC, the problem is for the 32 bit system, were the API wants the inode number to be 32 bit..
> 
> In the past the inode_cache was introduced because the inode often reached 2^32 [1]

Right, it makes sense for the 32bit case under some circumstances.
However I don't see where's "often reached 2^32" mentioned in the link.
If it's really 'often', then there's something wrong. A 32bit machine
serving large amounts of files? Or, it could reach the limit over time,
say a few years, so I'm not saying it can't happen in practice.

Then, inode_cache is the workaround for this usecase and can be turned on
if things are going to be bad.

Another workaround, that requires more manual intervention, is to create
a new subvolume and reflink all the files there.

> Because most modern hardware is 64 bit (with the exception of ARM ?),
> could be make sense to allow btrfs to work without inode_cache only on
> 64bit, loosing the possibility to be used on 32 bit system.
> 
> Instead when the inode_cache is used, btrfs still be compatible with
> both 32bit and 64bit systems.

There are 32bit machines in use and inode_cache brings some
compatibility assurance. Just that I would not recommend it to use by
default.

> How often a filesystem is moved between a 64bit and 32bit systems ?

Not so often I'd say, but for example the block device on a 32bit
machine can be exported via iscsi to a 64bit machine temporarily, then
the changes made on the 64bit host can make it unusable.

Thanks for the pointer, I think this is a documentation issue, we can
keep inode_cache code as-is.

Re: Fwd: [suggestion] Add verbose notification about inode-cache rebuild to kernel log

[Goffredo Baroncelli]

Hi David,

On 05/12/2014 04:39 PM, David Sterba wrote:
>> Because most modern hardware is 64 bit (with the exception of ARM ?),
>> > could be make sense to allow btrfs to work without inode_cache only on
>> > 64bit, loosing the possibility to be used on 32 bit system.
>> > 
>> > Instead when the inode_cache is used, btrfs still be compatible with
>> > both 32bit and 64bit systems.
> There are 32bit machines in use and inode_cache brings some
> compatibility assurance. Just that I would not recommend it to use by
> default.
> 
>> > How often a filesystem is moved between a 64bit and 32bit systems ?
> Not so often I'd say, but for example the block device on a 32bit
> machine can be exported via iscsi to a 64bit machine temporarily, then
> the changes made on the 64bit host can make it unusable.


To be clear, I would like to avoid inode_cache on 64bit machine.

In order to avoid the risk to exhaust the inode on 32bit and to be backward compatible with what already exists, we could add a flag to mkfs.btrfs to be only 64bit compatible and avoid inode_cache.

It it would be faster, I am sure that there are uses cases for that.

The major drawback that I see, is that it would be a code path less tested: I think that the 32bit system[*] would disappear soon

Finally I have a question: it is possible to disable inode_cache ? what means the flag "noinode_cache" ? It means disable the inode cache at all, or only avoid to store on disk the inode cache ?


[*] I means 32bit system were it is reasonable that btrfs would run.

BR
G.Baroncelli
-- 
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Fwd: [suggestion] Add verbose notification about inode-cache rebuild to kernel log

[Duncan]

Goffredo Baroncelli posted on Mon, 12 May 2014 20:18:04 +0200 as
excerpted:

> Finally I have a question: it is possible to disable inode_cache ? what
> means the flag "noinode_cache" ? It means disable the inode cache at
> all, or only avoid to store on disk the inode cache ?

Unlike space_cache, which is enabled by default these days, inode_cache 
must be specifically enabled every time it is to be used.  So simply not 
including the inode_cache mount option when doing the original mount 
means it's disabled.

Meanwhile, the noinode_cache flag is simply one of the mount-option 
negators, to allow turning off a mount option on an existing mount when 
it is remounted, to avoid having to fully umount and do a fresh mount, to 
deactivate the option.

So simply don't include inode_cache when doing the original mount to 
disable it, or if for whatever reason it's enabled and you want to 
disable it on a remount, use -o remount,noinode_cache for the remount, 
and it'll disable inode_cache if the filesystem was already mounted with 
it enabled.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Fwd: [suggestion] Add verbose notification about inode-cache rebuild to kernel log

[David Sterba]

On Mon, May 12, 2014 at 08:18:04PM +0200, Goffredo Baroncelli wrote:
> To be clear, I would like to avoid inode_cache on 64bit machine.
>
> In order to avoid the risk to exhaust the inode on 32bit and to be
> backward compatible with what already exists, we could add a flag to
> mkfs.btrfs to be only 64bit compatible and avoid inode_cache.

I thought about such bit as well but don't see a strong need for it, at
the moment the inode_cache is the only option that could prevent the
incompatibilities, but otherwise all features are designed to work on
32bits the same way as on 64bit.

So, my position here is not enforce anything, but document all the pros
and cons of the option and let the user decide.

Possibly, add more safety checks that would detect that a filesystem is
not safely mountable on 32bit. Eg. look for the highest inode num if
it's a 32bit system and inode_cache is not up-to-date, or warn early
that the inode space is running out.

> It it would be faster, I am sure that there are uses cases for that.
> 
> The major drawback that I see, is that it would be a code path less
> tested: I think that the 32bit system[*] would disappear soon

That's a valid point and for that reason I would not add more code.

> Finally I have a question: it is possible to disable inode_cache ?
> what means the flag "noinode_cache" ? It means disable the inode cache
> at all, or only avoid to store on disk the inode cache ?

The noinode_cache disables using the cache at runtime, it will be kept
on disk though (there's no "clear inode_cache" as for the space cache).