btrfs-ino-cache runs on every boot for 6 minutes

btrfs-ino-cache runs on every boot for 6 minutes

[Szőts Ákos]

Dear list members,

I'm not sure if this a bug or an intended behaviour, since I've yet to find a 
reliable source with Google search.

My problem is the following: with the 3.12 kernel on every single boot (even 
when the computer was shut down clear) the process [btrfs-ino-cache] is 
preventing the successful startup of X because it's running for 6 minutes.

I can see this in "iotop" and while that program runs the whole system is 
being blocked. After 6 minutes I restart X and everyting goes normal from that 
point on.

My question is: is that necessary to regenerate the inode cache on every boot 
or is this a bug? Of course if I remove the "inode_cache" option from fstab, 
this phenomenon disappears.

Versions:
- OS: openSUSE 13.1
- Kernel: 3.12.0-6.ge7c00d8-desktop #1 SMP PREEMPT Tue Nov 12 13:09:24 UTC 
2013 (e7c00d8)
- Mount options: compress=lzo,space_cache,inode_cache,noatime

Best regards,

Ákos

Re: btrfs-ino-cache runs on every boot for 6 minutes

[Duncan]

Szőts Ákos posted on Tue, 03 Dec 2013 12:28:26 +0100 as excerpted:

> Dear list members,
> 
> I'm not sure if this a bug or an intended behaviour, since I've yet to
> find a reliable source with Google search.
> 
> My problem is the following: with the 3.12 kernel on every single boot
> (even when the computer was shut down clear) the process
> [btrfs-ino-cache] is preventing the successful startup of X because it's
> running for 6 minutes.
> 
> I can see this in "iotop" and while that program runs the whole system
> is being blocked. After 6 minutes I restart X and everyting goes normal
> from that point on.
> 
> My question is: is that necessary to regenerate the inode cache on every
> boot or is this a bug? Of course if I remove the "inode_cache" option
> from fstab, this phenomenon disappears.
> 
> Versions:
> - OS: openSUSE 13.1 - Kernel: 3.12.0-6.ge7c00d8-desktop #1 SMP PREEMPT
> Tue Nov 12 13:09:24 UTC 2013 (e7c00d8)
> - Mount options: compress=lzo,space_cache,inode_cache,noatime

>From what I've seen (as a btrfs using admin and list regular, /not/ a 
dev, btrfs or otherwise), the inode_cache mount option isn't intended or 
recommended for general purpose use.  Its intended purpose is, I believe, 
for cases such as mail servers, for example, where a sluggish startup 
isn't a big issue, but where quickly and efficiently creating/deleting 
lots of little files is part of the job description.

The general recommendation thus appears to be to omit the inode_cache 
mount option unless your use-case requires it, and the X-based user 
desktop use-case almost certainly doesn't.


Meanwhile, briefly on the other options:  While continuous use of the 
space_cache option won't hurt anything, it really only needs turned on 
once, after which it's on permanently unless you specifically disable it 
again.  Noatime's a good choice on most filesystems, but especially btrfs 
if you're using snapshotting (as I believe OpenSuSE makes heavy use of), 
so good choice there.  And I'm using compress=lzo here too. =:^)

You may wish to consider adding autodefrag also, altho if you've not been 
running with it from the beginning, performance may be bad for awhile 
until it has caught back up with existing fragmentation, but then it 
should be better.  One way around that is to defrag (or rebalance, which 
rewrites everything and thus defrags in the process) everything, which 
could take awhile on a multi-terabyte spinning-rust drive, then mount 
with autodefrag from then on, to avoid heavy fragmentation happening 
again.  That used to require a rather convoluted command, but a new 
enough btrfs-progs and kernel should let you use the btrfs filesystem 
defrag -r (recursive) option. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs-ino-cache runs on every boot for 6 minutes

[Liu Bo]

On Tue, Dec 03, 2013 at 12:28:26PM +0100, Szőts Ákos wrote:
> Dear list members,
> 
> I'm not sure if this a bug or an intended behaviour, since I've yet to find a 
> reliable source with Google search.
> 
> My problem is the following: with the 3.12 kernel on every single boot (even 
> when the computer was shut down clear) the process [btrfs-ino-cache] is 
> preventing the successful startup of X because it's running for 6 minutes.
> 
> I can see this in "iotop" and while that program runs the whole system is 
> being blocked. After 6 minutes I restart X and everyting goes normal from that 
> point on.

6min is way too much, something may go wrongly,
could you please run "echo w > /proc/sysrq-trigger" when that happens?

-liubo

> 
> My question is: is that necessary to regenerate the inode cache on every boot 
> or is this a bug? Of course if I remove the "inode_cache" option from fstab, 
> this phenomenon disappears.
> 
> Versions:
> - OS: openSUSE 13.1
> - Kernel: 3.12.0-6.ge7c00d8-desktop #1 SMP PREEMPT Tue Nov 12 13:09:24 UTC 
> 2013 (e7c00d8)
> - Mount options: compress=lzo,space_cache,inode_cache,noatime
> 
> Best regards,
> 
> Ákos
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-ino-cache runs on every boot for 6 minutes

[Szőts Ákos]

Thank you for your answer.

Here are the "blocked states" logs. Since I had a plenty of time, I made 
multiple of them. The last two are the longest of them.

- http://paste.opensuse.org/view/raw/51654551
- http://paste.opensuse.org/view/raw/39005796
- http://paste.opensuse.org/view/raw/38028651
- http://paste.opensuse.org/view/raw/21592065
- http://paste.opensuse.org/view/raw/46655344
- http://paste.opensuse.org/view/raw/26821272

Ákos

Re: btrfs-ino-cache runs on every boot for 6 minutes

[Liu Bo]

On Wed, Dec 04, 2013 at 01:41:42PM +0100, Szőts Ákos wrote:
> Thank you for your answer.
> 
> Here are the "blocked states" logs. Since I had a plenty of time, I made 
> multiple of them. The last two are the longest of them.
> 
> - http://paste.opensuse.org/view/raw/51654551
> - http://paste.opensuse.org/view/raw/39005796
> - http://paste.opensuse.org/view/raw/38028651
> - http://paste.opensuse.org/view/raw/21592065
> - http://paste.opensuse.org/view/raw/46655344
> - http://paste.opensuse.org/view/raw/26821272
> 
> Ákos

Could you please try the following and show me the output of dmesg?

thanks,
-liubo


diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 057be95..8f7711e 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -2950,11 +2950,16 @@ int load_free_ino_cache(struct btrfs_fs_info *fs_info, struct btrfs_root *root)
 		return 0;
 
 	inode = lookup_free_ino_inode(root, path);
-	if (IS_ERR(inode))
+	if (IS_ERR(inode)) {
+		printk("%s: ino cache ino is NOT found\n", __func__);
 		goto out;
+	}
 
-	if (root_gen != BTRFS_I(inode)->generation)
+	if (root_gen != BTRFS_I(inode)->generation) {
+		printk("%s: gen mismatch: root_gen_mem=%llu, root_gen=%llu, ino's gen=%llu\n",
+			__func__, btrfs_root_generation(&root->root_item), root_gen, BTRFS_I(inode)->generation);
 		goto out_put;
+	}
 
 	ret = __load_free_space_cache(root, inode, ctl, path, 0);
 
diff --git a/fs/btrfs/inode-map.c b/fs/btrfs/inode-map.c
index ab485e5..6535139 100644
--- a/fs/btrfs/inode-map.c
+++ b/fs/btrfs/inode-map.c
@@ -473,7 +473,12 @@ again:
 	}
 
 	spin_lock(&root->cache_lock);
-	if (root->cached != BTRFS_CACHE_FINISHED) {
+
+	/* ino cache has 3 state:
+	 * BTRFS_CACHE_NO, BTRFS_CACHE_START, BTRFS_CACHE_FINISHED
+	 */
+	if (root->cached != BTRFS_CACHE_NO && root->cached != BTRFS_CACHE_FINISHED) {
+		printk("%s: bail out because it's not FINISHED, cached=%d\n", __func__, root->cached);
 		ret = -1;
 		spin_unlock(&root->cache_lock);
 		goto out_put;

Re: btrfs-ino-cache runs on every boot for 6 minutes

[Szőts Ákos]

A következőt írtad ekkor: 2013. december 6. 14:04:19
> Could you please try the following and show me the output of dmesg?
> 
> thanks,
> -liubo

The following happened:
- I upgraded to 3.13.0-rc2-6.ge7c00d8 (since the original 3.12 kernel with 
SUSE patches is not available anymore)
- Patched it and booted from that, compiled the NVIDIA driver
- and suddenly everything started to work fine. Now I don't have to wait 6 
minutes for the [btrfs-ino-cache] process.

One thing I noticed: before any logging would start, kernel prints out some 
(initramfs?) messages, like:
"Waiting for device /dev/disk/... to appear
[...]
fsck succeeded Mounting root read-write
Mounting root /dev/disk/...
mount -o rw,compress=lzo,space_cache,noatime,inode_cache /dev/... /root"

And logging starts from this point.

The interesting point is the initramfs mounting options: with the stock 3.12 
kernel I had only the following: rw,compress=lzo,space_cache. But now it's 
rw,compress=lzo,space_cache,noatime,inode_cache. In /etc/fstab the options are 
intact for a month now. Maybe I should have regenerated the initrd when I 
added the fstab mount options noatime and inode_cache?

I searched for your extra kernel messages in dmesg, but I didn't find them, 
although here is the dmesg output: http://paste.opensuse.org/30797527

Thank you,

Ákos

Re: btrfs-ino-cache runs on every boot for 6 minutes

[Liu Bo]

On Fri, Dec 06, 2013 at 01:46:10PM +0100, Szőts Ákos wrote:
> A következőt írtad ekkor: 2013. december 6. 14:04:19
> > Could you please try the following and show me the output of dmesg?
> > 
> > thanks,
> > -liubo
> 
> The following happened:
> - I upgraded to 3.13.0-rc2-6.ge7c00d8 (since the original 3.12 kernel with 
> SUSE patches is not available anymore)
> - Patched it and booted from that, compiled the NVIDIA driver
> - and suddenly everything started to work fine. Now I don't have to wait 6 
> minutes for the [btrfs-ino-cache] process.
> 
> One thing I noticed: before any logging would start, kernel prints out some 
> (initramfs?) messages, like:
> "Waiting for device /dev/disk/... to appear
> [...]
> fsck succeeded Mounting root read-write
> Mounting root /dev/disk/...
> mount -o rw,compress=lzo,space_cache,noatime,inode_cache /dev/... /root"
> 
> And logging starts from this point.
> 
> The interesting point is the initramfs mounting options: with the stock 3.12 
> kernel I had only the following: rw,compress=lzo,space_cache. But now it's 
> rw,compress=lzo,space_cache,noatime,inode_cache. In /etc/fstab the options are 
> intact for a month now. Maybe I should have regenerated the initrd when I 
> added the fstab mount options noatime and inode_cache?
> 
> I searched for your extra kernel messages in dmesg, but I didn't find them, 
> although here is the dmesg output: http://paste.opensuse.org/30797527

It seems that we should owe credit to the updated 3.13 kernel, could you
please try again 3.13 kernel without the patch?

thanks,
-liubo

Re: btrfs-ino-cache runs on every boot for 6 minutes

[Szőts Ákos]

Liu Bo a következőt írta ekkor: 2013. december 6. 21:30:34
> It seems that we should owe credit to the updated 3.13 kernel, could you
> please try again 3.13 kernel without the patch?
> 
> thanks,
> -liubo

I forcibly reinstalled the 3.13 kernel from the repository, booted from it 
twice and yes, it really seems that this bug is fixed now.

Thank you for your time.

Ákos

Re: btrfs-ino-cache runs on every boot for 6 minutes

[Szőts Ákos]

Dear Duncan,

Thank you for your advices!

Unfortunately I haven't found a good documentation on when I should use a 
cache_, that's why I thought "it's all the same, won't hurt" :).
Now, if this bug is fixed, I'll turn off the inode_cache, if it's unnecessary.

About autodefrag: yes, I'm considering running a total defragmentation on my 
whole drive ("/"), but unfortunately there is a bug which prevented me from 
doing it: it's a defrag panic which was fixed in 3.13.

So either I wait until that kernel is out or I start a rebalance. The latter 
seems better now :).

Thank you.

Ákos
Ps.: sorry for not replying to your letter in the thread but that didn't 
arrive in my mailbox.