2.6.17-ck1: fcache problem...

2.6.17-ck1: fcache problem...

[Paolo Ornati]

Hello,

I'm trying to get fcache (2.6.17-ck1) working on my machine, but it
works just once ;)

In Gentoo the root fs is remounted rw in "checkroot" init script, so
I've done this:

--- /etc/init.d/checkroot.orig  2006-06-24 18:47:41.000000000 +0200
+++ /etc/init.d/checkroot       2006-06-25 09:17:52.000000000 +0200
@@ -70,8 +70,10 @@
        if mount -vf -o remount / 2> /dev/null | \
           awk '{ if ($6 ~ /rw/) exit 0; else exit 1; }'
        then
-               ebegin "Remounting root filesystem read/write"
-               mount -n -o remount,rw / &> /dev/null
+               ebegin "Remounting root filesystem read/write (FCACHE)"
+               # try with fcache
+               mount -n -o remount,rw,fcache_dev=8/10,fcache_prime=0 / &> /dev/null || \
+                       mount -n -o remount,rw / &> /dev/null
                if [[ $? -ne 0 ]] ; then
                        eend 2 "Root filesystem could not be mounted read/write :("
                        if [[ ${RC_FORCE_AUTO} != "yes" ]] ; then


1) priming - it works

[  167.488268] fcache: ios r/w 8304/4747, hits 0, misses 0, overwrites 1217
[  167.882597] fcache: wrote 8304 extents, holding 347648 sectors of data
[  167.899555] fcache: wrote header (extents=8304,serial=33)

"remounting with priming=0"

[  167.905498] fcache: header looks valid (extents=8304 extents, serial=33)
[  167.928273] fcache: loaded 8304 extents
[  167.928320] fcache: sda10 opened successfully (not priming)


2) first boot with priming=0 - it works! Great speedup :)

[   37.845964] fcache: header looks valid (extents=8304 extents, serial=33)
[   37.874101] fcache: loaded 8304 extents
[   37.874105] fcache: sda10 opened successfully (not priming)


3) reboot - it doesn't work anymore :(

[   26.673525] fcache: found serial 33, expected 34.
[   26.673529] fcache: reprime the cache!
[   26.673535] ext3: failed to open fcache (err=-22)



I'm doing something wrong?

-- 
	Paolo Ornati
	Linux 2.6.17-ck1 on x86_64

Re: 2.6.17-ck1: fcache problem...

[Jens Axboe]

On Sun, Jun 25 2006, Paolo Ornati wrote:
> Hello,
> 
> I'm trying to get fcache (2.6.17-ck1) working on my machine, but it
> works just once ;)
> 
> In Gentoo the root fs is remounted rw in "checkroot" init script, so
> I've done this:
> 
> --- /etc/init.d/checkroot.orig  2006-06-24 18:47:41.000000000 +0200
> +++ /etc/init.d/checkroot       2006-06-25 09:17:52.000000000 +0200
> @@ -70,8 +70,10 @@
>         if mount -vf -o remount / 2> /dev/null | \
>            awk '{ if ($6 ~ /rw/) exit 0; else exit 1; }'
>         then
> -               ebegin "Remounting root filesystem read/write"
> -               mount -n -o remount,rw / &> /dev/null
> +               ebegin "Remounting root filesystem read/write (FCACHE)"
> +               # try with fcache
> +               mount -n -o remount,rw,fcache_dev=8/10,fcache_prime=0 / &> /dev/null || \
> +                       mount -n -o remount,rw / &> /dev/null
>                 if [[ $? -ne 0 ]] ; then
>                         eend 2 "Root filesystem could not be mounted read/write :("
>                         if [[ ${RC_FORCE_AUTO} != "yes" ]] ; then
> 
> 
> 1) priming - it works
> 
> [  167.488268] fcache: ios r/w 8304/4747, hits 0, misses 0, overwrites 1217
> [  167.882597] fcache: wrote 8304 extents, holding 347648 sectors of data
> [  167.899555] fcache: wrote header (extents=8304,serial=33)
> 
> "remounting with priming=0"
> 
> [  167.905498] fcache: header looks valid (extents=8304 extents, serial=33)
> [  167.928273] fcache: loaded 8304 extents
> [  167.928320] fcache: sda10 opened successfully (not priming)
> 
> 
> 2) first boot with priming=0 - it works! Great speedup :)
> 
> [   37.845964] fcache: header looks valid (extents=8304 extents, serial=33)
> [   37.874101] fcache: loaded 8304 extents
> [   37.874105] fcache: sda10 opened successfully (not priming)
> 
> 
> 3) reboot - it doesn't work anymore :(
> 
> [   26.673525] fcache: found serial 33, expected 34.
> [   26.673529] fcache: reprime the cache!
> [   26.673535] ext3: failed to open fcache (err=-22)

Hmm, and you are sure that the fs is properly umounted on reboot? Or is
it just remounted ro? It looks like fcache_close_dev() isn't being
called, so the cache serial doesn't match what we expect from the fs,
hence fcache bails out since it could indicate that the fs has been
changed without fcache being attached.

What kind of speedup did you see?

-- 
Jens Axboe

Re: 2.6.17-ck1: fcache problem...

[Paolo Ornati]

On Sun, 25 Jun 2006 12:28:39 +0200
Jens Axboe <axboe@suse.de> wrote:

> > [   26.673525] fcache: found serial 33, expected 34.
> > [   26.673529] fcache: reprime the cache!
> > [   26.673535] ext3: failed to open fcache (err=-22)
> 
> Hmm, and you are sure that the fs is properly umounted on reboot? Or is
> it just remounted ro? It looks like fcache_close_dev() isn't being
> called, so the cache serial doesn't match what we expect from the fs,
> hence fcache bails out since it could indicate that the fs has been
> changed without fcache being attached.

Ahh... it is the root fs and it's just remounted read-only by the
standard Gentoo scripts ;)

I don't think that unmounting it is trivial (you need to chroot to a
virtual FS or something...). Does any distro do it?

> 
> What kind of speedup did you see?

with cold caches...

NORMAL
from Grub to KDM login		42"5	(~6" to reach init)
KDE 3.5.3 startup		10"
Firefox				7"

FCACHE
from Grub to KDM login		31"4	(~6" to reach init)
KDE 3.5.3 startup		4"
Firefox				2"4


:)

-- 
	Paolo Ornati
	Linux 2.6.17.1 on x86_64

Re: 2.6.17-ck1: fcache problem...

[Jens Axboe]

On Sun, Jun 25 2006, Paolo Ornati wrote:
> On Sun, 25 Jun 2006 12:28:39 +0200
> Jens Axboe <axboe@suse.de> wrote:
> 
> > > [   26.673525] fcache: found serial 33, expected 34.
> > > [   26.673529] fcache: reprime the cache!
> > > [   26.673535] ext3: failed to open fcache (err=-22)
> > 
> > Hmm, and you are sure that the fs is properly umounted on reboot? Or is
> > it just remounted ro? It looks like fcache_close_dev() isn't being
> > called, so the cache serial doesn't match what we expect from the fs,
> > hence fcache bails out since it could indicate that the fs has been
> > changed without fcache being attached.
> 
> Ahh... it is the root fs and it's just remounted read-only by the
> standard Gentoo scripts ;)
> 
> I don't think that unmounting it is trivial (you need to chroot to a
> virtual FS or something...). Does any distro do it?

ro should be enough, something odd must be going on. I'll add it to the
list of things to test tomorrow.

> > What kind of speedup did you see?
> 
> with cold caches...
> 
> NORMAL
> from Grub to KDM login		42"5	(~6" to reach init)
> KDE 3.5.3 startup		10"
> Firefox				7"
> 
> FCACHE
> from Grub to KDM login		31"4	(~6" to reach init)
> KDE 3.5.3 startup		4"
> Firefox				2"4

Very nice!

-- 
Jens Axboe

Re: 2.6.17-ck1: fcache problem...

[Paolo Ornati]

On Sun, 25 Jun 2006 19:43:59 +0200
Jens Axboe <axboe@suse.de> wrote:

> > > Hmm, and you are sure that the fs is properly umounted on reboot? Or is
> > > it just remounted ro? It looks like fcache_close_dev() isn't being
> > > called, so the cache serial doesn't match what we expect from the fs,
> > > hence fcache bails out since it could indicate that the fs has been
> > > changed without fcache being attached.
> > 
> > Ahh... it is the root fs and it's just remounted read-only by the
> > standard Gentoo scripts ;)
> > 
> > I don't think that unmounting it is trivial (you need to chroot to a
> > virtual FS or something...). Does any distro do it?
> 
> ro should be enough, something odd must be going on. I'll add it to the
> list of things to test tomorrow.

Since "fcache_close_dev()" is called by "ext3_put_super()" I have added
this stupid printk:

--- fs/ext3/super.c.orig        2006-06-27 10:47:15.000000000 +0200
+++ fs/ext3/super.c     2006-06-27 10:50:36.000000000 +0200
@@ -422,6 +422,8 @@ static void ext3_put_super (struct super

        has_fcache = test_opt(sb, FCACHE);

+       printk("!!! ext3_put_super !!!   has_fcache=%d\n", has_fcache);
+
        ext3_xattr_put_super(sb);
        journal_destroy(sbi->s_journal);
        if (!(sb->s_flags & MS_RDONLY)) {


It triggers on unmount but it doesn't on remount "ro".

So the problem is that "fcache_close_dev()" have zero chances to run ;)

-- 
	Paolo Ornati
	Linux 2.6.17-ck1 on x86_64

Re: 2.6.17-ck1: fcache problem...

[Jens Axboe]

On Tue, Jun 27 2006, Paolo Ornati wrote:
> On Sun, 25 Jun 2006 19:43:59 +0200
> Jens Axboe <axboe@suse.de> wrote:
> 
> > > > Hmm, and you are sure that the fs is properly umounted on reboot? Or is
> > > > it just remounted ro? It looks like fcache_close_dev() isn't being
> > > > called, so the cache serial doesn't match what we expect from the fs,
> > > > hence fcache bails out since it could indicate that the fs has been
> > > > changed without fcache being attached.
> > > 
> > > Ahh... it is the root fs and it's just remounted read-only by the
> > > standard Gentoo scripts ;)
> > > 
> > > I don't think that unmounting it is trivial (you need to chroot to a
> > > virtual FS or something...). Does any distro do it?
> > 
> > ro should be enough, something odd must be going on. I'll add it to the
> > list of things to test tomorrow.
> 
> Since "fcache_close_dev()" is called by "ext3_put_super()" I have added
> this stupid printk:
> 
> --- fs/ext3/super.c.orig        2006-06-27 10:47:15.000000000 +0200
> +++ fs/ext3/super.c     2006-06-27 10:50:36.000000000 +0200
> @@ -422,6 +422,8 @@ static void ext3_put_super (struct super
> 
>         has_fcache = test_opt(sb, FCACHE);
> 
> +       printk("!!! ext3_put_super !!!   has_fcache=%d\n", has_fcache);
> +
>         ext3_xattr_put_super(sb);
>         journal_destroy(sbi->s_journal);
>         if (!(sb->s_flags & MS_RDONLY)) {
> 
> 
> It triggers on unmount but it doesn't on remount "ro".
> 
> So the problem is that "fcache_close_dev()" have zero chances to run ;)

Hmm, you seem to be using an older fcache version. I'll test this and
post an updated patch if it doesn't work with the newer version.

-- 
Jens Axboe

Re: 2.6.17-ck1: fcache problem...

[Paolo Ornati]

On Tue, 27 Jun 2006 11:54:44 +0200
Jens Axboe <axboe@suse.de> wrote:

> > So the problem is that "fcache_close_dev()" have zero chances to run ;)
> 
> Hmm, you seem to be using an older fcache version. I'll test this and
> post an updated patch if it doesn't work with the newer version.

I'm using the one that comes with "2.6.17-ck1" as the subject suggest :)

Anyway I've just seen this in your git tree:


@@ -2222,6 +2283,7 @@ static int ext3_remount (struct super_bl

[CUT]

        ext3_init_journal_params(sb, sbi->s_journal);

+       if (fcache_devnum) {
+               ext3_close_fcache(sb);
+               ext3_open_fcache(sb, fcache_devnum);
+       }


So I suppose that it's already fixed.

:)

-- 
	Paolo Ornati
	Linux 2.6.17-ck1 on x86_64

Re: 2.6.17-ck1: fcache problem...

[Paolo Ornati]

On Tue, 27 Jun 2006 12:24:57 +0200
Paolo Ornati <ornati@fastwebnet.it> wrote:

> So I suppose that it's already fixed.
> 
> :)

Oppss... even with your GIT tree it doesn't work :(

But this time I think I've found the problem:

--- fs/ext3/super.c ---

ext3_remount()
{
...
	unsigned long fcache_devnum = 0;
...
	if (!parse_options(data, sb, NULL, NULL, &fcache_devnum, &n_blocks_count, 1))
...
        if (fcache_devnum) {
                ext3_close_fcache(sb);
                ext3_open_fcache(sb, fcache_devnum);
        }
...
}

---------------------


If I understand correctly I have to pass the "fdev=..." option not only
when remounting "rw" but even when remounting "ro" before shutdown or
reboot.

Cannot the code figure out this himself al let "mount -o remount,ro"
work?

-- 
	Paolo Ornati
	Linux 2.6.17-gd2581eb4 on x86_64

Re: 2.6.17-ck1: fcache problem...

[Jens Axboe]

On Tue, Jun 27 2006, Paolo Ornati wrote:
> On Tue, 27 Jun 2006 12:24:57 +0200
> Paolo Ornati <ornati@fastwebnet.it> wrote:
> 
> > So I suppose that it's already fixed.
> > 
> > :)
> 
> Oppss... even with your GIT tree it doesn't work :(
> 
> But this time I think I've found the problem:
> 
> --- fs/ext3/super.c ---
> 
> ext3_remount()
> {
> ...
> 	unsigned long fcache_devnum = 0;
> ...
> 	if (!parse_options(data, sb, NULL, NULL, &fcache_devnum, &n_blocks_count, 1))
> ...
>         if (fcache_devnum) {
>                 ext3_close_fcache(sb);
>                 ext3_open_fcache(sb, fcache_devnum);
>         }
> ...
> }
> 
> ---------------------
> 
> 
> If I understand correctly I have to pass the "fdev=..." option not only
> when remounting "rw" but even when remounting "ro" before shutdown or
> reboot.
> 
> Cannot the code figure out this himself al let "mount -o remount,ro"
> work?

It could be fixed up, yes, but for now you have to always pass the fdev
option for it to work. Sorry, I thought you knew that, I think I wrote
that in the original mail as well.

But it needs to be fixed of course, also so you don't have to do it for
'rw' remounts (which I sometimes do just to check stats).

-- 
Jens Axboe

Re: 2.6.17-ck1: fcache problem...

[Paolo Ornati]

On Tue, 27 Jun 2006 15:10:33 +0200
Jens Axboe <axboe@suse.de> wrote:

> > Cannot the code figure out this himself al let "mount -o remount,ro"
> > work?
> 
> It could be fixed up, yes, but for now you have to always pass the fdev
> option for it to work. Sorry, I thought you knew that, I think I wrote
> that in the original mail as well.

No ;)

http://lkml.org/lkml/2006/5/15/46

you talked about remounting rw at boot (modifing distro script) and
remounting for stopping priming.

> 
> But it needs to be fixed of course, also so you don't have to do it for
> 'rw' remounts (which I sometimes do just to check stats).

I agree :)

-- 
	Paolo Ornati
	Linux 2.6.17-gd2581eb4 on x86_64

Re: 2.6.17-ck1: fcache problem...

[Jens Axboe]

On Tue, Jun 27 2006, Paolo Ornati wrote:
> On Tue, 27 Jun 2006 15:10:33 +0200
> Jens Axboe <axboe@suse.de> wrote:
> 
> > > Cannot the code figure out this himself al let "mount -o remount,ro"
> > > work?
> > 
> > It could be fixed up, yes, but for now you have to always pass the fdev
> > option for it to work. Sorry, I thought you knew that, I think I wrote
> > that in the original mail as well.
> 
> No ;)
> 
> http://lkml.org/lkml/2006/5/15/46
> 
> you talked about remounting rw at boot (modifing distro script) and
> remounting for stopping priming.

I guess not, I've not noticed since my modified suse boot.rootfs script
use FCACHE_ARGS for both mount rw and remount ro. I does tell you to use
fdev for rw remounts though, so it's not like I didn't hint at it at
least :-)

> > But it needs to be fixed of course, also so you don't have to do it for
> > 'rw' remounts (which I sometimes do just to check stats).
> 
> I agree :)

Will fix it up, for now just pass fdev= always.

-- 
Jens Axboe

Re: 2.6.17-ck1: fcache problem...

[Paolo Ornati]

On Tue, 27 Jun 2006 15:35:45 +0200
Jens Axboe <axboe@suse.de> wrote:

> > > But it needs to be fixed of course, also so you don't have to do it for
> > > 'rw' remounts (which I sometimes do just to check stats).
> > 
> > I agree :)
> 
> Will fix it up, for now just pass fdev= always.

Ok.

-- 
	Paolo Ornati
	Linux 2.6.17-gd2581eb4 on x86_64