fscache review comments, part 1

fscache review comments, part 1

[Christoph Hellwig]

Just to be on the same boat, all these comments are against
http://people.redhat.com/~dhowells/nfs/nfs+fscache-16.tar.bz2

This mail is just the core-touching paches, the fscache and cachefiles
code will get a review of it's own later on.

01-ino64.diff:
	ACK

02-ino64-nfs.diff:
	Unfortunately there's a lot of broken userspace that can't deal
	with 64bit inode numbers, so you need to make the lod behaviour
	a mount option at least, probably even the default.  Given that
	we're going to run into problems like that it might make sense
	to make the option VFS-level instead of just in nfs.  (Note:
	XFS already has an option like that)

03-fsmisc.diff:
	ACK

05-release-page.diff:
	ACK, though I'd wish someone could come up with nicer names for
	read_cache_pages_invalidate_page(s).

06-block-afs.diff:
	ACK.

09-cachefiles-ia64.diff:
	ACK.  (Although I remember at some point I wanted to unify
	copy_page and copy_highpage)

11-autofs-dcache.diff:
	ACK.  Should probably go in ASAP.

12-reiserfs-dcache.diff:
	ACK.  Again should probably go in ASAP.

	Note that reiserfs_kill_sb could be written a lot nicer by
	using a local variable.

static void reiserfs_kill_sb(struct super_block *sb)
{
	struct reiserfs_sb_info *si = REISERFS_SB(s);

	if (si) {
		if (si->xattr_root) {
			d_invalidate(si->xattr_root);
			dput(si->xattr_root);
		}
		if (si->priv_root) {
			d_invalidate(si->priv_root);
			dput(si->priv_root);
		}
	}

	kill_block_super(sb);
}

13-dcache-crunch.diff:
	I know Jan has been rather unhappy with this, and I tend to agree
	to at least the code duplication part.  I wish I had time to
	look into a better way to implement the rather nice idea behind
	shrink_dcache_for_umount.
	Why is this patch required for fscache anyway?
	(and why is it last in the series then?)

Re: fscache review comments, part 1

[Andreas Dilger]

On Sep 28, 2006  17:45 +0100, Christoph Hellwig wrote:
> 02-ino64-nfs.diff:
> 	Unfortunately there's a lot of broken userspace that can't deal
> 	with 64bit inode numbers, so you need to make the lod behaviour
> 	a mount option at least, probably even the default.  Given that
> 	we're going to run into problems like that it might make sense
> 	to make the option VFS-level instead of just in nfs.  (Note:
> 	XFS already has an option like that)

A general question - is there a known list of common applications that have
problems with 64-bit inodes?  It's been mentioned several times, but I'm
wondering if any effort is going toward fixing those apps so that 5 years
from now (or whenever) we CAN have 64-bit inodes.  If it relates to on-disk
formats like ustar, we can always work with something like star to ensure
that it is useful in the future.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: fscache review comments, part 1

[Christoph Hellwig]

On Thu, Sep 28, 2006 at 11:30:17AM -0600, Andreas Dilger wrote:
> On Sep 28, 2006  17:45 +0100, Christoph Hellwig wrote:
> > 02-ino64-nfs.diff:
> > 	Unfortunately there's a lot of broken userspace that can't deal
> > 	with 64bit inode numbers, so you need to make the lod behaviour
> > 	a mount option at least, probably even the default.  Given that
> > 	we're going to run into problems like that it might make sense
> > 	to make the option VFS-level instead of just in nfs.  (Note:
> > 	XFS already has an option like that)
> 
> A general question - is there a known list of common applications that have
> problems with 64-bit inodes?  It's been mentioned several times, but I'm
> wondering if any effort is going toward fixing those apps so that 5 years
> from now (or whenever) we CAN have 64-bit inodes.  If it relates to on-disk
> formats like ustar, we can always work with something like star to ensure
> that it is useful in the future.

The most important ones we've seen at SGI were various version of tar
and many commercial backup applications that silently (!) lost data
on restore with 64bit inode numbers.  I wasn't directly involved with
those issue, but maybe some people that were at SGI at that time can
fill in more details.

Re: fscache review comments, part 1

[Timothy Shimmin]

Christoph Hellwig wrote:
> On Thu, Sep 28, 2006 at 11:30:17AM -0600, Andreas Dilger wrote:
>> On Sep 28, 2006  17:45 +0100, Christoph Hellwig wrote:
>>> 02-ino64-nfs.diff:
>>> 	Unfortunately there's a lot of broken userspace that can't deal
>>> 	with 64bit inode numbers, so you need to make the lod behaviour
>>> 	a mount option at least, probably even the default.  Given that
>>> 	we're going to run into problems like that it might make sense
>>> 	to make the option VFS-level instead of just in nfs.  (Note:
>>> 	XFS already has an option like that)
>> A general question - is there a known list of common applications that have
>> problems with 64-bit inodes?  It's been mentioned several times, but I'm
>> wondering if any effort is going toward fixing those apps so that 5 years
>> from now (or whenever) we CAN have 64-bit inodes.  If it relates to on-disk
>> formats like ustar, we can always work with something like star to ensure
>> that it is useful in the future.
> 
> The most important ones we've seen at SGI were various version of tar
> and many commercial backup applications that silently (!) lost data
> on restore with 64bit inode numbers.  I wasn't directly involved with
> those issue, but maybe some people that were at SGI at that time can
> fill in more details.

I wasn't involved with this much either, but I know people have
cited Networker a number of times.

--Tim

Re: fscache review comments, part 1

[David Howells]

Christoph Hellwig <hch@infradead.org> wrote:

> 	Why is this patch required for fscache anyway?

It's not, it's just there in my patchset because that's where I developed it.
I should probably drop it from there as Andrew is now carrying it.

> 	(and why is it last in the series then?)

Because it is.

David

Re: fscache review comments, part 1

[David Howells]

Christoph Hellwig <hch@infradead.org> wrote:

> 	Unfortunately there's a lot of broken userspace that can't deal
> 	with 64bit inode numbers, so you need to make the lod behaviour
> 	a mount option at least, probably even the default.  Given that
> 	we're going to run into problems like that it might make sense
> 	to make the option VFS-level instead of just in nfs.  (Note:
> 	XFS already has an option like that)

This really needs fixing quite urgently.  We've seen it break ld.so/libdl (and
other things, but dynamic loading is one of the major pains as it's not
entirely obvious).

David

Re: fscache review comments, part 1

[Andrew Morton]

On Mon, 02 Oct 2006 14:40:54 +0100
David Howells <dhowells@redhat.com> wrote:

> Christoph Hellwig <hch@infradead.org> wrote:
> 
> > 	Unfortunately there's a lot of broken userspace that can't deal
> > 	with 64bit inode numbers, so you need to make the lod behaviour
> > 	a mount option at least, probably even the default.  Given that
> > 	we're going to run into problems like that it might make sense
> > 	to make the option VFS-level instead of just in nfs.  (Note:
> > 	XFS already has an option like that)
> 
> This really needs fixing quite urgently.  We've seen it break ld.so/libdl (and
> other things, but dynamic loading is one of the major pains as it's not
> entirely obvious).
> 

Speaking of which..  I've been having struggling a bit with 

vfs-make-filldir_t-and-struct-kstat-deal-in-64-bit-inode-numbers.patch
nfs-represent-64-bit-fileids-as-64-bit-inode-numbers-on-32-bit-systems.patch

I don't have a good understanding of what the implications are for userspace.
What are the risks and what are the advantages and what the level of urgency
is behind those patches.

Help?

Re: fscache review comments, part 1

[Christoph Hellwig]

On Mon, Oct 02, 2006 at 10:39:00AM -0700, Andrew Morton wrote:
> On Mon, 02 Oct 2006 14:40:54 +0100
> David Howells <dhowells@redhat.com> wrote:
> 
> > Christoph Hellwig <hch@infradead.org> wrote:
> > 
> > > 	Unfortunately there's a lot of broken userspace that can't deal
> > > 	with 64bit inode numbers, so you need to make the lod behaviour
> > > 	a mount option at least, probably even the default.  Given that
> > > 	we're going to run into problems like that it might make sense
> > > 	to make the option VFS-level instead of just in nfs.  (Note:
> > > 	XFS already has an option like that)
> > 
> > This really needs fixing quite urgently.  We've seen it break ld.so/libdl (and
> > other things, but dynamic loading is one of the major pains as it's not
> > entirely obvious).
> > 
> 
> Speaking of which..  I've been having struggling a bit with 
> 
> vfs-make-filldir_t-and-struct-kstat-deal-in-64-bit-inode-numbers.patch
> nfs-represent-64-bit-fileids-as-64-bit-inode-numbers-on-32-bit-systems.patch

Well, that's the two patches I've commented on above :)

> I don't have a good understanding of what the implications are for userspace.
> What are the risks and what are the advantages and what the level of urgency
> is behind those patches.

vfs-make-filldir_t-and-struct-kstat-deal-in-64-bit-inode-numbers.patch is
perfectly fine and harmless, seems fine for 2.6.19.

nfs-represent-64-bit-fileids-as-64-bit-inode-numbers-on-32-bit-systems.patch
is quite risky as I mentioned so I'd drop it for the time beeing - we
shouldn't put it in as long as we don't have a mount option for it.

Re: fscache review comments, part 1

[Al Viro]

On Thu, Sep 28, 2006 at 05:45:29PM +0100, Christoph Hellwig wrote:
> 02-ino64-nfs.diff:
> 	Unfortunately there's a lot of broken userspace that can't deal
> 	with 64bit inode numbers, so you need to make the lod behaviour
> 	a mount option at least, probably even the default.  Given that
> 	we're going to run into problems like that it might make sense
> 	to make the option VFS-level instead of just in nfs.  (Note:
> 	XFS already has an option like that)

I don't think it belongs at VFS level.  Sure, NFS might implement that,
but...  I'd still like to see real-life examples of such breakage before
bothering with extra complexity.

64-bit inode number issues

[David Howells]

Christoph Hellwig <hch@infradead.org> wrote:

> 01-ino64.diff:
> 	ACK
> 
> 02-ino64-nfs.diff:
> 	Unfortunately there's a lot of broken userspace that can't deal
> 	with 64bit inode numbers, so you need to make the lod behaviour
> 	a mount option at least, probably even the default.  Given that
> 	we're going to run into problems like that it might make sense
> 	to make the option VFS-level instead of just in nfs.  (Note:
> 	XFS already has an option like that)

This problem doesn't just apply to NFS.  It applies to any filesystem that can
generate 64-bit inode numbers - which includes Ext3, I believe - so not
applying the NFS patch doesn't really solve your problem (Al Viro asked me for
examples of that, btw) and leaves my problem unsolved.

I think any solution has to apply at the VFS level or higher.  I'm not sure Al
agrees though.  I think we have to do one of the following:

 (1) Turn on inode number compression (which tries to hide the problem).

 (2) Give an error to stat and getdents if the inode number can't be
     represented by 32-bits.

And I think we're going to do this, that we need to do this on a per-superblock
basis or on a per-system basis, and I think it should be _off_ by default.  It
might also be possible to do it on a per-process level, but I suspect that'd be
too much effort.

David

Re: 64-bit inode number issues

[David Howells]

David Howells <dhowells@redhat.com> wrote:

> This problem doesn't just apply to NFS.  It applies to any filesystem that can
> generate 64-bit inode numbers - which includes Ext3, I believe

Actually, Ext3 can't do 64-bit inode numbers.  I'd misremembered what Stephen
told me.

David

Re: 64-bit inode number issues

[Christoph Hellwig]

On Wed, Oct 04, 2006 at 03:18:21PM +0100, David Howells wrote:
> > 	Unfortunately there's a lot of broken userspace that can't deal
> > 	with 64bit inode numbers, so you need to make the lod behaviour
> > 	a mount option at least, probably even the default.  Given that
> > 	we're going to run into problems like that it might make sense
> > 	to make the option VFS-level instead of just in nfs.  (Note:
> > 	XFS already has an option like that)
> 
> This problem doesn't just apply to NFS.  It applies to any filesystem that can
> generate 64-bit inode numbers - which includes Ext3, I believe - so not
> applying the NFS patch doesn't really solve your problem (Al Viro asked me for
> examples of that, btw) and leaves my problem unsolved.

As you mention in your next mail ext3 is not affected.  XFS for example
is, that's why it has a 32bit inode numbers only mount option already.

> I think any solution has to apply at the VFS level or higher.  I'm not sure Al
> agrees though.  I think we have to do one of the following:
> 
>  (1) Turn on inode number compression (which tries to hide the problem).
> 
>  (2) Give an error to stat and getdents if the inode number can't be
>      represented by 32-bits.
> 
> And I think we're going to do this, that we need to do this on a per-superblock
> basis or on a per-system basis, and I think it should be _off_ by default.  It
> might also be possible to do it on a per-process level, but I suspect that'd be
> too much effort.

This definitly should be on a per-superblock basis, aka the mount option
I suggested above.    I don't think an actual scheme to deal with the
inode numbers at the VFS level makes sense, what I meant was only to do
a vfs-level mount option (numerical) instead of a per-filesystem one.

Al commented he doesn't like that.  He also think we should give the
64bit inode numbers a try, so I'd say:

 o add an inode32 option for nfs that mirrors the XFS option
 o turn it off by default in -mm and see what goes boom

Re: 64-bit inode number issues

[Theodore Tso]

On Wed, Oct 04, 2006 at 03:23:49PM +0100, David Howells wrote:
> David Howells <dhowells@redhat.com> wrote:
> 
> > This problem doesn't just apply to NFS.  It applies to any filesystem that can
> > generate 64-bit inode numbers - which includes Ext3, I believe
> 
> Actually, Ext3 can't do 64-bit inode numbers.  I'd misremembered what Stephen
> told me.

Nope, we've thought about using 64-bit inode numbers for ext4, however
--- it would solve a number of problems for us.  Unfortunately, given
issues with 64-bit inods at the VFS, glibc, and userspace layers, we
decided it wasn't worth the pain and suffering that would be
involved....

						- Ted

Re: 64-bit inode number issues

[Alexander Viro]

On Sat, Oct 07, 2006 at 10:00:35PM -0400, Theodore Tso wrote:
> On Wed, Oct 04, 2006 at 03:23:49PM +0100, David Howells wrote:
> > David Howells <dhowells@redhat.com> wrote:
> > 
> > > This problem doesn't just apply to NFS.  It applies to any filesystem that can
> > > generate 64-bit inode numbers - which includes Ext3, I believe
> > 
> > Actually, Ext3 can't do 64-bit inode numbers.  I'd misremembered what Stephen
> > told me.
> 
> Nope, we've thought about using 64-bit inode numbers for ext4, however
> --- it would solve a number of problems for us.  Unfortunately, given
> issues with 64-bit inods at the VFS, glibc, and userspace layers, we
> decided it wasn't worth the pain and suffering that would be
> involved....

VFS is fine with those...  glibc probably should be OK; userland might
be nasty, though.

Re: 64-bit inode number issues

[David Howells]

Christoph Hellwig <hch@infradead.org> wrote:

> Al commented he doesn't like that.  He also think we should give the
> 64bit inode numbers a try, so I'd say:
> 
>  o add an inode32 option for nfs that mirrors the XFS option
>  o turn it off by default in -mm and see what goes boom

I believe -mm is currently carrying the 64-bit inodes for NFS patch.

David

Re: 64-bit inode number issues

[David Chinner]

On Sat, Oct 07, 2006 at 10:01:32PM +0100, Christoph Hellwig wrote:
> I don't think an actual scheme to deal with the
> inode numbers at the VFS level makes sense, what I meant was only to do
> a vfs-level mount option (numerical) instead of a per-filesystem one.

Different fileystems might want different defaults - XFS (and every
other filesystem) will still want 32 bit inodes by default right
now, while NFS will probably want 64 bit inodes. That might
complicate a vfs-level mount option, especially given that we don't
want the XFS mount options to change.

> Al commented he doesn't like that.  He also think we should give the
> 64bit inode numbers a try, so I'd say:
> 
>  o add an inode32 option for nfs that mirrors the XFS option

XFS uses an "inode64" option to turn on 64 bit inodes. If that is
not specified it uses 32 bit inodes - there is no "inode32" option
in XFS.

>  o turn it off by default in -mm and see what goes boom

Will we even see the conditions for the NFS client to go boom in the
environments -mm kernels are typically run? i.e. does someone have a
test case that reliably triggers problems?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: 64-bit inode number issues

[David Howells]

David Chinner <dgc@sgi.com> wrote:

> >  o turn it off by default in -mm and see what goes boom
> 
> Will we even see the conditions for the NFS client to go boom in the
> environments -mm kernels are typically run? i.e. does someone have a
> test case that reliably triggers problems?

I don't know about that, but we (Red Hat) have had customers logging the
problem in our Bugzilla against NFS.

Coming up with a reliable test case is a little tricky, as it involves setting
up a server to generate fileids > 0xffffffff, and manipulating inode numbers
directly generally isn't allowed...  Maybe it can be done with a ramfs hack.

David

Re: fscache review comments, part 1

[David Howells]

Christoph Hellwig <hch@infradead.org> wrote:

> 05-release-page.diff:
> 	ACK, though I'd wish someone could come up with nicer names for
> 	read_cache_pages_invalidate_page(s).

They could always be nested:-).  That way they don't pollute the namespace.
This is one of the gcc extensions.  If you do nm on the output, you see the
nested function appear like this if not inlined:

	000000000000058f t invalidate_page.17245

That above derives from this:


int read_cache_pages(struct address_space *mapping, struct list_head *pages,
			int (*filler)(void *, struct page *), void *data)
{
	struct page *page;
	struct pagevec lru_pvec;
	int ret = 0;

	/* see if a page needs releasing upon read_cache_pages() failure */
	void invalidate_page(struct address_space *mapping, struct page *page)
	{
		if (PagePrivate(page)) {
			if (TestSetPageLocked(page))
				BUG();
			page->mapping = mapping;
			do_invalidatepage(page, 0);
			page->mapping = NULL;
			unlock_page(page);
		}
		page_cache_release(page);
	}

	/* release a list of pages, invalidating them first if need be */
	void invalidate_pages(struct address_space *mapping,
			      struct list_head *pages)
	{
		struct page *victim;

		while (!list_empty(pages)) {
			victim = list_to_page(pages);
			list_del(&victim->lru);
			invalidate_page(mapping, victim);
		}
	}

	pagevec_init(&lru_pvec, 0);

	while (!list_empty(pages)) {
		page = list_to_page(pages);
		list_del(&page->lru);
		if (add_to_page_cache(page, mapping, page->index, GFP_KERNEL)) {
			invalidate_page(mapping, page);
			continue;
		}
		ret = filler(data, page);
		if (!pagevec_add(&lru_pvec, page))
			__pagevec_lru_add(&lru_pvec);
		if (ret) {
			invalidate_pages(mapping, pages);
			break;
		}
	}
	pagevec_lru_add(&lru_pvec);
	return ret;
}

David

Re: 64-bit inode number issues

[Arjan van de Ven]

On Mon, 2006-10-09 at 12:32 +0100, David Howells wrote:
> David Chinner <dgc@sgi.com> wrote:
> 
> > >  o turn it off by default in -mm and see what goes boom
> > 
> > Will we even see the conditions for the NFS client to go boom in the
> > environments -mm kernels are typically run? i.e. does someone have a
> > test case that reliably triggers problems?
> 
> I don't know about that, but we (Red Hat) have had customers logging the
> problem in our Bugzilla against NFS.
> 
> Coming up with a reliable test case is a little tricky, as it involves setting
> up a server to generate fileids > 0xffffffff, and manipulating inode numbers
> directly generally isn't allowed...  Maybe it can be done with a ramfs hack.


you can do a debug hack inside the kernel, where on 32 bit inode fs's
you copy the lower 32 bit into the higher 32 bit (and maybe invert the
bits or something); this can then be verified for all kinds of things
all over the place ;)

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com

Re: 64-bit inode number issues

[David Chinner]

On Mon, Oct 09, 2006 at 12:32:14PM +0100, David Howells wrote:
> David Chinner <dgc@sgi.com> wrote:
> 
> > >  o turn it off by default in -mm and see what goes boom
> > 
> > Will we even see the conditions for the NFS client to go boom in the
> > environments -mm kernels are typically run? i.e. does someone have a
> > test case that reliably triggers problems?
> 
> I don't know about that, but we (Red Hat) have had customers logging the
> problem in our Bugzilla against NFS.
> 
> Coming up with a reliable test case is a little tricky, as it involves setting
> up a server to generate fileids > 0xffffffff, and manipulating inode numbers
> directly generally isn't allowed...  Maybe it can be done with a ramfs hack.

64 bit machine, >1TB sized XFS filesystem on the server, inode64
mount option, fill the filesystem to 99% full and then create lots
of inodes. You should get inodes above 1TB then, and that will give
you 64 bit inode numbers.

Typically the way we do this quickly is create an XFS filesystem
then use the ag-wipe tool (*) we have in our QA suite that uses
xfs_db to "fill" up all but the last AG in the filesystem by poking
the right numbers into on disk structures.  This is significantly
faster than filling a terabyte of disk space.....

(*)http://oss.sgi.com/cgi-bin/cvsweb.cgi/xfs-cmds/xfstests/tools/ag-wipe

FWIW, if you don't have more than a TB of disk space, then you might
be able to create an XFS filesystem on loopback using a sparse file
and then using the ag-wipe tool on the loopback filesystem.

HTH.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: fscache review comments, part 1

[Christoph Hellwig]

On Mon, Oct 09, 2006 at 02:46:40PM +0100, David Howells wrote:
> Christoph Hellwig <hch@infradead.org> wrote:
> 
> > 05-release-page.diff:
> > 	ACK, though I'd wish someone could come up with nicer names for
> > 	read_cache_pages_invalidate_page(s).
> 
> They could always be nested:-).  That way they don't pollute the namespace.
> This is one of the gcc extensions.  If you do nm on the output, you see the
> nested function appear like this if not inlined:
> 
> 	000000000000058f t invalidate_page.17245
> 
> That above derives from this:

This is C, not pascal - please don't do such fugly things..

Re: 64-bit inode number issues

[Benjamin LaHaise]

On Sat, Oct 07, 2006 at 10:01:32PM +0100, Christoph Hellwig wrote:
> Al commented he doesn't like that.  He also think we should give the
> 64bit inode numbers a try, so I'd say:
> 
>  o add an inode32 option for nfs that mirrors the XFS option
>  o turn it off by default in -mm and see what goes boom

Unfortunately, that technique will not find any problems -- the people 
running systems with 64 bit inode numbers do not overlap with those 
running the -mm tree.  The only real way to get any exercising of the 
relevant code paths would be to fake 64 bit inode numbers in a commonly 
used filesystem like ext3.  The biggest difficultly with LFS has been 
that the API breakage is relatively silent; old applications work fine 
unless they encounter a large file, resulting in many lax applications.

		-ben
-- 
"Time is of no importance, Mr. President, only life is important."
Don't Email: <dont@kvack.org>.

Re: 64-bit inode number issues

[Chris Wedgwood]

On Thu, Oct 12, 2006 at 02:32:32PM -0400, Benjamin LaHaise wrote:

> The only real way to get any exercising of the relevant code paths
> would be to fake 64 bit inode numbers in a commonly used filesystem
> like ext3.

Why not add a config option (or boot-time flash) or even mount option
to have *existing* inode numbers shifted up 32-bits so that any
userspace breakage will be detected really early on?