NFS and partitioned md

NFS and partitioned md

[Martin Filip]

Hello to LKML,

few days ago I've changed my sw RAID5 to use md_d devices, which are
"partitonable". (major 254, minor dependant on partiton no)

The problem is with kernel space NFS daemon. When I create loopback
device and export it - everything works OK, but when exported directory
is directly something goes really wrong and it's not possible to mount
it.

I'm getting "nfs: server 192.168.0.2 not responding, timed out" in my
kernel log, when I look on what's happening on network, the last thing
what's there are 3 retransmitted GETATTR calls without any response. 

I'm a little bit confused because I thought that NFS should work on
filesystem and should not care about devices, but it seems that it's not
true.

Is here someone who is interested in this problematic and know whether
this is bug or feature? I've done lots of googling, but had not found
anything relevant :(

Many thanks in advance...

-- 
Martin Filip
e-mail: nexus@smoula.net
ICQ#: 31531391
jabber: nexus@smoula.net

 ________________________________________ 
/ BOFH Excuse #274: It was OK before you \
\ touched it.                            /
 ---------------------------------------- 
       \   ,__,
        \  (oo)____
           (__)    )\
              ||--|| *

Re: NFS and partitioned md

[Trond Myklebust]

On Mon, 2006-06-26 at 22:52 +0200, Martin Filip wrote:
> Hello to LKML,
> 
> few days ago I've changed my sw RAID5 to use md_d devices, which are
> "partitonable". (major 254, minor dependant on partiton no)
> 
> The problem is with kernel space NFS daemon. When I create loopback
> device and export it - everything works OK, but when exported directory
> is directly something goes really wrong and it's not possible to mount
> it.

Could we at the very least see a copy of your /etc/exports
and /etc/fstab please?

Cheers,
  Trond

Re: NFS and partitioned md

[Martin Filip]

Trond Myklebust píše v Po 26. 06. 2006 v 16:58 -0400:
> On Mon, 2006-06-26 at 22:52 +0200, Martin Filip wrote:
> > Hello to LKML,
> > 
> > few days ago I've changed my sw RAID5 to use md_d devices, which are
> > "partitonable". (major 254, minor dependant on partiton no)
> > 
> > The problem is with kernel space NFS daemon. When I create loopback
> > device and export it - everything works OK, but when exported directory
> > is directly something goes really wrong and it's not possible to mount
> > it.
> 
> Could we at the very least see a copy of your /etc/exports
> and /etc/fstab please?
of course... thought It's irelevant when it works with other devices...

$ cat /etc/exports 
/mnt/data/public            *(ro,all_squash,async)

mounted via
mount -t nfs 192.168.0.2:/mnt/data/public /mnt/tmp/


> 
> Cheers,
>   Trond
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: NFS and partitioned md

[Trond Myklebust]

On Mon, 2006-06-26 at 23:20 +0200, Martin Filip wrote:
> Trond Myklebust píše v Po 26. 06. 2006 v 16:58 -0400:
> > On Mon, 2006-06-26 at 22:52 +0200, Martin Filip wrote:
> > > Hello to LKML,
> > > 
> > > few days ago I've changed my sw RAID5 to use md_d devices, which are
> > > "partitonable". (major 254, minor dependant on partiton no)
> > > 
> > > The problem is with kernel space NFS daemon. When I create loopback
> > > device and export it - everything works OK, but when exported directory
> > > is directly something goes really wrong and it's not possible to mount
> > > it.
> > 
> > Could we at the very least see a copy of your /etc/exports
> > and /etc/fstab please?
> of course... thought It's irelevant when it works with other devices...
> 
> $ cat /etc/exports 
> /mnt/data/public            *(ro,all_squash,async)
> 
> mounted via
> mount -t nfs 192.168.0.2:/mnt/data/public /mnt/tmp/

You are running with the 'all_squash' export option: does the anonymous
user actually have the required permissions to access /mnt/data/public?

Cheers,
  Trond

Re: NFS and partitioned md

[Martin Filip]

Trond Myklebust píše v Po 26. 06. 2006 v 17:51 -0400:

> > > > few days ago I've changed my sw RAID5 to use md_d devices, which are
> > > > "partitonable". (major 254, minor dependant on partiton no)
> > > > 
> > > > The problem is with kernel space NFS daemon. When I create loopback
> > > > device and export it - everything works OK, but when exported directory
> > > > is directly something goes really wrong and it's not possible to mount
> > > > it.
> > > 
> > > Could we at the very least see a copy of your /etc/exports
> > > and /etc/fstab please?
> > of course... thought It's irelevant when it works with other devices...
> > 
> > $ cat /etc/exports 
> > /mnt/data/public            *(ro,all_squash,async)
> > 
> > mounted via
> > mount -t nfs 192.168.0.2:/mnt/data/public /mnt/tmp/
> 
> You are running with the 'all_squash' export option: does the anonymous
> user actually have the required permissions to access /mnt/data/public?
> 

Yes, of course... that dir and everything in that is world readable
(+executable when it is directory)
Everything worked with same settings, versions and everything between
switching my multiple md devices into md_d. And even now it works on
other devices than md_d.

Re: NFS and partitioned md

[Trond Myklebust]

On Tue, 2006-06-27 at 00:14 +0200, Martin Filip wrote:

> Yes, of course... that dir and everything in that is world readable
> (+executable when it is directory)
> Everything worked with same settings, versions and everything between
> switching my multiple md devices into md_d. And even now it works on
> other devices than md_d.

So would you then please help us determine what the difference between
the two setups actually is?

knfsd knows bugger all about what device you are running your filesystem
on. All it cares about is that the type of filesystem is supported.

Cheers,
  Trond

Re: NFS and partitioned md

[Neil Brown]

On Monday June 26, bugtraq@smoula.net wrote:
> Hello to LKML,
> 
> few days ago I've changed my sw RAID5 to use md_d devices, which are
> "partitonable". (major 254, minor dependant on partiton no)
> 
> The problem is with kernel space NFS daemon. When I create loopback
> device and export it - everything works OK, but when exported directory
> is directly something goes really wrong and it's not possible to mount
> it.
> 
> I'm getting "nfs: server 192.168.0.2 not responding, timed out" in my
> kernel log, when I look on what's happening on network, the last thing
> what's there are 3 retransmitted GETATTR calls without any response. 

Odd.  It works fine for me (I've had this sort of configuration on
some machines for ages, and I just tested a bleeding edge kernel and
it still works).

So I suspect there is something else going on that has nothing to do
with the usage of partitioned md.... then again, maybe there is some
weird sign extension happening to '254' somewhere, though that would
be terribly strange.

So: details please.
 What md device exactly (major and minor)
 What filesystem.
 'tcpdump -s0' trace capturing all nfs/mountd/rpc packets from before
 you issue the mount command. e.g.  use 'rpcinfo -p' to find out what
 port mountd is listening on then,

   tcpdump -s0 -w /tmp/trace host CLIENT and host SERVER and \( \
    port 2049 or port 111 or port MOUNTDPORT \)  &

 Then try the mount.

 After the experiment, on the server
    grep . /proc/net/rpc/*/content
    cat /proc/fs/nfsd/exports

That should be enough detail to start with.

NeilBrown

Re: NFS and partitioned md

[Martin Filip]

Hi,

thx for your interrest,

Neil Brown píše v Út 27. 06. 2006 v 10:32 +1000:
> Odd.  It works fine for me (I've had this sort of configuration on
> some machines for ages, and I just tested a bleeding edge kernel and
> it still works).
> 
> So I suspect there is something else going on that has nothing to do
> with the usage of partitioned md.... then again, maybe there is some
> weird sign extension happening to '254' somewhere, though that would
> be terribly strange.
> 
> So: details please.
>  What md device exactly (major and minor)
# mount | grep /mnt/data
/dev/md_d64p9 on /mnt/data type ext3 (rw)
/dev/loop/0 on /mnt/data-loop type ext3 (rw)

# ls -al /dev/md_d64p9
brw-rw---- 1 root disk 254, 4105 2006-06-27 10:17 /dev/md_d64p9
# ls -al /dev/loop0
lrwxrwxrwx 1 root root 6 2006-06-27 19:44 /dev/loop0 -> loop/0
# ls -al /dev/loop/0
brw-rw---- 1 root disk 7, 0 2006-06-27 19:45 /dev/loop/0

(as I look on that it comes on my mind, that problem could be minor
longer than 1 byte)


>  What filesystem.
as you can see above - ext3

>  'tcpdump -s0' trace capturing all nfs/mountd/rpc packets from before
>  you issue the mount command. e.g.  use 'rpcinfo -p' to find out what
>  port mountd is listening on then,
# cat /etc/exports 
/mnt/data/public            *(ro,all_squash,async)
/mnt/data-loop/public       *(ro,all_squash,async)

# rpcinfo -p  | grep mountd
    100005    1   udp    879  mountd
    100005    1   tcp    882  mountd
    100005    2   udp    879  mountd
    100005    2   tcp    882  mountd
    100005    3   udp    879  mountd
    100005    3   tcp    882  mountd

>    tcpdump -s0 -w /tmp/trace host CLIENT and host SERVER and \( \
>     port 2049 or port 111 or port MOUNTDPORT \)  &
on 192.168.0.2 (server)
# tcpdump -s0 -w /tmp/trace-data host 172.16.0.14 and host 192.168.0.2
and \( port 2049 or port 111 or port 879 or port 882 \) &

on 172.16.0.14 (client) - no fw between them
mount -t nfs 192.168.0.2:/mnt/data/public /mnt/tmp
 - fails with errors like
  * nfs: server 192.168.0.2 not responding, timed out in dmesg
 - can't read superblock from mount


> 
>  Then try the mount.
> 
>  After the experiment, on the server
>     grep . /proc/net/rpc/*/content
>     cat /proc/fs/nfsd/exports
# grep . /proc/net/rpc/*/content
/proc/net/rpc/auth.unix.ip/content:#class IP domain
/proc/net/rpc/auth.unix.ip/content:nfsd 172.16.0.14 *
/proc/net/rpc/nfsd.export/content:#path domain(flags)
/proc/net/rpc/nfsd.export/content:/mnt/data/public
*(ro,root_squash,all_squash,async,wdelay)
/proc/net/rpc/nfsd.fh/content:#domain fsidtype fsid [path]
/proc/net/rpc/nfsd.fh/content:# * 3 0x0100fe09000d4001
/proc/net/rpc/nfs4.idtoname/content:#domain type id [name]
/proc/net/rpc/nfs4.nametoid/content:#domain type name [id]

file /proc/fs/nfsd/exports does not exists


on 192.168.0.2 (server)
# tcpdump -s0 -w /tmp/trace-data-loop host 172.16.0.14 and host
192.168.0.2 and \( port 2049 or port 111 or port 879 or port 882 \) &

on 172.16.0.14 (client) - no fw between them
mount -t nfs 192.168.0.2:/mnt/data-loop/public /mnt/tmp

works OK
# grep . /proc/net/rpc/*/content
/proc/net/rpc/auth.unix.ip/content:#class IP domain
/proc/net/rpc/auth.unix.ip/content:nfsd 172.16.0.14 *
/proc/net/rpc/nfsd.export/content:#path domain(flags)
/proc/net/rpc/nfsd.export/content:/mnt/data-loop/public
*(ro,root_squash,all_squash,async,wdelay)
/proc/net/rpc/nfsd.export/content:/mnt/data/public
*(ro,root_squash,all_squash,async,wdelay)
/proc/net/rpc/nfsd.fh/content:#domain fsidtype fsid [path]
/proc/net/rpc/nfsd.fh/content:* 0
0x0000070000013881 /mnt/data-loop/public
/proc/net/rpc/nfsd.fh/content:# * 3 0x0100fe09000d4001
/proc/net/rpc/nfs4.idtoname/content:#domain type id [name]
/proc/net/rpc/nfs4.nametoid/content:#domain type name [id]

file /proc/fs/nfsd/exports does not exists

both dump files could be downloaded from:
http://www.smoula.net/lkml-nfs/

> 
> That should be enough detail to start with.
> 

I hope :)))

again - thanks for your interrest

-- 
Martin Filip
e-mail: nexus@smoula.net
ICQ#: 31531391
jabber: nexus@smoula.net
www: http://www.smoula.net

 _______________________________ 
< BOFH Excuse #86: Runt packets >
 ------------------------------- 
       \   ,__,
        \  (oo)____
           (__)    )\
              ||--|| *

Re: NFS and partitioned md

[Neil Brown]

On Tuesday June 27, bugtraq@smoula.net wrote:
> Hi,
> 
> thx for your interrest,
> 
> Neil Brown píše v Út 27. 06. 2006 v 10:32 +1000:
> > Odd.  It works fine for me (I've had this sort of configuration on
> > some machines for ages, and I just tested a bleeding edge kernel and
> > it still works).
> > 
> > So I suspect there is something else going on that has nothing to do
> > with the usage of partitioned md.... then again, maybe there is some
> > weird sign extension happening to '254' somewhere, though that would
> > be terribly strange.
> > 
> > So: details please.
> >  What md device exactly (major and minor)
> # mount | grep /mnt/data
> /dev/md_d64p9 on /mnt/data type ext3 (rw)
> /dev/loop/0 on /mnt/data-loop type ext3 (rw)
> 
> # ls -al /dev/md_d64p9
> brw-rw---- 1 root disk 254, 4105 2006-06-27 10:17 /dev/md_d64p9
> # ls -al /dev/loop0
> lrwxrwxrwx 1 root root 6 2006-06-27 19:44 /dev/loop0 -> loop/0
> # ls -al /dev/loop/0
> brw-rw---- 1 root disk 7, 0 2006-06-27 19:45 /dev/loop/0
> 
> (as I look on that it comes on my mind, that problem could be minor
> longer than 1 byte)
> 

Exactly.  4105 > 256.  Such devices need a different format filehandle
which didn't work until very recently due to a bug (obviously no-one
tried it until recently).

The patch below fixes the kernel so that this will work.
Alternately use md_d0 md_d1, md_d2, or md_d3.  Then it will work with
no patches.

NeilBrown

-----------------------------
Fixing missing 'expkey' support for fsid type 3

From: Frank Filz <ffilzlnx@us.ibm.com>

Type '3' is used for the fsid in filehandles when the device number
of the device holding the filesystem has more than 8 bits in either
major or minor.
Unfortunately expkey_parse doesn't recognise type 3.  Fix this.

(Slighty modified from Frank's original)

Signed-Off-By: Frank Filz <ffilzlnx@us.ibm.com>
Signed-off-by: Neil Brown <neilb@suse.de>

### Diffstat output
 ./fs/nfsd/export.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff .prev/fs/nfsd/export.c ./fs/nfsd/export.c
--- .prev/fs/nfsd/export.c	2006-06-29 11:07:21.000000000 +1000
+++ ./fs/nfsd/export.c	2006-06-27 18:27:49.000000000 +1000
@@ -126,7 +126,7 @@ static int expkey_parse(struct cache_det
 	if (*ep)
 		goto out;
 	dprintk("found fsidtype %d\n", fsidtype);
-	if (fsidtype > 2)
+	if (key_len(fsidtype)==0) /* invalid type */
 		goto out;
 	if ((len=qword_get(&mesg, buf, PAGE_SIZE)) <= 0)
 		goto out;

Re: NFS and partitioned md

[Martin Filip]

Hi,

Neil Brown píše v Čt 29. 06. 2006 v 13:45 +1000:
> Exactly.  4105 > 256.  Such devices need a different format filehandle
> which didn't work until very recently due to a bug (obviously no-one
> tried it until recently).
> 
> The patch below fixes the kernel so that this will work.

I'm affraid the problem will be yet somewhere else. I've tried this
patch and the result is totaly same :(


> -----------------------------
> Fixing missing 'expkey' support for fsid type 3
> 
> From: Frank Filz <ffilzlnx@us.ibm.com>
> 
> Type '3' is used for the fsid in filehandles when the device number
> of the device holding the filesystem has more than 8 bits in either
> major or minor.
> Unfortunately expkey_parse doesn't recognise type 3.  Fix this.
> 
> (Slighty modified from Frank's original)
> 
> Signed-Off-By: Frank Filz <ffilzlnx@us.ibm.com>
> Signed-off-by: Neil Brown <neilb@suse.de>
> 
> ### Diffstat output
>  ./fs/nfsd/export.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff .prev/fs/nfsd/export.c ./fs/nfsd/export.c
> --- .prev/fs/nfsd/export.c	2006-06-29 11:07:21.000000000 +1000
> +++ ./fs/nfsd/export.c	2006-06-27 18:27:49.000000000 +1000
> @@ -126,7 +126,7 @@ static int expkey_parse(struct cache_det
>  	if (*ep)
>  		goto out;
>  	dprintk("found fsidtype %d\n", fsidtype);
> -	if (fsidtype > 2)
> +	if (key_len(fsidtype)==0) /* invalid type */
>  		goto out;
>  	if ((len=qword_get(&mesg, buf, PAGE_SIZE)) <= 0)
>  		goto out;
> -

-- 
Martin Filip
e-mail: nexus@smoula.net
ICQ#: 31531391
jabber: nexus@smoula.net
www: http://www.smoula.net

 _______________________________________ 
< BOFH Excuse #20: divide-by-zero error >
 --------------------------------------- 
       \   ,__,
        \  (oo)____
           (__)    )\
              ||--|| *

Re: NFS and partitioned md

[David Greaves]

Neil Brown wrote:
> On Tuesday June 27, bugtraq@smoula.net wrote:
>> Hi,
>>
>> thx for your interrest,
>>
>> Neil Brown píše v Út 27. 06. 2006 v 10:32 +1000:
>>> So I suspect there is something else going on that has nothing to do
>>> with the usage of partitioned md.... then again, maybe there is some
>>> weird sign extension happening to '254' somewhere, though that would
>>> be terribly strange.
>> (as I look on that it comes on my mind, that problem could be minor
>> longer than 1 byte)
>>
> 
> Exactly.  4105 > 256.  Such devices need a different format filehandle
> which didn't work until very recently due to a bug (obviously no-one
> tried it until recently).
> 
> The patch below fixes the kernel so that this will work.
> Alternately use md_d0 md_d1, md_d2, or md_d3.  Then it will work with
> no patches.

FWIW (and google) I have just encountered this problem on 2.6.16.9 server.

My error message with the NFS mount failing was:
mount teak:/media /mnt/test
mount: teak:/media: can't read superblock

teak:~# ll /dev/media*
brw-rw---- 1 root disk 254, 8128 2006-07-18 18:39 /dev/media
brw-rw---- 1 root disk 254, 8129 2006-07-18 18:39 /dev/media1
brw-rw---- 1 root disk 254, 8130 2006-07-18 18:39 /dev/media2
brw-rw---- 1 root disk 254, 8131 2006-07-18 18:39 /dev/media3
brw-rw---- 1 root disk 254, 8132 2006-07-18 18:39 /dev/media4

I rebooted to use /dev/md_d0 and /dev/md_d0p1 and it's fine.

David

Re: NFS and partitioned md

[Martin Filip]

Hi,

my solution was to use fsid parameter for exports... maybe some other
mechanism for selecting fsids could be created instead of fsid = device
minor

nx

David Greaves píše v Út 18. 07. 2006 v 19:36 +0100:
> Neil Brown wrote:
> > On Tuesday June 27, bugtraq@smoula.net wrote:
> >> Hi,
> >>
> >> thx for your interrest,
> >>
> >> Neil Brown píše v Út 27. 06. 2006 v 10:32 +1000:
> >>> So I suspect there is something else going on that has nothing to do
> >>> with the usage of partitioned md.... then again, maybe there is some
> >>> weird sign extension happening to '254' somewhere, though that would
> >>> be terribly strange.
> >> (as I look on that it comes on my mind, that problem could be minor
> >> longer than 1 byte)
> >>
> > 
> > Exactly.  4105 > 256.  Such devices need a different format filehandle
> > which didn't work until very recently due to a bug (obviously no-one
> > tried it until recently).
> > 
> > The patch below fixes the kernel so that this will work.
> > Alternately use md_d0 md_d1, md_d2, or md_d3.  Then it will work with
> > no patches.
> 
> FWIW (and google) I have just encountered this problem on 2.6.16.9 server.
> 
> My error message with the NFS mount failing was:
> mount teak:/media /mnt/test
> mount: teak:/media: can't read superblock
> 
> teak:~# ll /dev/media*
> brw-rw---- 1 root disk 254, 8128 2006-07-18 18:39 /dev/media
> brw-rw---- 1 root disk 254, 8129 2006-07-18 18:39 /dev/media1
> brw-rw---- 1 root disk 254, 8130 2006-07-18 18:39 /dev/media2
> brw-rw---- 1 root disk 254, 8131 2006-07-18 18:39 /dev/media3
> brw-rw---- 1 root disk 254, 8132 2006-07-18 18:39 /dev/media4
> 
> I rebooted to use /dev/md_d0 and /dev/md_d0p1 and it's fine.
> 
> David

Re: NFS and partitioned md

[Neil Brown]

On Tuesday July 18, bugtraq@smoula.net wrote:
> Hi,
> 
> my solution was to use fsid parameter for exports... maybe some other
> mechanism for selecting fsids could be created instead of fsid = device
> minor

Yes.  Better management of fsid is on my wishlist for nfs-utils.
Unfortunately I haven't had any really clever ideas yet.

NeilBrown

Re: NFS and partitioned md

[Frank van Maarseveen]

On Thu, Jul 20, 2006 at 10:30:01AM +1000, Neil Brown wrote:
> On Tuesday July 18, bugtraq@smoula.net wrote:
> > Hi,
> > 
> > my solution was to use fsid parameter for exports... maybe some other
> > mechanism for selecting fsids could be created instead of fsid = device
> > minor
> 
> Yes.  Better management of fsid is on my wishlist for nfs-utils.
> Unfortunately I haven't had any really clever ideas yet.

I'd like to "virtualize" exports such that it is possible to transplant
disks/partitions from one machine into another without having to bother
with device numbering. One step in that direction is to derive the fsid
from an IP address. The server machine needs an additional IP address
for every export entry. This IP address is determined by deriving
a hostname from the last pathname component of the export entry and
resolving it. E.g. something like:

/etc/exports:
	/exported/path/name	*(rw,sync,no_root_squash,no_subtree_check,fsid="nfs-%s")

This would set the fsid to the IP address of host "nfs-name".

-- 
Frank

Re: [NFS] NFS and partitioned md

[Neil Brown]

On Thursday July 20, frankvm@frankvm.com wrote:
> On Thu, Jul 20, 2006 at 10:30:01AM +1000, Neil Brown wrote:
> > On Tuesday July 18, bugtraq@smoula.net wrote:
> > > Hi,
> > > 
> > > my solution was to use fsid parameter for exports... maybe some other
> > > mechanism for selecting fsids could be created instead of fsid = device
> > > minor
> > 
> > Yes.  Better management of fsid is on my wishlist for nfs-utils.
> > Unfortunately I haven't had any really clever ideas yet.
> 
> I'd like to "virtualize" exports such that it is possible to transplant
> disks/partitions from one machine into another without having to bother
> with device numbering. One step in that direction is to derive the fsid
> from an IP address. The server machine needs an additional IP address
> for every export entry. This IP address is determined by deriving
> a hostname from the last pathname component of the export entry and
> resolving it. E.g. something like:
> 
> /etc/exports:
> 	/exported/path/name	*(rw,sync,no_root_squash,no_subtree_check,fsid="nfs-%s")
> 
> This would set the fsid to the IP address of host "nfs-name".

(I'm catching up on only mail - seems I missed this...)

I think that is very specific to your particular setup, but there
certainly is bits of a possibly usable idea in there.

As the fsid is limited in size, we really need some sort of lookup
table somewhere to make between fsid and some arbitrary name for the
filesystem.
You are suggesting using the DNS for this lookup.  
Maybe that make sense..... maybe.

My leaning is to make it somebody-elses-problem by enabling a
call-out.

i.e. we declare a program that will be used for mapping between fsid
and mount point.

So: when parsing /etc/exports, if we find "fsid=??", we run the
program passing the path get an fsid.
When we get a filehandle with an unknown fsid, we pass it to the
program which will return a pathname (possible auto-mounting something
or whatever).

You could quite easily make a script that does the mapping you
require.

Maybe one day...

NeilBrown