Linux client mount fails with Gluster NFSv3 server

Linux client mount fails with Gluster NFSv3 server

[Shehjar Tikoo]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]

Hi All

I am writing a NFSv3 server as part of the Gluster clustered FS.
To start with, I've implemented the Mountv3 protocol and am just
starting out with NFSv3. In NFSv3, the first thing I've implemented
is the FSINFO and GETATTR calls to support mounting with NFS client.

The problem I am facing is this. The Linux NFS client fails to mount
the remote export even though it is successfully receiving the file
handle from the MNT request and the result of the FSINFO call. This
is shown in the attached pcap file, which would be best viewed through
wireshark with "rpc" as the display filter.

The command line output is shown below:

root@indus:statcache# mount 127.0.0.1:/pos1 /mnt -o noacl,nolock
mount.nfs: mounting 127.0.0.1:/pos1 failed, reason given by server:
   No such file or directory

This happens even though, we're told the following by showmount.
root@indus:statcache# showmount -e
Export list for indus:
/pos1 (everyone)
/pos2 (everyone)
/pos3 (everyone)
/pos4 (everyone)
root@indus:statcache#

..where /pos1, /pos2, etc are exports from the locally running Gluster
NFS server.

As you'll notice in the trace, there is no NFSv3 request after
the FSINFO, so I've a feeling it could be that some field in the
FSINFO reply is not what the Linux NFS client is expecting. Could that
be the reason for the mount failure?

What else should I be looking into to investigate this further?

The client is a 2.6.18-5 kernel supplied with Debian on an AMD64 box.
nfs-utils is version 1.1.4.

Many thanks,
-Shehjar

[-- Attachment #2: mount.dump --]
[-- Type: application/octet-stream, Size: 6718 bytes --]

Re: Linux client mount fails with Gluster NFSv3 server

[Trond Myklebust]

On Mon, 2009-08-31 at 19:37 +0530, Shehjar Tikoo wrote:
> Hi All
> 
> I am writing a NFSv3 server as part of the Gluster clustered FS.
> To start with, I've implemented the Mountv3 protocol and am just
> starting out with NFSv3. In NFSv3, the first thing I've implemented
> is the FSINFO and GETATTR calls to support mounting with NFS client.
> 
> The problem I am facing is this. The Linux NFS client fails to mount
> the remote export even though it is successfully receiving the file
> handle from the MNT request and the result of the FSINFO call. This
> is shown in the attached pcap file, which would be best viewed through
> wireshark with "rpc" as the display filter.
> 
> The command line output is shown below:
> 
> root@indus:statcache# mount 127.0.0.1:/pos1 /mnt -o noacl,nolock
> mount.nfs: mounting 127.0.0.1:/pos1 failed, reason given by server:
>    No such file or directory
> 
> This happens even though, we're told the following by showmount.
> root@indus:statcache# showmount -e
> Export list for indus:
> /pos1 (everyone)
> /pos2 (everyone)
> /pos3 (everyone)
> /pos4 (everyone)
> root@indus:statcache#
> 
> ..where /pos1, /pos2, etc are exports from the locally running Gluster
> NFS server.
> 
> As you'll notice in the trace, there is no NFSv3 request after
> the FSINFO, so I've a feeling it could be that some field in the
> FSINFO reply is not what the Linux NFS client is expecting. Could that
> be the reason for the mount failure?
> 
> What else should I be looking into to investigate this further?
> 
> The client is a 2.6.18-5 kernel supplied with Debian on an AMD64 box.
> nfs-utils is version 1.1.4.
> 
> Many thanks,
> -Shehjar

Wireshark fails to decode your server's reply too. I'd start looking
there...

Cheers
  Trond

Re: Linux client mount fails with Gluster NFSv3 server

[Shehjar Tikoo]

Trond Myklebust wrote:
> On Mon, 2009-08-31 at 19:37 +0530, Shehjar Tikoo wrote:
>> Hi All
>>
>> I am writing a NFSv3 server as part of the Gluster clustered FS.
>> To start with, I've implemented the Mountv3 protocol and am just
>> starting out with NFSv3. In NFSv3, the first thing I've implemented
>> is the FSINFO and GETATTR calls to support mounting with NFS client.
>>
>> The problem I am facing is this. The Linux NFS client fails to mount
>> the remote export even though it is successfully receiving the file
>> handle from the MNT request and the result of the FSINFO call. This
>> is shown in the attached pcap file, which would be best viewed through
>> wireshark with "rpc" as the display filter.
>>
>> The command line output is shown below:
>>
>> root@indus:statcache# mount 127.0.0.1:/pos1 /mnt -o noacl,nolock
>> mount.nfs: mounting 127.0.0.1:/pos1 failed, reason given by server:
>>    No such file or directory
>>
>> This happens even though, we're told the following by showmount.
>> root@indus:statcache# showmount -e
>> Export list for indus:
>> /pos1 (everyone)
>> /pos2 (everyone)
>> /pos3 (everyone)
>> /pos4 (everyone)
>> root@indus:statcache#
>>
>> ..where /pos1, /pos2, etc are exports from the locally running Gluster
>> NFS server.
>>
>> As you'll notice in the trace, there is no NFSv3 request after
>> the FSINFO, so I've a feeling it could be that some field in the
>> FSINFO reply is not what the Linux NFS client is expecting. Could that
>> be the reason for the mount failure?
>>
>> What else should I be looking into to investigate this further?
>>
>> The client is a 2.6.18-5 kernel supplied with Debian on an AMD64 box.
>> nfs-utils is version 1.1.4.
>>
>> Many thanks,
>> -Shehjar
> 
> Wireshark fails to decode your server's reply too. I'd start looking
> there...
> 

Bruce, Trond,

I am able to view the packets just fine using wireshark Version 1.0.6.
It is possible that the default options for you for TCP and RPC are
not same as the ones below.
Could you please try viewing the dump with the following options set 
in the wireshark Protocol preferences pane.

Press <Ctrl> + <Shift> + p to bring up the protocol preferences
window.

First, expand the "Protocol" section header in the window that pops
up. Then look for "TCP" section. In the TCP section, please check the 
following option:

"Allow subdissector to reassemble TCP streams"

Then, search for the "RPC" section under "Protocols". For RPC, please
check the following option:

"Reassemble RPC over TCP message spanning multiple TCP segments"

This should make the RPC records visible properly.

Thanks
Shehjar


> Cheers
>   Trond
> 

Re: Linux client mount fails with Gluster NFSv3 server

[Trond Myklebust]

On Tue, 2009-09-01 at 12:09 +0530, Shehjar Tikoo wrote:
> Trond Myklebust wrote:
> > On Mon, 2009-08-31 at 19:37 +0530, Shehjar Tikoo wrote:
> >> Hi All
> >>
> >> I am writing a NFSv3 server as part of the Gluster clustered FS.
> >> To start with, I've implemented the Mountv3 protocol and am just
> >> starting out with NFSv3. In NFSv3, the first thing I've implemented
> >> is the FSINFO and GETATTR calls to support mounting with NFS client.
> >>
> >> The problem I am facing is this. The Linux NFS client fails to mount
> >> the remote export even though it is successfully receiving the file
> >> handle from the MNT request and the result of the FSINFO call. This
> >> is shown in the attached pcap file, which would be best viewed through
> >> wireshark with "rpc" as the display filter.
> >>
> >> The command line output is shown below:
> >>
> >> root@indus:statcache# mount 127.0.0.1:/pos1 /mnt -o noacl,nolock
> >> mount.nfs: mounting 127.0.0.1:/pos1 failed, reason given by server:
> >>    No such file or directory
> >>
> >> This happens even though, we're told the following by showmount.
> >> root@indus:statcache# showmount -e
> >> Export list for indus:
> >> /pos1 (everyone)
> >> /pos2 (everyone)
> >> /pos3 (everyone)
> >> /pos4 (everyone)
> >> root@indus:statcache#
> >>
> >> ..where /pos1, /pos2, etc are exports from the locally running Gluster
> >> NFS server.
> >>
> >> As you'll notice in the trace, there is no NFSv3 request after
> >> the FSINFO, so I've a feeling it could be that some field in the
> >> FSINFO reply is not what the Linux NFS client is expecting. Could that
> >> be the reason for the mount failure?
> >>
> >> What else should I be looking into to investigate this further?
> >>
> >> The client is a 2.6.18-5 kernel supplied with Debian on an AMD64 box.
> >> nfs-utils is version 1.1.4.
> >>
> >> Many thanks,
> >> -Shehjar
> > 
> > Wireshark fails to decode your server's reply too. I'd start looking
> > there...
> > 
> 
> Bruce, Trond,
> 
> I am able to view the packets just fine using wireshark Version 1.0.6.
> It is possible that the default options for you for TCP and RPC are
> not same as the ones below.
> Could you please try viewing the dump with the following options set 
> in the wireshark Protocol preferences pane.
> 
> Press <Ctrl> + <Shift> + p to bring up the protocol preferences
> window.
> 
> First, expand the "Protocol" section header in the window that pops
> up. Then look for "TCP" section. In the TCP section, please check the 
> following option:
> 
> "Allow subdissector to reassemble TCP streams"
> 
> Then, search for the "RPC" section under "Protocols". For RPC, please
> check the following option:
> 
> "Reassemble RPC over TCP message spanning multiple TCP segments"
> 
> This should make the RPC records visible properly.

I always run with those options enabled, and they were able to
reconstruct most of the RPC records correctly, but not the reply to the
FSINFO call.
Furthermore, when I looked at the binary contents, it seemed to me that
the post-op attributes contained some fishy information, such as
nlink==0. That alone would cause the NFS client to give up.

Trond

Re: Linux client mount fails with Gluster NFSv3 server

[Shehjar Tikoo]

Trond Myklebust wrote:
> On Tue, 2009-09-01 at 12:09 +0530, Shehjar Tikoo wrote:
>> Trond Myklebust wrote:
>>> On Mon, 2009-08-31 at 19:37 +0530, Shehjar Tikoo wrote:
>>>> Hi All
>>>>
>>>> I am writing a NFSv3 server as part of the Gluster clustered FS.
>>>> To start with, I've implemented the Mountv3 protocol and am just
>>>> starting out with NFSv3. In NFSv3, the first thing I've implemented
>>>> is the FSINFO and GETATTR calls to support mounting with NFS client.
>>>>
>>>> The problem I am facing is this. The Linux NFS client fails to mount
>>>> the remote export even though it is successfully receiving the file
>>>> handle from the MNT request and the result of the FSINFO call. This
>>>> is shown in the attached pcap file, which would be best viewed through
>>>> wireshark with "rpc" as the display filter.
>>>>
>>>> The command line output is shown below:
>>>>
>>>> root@indus:statcache# mount 127.0.0.1:/pos1 /mnt -o noacl,nolock
>>>> mount.nfs: mounting 127.0.0.1:/pos1 failed, reason given by server:
>>>>    No such file or directory
>>>>
>>>> This happens even though, we're told the following by showmount.
>>>> root@indus:statcache# showmount -e
>>>> Export list for indus:
>>>> /pos1 (everyone)
>>>> /pos2 (everyone)
>>>> /pos3 (everyone)
>>>> /pos4 (everyone)
>>>> root@indus:statcache#
>>>>
>>>> ..where /pos1, /pos2, etc are exports from the locally running Gluster
>>>> NFS server.
>>>>
>>>> As you'll notice in the trace, there is no NFSv3 request after
>>>> the FSINFO, so I've a feeling it could be that some field in the
>>>> FSINFO reply is not what the Linux NFS client is expecting. Could that
>>>> be the reason for the mount failure?
>>>>
>>>> What else should I be looking into to investigate this further?
>>>>
>>>> The client is a 2.6.18-5 kernel supplied with Debian on an AMD64 box.
>>>> nfs-utils is version 1.1.4.
>>>>
>>>> Many thanks,
>>>> -Shehjar
>>> Wireshark fails to decode your server's reply too. I'd start looking
>>> there...
>>>
>> Bruce, Trond,
>>
>> I am able to view the packets just fine using wireshark Version 1.0.6.
>> It is possible that the default options for you for TCP and RPC are
>> not same as the ones below.
>> Could you please try viewing the dump with the following options set 
>> in the wireshark Protocol preferences pane.
>>
>> Press <Ctrl> + <Shift> + p to bring up the protocol preferences
>> window.
>>
>> First, expand the "Protocol" section header in the window that pops
>> up. Then look for "TCP" section. In the TCP section, please check the 
>> following option:
>>
>> "Allow subdissector to reassemble TCP streams"
>>
>> Then, search for the "RPC" section under "Protocols". For RPC, please
>> check the following option:
>>
>> "Reassemble RPC over TCP message spanning multiple TCP segments"
>>
>> This should make the RPC records visible properly.
> 
> I always run with those options enabled, and they were able to
> reconstruct most of the RPC records correctly, but not the reply to the
> FSINFO call.
> Furthermore, when I looked at the binary contents, it seemed to me that
> the post-op attributes contained some fishy information, such as
> nlink==0. That alone would cause the NFS client to give up.
> 

Got it! That was the problem. I missed setting the nlink field
in FSINFO. Mount works fine now.

Thanks a ton.
-Shehjar

> Trond
> 

Re: Linux client mount fails with Gluster NFSv3 server

[J. Bruce Fields]

On Tue, Sep 01, 2009 at 08:26:37AM -0400, Trond Myklebust wrote:
> On Tue, 2009-09-01 at 12:09 +0530, Shehjar Tikoo wrote:
> > I am able to view the packets just fine using wireshark Version 1.0.6.
> > It is possible that the default options for you for TCP and RPC are
> > not same as the ones below.
> > Could you please try viewing the dump with the following options set 
> > in the wireshark Protocol preferences pane.
> > 
> > Press <Ctrl> + <Shift> + p to bring up the protocol preferences
> > window.
> > 
> > First, expand the "Protocol" section header in the window that pops
> > up. Then look for "TCP" section. In the TCP section, please check the 
> > following option:
> > 
> > "Allow subdissector to reassemble TCP streams"
> > 
> > Then, search for the "RPC" section under "Protocols". For RPC, please
> > check the following option:
> > 
> > "Reassemble RPC over TCP message spanning multiple TCP segments"
> > 
> > This should make the RPC records visible properly.
> 
> I always run with those options enabled, and they were able to
> reconstruct most of the RPC records correctly, but not the reply to the
> FSINFO call.

Same here--both options set, and I don't normally have this problem.
The segments look OK, though (the sequence numbers are what you'd
expect, anyway)--if the xdr was screwed up you'd think wireshark would
still attempt to parse it.  I'm using 1.0.7 (from 1.0.7-1ubuntu1), but
too lazy to try to debug wireshark....

--b.

> Furthermore, when I looked at the binary contents, it seemed to me that
> the post-op attributes contained some fishy information, such as
> nlink==0. That alone would cause the NFS client to give up.

Re: Linux client mount fails with Gluster NFSv3 server

[J. Bruce Fields]

On Mon, Aug 31, 2009 at 07:37:07PM +0530, Shehjar Tikoo wrote:
> Hi All
>
> I am writing a NFSv3 server as part of the Gluster clustered FS.
> To start with, I've implemented the Mountv3 protocol and am just
> starting out with NFSv3. In NFSv3, the first thing I've implemented
> is the FSINFO and GETATTR calls to support mounting with NFS client.
>
> The problem I am facing is this. The Linux NFS client fails to mount
> the remote export even though it is successfully receiving the file
> handle from the MNT request and the result of the FSINFO call. This
> is shown in the attached pcap file, which would be best viewed through
> wireshark with "rpc" as the display filter.
>
> The command line output is shown below:
>
> root@indus:statcache# mount 127.0.0.1:/pos1 /mnt -o noacl,nolock
> mount.nfs: mounting 127.0.0.1:/pos1 failed, reason given by server:
>   No such file or directory
>
> This happens even though, we're told the following by showmount.
> root@indus:statcache# showmount -e
> Export list for indus:
> /pos1 (everyone)
> /pos2 (everyone)
> /pos3 (everyone)
> /pos4 (everyone)
> root@indus:statcache#
>
> ..where /pos1, /pos2, etc are exports from the locally running Gluster
> NFS server.
>
> As you'll notice in the trace, there is no NFSv3 request after
> the FSINFO, so I've a feeling it could be that some field in the
> FSINFO reply is not what the Linux NFS client is expecting. Could that
> be the reason for the mount failure?
>
> What else should I be looking into to investigate this further?
>
> The client is a 2.6.18-5 kernel supplied with Debian on an AMD64 box.
> nfs-utils is version 1.1.4.

Ugh. For some reason, the version of wireshark I'm running is refusing
to reassemble those tcp replies, which are all coming in two segments.

--b.