klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

[Hans-Peter Jansen]

[-- Attachment #1: Type: text/plain, Size: 2248 bytes --]

Hi,

I suffer from a problem, where my diskless environment doesn't work anymore 
after upgrading the server from openSUSE 11.0 (2.6.25.20) to openSUSE 11.1 
(2.6.27.21).

The diskless environment gets set up with an customized initrd via pxelinux.
The initrd uses klibc's nfsmount, and I've tried both protocols: udp and 
tcp. It's not an iptables issue, since I can mount these directories from 
other (fully booted) systems just fine. 

Using the debug version of klibc's nfsmount prints these messages:
short read: 0 < 28
Port for 100003/3[udp]: 0
short read: 0 < 28
Port for 100005/3[udp]: 0
NFS params:
  server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp,
  mount_port = 627, nfs_port = 2049, flags = 00010282,
  rsize = 0, wsize = 0, timeo = 7, retrans = 3,
  acreg (min,max) = (3,60), acdir (min,max) = (30,60),
  soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0

tcpdumping the communication shows, that for whatever reason, the client 
sends the mount request from source port 798 to destination port 627, but 
the server doesn't listen on 627:
# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  41656  mountd
    100005    1   tcp  49063  mountd
    100005    2   udp  41656  mountd
    100005    2   tcp  49063  mountd
    100005    3   udp  41656  mountd
    100005    3   tcp  49063  mountd
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100021    1   udp  33144  nlockmgr
    100021    3   udp  33144  nlockmgr
    100021    4   udp  33144  nlockmgr
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   tcp  57268  nlockmgr
    100021    3   tcp  57268  nlockmgr
    100021    4   tcp  57268  nlockmgr
    100024    1   udp  39508  status
    100024    1   tcp  45430  status


I've attached the relevant tcpdump section.

I would be very grateful, if some kind soul could enlight me here, as I'm 
fully puzzled..

Pete

[-- Attachment #2: shark-nfs-mount.dump --]
[-- Type: application/octet-stream, Size: 4824 bytes --]

Re: klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

[Trond Myklebust]

On Wed, 2009-04-15 at 12:50 +0200, Hans-Peter Jansen wrote:
> Hi,
> 
> I suffer from a problem, where my diskless environment doesn't work anymore 
> after upgrading the server from openSUSE 11.0 (2.6.25.20) to openSUSE 11.1 
> (2.6.27.21).
> 
> The diskless environment gets set up with an customized initrd via pxelinux.
> The initrd uses klibc's nfsmount, and I've tried both protocols: udp and 
> tcp. It's not an iptables issue, since I can mount these directories from 
> other (fully booted) systems just fine. 
> 
> Using the debug version of klibc's nfsmount prints these messages:
> short read: 0 < 28
> Port for 100003/3[udp]: 0
> short read: 0 < 28
> Port for 100005/3[udp]: 0
> NFS params:
>   server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp,
>   mount_port = 627, nfs_port = 2049, flags = 00010282,
    ^^^^^^^^^^^^^^^^

>   rsize = 0, wsize = 0, timeo = 7, retrans = 3,
>   acreg (min,max) = (3,60), acdir (min,max) = (30,60),
>   soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0
> 
> tcpdumping the communication shows, that for whatever reason, the client 
> sends the mount request from source port 798 to destination port 627, but 
> the server doesn't listen on 627:

The question is why is klibc specifying mount port = 627?

Trond

Re: klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

[Hans-Peter Jansen]

Am Mittwoch, 15. April 2009 schrieb Trond Myklebust:
> On Wed, 2009-04-15 at 12:50 +0200, Hans-Peter Jansen wrote:
> > Hi,
> >
> > I suffer from a problem, where my diskless environment doesn't work
> > anymore after upgrading the server from openSUSE 11.0 (2.6.25.20) to
> > openSUSE 11.1 (2.6.27.21).
> >
> > The diskless environment gets set up with an customized initrd via
> > pxelinux. The initrd uses klibc's nfsmount, and I've tried both
> > protocols: udp and tcp. It's not an iptables issue, since I can mount
> > these directories from other (fully booted) systems just fine.
> >
> > Using the debug version of klibc's nfsmount prints these messages:
> > short read: 0 < 28
> > Port for 100003/3[udp]: 0
> > short read: 0 < 28
> > Port for 100005/3[udp]: 0
> > NFS params:
> >   server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp,
> >   mount_port = 627, nfs_port = 2049, flags = 00010282,
>
>     ^^^^^^^^^^^^^^^^
>
> >   rsize = 0, wsize = 0, timeo = 7, retrans = 3,
> >   acreg (min,max) = (3,60), acdir (min,max) = (30,60),
> >   soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0
> >
> > tcpdumping the communication shows, that for whatever reason, the
> > client sends the mount request from source port 798 to destination port
> > 627, but the server doesn't listen on 627:
>
> The question is why is klibc specifying mount port = 627?

Well, this is exactly my problem. That's the mount port, but this one cannot 
be set to a manual value, thus the rpc stuff infront of it must have gone 
wrong.

[Update]
Issue solved. It was that silly rpcbind package (0.1.6+git20080930-5.2), 
that has gone wrong. Replacing it with portmap (6.0+git20070716-31.37) did 
it. Looks like this deserves a new bugzilla record.

Sorry for the churn and thanks for your care, Trond.

Pete

Re: klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

[Chuck Lever]

On Apr 15, 2009, at 9:48 AM, Hans-Peter Jansen wrote:
> Am Mittwoch, 15. April 2009 schrieb Trond Myklebust:
>> On Wed, 2009-04-15 at 12:50 +0200, Hans-Peter Jansen wrote:
>>> Hi,
>>>
>>> I suffer from a problem, where my diskless environment doesn't work
>>> anymore after upgrading the server from openSUSE 11.0 (2.6.25.20) to
>>> openSUSE 11.1 (2.6.27.21).
>>>
>>> The diskless environment gets set up with an customized initrd via
>>> pxelinux. The initrd uses klibc's nfsmount, and I've tried both
>>> protocols: udp and tcp. It's not an iptables issue, since I can  
>>> mount
>>> these directories from other (fully booted) systems just fine.
>>>
>>> Using the debug version of klibc's nfsmount prints these messages:
>>> short read: 0 < 28
>>> Port for 100003/3[udp]: 0
>>> short read: 0 < 28
>>> Port for 100005/3[udp]: 0
>>> NFS params:
>>>  server = xxx.xx.xx.xx, path = "/roroot", version = 3, proto = udp,
>>>  mount_port = 627, nfs_port = 2049, flags = 00010282,
>>
>>    ^^^^^^^^^^^^^^^^
>>
>>>  rsize = 0, wsize = 0, timeo = 7, retrans = 3,
>>>  acreg (min,max) = (3,60), acdir (min,max) = (30,60),
>>>  soft = 0, intr = 1, posix = 0, nocto = 0, noac = 0
>>>
>>> tcpdumping the communication shows, that for whatever reason, the
>>> client sends the mount request from source port 798 to destination  
>>> port
>>> 627, but the server doesn't listen on 627:
>>
>> The question is why is klibc specifying mount port = 627?

nfs_root_ports() (in fs/nfs/nfsroot.c) sets the default mountd port to  
627, I think only if the rpcbind doesn't work.

> Well, this is exactly my problem. That's the mount port, but this  
> one cannot
> be set to a manual value, thus the rpc stuff infront of it must have  
> gone
> wrong.
>
> [Update]
> Issue solved. It was that silly rpcbind package  
> (0.1.6+git20080930-5.2),
> that has gone wrong. Replacing it with portmap  
> (6.0+git20070716-31.37) did
> it. Looks like this deserves a new bugzilla record.

When using rpcbind instead of portmapper, what does the output of  
"rpcinfo" look like on the server?

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

[Hans-Peter Jansen]

Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
>
> When using rpcbind instead of portmapper, what does the output of
> "rpcinfo" look like on the server?

It's dumped in the mail starting this thread.

Thanks, 
Pete

Re: klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

[Chuck Lever]

On Apr 15, 2009, at 1:23 PM, Hans-Peter Jansen wrote:
> Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
>>
>> When using rpcbind instead of portmapper, what does the output of
>> "rpcinfo" look like on the server?
>
> It's dumped in the mail starting this thread.

I don't see why the client's rpcbind attempt for the server's mountd  
service should have failed.

Would it be possible for you to capture a packet trace of the client's  
attempt to mount it's root file system?  (You will likely need to do  
this for a bugzilla report, anyway).

On the server you can use "tcpdump -s0 -w /tmp/raw host " followed by  
the hostname of your client.  It might be useful to capture both a  
failed session (with rpcbind) and a working session (with  
portmapper).  We want the raw (binary) trace dumps, not text output.

Also let us know what's running on your clients (distribution, kernel  
version, etc).

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

[Hans-Peter Jansen]

[-- Attachment #1: Type: text/plain, Size: 2290 bytes --]

Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
> On Apr 15, 2009, at 1:23 PM, Hans-Peter Jansen wrote:
> > Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
> >> When using rpcbind instead of portmapper, what does the output of
> >> "rpcinfo" look like on the server?
> >
> > It's dumped in the mail starting this thread.
>
> I don't see why the client's rpcbind attempt for the server's mountd
> service should have failed.

For completeness, here's the current rpcinfo view with portmap:

# rpcinfo -p 172.16.23.110
   Program Vers Proto   Port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  54838  mountd
    100005    1   tcp  32772  mountd
    100005    2   udp  54838  mountd
    100005    2   tcp  32772  mountd
    100005    3   udp  54838  mountd
    100005    3   tcp  32772  mountd
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100021    1   udp  35501  nlockmgr
    100021    3   udp  35501  nlockmgr
    100021    4   udp  35501  nlockmgr
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   tcp  54766  nlockmgr
    100021    3   tcp  54766  nlockmgr
    100021    4   tcp  54766  nlockmgr
    100024    1   udp  44650  status
    100024    1   tcp  39765  status


> Would it be possible for you to capture a packet trace of the client's
> attempt to mount it's root file system?  (You will likely need to do
> this for a bugzilla report, anyway).

Ähem, Chuck, may I ask you to look into the initial mail again. The failing 
case is attached there. Here I've attached the good one. Since I couldn't 
locate any mount attempt in the dump, I've left a few more nfs 
transactions.

> Also let us know what's running on your clients (distribution, kernel
> version, etc).

Hmm, sure. The client setup is a legacy SuSE 9.3 diskless environment 
(unfortunately Novell didn't manage to create a distribution with similar 
stability since then..., being a rpm junke, I will soon check Cent-OS 
(again)).

Client (relevant) versions:
Kernel: 2.6.11.4
Udev (nfsmount): 053

Let me know, what more I can provide, please.

THanks,
Pete

[-- Attachment #2: shark-nfs-mount-ok.dump --]
[-- Type: application/octet-stream, Size: 11534 bytes --]

Re: klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

[Hans-Peter Jansen]

[-- Attachment #1: Type: text/plain, Size: 2290 bytes --]

Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
> On Apr 15, 2009, at 1:23 PM, Hans-Peter Jansen wrote:
> > Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
> >> When using rpcbind instead of portmapper, what does the output of
> >> "rpcinfo" look like on the server?
> >
> > It's dumped in the mail starting this thread.
>
> I don't see why the client's rpcbind attempt for the server's mountd
> service should have failed.

For completeness, here's the current rpcinfo view with portmap:

# rpcinfo -p 172.16.23.110
   Program Vers Proto   Port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  54838  mountd
    100005    1   tcp  32772  mountd
    100005    2   udp  54838  mountd
    100005    2   tcp  32772  mountd
    100005    3   udp  54838  mountd
    100005    3   tcp  32772  mountd
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100021    1   udp  35501  nlockmgr
    100021    3   udp  35501  nlockmgr
    100021    4   udp  35501  nlockmgr
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   tcp  54766  nlockmgr
    100021    3   tcp  54766  nlockmgr
    100021    4   tcp  54766  nlockmgr
    100024    1   udp  44650  status
    100024    1   tcp  39765  status


> Would it be possible for you to capture a packet trace of the client's
> attempt to mount it's root file system?  (You will likely need to do
> this for a bugzilla report, anyway).

Ähem, Chuck, may I ask you to look into the initial mail again. The failing 
case is attached there. Here I've attached the good one. Since I couldn't 
locate any mount attempt in the dump, I've left a few more nfs 
transactions.

> Also let us know what's running on your clients (distribution, kernel
> version, etc).

Hmm, sure. The client setup is a legacy SuSE 9.3 diskless environment 
(unfortunately Novell didn't manage to create a distribution with similar 
stability since then..., being a rpm junke, I will soon check Cent-OS 
(again)).

Client (relevant) versions:
Kernel: 2.6.11.4
Udev (nfsmount): 053

Let me know, what more I can provide, please.

THanks,
Pete

[-- Attachment #2: shark-nfs-mount-ok.dump --]
[-- Type: application/octet-stream, Size: 11534 bytes --]

Re: klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

[Chuck Lever]

On Apr 15, 2009, at 3:17 PM, Hans-Peter Jansen wrote:
> Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
>> On Apr 15, 2009, at 1:23 PM, Hans-Peter Jansen wrote:
>>> Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
>>>> When using rpcbind instead of portmapper, what does the output of
>>>> "rpcinfo" look like on the server?
>>>
>>> It's dumped in the mail starting this thread.
>>
>> I don't see why the client's rpcbind attempt for the server's mountd
>> service should have failed.
>
> For completeness, here's the current rpcinfo view with portmap:
>
> # rpcinfo -p 172.16.23.110
>   Program Vers Proto   Port
>    100000    2   tcp    111  portmapper
>    100000    2   udp    111  portmapper
>    100005    1   udp  54838  mountd
>    100005    1   tcp  32772  mountd
>    100005    2   udp  54838  mountd
>    100005    2   tcp  32772  mountd
>    100005    3   udp  54838  mountd
>    100005    3   tcp  32772  mountd
>    100003    2   udp   2049  nfs
>    100003    3   udp   2049  nfs
>    100003    4   udp   2049  nfs
>    100021    1   udp  35501  nlockmgr
>    100021    3   udp  35501  nlockmgr
>    100021    4   udp  35501  nlockmgr
>    100003    2   tcp   2049  nfs
>    100003    3   tcp   2049  nfs
>    100003    4   tcp   2049  nfs
>    100021    1   tcp  54766  nlockmgr
>    100021    3   tcp  54766  nlockmgr
>    100021    4   tcp  54766  nlockmgr
>    100024    1   udp  44650  status
>    100024    1   tcp  39765  status
>
>
>> Would it be possible for you to capture a packet trace of the =20
>> client's
>> attempt to mount it's root file system?  (You will likely need to do
>> this for a bugzilla report, anyway).
>
> =C4hem, Chuck, may I ask you to look into the initial mail again. The=
 =20
> failing
> case is attached there. Here I've attached the good one. Since I =20
> couldn't
> locate any mount attempt in the dump, I've left a few more nfs
> transactions.

The client makes a PMAP_GETPORT request via TCP.  The server's rpcbind =
=20
drops the connection without replying after receiving the request.  I =20
didn't see anything immediately wrong with the request, although =20
wireshark didn't like it either.  I had to decode it by hand.

Restarting rpcbind usually means you lose all your rpc service =20
registrations until you restart those services, but it would be worth =20
trying this: stop the server's rpcbind service, then run rpcbind in a =20
terminal session with "-d" to see what it thinks the problem is when =20
it drops the connection.

Please attach a copy of your /etc/netconfig to reply.

>> Also let us know what's running on your clients (distribution, kerne=
l
>> version, etc).
>
> Hmm, sure. The client setup is a legacy SuSE 9.3 diskless environment
> (unfortunately Novell didn't manage to create a distribution with =20
> similar
> stability since then..., being a rpm junke, I will soon check Cent-OS
> (again)).
>
> Client (relevant) versions:
> Kernel: 2.6.11.4
> Udev (nfsmount): 053
>
> Let me know, what more I can provide, please.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: klibc's nfsmount failure with 2.6.27.21, while 2.6.25.20 was fine

[Hans-Peter Jansen]

Am Mittwoch, 15. April 2009 schrieb Chuck Lever:
> On Apr 15, 2009, at 3:17 PM, Hans-Peter Jansen wrote:
> >
> > =C4hem, Chuck, may I ask you to look into the initial mail again. T=
he
> > failing
> > case is attached there. Here I've attached the good one. Since I
> > couldn't
> > locate any mount attempt in the dump, I've left a few more nfs
> > transactions.
>
> The client makes a PMAP_GETPORT request via TCP.  The server's rpcbin=
d
> drops the connection without replying after receiving the request.  I
> didn't see anything immediately wrong with the request, although
> wireshark didn't like it either.  I had to decode it by hand.

	8[

> Restarting rpcbind usually means you lose all your rpc service
> registrations until you restart those services, but it would be worth
> trying this: stop the server's rpcbind service, then run rpcbind in a
> terminal session with "-d" to see what it thinks the problem is when
> it drops the connection.
>
> Please attach a copy of your /etc/netconfig to reply.

Okay, but since it requires yet another portmap <-> rpcbind swap, pleas=
e=20
give me a few days to find a appropriate slot.

Pete