Possible bug in NFSv4 server regarding to looking for right path for clients to mount

Possible bug in NFSv4 server regarding to looking for right path for clients to mount

[Song Gao]

Hello,

There's possibly a bug related NFSv4 server's behavior on looking for
path on /export .

I have a 32 nodes KVM cluster. NFS, host filesystem, KVM storage pool,
are all placed on SSD. KVM guests are connected to host through virtio
drived GB ethernet.

I followed this guide to configure NFS:
https://help.ubuntu.com/community/SettingUpNFSHowTo

The problem was that, when I execute a program simultaneously (through
cssh) on 32 nodes, there were always a number of nodes that
encountered "input/output error". The /var/log/messages on client
shows "state manager failed on NFSv4 server with error 10022".

I tried different options in client /etc/fstab as well as export
settings in server but nothing helps. Interestingly, by changing the
item in /etc/fstab from:
    some-host:/tigers  nfs    rw,bg,hard,intr    0    0
into:
    some-host:/export/tigers    /nfs    nfs    rw,bg,hard,intr    0    0
fixed the problem.

I'm not familiar with it but I guess that it's probably caused by some
problem in NFS server when it tries to find the right path when all 32
clients are trying to mount NFS at the same time.

Here's some more information:
client: debian "squeeze", 2.6.32-5-amd64
server: Ubuntu Server 12.04, 3.2.0-23-generic
nfs version: NFSv4
no security option
results of `exports -v`:
    /export 192.168.100.0/24(rw,wdelay,insecure,root_squash,no_subtree_check,fsid=0)
    /export/tigers
192.168.100.0/24(rw,wdelay,nohide,insecure,root_squash,no_subtree_check)

Please let me know if there's any other information I need to provide.

Thanks!


Kind regards,
-- 
Song Gao
https://profiles.google.com/song.gao.beta/about

Re: Possible bug in NFSv4 server regarding to looking for right path for clients to mount

[J. Bruce Fields]

On Mon, Jul 02, 2012 at 01:11:39AM -0500, Song Gao wrote:
> Hello,
> 
> There's possibly a bug related NFSv4 server's behavior on looking for
> path on /export .
> 
> I have a 32 nodes KVM cluster. NFS, host filesystem, KVM storage pool,
> are all placed on SSD. KVM guests are connected to host through virtio
> drived GB ethernet.
> 
> I followed this guide to configure NFS:
> https://help.ubuntu.com/community/SettingUpNFSHowTo
> 
> The problem was that, when I execute a program simultaneously (through
> cssh) on 32 nodes, there were always a number of nodes that
> encountered "input/output error". The /var/log/messages on client
> shows "state manager failed on NFSv4 server with error 10022".

That's NFS4ERR_STALE_CLIENTID.

That's an odd thing to happen.  I'd be curious to know which operation
exactly the server was returning that error to.  Normally that should
only happen if the server rebooted.

> I tried different options in client /etc/fstab as well as export
> settings in server but nothing helps. Interestingly, by changing the
> item in /etc/fstab from:
>     some-host:/tigers  nfs    rw,bg,hard,intr    0    0
> into:
>     some-host:/export/tigers    /nfs    nfs    rw,bg,hard,intr    0    0
> fixed the problem.

The advice to use "fsid=0" in the ubuntu howto is out of date.  With a
recent server you should just export and mount /export/tigers as you
would with NFSv3.

If you do as you've done above (and as is sugested in that howto), then
clients see different paths depending on whether they're using v3 or v4.

When you changed the client to mount /export/tigers, it probably
couldn't find that path, so tried v3 instead.  That works around the v4
bug.

If you change the exports to remove fsid=0, then you'll probably want to
mount with nfsvers=3 to continue working around the v4 bug.

> I'm not familiar with it but I guess that it's probably caused by some
> problem in NFS server when it tries to find the right path when all 32
> clients are trying to mount NFS at the same time.
> 
> Here's some more information:
> client: debian "squeeze", 2.6.32-5-amd64
> server: Ubuntu Server 12.04, 3.2.0-23-generic

Might be worth trying more recent versions.  The client especially has
seen a lot of v4 changes since 2.6.32.

--b.

> nfs version: NFSv4
> no security option
> results of `exports -v`:
>     /export 192.168.100.0/24(rw,wdelay,insecure,root_squash,no_subtree_check,fsid=0)
>     /export/tigers
> 192.168.100.0/24(rw,wdelay,nohide,insecure,root_squash,no_subtree_check)
> 
> Please let me know if there's any other information I need to provide.
> 
> Thanks!
> 
> 
> Kind regards,
> -- 
> Song Gao
> https://profiles.google.com/song.gao.beta/about
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Possible bug in NFSv4 server regarding to looking for right path for clients to mount

[Song Gao]

Thanks Bruce.

The error was raised when I was executing fish-shell simultaneously
from 32 nodes. The fish-shell requires several configuration/function
files which were placed on NFS.

It seems that the problem might be caused by network instead of
/export. Later I moved mounting from /etc/fstab to /etc/rc.local and
since then I've never seen the problem again, no matter NFSv4 or
NFSv2.

Out of curiosity, I moved mounting configuration back to /etc/fstab
again, and the same problem occurred again. Among those clients that
failed to load files, some failed to mount, which is probably because
the NFS was up later than DHCP. However, there are some that succeeded
to mount the NFS, but still failed to load some of the files. (They
did succeed to load some files, but not all.) For these nodes, if I
open a new window (in tmux), and execute the fish-shell again (without
remounting NFS), everything just works.

This is still quite strange and I could not think of an explaination.


On Thu, Jul 5, 2012 at 4:56 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Mon, Jul 02, 2012 at 01:11:39AM -0500, Song Gao wrote:
> > Hello,
> >
> > There's possibly a bug related NFSv4 server's behavior on looking for
> > path on /export .
> >
> > I have a 32 nodes KVM cluster. NFS, host filesystem, KVM storage pool,
> > are all placed on SSD. KVM guests are connected to host through virtio
> > drived GB ethernet.
> >
> > I followed this guide to configure NFS:
> > https://help.ubuntu.com/community/SettingUpNFSHowTo
> >
> > The problem was that, when I execute a program simultaneously (through
> > cssh) on 32 nodes, there were always a number of nodes that
> > encountered "input/output error". The /var/log/messages on client
> > shows "state manager failed on NFSv4 server with error 10022".
>
> That's NFS4ERR_STALE_CLIENTID.
>
> That's an odd thing to happen.  I'd be curious to know which operation
> exactly the server was returning that error to.  Normally that should
> only happen if the server rebooted.
>
> > I tried different options in client /etc/fstab as well as export
> > settings in server but nothing helps. Interestingly, by changing the
> > item in /etc/fstab from:
> >     some-host:/tigers  nfs    rw,bg,hard,intr    0    0
> > into:
> >     some-host:/export/tigers    /nfs    nfs    rw,bg,hard,intr    0    0
> > fixed the problem.
>
> The advice to use "fsid=0" in the ubuntu howto is out of date.  With a
> recent server you should just export and mount /export/tigers as you
> would with NFSv3.
>
> If you do as you've done above (and as is sugested in that howto), then
> clients see different paths depending on whether they're using v3 or v4.
>
> When you changed the client to mount /export/tigers, it probably
> couldn't find that path, so tried v3 instead.  That works around the v4
> bug.
It turned out to be NFSv2 on client. (I used nfsstat). I also noticed
that, for NFSv4 it mshows as "Client fps stats" without indication v4.
Is this normal?
>
> If you change the exports to remove fsid=0, then you'll probably want to
> mount with nfsvers=3 to continue working around the v4 bug.
>
> > I'm not familiar with it but I guess that it's probably caused by some
> > problem in NFS server when it tries to find the right path when all 32
> > clients are trying to mount NFS at the same time.
> >
> > Here's some more information:
> > client: debian "squeeze", 2.6.32-5-amd64
> > server: Ubuntu Server 12.04, 3.2.0-23-generic
>
> Might be worth trying more recent versions.  The client especially has
> seen a lot of v4 changes since 2.6.32.
>
> --b.
>
> > nfs version: NFSv4
> > no security option
> > results of `exports -v`:
> >     /export 192.168.100.0/24(rw,wdelay,insecure,root_squash,no_subtree_check,fsid=0)
> >     /export/tigers
> > 192.168.100.0/24(rw,wdelay,nohide,insecure,root_squash,no_subtree_check)
> >
> > Please let me know if there's any other information I need to provide.
> >
> > Thanks!
> >
> >
> > Kind regards,
> > --
> > Song Gao
> > https://profiles.google.com/song.gao.beta/about
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Possible bug in NFSv4 server regarding to looking for right path for clients to mount

[Song Gao]

Hello all,

There's possibly a bug related NFSv4 server's behavior on looking for
path on /export .

I have a 32 nodes KVM cluster. NFS, host filesystem, KVM storage pool,
are all placed on SSD. KVM guests are connected to host through virtio
drived GB ethernet.

I followed this guide to configure NFS:
https://help.ubuntu.com/community/SettingUpNFSHowTo

The problem was that, when I execute a program simultaneously (through
cssh) on 32 nodes, there were always a number of nodes that
encountered "input/output error". The /var/log/messages on client
shows "state manager failed on NFSv4 server with error 10022".

I tried different options in client /etc/fstab as well as export
settings in server but nothing helps. Interestingly, by changing the
item in /etc/fstab from:
    some-host:/tigers  nfs    rw,bg,hard,intr    0    0
into:
    some-host:/export/tigers    /nfs    nfs    rw,bg,hard,intr    0    0
fixed the problem.

I'm not familiar with it but I guess that it's probably caused by some
problem in NFS server when it tries to find the right path when all 32
clients are trying to mount NFS at the same time.

Here's some more information:
client: debian "squeeze", 2.6.32-5-amd64
server: Ubuntu Server 12.04, 3.2.0-23-generic
nfs version: NFSv4
no security option
results of `exports -v`:
    /export
192.168.100.0/24(rw,wdelay,insecure,root_squash,no_subtree_check,fsid=0)
    /export/tigers
192.168.100.0/24(rw,wdelay,nohide,insecure,root_squash,no_subtree_check)

Please let me know if there's any other information I need to provide.

Thanks!

Kind regards,
-- 
Song Gao
https://profiles.google.com/song.gao.beta/about