2.6.21.3: NFS: Buggy server - nlink == 0!

2.6.21.3: NFS: Buggy server - nlink == 0!

[Ravikiran G Thirumalai]

While running a dbench stress test on  a nfs mounted file system, I notice
the subject error message on the client machine.  The client machine is a 48
core box with NUMA characteristics and 1024 dbench processes running
continuously in a loop, while another memory hog application runs in parallel.
The client is on 2.6.21.3.  The server is booted up with 2.6.21.3 as well.
Attached is the server configuration.  Same test on a 2.6.16 client does not
spew out these messages.  Is this really the server issue, or is the NFS
client to be blamed here?

fstab on the client goes as:
vus2:/mnt/sda5          /nfstest                nfs  udp,wsize=32768,rsize=32768 0 0

/etc/exports on the server looks like:
/mnt/sda5 *(rw,no_root_squash,sync)

I will be happy to post full config/dmesg if required.

The client side NFS config goes as:
#
# Network File Systems
#
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
# CONFIG_NFS_V4 is not set
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
# CONFIG_NFSD_V4 is not set
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
# CONFIG_RPCSEC_GSS_KRB5 is not set
# CONFIG_RPCSEC_GSS_SPKM3 is not set

The server side NFS config goes as:

#
# Network File Systems
#
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
# CONFIG_NFS_V4 is not set
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m

Thanks,
Kiran

Re: 2.6.21.3: NFS: Buggy server - nlink == 0!

[Peter Åstrand]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 975 bytes --]

On Wed, 6 Jun 2007, Ravikiran G Thirumalai wrote:

> While running a dbench stress test on  a nfs mounted file system, I notice
> the subject error message on the client machine.  The client machine is a 48
> core box with NUMA characteristics and 1024 dbench processes running
> continuously in a loop, while another memory hog application runs in parallel.
> The client is on 2.6.21.3.  The server is booted up with 2.6.21.3 as well.
> Attached is the server configuration.  Same test on a 2.6.16 client does not
> spew out these messages.  Is this really the server issue, or is the NFS
> client to be blamed here?

I've seen this message with the unfs3 server as well, and it contains some 
explicit checks to avoid sending nlink=0. So this is another indication 
that this is actually a client problem.

Regards, 
---
Peter Åstrand		ThinLinc Chief Developer
Cendio AB		http://www.cendio.se
Wallenbergs gata 4
583 30 Linköping	Phone: +46-13-21 46 00

[-- Attachment #2: Type: text/plain, Size: 286 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[-- Attachment #3: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.6.21.3: NFS: Buggy server - nlink == 0!

[Trond Myklebust]

T24gV2VkLCAyMDA3LTA2LTA2IGF0IDEyOjI1ICswMjAwLCBQZXRlciDDhXN0cmFuZCB3cm90ZToK
PiBPbiBXZWQsIDYgSnVuIDIwMDcsIFJhdmlraXJhbiBHIFRoaXJ1bWFsYWkgd3JvdGU6Cj4gCj4g
PiBXaGlsZSBydW5uaW5nIGEgZGJlbmNoIHN0cmVzcyB0ZXN0IG9uICBhIG5mcyBtb3VudGVkIGZp
bGUgc3lzdGVtLCBJIG5vdGljZQo+ID4gdGhlIHN1YmplY3QgZXJyb3IgbWVzc2FnZSBvbiB0aGUg
Y2xpZW50IG1hY2hpbmUuICBUaGUgY2xpZW50IG1hY2hpbmUgaXMgYSA0OAo+ID4gY29yZSBib3gg
d2l0aCBOVU1BIGNoYXJhY3RlcmlzdGljcyBhbmQgMTAyNCBkYmVuY2ggcHJvY2Vzc2VzIHJ1bm5p
bmcKPiA+IGNvbnRpbnVvdXNseSBpbiBhIGxvb3AsIHdoaWxlIGFub3RoZXIgbWVtb3J5IGhvZyBh
cHBsaWNhdGlvbiBydW5zIGluIHBhcmFsbGVsLgo+ID4gVGhlIGNsaWVudCBpcyBvbiAyLjYuMjEu
My4gIFRoZSBzZXJ2ZXIgaXMgYm9vdGVkIHVwIHdpdGggMi42LjIxLjMgYXMgd2VsbC4KPiA+IEF0
dGFjaGVkIGlzIHRoZSBzZXJ2ZXIgY29uZmlndXJhdGlvbi4gIFNhbWUgdGVzdCBvbiBhIDIuNi4x
NiBjbGllbnQgZG9lcyBub3QKPiA+IHNwZXcgb3V0IHRoZXNlIG1lc3NhZ2VzLiAgSXMgdGhpcyBy
ZWFsbHkgdGhlIHNlcnZlciBpc3N1ZSwgb3IgaXMgdGhlIE5GUwo+ID4gY2xpZW50IHRvIGJlIGJs
YW1lZCBoZXJlPwo+IAo+IEkndmUgc2VlbiB0aGlzIG1lc3NhZ2Ugd2l0aCB0aGUgdW5mczMgc2Vy
dmVyIGFzIHdlbGwsIGFuZCBpdCBjb250YWlucyBzb21lIAo+IGV4cGxpY2l0IGNoZWNrcyB0byBh
dm9pZCBzZW5kaW5nIG5saW5rPTAuIFNvIHRoaXMgaXMgYW5vdGhlciBpbmRpY2F0aW9uIAo+IHRo
YXQgdGhpcyBpcyBhY3R1YWxseSBhIGNsaWVudCBwcm9ibGVtLgoKRmVlbCBmcmVlIHRvIHNlbmQg
YSB0Y3BkdW1wIHRvIHByb3ZlIGl0LiBTbyBmYXIgYWxsIGNhc2VzIEkndmUgZXZlciBzZWVuCm9m
IHRoaXMgZXJyb3IgbWVzc2FnZSBoYXZlIGJlZW4gc2VydmVycyBzZW5kaW5nIGNyYXAgYmFjayBm
b3IgTE9PS1VQCmNhbGxzLiBUaGUgY2xpZW50IGRvZXMgbm90IGV2ZXIgbW9kaWZ5IHRoZSBzdHJ1
Y3QgbmZzX2ZhdHRyIGFmdGVyIGl0IGhhcwpkZWNvZGVkIHRoZSBzZXJ2ZXIgcmVwbHkuCgpUcm9u
ZAoKCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0KVGhpcyBTRi5uZXQgZW1haWwgaXMgc3BvbnNvcmVkIGJ5IERC
MiBFeHByZXNzCkRvd25sb2FkIERCMiBFeHByZXNzIEMgLSB0aGUgRlJFRSB2ZXJzaW9uIG9mIERC
MiBleHByZXNzIGFuZCB0YWtlCmNvbnRyb2wgb2YgeW91ciBYTUwuIE5vIGxpbWl0cy4gSnVzdCBk
YXRhLiBDbGljayB0byBnZXQgaXQgbm93LgpodHRwOi8vc291cmNlZm9yZ2UubmV0L3Bvd2VyYmFy
L2RiMi8KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KTkZT
IG1haWxsaXN0ICAtICBORlNAbGlzdHMuc291cmNlZm9yZ2UubmV0Cmh0dHBzOi8vbGlzdHMuc291
cmNlZm9yZ2UubmV0L2xpc3RzL2xpc3RpbmZvL25mcwo=

Re: 2.6.21.3: NFS: Buggy server - nlink == 0!

[Ravikiran G Thirumalai]

On Wed, Jun 06, 2007 at 08:33:17AM -0400, Trond Myklebust wrote:
> On Wed, 2007-06-06 at 12:25 +0200, Peter =C5strand wrote:
> > On Wed, 6 Jun 2007, Ravikiran G Thirumalai wrote:
> > =

> > > While running a dbench stress test on  a nfs mounted file system, I n=
otice
> > > the subject error message on the client machine.  The client machine =
is a 48
> > > core box with NUMA characteristics and 1024 dbench processes running
> > > continuously in a loop, while another memory hog application runs in =
parallel.
> > > The client is on 2.6.21.3.  The server is booted up with 2.6.21.3 as =
well.
> > > Attached is the server configuration.  Same test on a 2.6.16 client d=
oes not
> > > spew out these messages.  Is this really the server issue, or is the =
NFS
> > > client to be blamed here?
> > =

> > I've seen this message with the unfs3 server as well, and it contains s=
ome =

> > explicit checks to avoid sending nlink=3D0. So this is another indicati=
on =

> > that this is actually a client problem.
> =

> Feel free to send a tcpdump to prove it. So far all cases I've ever seen
> of this error message have been servers sending crap back for LOOKUP
> calls. The client does not ever modify the struct nfs_fattr after it has
> decoded the server reply.

I think Trond is right.  I tried the same test with the server running
mainline 2.6.18, and I did not see these error messages on the client.  This
seems to be a regression in the 2.6.21 nfs server code.

Thanks,
Kiran

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.6.21.3: NFS: Buggy server - nlink == 0!

[Trond Myklebust]

The server is returning a deleted file to the client as the result of a
lookup() call. I'd call that a server bug.

Trond


On Wed, 2007-06-06 at 01:13 -0700, Ravikiran G Thirumalai wrote:
> While running a dbench stress test on  a nfs mounted file system, I notice
> the subject error message on the client machine.  The client machine is a 48
> core box with NUMA characteristics and 1024 dbench processes running
> continuously in a loop, while another memory hog application runs in parallel.
> The client is on 2.6.21.3.  The server is booted up with 2.6.21.3 as well.
> Attached is the server configuration.  Same test on a 2.6.16 client does not
> spew out these messages.  Is this really the server issue, or is the NFS
> client to be blamed here?
> 
> fstab on the client goes as:
> vus2:/mnt/sda5          /nfstest                nfs  udp,wsize=32768,rsize=32768 0 0
> 
> /etc/exports on the server looks like:
> /mnt/sda5 *(rw,no_root_squash,sync)
> 
> I will be happy to post full config/dmesg if required.
> 
> The client side NFS config goes as:
> #
> # Network File Systems
> #
> CONFIG_NFS_FS=y
> CONFIG_NFS_V3=y
> # CONFIG_NFS_V3_ACL is not set
> # CONFIG_NFS_V4 is not set
> CONFIG_NFS_DIRECTIO=y
> CONFIG_NFSD=y
> CONFIG_NFSD_V3=y
> # CONFIG_NFSD_V3_ACL is not set
> # CONFIG_NFSD_V4 is not set
> CONFIG_NFSD_TCP=y
> CONFIG_LOCKD=y
> CONFIG_LOCKD_V4=y
> CONFIG_EXPORTFS=y
> CONFIG_NFS_COMMON=y
> CONFIG_SUNRPC=y
> # CONFIG_RPCSEC_GSS_KRB5 is not set
> # CONFIG_RPCSEC_GSS_SPKM3 is not set
> 
> The server side NFS config goes as:
> 
> #
> # Network File Systems
> #
> CONFIG_NFS_FS=m
> CONFIG_NFS_V3=y
> CONFIG_NFS_V3_ACL=y
> # CONFIG_NFS_V4 is not set
> CONFIG_NFS_DIRECTIO=y
> CONFIG_NFSD=m
> CONFIG_NFSD_V2_ACL=y
> CONFIG_NFSD_V3=y
> CONFIG_NFSD_V3_ACL=y
> CONFIG_NFSD_V4=y
> CONFIG_NFSD_TCP=y
> CONFIG_LOCKD=m
> CONFIG_LOCKD_V4=y
> CONFIG_EXPORTFS=m
> CONFIG_NFS_ACL_SUPPORT=m
> CONFIG_NFS_COMMON=y
> CONFIG_SUNRPC=m
> 
> Thanks,
> Kiran
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs