troubling interaction with quotas and nfs

troubling interaction with quotas and nfs

[seth vidal]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

Hi all,
 I'm running red hat 7.2 with the 2.4.9-31 kernel.
Its an ext3 file system, exported for nfsv3 udp and nfsv2 clients.

nothing fancy at all.

I have quotas set for my users. If a user hits their hard quota and the
process continues to attempt to write to a file in their  quota'd space
- the process load on the server shoots through the roof. 

I can reliably recreate this by setting a hard quota, having the system
hit the hard quota and continue to attempt to write to that file - the
server becomes terribly unhappy.

I never saw this problem on older 2.2.X servers and I'm not sure why I'm
seeing it now.

Any places I should be looking first?

thanks
-sv

-- 
GPG Public Key: http://www.phy.duke.edu/~skvidal/skvidal.gpg

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

Re: troubling interaction with quotas and nfs

[Neil Brown]

On  April 9, skvidal@phy.duke.edu wrote:
> Hi all,
>  I'm running red hat 7.2 with the 2.4.9-31 kernel.
> Its an ext3 file system, exported for nfsv3 udp and nfsv2 clients.
> 
> nothing fancy at all.
> 
> I have quotas set for my users. If a user hits their hard quota and the
> process continues to attempt to write to a file in their  quota'd space
> - the process load on the server shoots through the roof. 
> 
> I can reliably recreate this by setting a hard quota, having the system
> hit the hard quota and continue to attempt to write to that file - the
> server becomes terribly unhappy.
> 
> I never saw this problem on older 2.2.X servers and I'm not sure why I'm
> seeing it now.
> 
> Any places I should be looking first?
> 

tcpdump?
The server should return nfserr_dquot (69 == 0x45) and the client
should return this to the application.
But it sounds like the client is repeating the request for some
reason.

You could also 
   echo 32 > /proc/sys/sunrpc/nfsd_debug
and watch for
    "nfsd: write complete err=%d"
messages in the kernel log.  These will tell you whether nfsd thinks
it got an error for the write.

NeilBrown

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: troubling interaction with quotas and nfs

[seth vidal]

[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]

On Tue, 2002-04-09 at 00:45, Neil Brown wrote:
> On  April 9, skvidal@phy.duke.edu wrote:
> > Hi all,
> >  I'm running red hat 7.2 with the 2.4.9-31 kernel.
> > Its an ext3 file system, exported for nfsv3 udp and nfsv2 clients.
> > 
> > nothing fancy at all.
> > 
> > I have quotas set for my users. If a user hits their hard quota and the
> > process continues to attempt to write to a file in their  quota'd space
> > - the process load on the server shoots through the roof. 
> > 
> > I can reliably recreate this by setting a hard quota, having the system
> > hit the hard quota and continue to attempt to write to that file - the
> > server becomes terribly unhappy.
> > 
> > I never saw this problem on older 2.2.X servers and I'm not sure why I'm
> > seeing it now.
> > 
> > Any places I should be looking first?
> > 
> 
> tcpdump?
> The server should return nfserr_dquot (69 == 0x45) and the client
> should return this to the application.
> But it sounds like the client is repeating the request for some
> reason.
> 
> You could also 
>    echo 32 > /proc/sys/sunrpc/nfsd_debug
> and watch for
>     "nfsd: write complete err=%d"
> messages in the kernel log.  These will tell you whether nfsd thinks
> it got an error for the write.
> 

It appears to return -122

However, the program will exit correctly if it catches that problem. But
the system does not respond well if there is a continued attempt to
write a lot of data when the quota is filled.

That doesn't seem to make sense.

-sv


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

Re: troubling interaction with quotas and nfs

[Andrew Ryan]

Neil Brown wrote:

>
> You could also
>    echo 32 > /proc/sys/sunrpc/nfsd_debug
> and watch for
>     "nfsd: write complete err=%d"
> messages in the kernel log.  These will tell you whether nfsd thinks
> it got an error for the write.

Can you elaborate on what the various values one could write to
/proc/sys/sunrpc/nfsd_debug and what you would get with different values in
terms of logging? This sounds like something that could be useful to many
others and included in the HOWTO or FAQ as well. I've checked the kernel docs
and they basically say to 'use the source,' but since we have some of the
core NFS client/server developers here perhaps someone can offer a few hints
for people debugging NFS configurations.

Also, is there any known or performance or stability hit to running with
NFS debug on?


thanks,
andrew


_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: troubling interaction with quotas and nfs

[Neil Brown]

On Tuesday April 9, andrewr@collab.net wrote:
> Neil Brown wrote:
> 
> >
> > You could also
> >    echo 32 > /proc/sys/sunrpc/nfsd_debug
> > and watch for
> >     "nfsd: write complete err=%d"
> > messages in the kernel log.  These will tell you whether nfsd thinks
> > it got an error for the write.
> 
> Can you elaborate on what the various values one could write to
> /proc/sys/sunrpc/nfsd_debug and what you would get with different values in
> terms of logging? This sounds like something that could be useful to many
> others and included in the HOWTO or FAQ as well. I've checked the kernel docs
> and they basically say to 'use the source,' but since we have some of the
> core NFS client/server developers here perhaps someone can offer a few hints
> for people debugging NFS configurations.

"use the source" is really the right answer :-)
If you look in the source, it is fairly easy to see what to put in
/proc/sys/sunrpc/nfsd_debug, and you can see what the results means.

If you don't look at the source, then the debugging output probably
isn't very helpful.

However:
 If you look in
    include/linux/nfsd/debug.h
 you will see:
/*
 * knfsd debug flags
 */
#define NFSDDBG_SOCK		0x0001
#define NFSDDBG_FH		0x0002
#define NFSDDBG_EXPORT		0x0004
#define NFSDDBG_SVC		0x0008
#define NFSDDBG_PROC		0x0010
#define NFSDDBG_FILEOP		0x0020
#define NFSDDBG_AUTH		0x0040
#define NFSDDBG_REPCACHE	0x0080
#define NFSDDBG_XDR		0x0100
#define NFSDDBG_LOCKD		0x0200

So if you
   echo 1 > /proc/sys/sunrpc/nfsd_debug
it will turn on "SOCK" debugging and turn the others off.
If you
   echo 1023 > /proc/sys/sunrpc/nfsd_debug
it will turn on all debugging.

Each file in fs/nfsd/*.c has something like
#define NFSDDBG_FACILITY		NFSDDBG_FILEOP

so all dprintk calls in that file will use that facility and 
will only be printed if the right number has been written to
 ..../nfsd_debug

 
> 
> Also, is there any known or performance or stability hit to running with
> NFS debug on?

It will generate lots and lots (and lots) of messages which will
definately hurt performance, just scrolling the console if nothing
else.

I only turn it on for a few seconds at a time, or when I am
generating a very small amount of carefully controlled traffic.

NeilBrown

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs