Re: NFS4ERR_STALE_CLIENTID loop

[Trond Myklebust <Trond.Myklebust@netapp.com>]

On Sat, 2011-10-29 at 00:25 +0000, David Flynn wrote: 
> * David Flynn (davidf@rd.bbc.co.uk) wrote:
> > * Trond Myklebust (Trond.Myklebust@netapp.com) wrote:
> > > Do you have an example of the stateid argument's value? Does it change
> > > at all between separate WRITE attempts?
> > 
> > Further to all this, i've just had a similar fault on another machine,
> 
> Using the same kernel, same mountpoint as before, we're currently
> experiencing a loop involving NFS4ERR_STALE_CLIENTID.
> Trace:
>   ftp://ftp.kw.bbc.co.uk/davidf/priv/saesheil.pcap
> 
> Unfortunately, this is resulting in about 40 nodes doing their best to
> kill the poor solaris server.  Generating a combined total of
> 250Mbit/sec towards the NFS server (collecting a little under
> 200Mbit/sec of replies).
> 
> Have we not heard of exponential backoff?
> 
> This seems to require major attention, given that this amounted to a
> site wide DoS: going round all the machines and killing the processes
> that were having major problems brought the situation back under
> control.  Frankly i'd rather that you panicked the kernel than this.

OK. This is the first time I've seen this tcpdump.

The problem seems like a split-brain issue on the server... On the one
hand, it is happily telling us that our lease is OK when we RENEW. Then
when we try to use said lease in an OPEN, it is replying with
STALE_CLIENTID.

IOW: This isn't a problem I can fix on the client whether or not I add
exponential backoff. The problem needs to be addressed on the server by
the Solaris folks....

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com