DoS with NFSv4.1 client

["Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>]

Hi,

last night we got a DoS attack with one of the NFS clients.
The farm node, which was accessing data with pNFS,
went mad and have tried to kill dCache NFS server. As usually
this have happened over night and we was not able to
get a network traffic or bump the debug level.

The symptoms are:

client starts to bombard the MDS with OPEN requests. As we see
state created on the server side, the requests was processed by
server. Nevertheless, for some reason, client did not like it. Here
is the result of mountstats:

OPEN:
	17087065 ops (99%) 	1 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 356	avg bytes received per op: 455
	backlog wait: 0.014707 	RTT: 4.535704 	total execute time: 4.574094 (milliseconds)
CLOSE:
	290 ops (0%) 	0 retrans (0%) 	0 major timeouts
	avg bytes sent per op: 247	avg bytes received per op: 173
	backlog wait: 308.827586 	RTT: 1748.479310 	total execute time: 2057.365517 (milliseconds)


As you can see there is a quite a big difference between number of open and close requests.
The same picture we can see on the server side as well:

NFSServerV41 Stats:                   average±stderr(ns)       min(ns)     max(ns)            Sampes 
  DESTROY_SESSION                          26056±4511.89        13000        97000                17
  OPEN                                    1197297±  0.00       816000  31924558000          54398533
  RESTOREFH                                     0±  0.00            0  25018778000          54398533
  SEQUENCE                                   1000±  0.00         1000  26066722000          55601046
  LOOKUP                                  4607959±  0.00       375000  26977455000             32118
  GETDEVICEINFO                             13158±100.88         4000       655000             11378
  CLOSE                                  16236211±  0.00         5000  21021819000             20420
  LAYOUTGET                             271736361±  0.00     10003000  68414723000             21095

The last column is the number of requests.

This is with RHEL6.4 as the client. By looking at the code,
I can see a loop at nfs4proc.c#nfs4_do_open() which can be
the cause of the problem. Nevertheless, I can't
fine any reason why this look turned into an 'infinite' one.

At the and our server ran out of memory and we have returned
NFSERR_SERVERFAULT to the client. This triggered client to
reestablish the session and all open state ids was
invalidated and cleaned up.

I am still trying to reproduce this behavior (on client
and server) and any hint is welcome.

Tigran.