rpc.mountd can be blocked by a bad client

rpc.mountd can be blocked by a bad client

[Strösser, Bodo]

Hello,

a few days ago we had some trouble with a NFS server. The clients most of the time no longer
could mount any shares, but in rare cases they had success.

We found out, that during the times when mounts failed, rpc.mountd hung on a write() to a TCP
socket. netstat showed, that Send-Q was full and Recv-Q counted up slowly. After a long time
the write ended with an error ("TCP timeout" IIRC) and rpc.mountd worked normally for a short
while until it again hung on write() for the same reason. The problem was caused by a MTU size
configured wrong. So, one single bad client (or as much clients as the number of threads used
by rpc.mountd) can block rpc.mountd entirely.

But what will happen, if someone intentionally sends RPC requests, but doesn't read() the
answers? I wrote a small tool to test this situation. It fires DUMP requests to rpc.mountd as
fast as possible, but does not read from the socket. The result is the same as with the
problem above: rpc.mountd hangs in write() and no longer responds to other requests while no
TCP timeout breaks up this situation.

So it's quite easy to intentionally block rpc.mountd from remote.

Please CC me, I'm not on the list.

Best regards,
Bodo

Re: rpc.mountd can be blocked by a bad client

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2183 bytes --]

On Wed, 24 Sep 2014 12:57:09 +0200 "Strösser, Bodo"
<bodo.stroesser@ts.fujitsu.com> wrote:

> Hello,
> 
> a few days ago we had some trouble with a NFS server. The clients most of the time no longer
> could mount any shares, but in rare cases they had success.
> 
> We found out, that during the times when mounts failed, rpc.mountd hung on a write() to a TCP
> socket. netstat showed, that Send-Q was full and Recv-Q counted up slowly. After a long time
> the write ended with an error ("TCP timeout" IIRC) and rpc.mountd worked normally for a short
> while until it again hung on write() for the same reason. The problem was caused by a MTU size
> configured wrong. So, one single bad client (or as much clients as the number of threads used
> by rpc.mountd) can block rpc.mountd entirely.
> 
> But what will happen, if someone intentionally sends RPC requests, but doesn't read() the
> answers? I wrote a small tool to test this situation. It fires DUMP requests to rpc.mountd as
> fast as possible, but does not read from the socket. The result is the same as with the
> problem above: rpc.mountd hangs in write() and no longer responds to other requests while no
> TCP timeout breaks up this situation.
> 
> So it's quite easy to intentionally block rpc.mountd from remote.

That's rather nasty.
We could possibly set the socket to be non-blocking, or we could set an alarm
just before handling a request.
Probably rpc_dispatch() in support/nfs/rpcdispatch.c would be the best place
to put the timeout.
 catch SIGALRM (don't set SA_RESTART)
 alarm(10);
 call svc_sendreply
 alarm(0);

if the alarm fires while svc_sendreply is writing to the socket it should get
an error and close the connection.

This would only fix mountd (as it is the only process to use rpc_dispatch).
Is a similar thing needed for statd I wonder??  It isn't so important.

NeilBrown

> 
> Please CC me, I'm not on the list.
> 
> Best regards,
> Bodo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

RE: rpc.mountd can be blocked by a bad client

[Strösser, Bodo]

PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBOZWlsQnJvd24gW21haWx0bzpu
ZWlsYkBzdXNlLmRlXQ0KPiBTZW50OiBUaHVyc2RheSwgU2VwdGVtYmVyIDI1LCAyMDE0IDI6MzIg
QU0NCj4gVG86IFN0csO2c3NlciwgQm9kbw0KPiBDYzogbGludXgtbmZzQHZnZXIua2VybmVsLm9y
ZzsgYmZpZWxkc0BmaWVsZHNlcy5vcmcNCj4gU3ViamVjdDogUmU6IHJwYy5tb3VudGQgY2FuIGJl
IGJsb2NrZWQgYnkgYSBiYWQgY2xpZW50DQo+IA0KPiBPbiBXZWQsIDI0IFNlcCAyMDE0IDEyOjU3
OjA5ICswMjAwICJTdHLDtnNzZXIsIEJvZG8iDQo+IDxib2RvLnN0cm9lc3NlckB0cy5mdWppdHN1
LmNvbT4gd3JvdGU6DQo+IA0KPiA+IEhlbGxvLA0KPiA+DQo+ID4gYSBmZXcgZGF5cyBhZ28gd2Ug
aGFkIHNvbWUgdHJvdWJsZSB3aXRoIGEgTkZTIHNlcnZlci4gVGhlIGNsaWVudHMgbW9zdCBvZiB0
aGUgdGltZSBubw0KPiBsb25nZXINCj4gPiBjb3VsZCBtb3VudCBhbnkgc2hhcmVzLCBidXQgaW4g
cmFyZSBjYXNlcyB0aGV5IGhhZCBzdWNjZXNzLg0KPiA+DQo+ID4gV2UgZm91bmQgb3V0LCB0aGF0
IGR1cmluZyB0aGUgdGltZXMgd2hlbiBtb3VudHMgZmFpbGVkLCBycGMubW91bnRkIGh1bmcgb24g
YSB3cml0ZSgpIHRvDQo+IGEgVENQDQo+ID4gc29ja2V0LiBuZXRzdGF0IHNob3dlZCwgdGhhdCBT
ZW5kLVEgd2FzIGZ1bGwgYW5kIFJlY3YtUSBjb3VudGVkIHVwIHNsb3dseS4gQWZ0ZXIgYSBsb25n
DQo+IHRpbWUNCj4gPiB0aGUgd3JpdGUgZW5kZWQgd2l0aCBhbiBlcnJvciAoIlRDUCB0aW1lb3V0
IiBJSVJDKSBhbmQgcnBjLm1vdW50ZCB3b3JrZWQgbm9ybWFsbHkgZm9yIGENCj4gc2hvcnQNCj4g
PiB3aGlsZSB1bnRpbCBpdCBhZ2FpbiBodW5nIG9uIHdyaXRlKCkgZm9yIHRoZSBzYW1lIHJlYXNv
bi4gVGhlIHByb2JsZW0gd2FzIGNhdXNlZCBieSBhDQo+IE1UVSBzaXplDQo+ID4gY29uZmlndXJl
ZCB3cm9uZy4gU28sIG9uZSBzaW5nbGUgYmFkIGNsaWVudCAob3IgYXMgbXVjaCBjbGllbnRzIGFz
IHRoZSBudW1iZXIgb2YgdGhyZWFkcw0KPiB1c2VkDQo+ID4gYnkgcnBjLm1vdW50ZCkgY2FuIGJs
b2NrIHJwYy5tb3VudGQgZW50aXJlbHkuDQo+ID4NCj4gPiBCdXQgd2hhdCB3aWxsIGhhcHBlbiwg
aWYgc29tZW9uZSBpbnRlbnRpb25hbGx5IHNlbmRzIFJQQyByZXF1ZXN0cywgYnV0IGRvZXNuJ3Qg
cmVhZCgpDQo+IHRoZQ0KPiA+IGFuc3dlcnM/IEkgd3JvdGUgYSBzbWFsbCB0b29sIHRvIHRlc3Qg
dGhpcyBzaXR1YXRpb24uIEl0IGZpcmVzIERVTVAgcmVxdWVzdHMgdG8NCj4gcnBjLm1vdW50ZCBh
cw0KPiA+IGZhc3QgYXMgcG9zc2libGUsIGJ1dCBkb2VzIG5vdCByZWFkIGZyb20gdGhlIHNvY2tl
dC4gVGhlIHJlc3VsdCBpcyB0aGUgc2FtZSBhcyB3aXRoIHRoZQ0KPiA+IHByb2JsZW0gYWJvdmU6
IHJwYy5tb3VudGQgaGFuZ3MgaW4gd3JpdGUoKSBhbmQgbm8gbG9uZ2VyIHJlc3BvbmRzIHRvIG90
aGVyIHJlcXVlc3RzDQo+IHdoaWxlIG5vDQo+ID4gVENQIHRpbWVvdXQgYnJlYWtzIHVwIHRoaXMg
c2l0dWF0aW9uLg0KPiA+DQo+ID4gU28gaXQncyBxdWl0ZSBlYXN5IHRvIGludGVudGlvbmFsbHkg
YmxvY2sgcnBjLm1vdW50ZCBmcm9tIHJlbW90ZS4NCj4gDQo+IFRoYXQncyByYXRoZXIgbmFzdHku
DQo+IFdlIGNvdWxkIHBvc3NpYmx5IHNldCB0aGUgc29ja2V0IHRvIGJlIG5vbi1ibG9ja2luZywg
b3Igd2UgY291bGQgc2V0IGFuIGFsYXJtDQo+IGp1c3QgYmVmb3JlIGhhbmRsaW5nIGEgcmVxdWVz
dC4NCj4gUHJvYmFibHkgcnBjX2Rpc3BhdGNoKCkgaW4gc3VwcG9ydC9uZnMvcnBjZGlzcGF0Y2gu
YyB3b3VsZCBiZSB0aGUgYmVzdCBwbGFjZQ0KPiB0byBwdXQgdGhlIHRpbWVvdXQuDQo+ICBjYXRj
aCBTSUdBTFJNIChkb24ndCBzZXQgU0FfUkVTVEFSVCkNCj4gIGFsYXJtKDEwKTsNCj4gIGNhbGwg
c3ZjX3NlbmRyZXBseQ0KPiAgYWxhcm0oMCk7DQo+IA0KDQpJIGFsc28gdGhvdWdodCBhYm91dCBj
aGFuZ2luZyB0aGUgc29ja2V0IHRvIG5vbi1ibG9ja2luZy4gQnV0IEknbSBub3Qgc3VyZTogaXMg
aXQNCnBvc3NpYmxlIHRvIGhhdmUgc3VjaCBiaWcgUlBDIHJlcGxpZXMsIHRoYXQgdGhleSBkb24n
dCBmaXQgaW50byB0aGUgc29ja2V0DQpidWZmZXI/IElmIHNvLCB3cml0ZSgpIHdvdWxkIHB1dCB0
aGUgZmlyc3QgcGFydCBpbnRvIHRoZSBidWZmZXIgYW5kIGEgc2Vjb25kDQp3cml0ZSBmb3IgdGhl
IHJlc3Qgd291bGQgZmFpbCwgYXMgcHJvYmFibHkgdGhlIGZpcnN0IHBhcnQgaXNuJ3QgYWNrZWQg
eWV0LCByaWdodD8NClNvLCBub24tYmxvY2tpbmcgbmVlZHMgdG8gYmUgY29tYmluZWQgd2l0aCBh
IGhhbmRsaW5nIG9mIGJ1ZmZlci1mdWxsIHNpdHVhdGlvbnMsDQpJIGd1ZXNzLiBTdWNoIGEgaGFu
ZGxpbmcgdG9nZXRoZXIgd2l0aCBhIHRpbWVvdXQgZm9yIHN0YXJ2aW5nIGNvbm5lY3Rpb25zIHdv
dWxkDQpiZSBhIGNsZWFuIHNvbHV0aW9uLg0KVG8gZG8gdGhhdCwgb25lIHdvdWxkIGhhdmUgdG8g
cmVwbGFjZSB0aGUgdGNwIHdyaXRlIHJvdXRpbmUgb2YgdGhlIHJwYyBsaWJyYXJ5Lg0KVGhhdCBt
ZWFucyB0byBjaGFuZ2UgdGhlIHhkcnMncyBwb2ludGVyIHRvIHRoZSB3cml0ZSBmdW5jdGlvbi4g
SSBkb24ndCBrbm93LA0Kd2hldGhlciB0aGF0IGNhbiBiZSBkb25lIGluIGEgcG9ydGFibGUgd2F5
LCB3aGljaCB3b3JrcyBhdCB0aGUgZGlmZmVyZW50IHBsYXRmb3Jtcy4NCg0KQWJvdXQgc2V0dGlu
ZyBhIGFsYXJtIHRpbWVvdXQ6IEknbSBub3Qgc3VyZSwgdGhhdCBycGNfZGlzcGF0Y2goKSBpcyB0
aGUgcmlnaHQNCnBsYWNlIGZvciBpdC4gbW91bnRkIHVzZXMgbW91bnRfZGlzcGF0Y2goKSB3aGlj
aCBoYXMgYW4gZXhpdCB2aWEgc3ZjZXJyX2F1dGgoKSwNCnRoYXQgYWdhaW4gc2VuZHMgYSByZXBs
eS4gU28gdGhlIHRpbWVvdXQgeW91IHN1Z2dlc3Qgc2hvdWxkIGJlIGluc2VydGVkIGluDQptb3Vu
dF9kaXNwYXRjaCgpLCBJIHRoaW5rLg0KT1RPSCwgYSB0aW1lb3V0IHdpbGwgc2hvcnRlbiB0aGUg
aGFuZywgYnV0IGJhZCBjbGllbnRzIGNhbiBzdGlsbCBzbG93IGRvd24gbW91bnRkDQpleHRyZW1l
bHkuDQoNCkJUVzogQUZBSUNTIG9uIExpbnV4IHdpdGggbGlidGlycGMsIHVzaW5nIHRoZSBjb250
cm9sIFNWQ0dFVF9DT05OTUFYUkVDLCB0aGUgc29ja2V0DQppbmRpcmVjdGx5IGNhbiBzZXQgdG8g
bm9uLWJsb2NraW5nLiBUaGF0IHNlZW1zIHRvIHJlc3VsdCBpbiB3cml0ZV92YygpIGRvaW5nIGEg
bWF4Lg0KMiBzZWNvbmQgbG9vcCBvZiB3cml0ZSgpIHVudGlsIGl0IGdpdmVzIHVwLg0KDQpPbmUg
b3RoZXIgcG9pbnQ6IEFGQUlDUyBvbiBMaW51eCB3aXRoIGxpYnRpcnBjIHRoZSBsaXN0ZW5pbmcg
c29ja2V0IG9mIG1vdW50ZCBpcw0KaW4gYmxvY2tpbmcgbW9kZS4gV291bGQgdGhhdCBiZSBhIHBy
b2JsZW0gd2hlbiBydW5uaW5nIG11bHRpcGxlICJ0aHJlYWRzIj8NClRoZSBjb21tZW50IGluIHN2
Y19zb2NrZXQuYy9zdmNfc29ja2V0KCksIHdoZXJlIHRoZSBsaXN0ZW5pbmcgc29ja2V0IGlzIHNl
dCB0bw0Kbm9uLWJsb2NraW5nLCBzb3VuZHMgdmVyeSByZWFzb25hYmxlLiBCdXQgQUZBSUNTIGlm
IGxpYnRpcnBjIGlzIHVzZWQsIE9fTk9OQkxPQ0sNCmN1cnJlbnRseSBpc24ndCBzZXQuDQoNCkJv
ZG8gU3Ryb2Vzc2VyDQoNCg0KPiBpZiB0aGUgYWxhcm0gZmlyZXMgd2hpbGUgc3ZjX3NlbmRyZXBs
eSBpcyB3cml0aW5nIHRvIHRoZSBzb2NrZXQgaXQgc2hvdWxkIGdldA0KPiBhbiBlcnJvciBhbmQg
Y2xvc2UgdGhlIGNvbm5lY3Rpb24uDQo+IA0KPiBUaGlzIHdvdWxkIG9ubHkgZml4IG1vdW50ZCAo
YXMgaXQgaXMgdGhlIG9ubHkgcHJvY2VzcyB0byB1c2UgcnBjX2Rpc3BhdGNoKS4NCj4gSXMgYSBz
aW1pbGFyIHRoaW5nIG5lZWRlZCBmb3Igc3RhdGQgSSB3b25kZXI/PyAgSXQgaXNuJ3Qgc28gaW1w
b3J0YW50Lg0KPiANCj4gTmVpbEJyb3duDQo+IA0KPiA+DQo+ID4gUGxlYXNlIENDIG1lLCBJJ20g
bm90IG9uIHRoZSBsaXN0Lg0KPiA+DQo+ID4gQmVzdCByZWdhcmRzLA0KPiA+IEJvZG8NCj4gPiAt
LQ0KPiA+IFRvIHVuc3Vic2NyaWJlIGZyb20gdGhpcyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1
YnNjcmliZSBsaW51eC1uZnMiIGluDQo+ID4gdGhlIGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9y
ZG9tb0B2Z2VyLmtlcm5lbC5vcmcNCj4gPiBNb3JlIG1ham9yZG9tbyBpbmZvIGF0ICBodHRwOi8v
dmdlci5rZXJuZWwub3JnL21ham9yZG9tby1pbmZvLmh0bWwNCg0K

Re: rpc.mountd can be blocked by a bad client

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]


Hi Bodo,

 I just discovered that you sent some patches to address this.
I found them here:
  http://www.spinics.net/lists/linux-nfs/msg47235.html

but I don't remember ever seeing them (probably my fault), and they are all
in one big email message which doesn't immediately look like it contains
patches.

Could you please resend the patches to the list and separate emails.  That
will make it a lot easier to review and discuss.

Thanks,
NeilBrown

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

RE: rpc.mountd can be blocked by a bad client

[Strösser, Bodo]

Hi Neil,

yes, I'll do.

I know, it wasn't a good idea to mix text and patches together in
one big mail. I did so, because I have been in a great hurry and
my normal environment for sending patches wasn't available ...

So, sorry for the trouble.

Best regards,
Bodo

> -----Original Message-----
> From: NeilBrown [mailto:neilb@suse.de]
> Sent: Wednesday, November 05, 2014 6:15 AM
> To: Strösser, Bodo
> Cc: linux-nfs@vger.kernel.org; bfields@fieldses.org
> Subject: Re: rpc.mountd can be blocked by a bad client
> 
> 
> Hi Bodo,
> 
>  I just discovered that you sent some patches to address this.
> I found them here:
>   http://www.spinics.net/lists/linux-nfs/msg47235.html
> 
> but I don't remember ever seeing them (probably my fault), and they are all
> in one big email message which doesn't immediately look like it contains
> patches.
> 
> Could you please resend the patches to the list and separate emails.  That
> will make it a lot easier to review and discuss.
> 
> Thanks,
> NeilBrown