NFSv4 callback: too many open connections, consider increasing the number of threads.

NFSv4 callback: too many open connections, consider increasing the number of threads.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1529 bytes --]


[resending - with more current address for trond]

Hi Trond et al.

 we have a customer who reports getting the message:
        [10879512.275305] NFSv4 callback: too many open  connections,
 consider increasing the number of threads. [10879512.275315] NFSv4 callback:
 too many open  connections, consider increasing the number of threads.
 [10879517.276133] __ratelimit: 133011 callbacks suppressed

Of course you cannot increase the number of threads handling callbacks.

The patch below will get rid of the message - which is presumably mostly just
noise - or possibly change it to
   NFSv4 callback: too many open connections, consider increasing the max
number of connections if the number of connections gets too high.

This isn't much good as this patch doesn't allow the max number to be
increased.
Do we want to make the max number of callback connections configurable, or
should we find a way to silence that message.
(For lockd that message is valid as there is a sysctl to change the max
number of connections for lockd).

Thanks,
NeilBrown


diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c
index 23ff18f..1129c42 100644
--- a/fs/nfs/callback.c
+++ b/fs/nfs/callback.c
@@ -265,6 +265,10 @@ int nfs_callback_up(u32 minorversion, struct rpc_xprt
*xprt) ret = -ENOMEM;
 		goto out_err;
 	}
+	/* As there is only one thread so we need to over-ride the
+	 * default maximum of 80 connections
+	 */
+	serv->sv_maxconn = 1024;
 
 	ret = svc_bind(serv, net);
 	if (ret < 0) {

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: NFSv4 callback: too many open connections, consider increasing the number of threads.

[Myklebust, Trond]

T24gVGh1LCAyMDEyLTA3LTE5IGF0IDE2OjQ2ICsxMDAwLCBOZWlsQnJvd24gd3JvdGU6DQo+IFty
ZXNlbmRpbmcgLSB3aXRoIG1vcmUgY3VycmVudCBhZGRyZXNzIGZvciB0cm9uZF0NCj4gDQo+IEhp
IFRyb25kIGV0IGFsLg0KPiANCj4gIHdlIGhhdmUgYSBjdXN0b21lciB3aG8gcmVwb3J0cyBnZXR0
aW5nIHRoZSBtZXNzYWdlOg0KPiAgICAgICAgIFsxMDg3OTUxMi4yNzUzMDVdIE5GU3Y0IGNhbGxi
YWNrOiB0b28gbWFueSBvcGVuICBjb25uZWN0aW9ucywNCj4gIGNvbnNpZGVyIGluY3JlYXNpbmcg
dGhlIG51bWJlciBvZiB0aHJlYWRzLiBbMTA4Nzk1MTIuMjc1MzE1XSBORlN2NCBjYWxsYmFjazoN
Cj4gIHRvbyBtYW55IG9wZW4gIGNvbm5lY3Rpb25zLCBjb25zaWRlciBpbmNyZWFzaW5nIHRoZSBu
dW1iZXIgb2YgdGhyZWFkcy4NCj4gIFsxMDg3OTUxNy4yNzYxMzNdIF9fcmF0ZWxpbWl0OiAxMzMw
MTEgY2FsbGJhY2tzIHN1cHByZXNzZWQNCj4gDQo+IE9mIGNvdXJzZSB5b3UgY2Fubm90IGluY3Jl
YXNlIHRoZSBudW1iZXIgb2YgdGhyZWFkcyBoYW5kbGluZyBjYWxsYmFja3MuDQo+IA0KPiBUaGUg
cGF0Y2ggYmVsb3cgd2lsbCBnZXQgcmlkIG9mIHRoZSBtZXNzYWdlIC0gd2hpY2ggaXMgcHJlc3Vt
YWJseSBtb3N0bHkganVzdA0KPiBub2lzZSAtIG9yIHBvc3NpYmx5IGNoYW5nZSBpdCB0bw0KPiAg
ICBORlN2NCBjYWxsYmFjazogdG9vIG1hbnkgb3BlbiBjb25uZWN0aW9ucywgY29uc2lkZXIgaW5j
cmVhc2luZyB0aGUgbWF4DQo+IG51bWJlciBvZiBjb25uZWN0aW9ucyBpZiB0aGUgbnVtYmVyIG9m
IGNvbm5lY3Rpb25zIGdldHMgdG9vIGhpZ2guDQo+IA0KPiBUaGlzIGlzbid0IG11Y2ggZ29vZCBh
cyB0aGlzIHBhdGNoIGRvZXNuJ3QgYWxsb3cgdGhlIG1heCBudW1iZXIgdG8gYmUNCj4gaW5jcmVh
c2VkLg0KPiBEbyB3ZSB3YW50IHRvIG1ha2UgdGhlIG1heCBudW1iZXIgb2YgY2FsbGJhY2sgY29u
bmVjdGlvbnMgY29uZmlndXJhYmxlLCBvcg0KPiBzaG91bGQgd2UgZmluZCBhIHdheSB0byBzaWxl
bmNlIHRoYXQgbWVzc2FnZS4NCj4gKEZvciBsb2NrZCB0aGF0IG1lc3NhZ2UgaXMgdmFsaWQgYXMg
dGhlcmUgaXMgYSBzeXNjdGwgdG8gY2hhbmdlIHRoZSBtYXgNCj4gbnVtYmVyIG9mIGNvbm5lY3Rp
b25zIGZvciBsb2NrZCkuDQo+IA0KPiBUaGFua3MsDQo+IE5laWxCcm93bg0KPiANCj4gDQo+IGRp
ZmYgLS1naXQgYS9mcy9uZnMvY2FsbGJhY2suYyBiL2ZzL25mcy9jYWxsYmFjay5jDQo+IGluZGV4
IDIzZmYxOGYuLjExMjljNDIgMTAwNjQ0DQo+IC0tLSBhL2ZzL25mcy9jYWxsYmFjay5jDQo+ICsr
KyBiL2ZzL25mcy9jYWxsYmFjay5jDQo+IEBAIC0yNjUsNiArMjY1LDEwIEBAIGludCBuZnNfY2Fs
bGJhY2tfdXAodTMyIG1pbm9ydmVyc2lvbiwgc3RydWN0IHJwY194cHJ0DQo+ICp4cHJ0KSByZXQg
PSAtRU5PTUVNOw0KPiAgCQlnb3RvIG91dF9lcnI7DQo+ICAJfQ0KPiArCS8qIEFzIHRoZXJlIGlz
IG9ubHkgb25lIHRocmVhZCBzbyB3ZSBuZWVkIHRvIG92ZXItcmlkZSB0aGUNCj4gKwkgKiBkZWZh
dWx0IG1heGltdW0gb2YgODAgY29ubmVjdGlvbnMNCj4gKwkgKi8NCj4gKwlzZXJ2LT5zdl9tYXhj
b25uID0gMTAyNDsNCj4gIA0KPiAgCXJldCA9IHN2Y19iaW5kKHNlcnYsIG5ldCk7DQo+ICAJaWYg
KHJldCA8IDApIHsNCg0KSGkgTmVpbCwNCg0KQ291bGQgeW91IHBsZWFzZSBhZGQgdGhlIG1pc3Np
bmcgcy1vLWI/DQoNCkNoZWVycw0KICBUcm9uZA0K

Re: NFSv4 callback: too many open connections, consider increasing the number of threads.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1549 bytes --]

On Mon, 30 Jul 2012 23:16:23 +0000 "Myklebust, Trond"
<Trond.Myklebust@netapp.com> wrote:

> Hi Neil,
> 
> Could you please add the missing s-o-b?
> 
> Cheers
>   Trond

Here 'tis.  Thanks.

NeilBrown

From: NeilBrown <neilb@suse.de>
Date: Tue, 31 Jul 2012 14:33:39 +1000
Subject: [PATCH] nfs: increase number of permitted callback connections.

By default a sunrpc service is limited to (N+3)*20 connections
where N is the number of threads.  This is 80 when N==1.
If this number is exceeded a warning is printed suggesting that
the number of threads be increased.  However with services which
run a single thread, this is impossible.

For such services there is a ->sv_maxconn setting that can be
used to forcibly increase the limit, and silence the message.
This is used by lockd.

The nfs client uses a sunrpc service to handle callbacks and
it too is single-threaded, so to avoid the useless messages,
and to allow a reasonable number of concurrent connections,
we need to set ->sv_maxconn.  1024 seems like a good number.

Signed-off-by: NeilBrown <neilb@suse.de>

diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c
index 23ff18f..9cb1215 100644
--- a/fs/nfs/callback.c
+++ b/fs/nfs/callback.c
@@ -265,6 +265,10 @@ int nfs_callback_up(u32 minorversion, struct rpc_xprt *xprt)
 		ret = -ENOMEM;
 		goto out_err;
 	}
+	/* As there is only one thread we need to over-ride the
+	 * default maximum of 80 connections
+	 */
+	serv->sv_maxconn = 1024;
 
 	ret = svc_bind(serv, net);
 	if (ret < 0) {

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

NFSv4 callback: too many open connections, consider increasing the number of threads.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1491 bytes --]


Hi Trond et al.

 we have a customer who reports getting the message:
        [10879512.275305] NFSv4 callback: too many open  connections, consider increasing the number of threads.
        [10879512.275315] NFSv4 callback: too many open  connections, consider increasing the number of threads.
        [10879517.276133] __ratelimit: 133011 callbacks suppressed

Of course you cannot increase the number of threads handling callbacks.

The patch below will get rid of the message - which is presumably mostly just
noise - or possibly change it to
   NFSv4 callback: too many open connections, consider increasing the max number of connections
if the number of connections gets too high.

This isn't much good as this patch doesn't allow the max number to be
increased.
Do we want to make the max number of callback connections configurable, or
should we find a way to silence that message.
(For lockd that message is valid as there is a sysctl to change the max
number of connections for lockd).

Thanks,
NeilBrown


diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c
index 23ff18f..1129c42 100644
--- a/fs/nfs/callback.c
+++ b/fs/nfs/callback.c
@@ -265,6 +265,10 @@ int nfs_callback_up(u32 minorversion, struct rpc_xprt *xprt)
 		ret = -ENOMEM;
 		goto out_err;
 	}
+	/* As there is only one thread so we need to over-ride the
+	 * default maximum of 80 connections
+	 */
+	serv->sv_maxconn = 1024;
 
 	ret = svc_bind(serv, net);
 	if (ret < 0) {

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]