rdma connection close/reopen every 5 mins

rdma connection close/reopen every 5 mins

[scar]

Hello, I have setup NFS/RDMA on our infiniband network.  The OS is 
CentOS 6.8 with kernel 2.6.32-696.1.1.el6.x86_64 and I am using 
rdma/nfs-rdma packages from the OS version 6.9_4.1.  The connection is 
successful:

10.10.11.249:/newwing on /newwing type nfs 
(rw,rdma,port=20049,addr=10.10.11.249)

except in /var/log/messages I am always seeing this exactly every 5 minutes:
  kernel: rpcrdma: connection to 10.10.11.249:20049 closed (-103)
  kernel: rpcrdma: connection to 10.10.11.249:20049 on mlx4_0, memreg 6 
slots 32 ird 16

I suppose it is from some service or cron job.... But I wonder how I 
could keep the connection open so this isn't happening every 5 minutes?

Thanks

Re: rdma connection close/reopen every 5 mins

[Chuck Lever]

> On Apr 26, 2017, at 10:23 AM, scar <scar@drigon.com> wrote:
> 
> Hello, I have setup NFS/RDMA on our infiniband network.  The OS is CentOS 6.8 with kernel 2.6.32-696.1.1.el6.x86_64 and I am using rdma/nfs-rdma packages from the OS version 6.9_4.1.  The connection is successful:
> 
> 10.10.11.249:/newwing on /newwing type nfs (rw,rdma,port=20049,addr=10.10.11.249)
> 
> except in /var/log/messages I am always seeing this exactly every 5 minutes:
> kernel: rpcrdma: connection to 10.10.11.249:20049 closed (-103)
> kernel: rpcrdma: connection to 10.10.11.249:20049 on mlx4_0, memreg 6 slots 32 ird 16

> I suppose it is from some service or cron job.... But I wonder how I could keep the connection open so this isn't happening every 5 minutes?

If it's every 5 minutes, that's probably the RPC idle timeout
that is closing the connection.

So this should be entirely harmless.


--
Chuck Lever

Re: rdma connection close/reopen every 5 mins

[scar]

Chuck Lever wrote on 04/26/2017 01:32 PM:
> If it's every 5 minutes, that's probably the RPC idle timeout
> that is closing the connection.
>
> So this should be entirely harmless.

Yes so far it seems harmless, but it is annoying and fills up the log 
with spam, plus it's not something I see when using NFS/TCP.

thanks

Re: rdma connection close/reopen every 5 mins

[Chuck Lever]

> On Apr 26, 2017, at 3:47 PM, scar <scar@drigon.com> wrote:
> 
> Chuck Lever wrote on 04/26/2017 01:32 PM:
>> If it's every 5 minutes, that's probably the RPC idle timeout
>> that is closing the connection.
>> 
>> So this should be entirely harmless.
> 
> Yes so far it seems harmless, but it is annoying and fills up the log with spam, plus it's not something I see when using NFS/TCP.

That's correct, NFS/TCP does not have the "connected / closed"
messages, but the idle timeout behavior is the exactly the same.

I tried to remove those messages a few years ago and was told
to go pound sand.

The best you can do is create a mechanism to access a file on
those mount points more frequently than the idle timeout. That
should prevent the connection from closing.


--
Chuck Lever

Re: rdma connection close/reopen every 5 mins

[Trond Myklebust]

T24gV2VkLCAyMDE3LTA0LTI2IGF0IDE1OjUyIC0wNzAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
PiBPbiBBcHIgMjYsIDIwMTcsIGF0IDM6NDcgUE0sIHNjYXIgPHNjYXJAZHJpZ29uLmNvbT4gd3Jv
dGU6DQo+ID4gDQo+ID4gQ2h1Y2sgTGV2ZXIgd3JvdGUgb24gMDQvMjYvMjAxNyAwMTozMiBQTToN
Cj4gPiA+IElmIGl0J3MgZXZlcnkgNSBtaW51dGVzLCB0aGF0J3MgcHJvYmFibHkgdGhlIFJQQyBp
ZGxlIHRpbWVvdXQNCj4gPiA+IHRoYXQgaXMgY2xvc2luZyB0aGUgY29ubmVjdGlvbi4NCj4gPiA+
IA0KPiA+ID4gU28gdGhpcyBzaG91bGQgYmUgZW50aXJlbHkgaGFybWxlc3MuDQo+ID4gDQo+ID4g
WWVzIHNvIGZhciBpdCBzZWVtcyBoYXJtbGVzcywgYnV0IGl0IGlzIGFubm95aW5nIGFuZCBmaWxs
cyB1cCB0aGUNCj4gPiBsb2cgd2l0aCBzcGFtLCBwbHVzIGl0J3Mgbm90IHNvbWV0aGluZyBJIHNl
ZSB3aGVuIHVzaW5nIE5GUy9UQ1AuDQo+IA0KPiBUaGF0J3MgY29ycmVjdCwgTkZTL1RDUCBkb2Vz
IG5vdCBoYXZlIHRoZSAiY29ubmVjdGVkIC8gY2xvc2VkIg0KPiBtZXNzYWdlcywgYnV0IHRoZSBp
ZGxlIHRpbWVvdXQgYmVoYXZpb3IgaXMgdGhlIGV4YWN0bHkgdGhlIHNhbWUuDQo+IA0KPiBJIHRy
aWVkIHRvIHJlbW92ZSB0aG9zZSBtZXNzYWdlcyBhIGZldyB5ZWFycyBhZ28gYW5kIHdhcyB0b2xk
DQo+IHRvIGdvIHBvdW5kIHNhbmQuDQo+IA0KIA0KDQpCeSBtZT8gSeKAmWQgYmUgaGFwcHkgdG8g
dGFrZSBhIHBhdGNoIHRvIHJlbW92ZSB0aG9zZSBtZXNzYWdlcy4gSSBmaW5kIGl0DQphYnNvbHV0
ZWx5IHJpZGljdWxvdXMgdG8gYmUgbG9nZ2luZyBmdWxseSBleHBlY3RlZCBiZWhhdmlvdXIgbGlr
ZSB0aGlzLg0KSXQgc2hvdWxkIGF0IGJlc3QgYmUgYSBkcHJpbnRrKCkgYW5kIEknZCBwcmVmZXIg
dG8gcmVwbGFjZSBpdCB3aXRoDQp0cmFjZXBvaW50cyBsaWtlIHdlIGRvIGZvciB0aGUgc29ja2V0
IGNvZGUuDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWlu
ZXIsIFByaW1hcnlEYXRhDQp0cm9uZC5teWtsZWJ1c3RAcHJpbWFyeWRhdGEuY29tDQo=

Re: rdma connection close/reopen every 5 mins

[Chuck Lever]

> On Apr 26, 2017, at 6:46 PM, Trond Myklebust <trondmy@primarydata.com> wrote:
> 
> On Wed, 2017-04-26 at 15:52 -0700, Chuck Lever wrote:
>>> On Apr 26, 2017, at 3:47 PM, scar <scar@drigon.com> wrote:
>>> 
>>> Chuck Lever wrote on 04/26/2017 01:32 PM:
>>>> If it's every 5 minutes, that's probably the RPC idle timeout
>>>> that is closing the connection.
>>>> 
>>>> So this should be entirely harmless.
>>> 
>>> Yes so far it seems harmless, but it is annoying and fills up the
>>> log with spam, plus it's not something I see when using NFS/TCP.
>> 
>> That's correct, NFS/TCP does not have the "connected / closed"
>> messages, but the idle timeout behavior is the exactly the same.
>> 
>> I tried to remove those messages a few years ago and was told
>> to go pound sand.
>> 
> 
> 
> By me?

No, I filed a bug on bugzilla.kernel.org and I think it was
Alan Cox (!) who told me to leave these alone.


> I’d be happy to take a patch to remove those messages. I find it
> absolutely ridiculous to be logging fully expected behaviour like this.
> It should at best be a dprintk() and I'd prefer to replace it with
> tracepoints like we do for the socket code.

I can send a patch for 4.13 that converts these to
dprintks for now. Tracepoints are definitely in plan...