[PATCH] svcrdma: Fence LOCAL_INV work requests

[PATCH] svcrdma: Fence LOCAL_INV work requests

[Steve Wise]

This applies on top of:

	commit e5a070216356dbcb03607cb264cc3104e17339b3
	Author: Steve Wise <swise@opengridcomputing.com>
	Date:   Wed May 28 15:12:01 2014 -0500

	    svcrdma: refactor marshalling logic

Fencing forces the invalidate to only happen after all subsequent send
work requests have been completed.

Signed-off-by: Steve Wise <swise@opengridcomputing.com>
---

 net/sunrpc/xprtrdma/svc_rdma_recvfrom.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
index 52d9f2c..8f92a61 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
@@ -338,7 +338,7 @@ static int rdma_read_chunk_frmr(struct svcxprt_rdma *xprt,
 		memset(&inv_wr, 0, sizeof(inv_wr));
 		inv_wr.wr_id = (unsigned long)ctxt;
 		inv_wr.opcode = IB_WR_LOCAL_INV;
-		inv_wr.send_flags = IB_SEND_SIGNALED;
+		inv_wr.send_flags = IB_SEND_SIGNALED | IB_SEND_FENCE;
 		inv_wr.ex.invalidate_rkey = frmr->mr->lkey;
 	}
 	ctxt->wr_op = read_wr.opcode;

RE: [PATCH] svcrdma: Fence LOCAL_INV work requests

[Devesh Sharma]

UmVwb3J0ZWQgYnkgOiBEZXZlc2ggU2hhcm1hDQpBY2snZWQgYnk6IERldmVzaCBTaGFybWENCg0K
LVJlZ2FyZHMNCiBEZXZlc2gNCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9t
OiBsaW51eC1yZG1hLW93bmVyQHZnZXIua2VybmVsLm9yZyBbbWFpbHRvOmxpbnV4LXJkbWEtDQo+
IG93bmVyQHZnZXIua2VybmVsLm9yZ10gT24gQmVoYWxmIE9mIFN0ZXZlIFdpc2UNCj4gU2VudDog
TW9uZGF5LCBKdW5lIDAyLCAyMDE0IDEwOjQ1IFBNDQo+IFRvOiBiZmllbGRzQGZpZWxkc2VzLm9y
Zw0KPiBDYzogbGludXgtbmZzQHZnZXIua2VybmVsLm9yZzsgbGludXgtcmRtYUB2Z2VyLmtlcm5l
bC5vcmc7DQo+IHRvbUBvcGVuZ3JpZGNvbXB1dGluZy5jb20NCj4gU3ViamVjdDogW1BBVENIXSBz
dmNyZG1hOiBGZW5jZSBMT0NBTF9JTlYgd29yayByZXF1ZXN0cw0KPiANCj4gVGhpcyBhcHBsaWVz
IG9uIHRvcCBvZjoNCj4gDQo+IAljb21taXQgZTVhMDcwMjE2MzU2ZGJjYjAzNjA3Y2IyNjRjYzMx
MDRlMTczMzliMw0KPiAJQXV0aG9yOiBTdGV2ZSBXaXNlIDxzd2lzZUBvcGVuZ3JpZGNvbXB1dGlu
Zy5jb20+DQo+IAlEYXRlOiAgIFdlZCBNYXkgMjggMTU6MTI6MDEgMjAxNCAtMDUwMA0KPiANCj4g
CSAgICBzdmNyZG1hOiByZWZhY3RvciBtYXJzaGFsbGluZyBsb2dpYw0KPiANCj4gRmVuY2luZyBm
b3JjZXMgdGhlIGludmFsaWRhdGUgdG8gb25seSBoYXBwZW4gYWZ0ZXIgYWxsIHN1YnNlcXVlbnQg
c2VuZCB3b3JrDQo+IHJlcXVlc3RzIGhhdmUgYmVlbiBjb21wbGV0ZWQuDQo+IA0KPiBTaWduZWQt
b2ZmLWJ5OiBTdGV2ZSBXaXNlIDxzd2lzZUBvcGVuZ3JpZGNvbXB1dGluZy5jb20+DQo+IC0tLQ0K
PiANCj4gIG5ldC9zdW5ycGMveHBydHJkbWEvc3ZjX3JkbWFfcmVjdmZyb20uYyB8ICAgIDIgKy0N
Cj4gIDEgZmlsZXMgY2hhbmdlZCwgMSBpbnNlcnRpb25zKCspLCAxIGRlbGV0aW9ucygtKQ0KPiAN
Cj4gZGlmZiAtLWdpdCBhL25ldC9zdW5ycGMveHBydHJkbWEvc3ZjX3JkbWFfcmVjdmZyb20uYw0K
PiBiL25ldC9zdW5ycGMveHBydHJkbWEvc3ZjX3JkbWFfcmVjdmZyb20uYw0KPiBpbmRleCA1MmQ5
ZjJjLi44ZjkyYTYxIDEwMDY0NA0KPiAtLS0gYS9uZXQvc3VucnBjL3hwcnRyZG1hL3N2Y19yZG1h
X3JlY3Zmcm9tLmMNCj4gKysrIGIvbmV0L3N1bnJwYy94cHJ0cmRtYS9zdmNfcmRtYV9yZWN2ZnJv
bS5jDQo+IEBAIC0zMzgsNyArMzM4LDcgQEAgc3RhdGljIGludCByZG1hX3JlYWRfY2h1bmtfZnJt
cihzdHJ1Y3Qgc3ZjeHBydF9yZG1hDQo+ICp4cHJ0LA0KPiAgCQltZW1zZXQoJmludl93ciwgMCwg
c2l6ZW9mKGludl93cikpOw0KPiAgCQlpbnZfd3Iud3JfaWQgPSAodW5zaWduZWQgbG9uZyljdHh0
Ow0KPiAgCQlpbnZfd3Iub3Bjb2RlID0gSUJfV1JfTE9DQUxfSU5WOw0KPiAtCQlpbnZfd3Iuc2Vu
ZF9mbGFncyA9IElCX1NFTkRfU0lHTkFMRUQ7DQo+ICsJCWludl93ci5zZW5kX2ZsYWdzID0gSUJf
U0VORF9TSUdOQUxFRCB8IElCX1NFTkRfRkVOQ0U7DQo+ICAJCWludl93ci5leC5pbnZhbGlkYXRl
X3JrZXkgPSBmcm1yLT5tci0+bGtleTsNCj4gIAl9DQo+ICAJY3R4dC0+d3Jfb3AgPSByZWFkX3dy
Lm9wY29kZTsNCj4gDQo+IC0tDQo+IFRvIHVuc3Vic2NyaWJlIGZyb20gdGhpcyBsaXN0OiBzZW5k
IHRoZSBsaW5lICJ1bnN1YnNjcmliZSBsaW51eC1yZG1hIiBpbiB0aGUNCj4gYm9keSBvZiBhIG1l
c3NhZ2UgdG8gbWFqb3Jkb21vQHZnZXIua2VybmVsLm9yZyBNb3JlIG1ham9yZG9tbyBpbmZvIGF0
DQo+IGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0K

Re: [PATCH] svcrdma: Fence LOCAL_INV work requests

[Bart Van Assche]

On 06/02/14 19:14, Steve Wise wrote:
> Fencing forces the invalidate to only happen after all subsequent send
> work requests have been completed.

Shouldn't that read "prior" instead of "subsequent" ?

Bart.

RE: [PATCH] svcrdma: Fence LOCAL_INV work requests

[Steve Wise]

> -----Original Message-----
> From: Bart Van Assche [mailto:bvanassche@acm.org]
> Sent: Thursday, June 05, 2014 2:23 AM
> To: Steve Wise; bfields@fieldses.org
> Cc: linux-nfs@vger.kernel.org; linux-rdma@vger.kernel.org; tom@opengridcomputing.com
> Subject: Re: [PATCH] svcrdma: Fence LOCAL_INV work requests
> 
> On 06/02/14 19:14, Steve Wise wrote:
> > Fencing forces the invalidate to only happen after all subsequent send
> > work requests have been completed.
> 
> Shouldn't that read "prior" instead of "subsequent" ?
>

Yes.  
 

Re: [PATCH] svcrdma: Fence LOCAL_INV work requests

[Steve Wise]

On 6/5/2014 8:55 AM, Steve Wise wrote
>> -----Original Message-----
>> From: Bart Van Assche [mailto:bvanassche@acm.org]
>> Sent: Thursday, June 05, 2014 2:23 AM
>> To: Steve Wise; bfields@fieldses.org
>> Cc: linux-nfs@vger.kernel.org; linux-rdma@vger.kernel.org; tom@opengridcomputing.com
>> Subject: Re: [PATCH] svcrdma: Fence LOCAL_INV work requests
>>
>> On 06/02/14 19:14, Steve Wise wrote:
>>> Fencing forces the invalidate to only happen after all subsequent send
>>> work requests have been completed.
>> Shouldn't that read "prior" instead of "subsequent" ?
>>
> Yes.
>   

I posted V2 of the patch to fix the comment.

Thanks,

Steve.