From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: [PATCHv2] ibverbs: add a new IBV_ACCESS_GIFT option Date: Sun, 24 Mar 2013 17:45:52 +0200 Message-ID: <20130324154552.GA16346@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Jason Gunthorpe Cc: Roland Dreier , "Michael R. Hines" , Sean Hefty , Hal Rosenstock , Yishai Hadas , Christoph Lameter , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , LKML , qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org List-Id: linux-rdma@vger.kernel.org At the moment registering an MR breaks COW. If the application does not care that adapter sees stale data (for example, it tracks writes reregisters and resends), it can use a new IBV_ACCESS_GIFT flag to prevent registration from breaking COW. The semantics are similar to that of SPLICE_F_GIFT thus the name. Signed-off-by: Michael S. Tsirkin --- This is compiled but untested. Michael, could you please try this patch (together with the kernel patch I'm sending separately) and report whether setting this flag unbreaks overcommit for you? include/infiniband/verbs.h | 3 ++- man/ibv_reg_mr.3 | 5 +++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h index 6acfc81..3290ec9 100644 --- a/include/infiniband/verbs.h +++ b/include/infiniband/verbs.h @@ -290,7 +290,8 @@ enum ibv_access_flags { IBV_ACCESS_REMOTE_WRITE = (1<<1), IBV_ACCESS_REMOTE_READ = (1<<2), IBV_ACCESS_REMOTE_ATOMIC = (1<<3), - IBV_ACCESS_MW_BIND = (1<<4) + IBV_ACCESS_MW_BIND = (1<<4), + IBV_ACCESS_GIFT = (1<<6) }; struct ibv_pd { diff --git a/man/ibv_reg_mr.3 b/man/ibv_reg_mr.3 index 7723771..3c302f0 100644 --- a/man/ibv_reg_mr.3 +++ b/man/ibv_reg_mr.3 @@ -34,6 +34,8 @@ describes the desired memory protection attributes; it is either 0 or the bitwis .B IBV_ACCESS_REMOTE_ATOMIC\fR Enable Remote Atomic Operation Access (if supported) .TP .B IBV_ACCESS_MW_BIND\fR Enable Memory Window Binding +.TP +.B IBV_ACCESS_GIFT\fR This memory is a gift to the adapter: if memory is modified after registration, the local version and data seen by the adapter through this region rkey may differ .PP If .B IBV_ACCESS_REMOTE_WRITE @@ -43,6 +45,9 @@ is set, then .B IBV_ACCESS_LOCAL_WRITE must be set too. .PP +.B IBV_ACCESS_GIFT +is only legal with remote or local read access. +.PP Local read access is always enabled for the MR. .PP .B ibv_dereg_mr() -- MST -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754151Ab3CXPpa (ORCPT ); Sun, 24 Mar 2013 11:45:30 -0400 Received: from mx1.redhat.com ([209.132.183.28]:11188 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753944Ab3CXPp3 (ORCPT ); Sun, 24 Mar 2013 11:45:29 -0400 Date: Sun, 24 Mar 2013 17:45:52 +0200 From: "Michael S. Tsirkin" To: Jason Gunthorpe Cc: Roland Dreier , "Michael R. Hines" , Sean Hefty , Hal Rosenstock , Yishai Hadas , Christoph Lameter , "linux-rdma@vger.kernel.org" , LKML , qemu-devel@nongnu.org Subject: [PATCHv2] ibverbs: add a new IBV_ACCESS_GIFT option Message-ID: <20130324154552.GA16346@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org At the moment registering an MR breaks COW. If the application does not care that adapter sees stale data (for example, it tracks writes reregisters and resends), it can use a new IBV_ACCESS_GIFT flag to prevent registration from breaking COW. The semantics are similar to that of SPLICE_F_GIFT thus the name. Signed-off-by: Michael S. Tsirkin --- This is compiled but untested. Michael, could you please try this patch (together with the kernel patch I'm sending separately) and report whether setting this flag unbreaks overcommit for you? include/infiniband/verbs.h | 3 ++- man/ibv_reg_mr.3 | 5 +++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h index 6acfc81..3290ec9 100644 --- a/include/infiniband/verbs.h +++ b/include/infiniband/verbs.h @@ -290,7 +290,8 @@ enum ibv_access_flags { IBV_ACCESS_REMOTE_WRITE = (1<<1), IBV_ACCESS_REMOTE_READ = (1<<2), IBV_ACCESS_REMOTE_ATOMIC = (1<<3), - IBV_ACCESS_MW_BIND = (1<<4) + IBV_ACCESS_MW_BIND = (1<<4), + IBV_ACCESS_GIFT = (1<<6) }; struct ibv_pd { diff --git a/man/ibv_reg_mr.3 b/man/ibv_reg_mr.3 index 7723771..3c302f0 100644 --- a/man/ibv_reg_mr.3 +++ b/man/ibv_reg_mr.3 @@ -34,6 +34,8 @@ describes the desired memory protection attributes; it is either 0 or the bitwis .B IBV_ACCESS_REMOTE_ATOMIC\fR Enable Remote Atomic Operation Access (if supported) .TP .B IBV_ACCESS_MW_BIND\fR Enable Memory Window Binding +.TP +.B IBV_ACCESS_GIFT\fR This memory is a gift to the adapter: if memory is modified after registration, the local version and data seen by the adapter through this region rkey may differ .PP If .B IBV_ACCESS_REMOTE_WRITE @@ -43,6 +45,9 @@ is set, then .B IBV_ACCESS_LOCAL_WRITE must be set too. .PP +.B IBV_ACCESS_GIFT +is only legal with remote or local read access. +.PP Local read access is always enabled for the MR. .PP .B ibv_dereg_mr() -- MST From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:35270) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UJn6s-0001MH-LL for qemu-devel@nongnu.org; Sun, 24 Mar 2013 11:45:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UJn6p-0004JD-Tc for qemu-devel@nongnu.org; Sun, 24 Mar 2013 11:45:26 -0400 Received: from mx1.redhat.com ([209.132.183.28]:60536) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UJn6p-0004IC-KU for qemu-devel@nongnu.org; Sun, 24 Mar 2013 11:45:23 -0400 Date: Sun, 24 Mar 2013 17:45:52 +0200 From: "Michael S. Tsirkin" Message-ID: <20130324154552.GA16346@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Subject: [Qemu-devel] [PATCHv2] ibverbs: add a new IBV_ACCESS_GIFT option List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jason Gunthorpe Cc: Roland Dreier , qemu-devel@nongnu.org, "linux-rdma@vger.kernel.org" , Yishai Hadas , LKML , "Michael R. Hines" , Hal Rosenstock , Sean Hefty , Christoph Lameter At the moment registering an MR breaks COW. If the application does not care that adapter sees stale data (for example, it tracks writes reregisters and resends), it can use a new IBV_ACCESS_GIFT flag to prevent registration from breaking COW. The semantics are similar to that of SPLICE_F_GIFT thus the name. Signed-off-by: Michael S. Tsirkin --- This is compiled but untested. Michael, could you please try this patch (together with the kernel patch I'm sending separately) and report whether setting this flag unbreaks overcommit for you? include/infiniband/verbs.h | 3 ++- man/ibv_reg_mr.3 | 5 +++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h index 6acfc81..3290ec9 100644 --- a/include/infiniband/verbs.h +++ b/include/infiniband/verbs.h @@ -290,7 +290,8 @@ enum ibv_access_flags { IBV_ACCESS_REMOTE_WRITE = (1<<1), IBV_ACCESS_REMOTE_READ = (1<<2), IBV_ACCESS_REMOTE_ATOMIC = (1<<3), - IBV_ACCESS_MW_BIND = (1<<4) + IBV_ACCESS_MW_BIND = (1<<4), + IBV_ACCESS_GIFT = (1<<6) }; struct ibv_pd { diff --git a/man/ibv_reg_mr.3 b/man/ibv_reg_mr.3 index 7723771..3c302f0 100644 --- a/man/ibv_reg_mr.3 +++ b/man/ibv_reg_mr.3 @@ -34,6 +34,8 @@ describes the desired memory protection attributes; it is either 0 or the bitwis .B IBV_ACCESS_REMOTE_ATOMIC\fR Enable Remote Atomic Operation Access (if supported) .TP .B IBV_ACCESS_MW_BIND\fR Enable Memory Window Binding +.TP +.B IBV_ACCESS_GIFT\fR This memory is a gift to the adapter: if memory is modified after registration, the local version and data seen by the adapter through this region rkey may differ .PP If .B IBV_ACCESS_REMOTE_WRITE @@ -43,6 +45,9 @@ is set, then .B IBV_ACCESS_LOCAL_WRITE must be set too. .PP +.B IBV_ACCESS_GIFT +is only legal with remote or local read access. +.PP Local read access is always enabled for the MR. .PP .B ibv_dereg_mr() -- MST