From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from quartz.orcorp.ca ([184.70.90.242]:48572 "EHLO quartz.orcorp.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932958AbbGUWyj (ORCPT ); Tue, 21 Jul 2015 18:54:39 -0400 Date: Tue, 21 Jul 2015 16:54:33 -0600 From: Jason Gunthorpe To: Steve Wise Cc: "'Tom Talpey'" , "'Chuck Lever'" , linux-rdma@vger.kernel.org, linux-nfs@vger.kernel.org Subject: Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site Message-ID: <20150721225433.GA25724@obsidianresearch.com> References: <20150720185624.10997.51574.stgit@manet.1015granger.net> <20150720190311.10997.12636.stgit@manet.1015granger.net> <55AD5B48.3010906@talpey.com> <20150720210544.GA9655@obsidianresearch.com> <015101d0c331$69e31d10$3da95730$@opengridcomputing.com> <55AD7065.8040809@talpey.com> <015701d0c33d$36b53110$a41f9330$@opengridcomputing.com> <20150720224244.GB12278@obsidianresearch.com> <55AECA92.3040105@opengridcomputing.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <55AECA92.3040105@opengridcomputing.com> Sender: linux-nfs-owner@vger.kernel.org List-ID: On Tue, Jul 21, 2015 at 05:41:22PM -0500, Steve Wise wrote: > On 7/20/2015 5:42 PM, Jason Gunthorpe wrote: > >On Mon, Jul 20, 2015 at 05:41:27PM -0500, Steve Wise wrote: > >>>B) why bother to check? Are machines with <4GB interesting, and worth > >>>supporting a special optimization? > >>No, but cxgb3 is still interesting to user applications, and perhaps NFSRDMA using FRMRs. > >Doesn't look like the NFS client will work. It requires an all > >physical memory lkey for SEND and RECV buffers.. > > > >Jason > > Looks like cxgb3 supports LOCAL_DMA_LKEY and MEM_MGT_EXTENSIONS so dma mrs > aren't required for NFSRDMA: > > t4:~/linux-2.6/drivers/infiniband/hw/cxgb3 # grep IB_DEVICE_ iwch_provider.c > strlcpy(dev->ibdev.name, "cxgb3_%d", IB_DEVICE_NAME_MAX); > dev->device_cap_flags = IB_DEVICE_LOCAL_DMA_LKEY | > IB_DEVICE_MEM_WINDOW | > IB_DEVICE_MEM_MGT_EXTENSIONS; Neat. Is the dma_mask set properly (I don't see any set at all)? > So cxgb3 can still support NFSRDMA and user verbs w/o get_dma_mr(). I'll > submit a patch soon to only support get_dma_mr() if unsigned long is 4 > bytes... So, NFS and RDS seem to be the only iWarp compatible ULPs? NFS has worked, and will continue to work with the global lkey. RDS looks like it relies on an insecure all physical rkey, so it won't work until that is fixed. So, I'd just use sizeof(physaddr_t) > 4 as the test. The only people that could be impacted are RDS users using distro kernels on machines with less than 4G of ram. I somehow doubt there are any of those... Jason From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Gunthorpe Subject: Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site Date: Tue, 21 Jul 2015 16:54:33 -0600 Message-ID: <20150721225433.GA25724@obsidianresearch.com> References: <20150720185624.10997.51574.stgit@manet.1015granger.net> <20150720190311.10997.12636.stgit@manet.1015granger.net> <55AD5B48.3010906@talpey.com> <20150720210544.GA9655@obsidianresearch.com> <015101d0c331$69e31d10$3da95730$@opengridcomputing.com> <55AD7065.8040809@talpey.com> <015701d0c33d$36b53110$a41f9330$@opengridcomputing.com> <20150720224244.GB12278@obsidianresearch.com> <55AECA92.3040105@opengridcomputing.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <55AECA92.3040105-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Steve Wise Cc: 'Tom Talpey' , 'Chuck Lever' , linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: linux-rdma@vger.kernel.org On Tue, Jul 21, 2015 at 05:41:22PM -0500, Steve Wise wrote: > On 7/20/2015 5:42 PM, Jason Gunthorpe wrote: > >On Mon, Jul 20, 2015 at 05:41:27PM -0500, Steve Wise wrote: > >>>B) why bother to check? Are machines with <4GB interesting, and worth > >>>supporting a special optimization? > >>No, but cxgb3 is still interesting to user applications, and perhaps NFSRDMA using FRMRs. > >Doesn't look like the NFS client will work. It requires an all > >physical memory lkey for SEND and RECV buffers.. > > > >Jason > > Looks like cxgb3 supports LOCAL_DMA_LKEY and MEM_MGT_EXTENSIONS so dma mrs > aren't required for NFSRDMA: > > t4:~/linux-2.6/drivers/infiniband/hw/cxgb3 # grep IB_DEVICE_ iwch_provider.c > strlcpy(dev->ibdev.name, "cxgb3_%d", IB_DEVICE_NAME_MAX); > dev->device_cap_flags = IB_DEVICE_LOCAL_DMA_LKEY | > IB_DEVICE_MEM_WINDOW | > IB_DEVICE_MEM_MGT_EXTENSIONS; Neat. Is the dma_mask set properly (I don't see any set at all)? > So cxgb3 can still support NFSRDMA and user verbs w/o get_dma_mr(). I'll > submit a patch soon to only support get_dma_mr() if unsigned long is 4 > bytes... So, NFS and RDS seem to be the only iWarp compatible ULPs? NFS has worked, and will continue to work with the global lkey. RDS looks like it relies on an insecure all physical rkey, so it won't work until that is fixed. So, I'd just use sizeof(physaddr_t) > 4 as the test. The only people that could be impacted are RDS users using distro kernels on machines with less than 4G of ram. I somehow doubt there are any of those... Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html