From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Message-ID: <560F01CD.7060504@deltatee.com> Date: Fri, 02 Oct 2015 16:14:37 -0600 From: Logan Gunthorpe MIME-Version: 1.0 References: <20150923043737.36490.70547.stgit@dwillia2-desk3.jf.intel.com> <20150923044227.36490.99741.stgit@dwillia2-desk3.jf.intel.com> <20151002212137.GB30448@deltatee.com> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [PATCH 14/15] mm, dax, pmem: introduce {get|put}_dev_pagemap() for dax-gup Sender: owner-linux-mm@kvack.org To: Dan Williams Cc: Andrew Morton , Dave Hansen , "linux-nvdimm@lists.01.org" , "linux-kernel@vger.kernel.org" , Linux MM , Alexander Viro , linux-fsdevel , Matthew Wilcox , Ross Zwisler , Stephen Bates List-ID: Hi Dan, Good to know you've already addressed the struct page issue. We'll watch out for an updated patchset to try. On 02/10/15 03:53 PM, Dan Williams wrote: > Hmm, I didn't have peer-to-peer PCI-E in mind for this mechanism, but > the test report is welcome nonetheless. The definition of dma_addr_t > is the device view of host memory, not necessarily the device view of > a peer device's memory range, so I expect you'll run into issues with > IOMMUs and other parts of the kernel that assume this definition. Yeah, we've actually been doing this with a number of more "hacky" techniques for some time. ZONE_DEVICE just provides us with a much cleaner way to set this up that doesn't require patching around get_user_pages in various places in the kernel. We've never had any issues with the IOMMU getting in the way (at least on Intel x86). My understanding always was that the IOMMU sits between a PCI card and main memory; it doesn't get in the way of peer-to-peer transfers. Though admittedly, I don't have a complete understanding of how the IOMMU works in the kernel. I'm just speaking from experimental experience. We've never actually tried this on other architectures. Thanks, Logan -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Logan Gunthorpe Subject: Re: [PATCH 14/15] mm, dax, pmem: introduce {get|put}_dev_pagemap() for dax-gup Date: Fri, 02 Oct 2015 16:14:37 -0600 Message-ID: <560F01CD.7060504@deltatee.com> References: <20150923043737.36490.70547.stgit@dwillia2-desk3.jf.intel.com> <20150923044227.36490.99741.stgit@dwillia2-desk3.jf.intel.com> <20151002212137.GB30448@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Andrew Morton , Dave Hansen , "linux-nvdimm@lists.01.org" , "linux-kernel@vger.kernel.org" , Linux MM , Alexander Viro , linux-fsdevel , Matthew Wilcox , Ross Zwisler , Stephen Bates To: Dan Williams Return-path: In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org Hi Dan, Good to know you've already addressed the struct page issue. We'll watch out for an updated patchset to try. On 02/10/15 03:53 PM, Dan Williams wrote: > Hmm, I didn't have peer-to-peer PCI-E in mind for this mechanism, but > the test report is welcome nonetheless. The definition of dma_addr_t > is the device view of host memory, not necessarily the device view of > a peer device's memory range, so I expect you'll run into issues with > IOMMUs and other parts of the kernel that assume this definition. Yeah, we've actually been doing this with a number of more "hacky" techniques for some time. ZONE_DEVICE just provides us with a much cleaner way to set this up that doesn't require patching around get_user_pages in various places in the kernel. We've never had any issues with the IOMMU getting in the way (at least on Intel x86). My understanding always was that the IOMMU sits between a PCI card and main memory; it doesn't get in the way of peer-to-peer transfers. Though admittedly, I don't have a complete understanding of how the IOMMU works in the kernel. I'm just speaking from experimental experience. We've never actually tried this on other architectures. Thanks, Logan From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752037AbbJBWOn (ORCPT ); Fri, 2 Oct 2015 18:14:43 -0400 Received: from ale.deltatee.com ([207.54.116.67]:46129 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751353AbbJBWOm (ORCPT ); Fri, 2 Oct 2015 18:14:42 -0400 Message-ID: <560F01CD.7060504@deltatee.com> Date: Fri, 02 Oct 2015 16:14:37 -0600 From: Logan Gunthorpe User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Icedove/31.8.0 MIME-Version: 1.0 To: Dan Williams CC: Andrew Morton , Dave Hansen , "linux-nvdimm@lists.01.org" , "linux-kernel@vger.kernel.org" , Linux MM , Alexander Viro , linux-fsdevel , Matthew Wilcox , Ross Zwisler , Stephen Bates References: <20150923043737.36490.70547.stgit@dwillia2-desk3.jf.intel.com> <20150923044227.36490.99741.stgit@dwillia2-desk3.jf.intel.com> <20151002212137.GB30448@deltatee.com> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.111 X-SA-Exim-Rcpt-To: Stephen.Bates@pmcs.com, ross.zwisler@linux.intel.com, willy@linux.intel.com, linux-fsdevel@vger.kernel.org, viro@zeniv.linux.org.uk, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, dave@sr71.net, akpm@linux-foundation.org, dan.j.williams@intel.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH 14/15] mm, dax, pmem: introduce {get|put}_dev_pagemap() for dax-gup X-SA-Exim-Version: 4.2.1 (built Mon, 26 Dec 2011 16:24:06 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Dan, Good to know you've already addressed the struct page issue. We'll watch out for an updated patchset to try. On 02/10/15 03:53 PM, Dan Williams wrote: > Hmm, I didn't have peer-to-peer PCI-E in mind for this mechanism, but > the test report is welcome nonetheless. The definition of dma_addr_t > is the device view of host memory, not necessarily the device view of > a peer device's memory range, so I expect you'll run into issues with > IOMMUs and other parts of the kernel that assume this definition. Yeah, we've actually been doing this with a number of more "hacky" techniques for some time. ZONE_DEVICE just provides us with a much cleaner way to set this up that doesn't require patching around get_user_pages in various places in the kernel. We've never had any issues with the IOMMU getting in the way (at least on Intel x86). My understanding always was that the IOMMU sits between a PCI card and main memory; it doesn't get in the way of peer-to-peer transfers. Though admittedly, I don't have a complete understanding of how the IOMMU works in the kernel. I'm just speaking from experimental experience. We've never actually tried this on other architectures. Thanks, Logan