Re: [PATCH 14/15] mm, dax, pmem: introduce {get|put}_dev_pagemap() for dax-gup

[Logan Gunthorpe <logang@deltatee.com>]

Hi Dan,

Good to know you've already addressed the struct page issue. We'll watch 
out for an updated patchset to try.


On 02/10/15 03:53 PM, Dan Williams wrote:
> Hmm, I didn't have peer-to-peer PCI-E in mind for this mechanism, but
> the test report is welcome nonetheless.  The definition of dma_addr_t
> is the device view of host memory, not necessarily the device view of
> a peer device's memory range, so I expect you'll run into issues with
> IOMMUs and other parts of the kernel that assume this definition.

Yeah, we've actually been doing this with a number of more "hacky" 
techniques for some time. ZONE_DEVICE just provides us with a much 
cleaner way to set this up that doesn't require patching around 
get_user_pages in various places in the kernel.

We've never had any issues with the IOMMU getting in the way (at least 
on Intel x86). My understanding always was that the IOMMU sits between a 
PCI card and main memory; it doesn't get in the way of peer-to-peer 
transfers. Though admittedly, I don't have a complete understanding of 
how the IOMMU works in the kernel. I'm just speaking from experimental 
experience. We've never actually tried this on other architectures.

Thanks,

Logan