Re: direct_access, pinning and truncation

[Matthew Wilcox <willy@linux.intel.com>]

On Thu, Oct 09, 2014 at 12:10:38PM +1100, Dave Chinner wrote:
> On Wed, Oct 08, 2014 at 03:05:23PM -0400, Matthew Wilcox wrote:
> > 
> > One of the things on my todo list is making O_DIRECT work to a
> > memory-mapped direct_access file.
> 
> I don't understand the motivation or the use case: O_DIRECT is
> purely for bypassing the page cache, and DAX already bypasses the
> page cache.  What difference is there between the DAX read/write
> path and a DAX-based O_DIRECT IO path, and why doesn't just ignoring
> O_DIRECT for DAX enabled filesystems simply do what you need?

There are two filesystems involved ... if both (or neither!) are DAX,
everything's fine.  The problem comes when you do things this way around:

int cachefd = open("/dax/cache", O_RDWR);
int datafd = open("/nfs/bigdata", O_RDWR | O_DIRECT);
void *cache = mmap(NULL, 1024 * 1024 * 1024, PROT_READ | PROT_WRITE,
		MAP_SHARED, cachefd, 0);
read(datafd, cache, 1024 * 1024);

The non-DAX filesystem needs to pin pages from the DAX filesystem while
they're under I/O.


Another attempt to solve this problem might be to turn the O_DIRECT
read into a read into a page of DRAM, followed by a copy from DRAM
to PMEM.  Conversely, writes could be done as a copy to DRAM followed
by a page-based write.


You also elided the paragraphs where I point out that this is an example
of a more general problem; there really are people who want to do RDMA
to DAX memory (the HPC crowd, of course), and we need to not open up
security holes when enabling that.  Since it's a potentially long-duration
and bi-directional mapping, the copy solution isn't going to work here
(without going all the way to solution 1).