direct_access, pinning and truncation

[Matthew Wilcox <willy@linux.intel.com>]

One of the things on my todo list is making O_DIRECT work to a
memory-mapped direct_access file.  Right now, it simply doesn't work
because there's no struct page for the memory, so get_user_pages() fails.
Boaz has posted a patch to create struct pages for direct_access files,
which is certainly one way of solving the immediate problem, but it
ignores the deeper problem.

For normal files, get_user_pages() elevates the reference count on
the pages.  If those pages are subsequently truncated from the file,
the underlying file blocks are released to the filesystem's free pool.
The pages are removed from the page cache and the process's address space,
but hang around until the caller of get_user_pages() calls put_page() on
them again at which point they are released into the pool of free pages.

Once we have a struct page for (or some other way to handle pinning of)
persistent memory blocks, truncating a file that has pinned pages will
still cause the disk blocks to be released to the free pool.  But there
weren't any pages of DRAM between the filesystem and the application!
So those blocks are "freed" while still referenced.  And that reference
might well be programmed into a piece of hardware that's doing DMA;
it can't be stopped.

I see three solutions here:

1. If get_user_pages() is called, copy from PMEM into DRAM, and provide
the caller with the struct pages of the DRAM.  Modify DAX to handle some
file pages being in the page cache, and make sure that we know whether
the PMEM or DRAM is up to date.  This has the obvious downside that
get_user_pages() becomes slow.

2. Modify filesystems that support DAX to handle pinning blocks.
Some filesystems (that support COW and snapshots) already support
reference-counting individual blocks.  We may be ale to do better by
using a tree of pinned extents or something.  This makes it much harder
to modify a filesystem to support DAX, and I don't see patches adding
this capability to ext2 being warmly welcomed.

3. Make truncate() block if it hits a pinned page.  There's really no
good reason to truncate a file that has pinned pages; it's either a bug
or you're trying to be nasty to someone.  We actually already have code
for this; inode_dio_wait() / inode_dio_done().  But page pinning isn't
just for O_DIRECT I/Os and other transient users like crypto, it's also
for long-lived things like RDMA, where we could potentially block for
an indefinite time.

Does option 3 open up a new attack surface?  I'm thinking about somebody
opening a large file that's publically readable, and pinning part of
it by handing part of it to an RDMA card.  That would prevent the owner
from truncating it.

One thing that option 3 doesn't do is affect whether a file can be
removed.  Just having the file mmaped is enough to prevent the file blocks
from being reused, even if all names for that file have been removed.

I'm open to other solutions ...