Re: [PATCH v9 0/6] MAP_DIRECT for DAX userspace flush

[Christoph Hellwig <hch@lst.de>]

On Fri, Oct 13, 2017 at 11:22:21AM -0700, Dan Williams wrote:
> So, here's a strawman can ibv_poll_cq() start returning ibv_wc_status
> == IBV_WC_LOC_PROT_ERR when file coherency is lost. This would make
> the solution generic across DAX and non-DAX. What's you're feeling for
> how well applications are prepared to deal with that status return?

The problem aren't local protection errors, but remote protection errors
when we modify a MR with an rkey that the remote side accesses.

> >  - How lease break can be done hitlessly, so the library user never
> >    needs to know it is happening or see failed/missed transfers
> 
> iommu redirect should be hit less and behave like the page cache case
> where RDMA targets pages that are no longer part of the file.

But systems that care about performance (e.g. the usual RDMA users) usually
don't use an IOMMU due to the performance impact.  Especially as HCAs
already have their own built-in iommus (aka the MR mechanism).

Note that file systems already have a mechanism like you mention above
to keep extents that are busy from being reallocated.  E.g. take a look at
fs/xfs/xfs_extent_busy.c.  The downside is that this could lock down
a massive amount of space in the busy list if we for example have a MR
covering a huge file that is truncated down.  So even if we'd want that
scheme we'd need some sort of ulmit for the amount of DAX pages locked
down in get_user_pages.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>