Re: [RFC 0/2] New MAP_PMEM_AWARE mmap flag

[Phil Terry <pterry@inphi.com>]

On 02/25/2016 12:15 PM, Dave Chinner wrote:
> On Thu, Feb 25, 2016 at 02:11:49PM -0500, Jeff Moyer wrote:
>> Jeff Moyer <jmoyer@redhat.com> writes:
>>
>>>> The big issue we have right now is that we haven't made the DAX/pmem
>>>> infrastructure work correctly and reliably for general use.  Hence
>>>> adding new APIs to workaround cases where we haven't yet provided
>>>> correct behaviour, let alone optimised for performance is, quite
>>>> frankly, a clear case premature optimisation.
>>> Again, I see the two things as separate issues.  You need both.
>>> Implementing MAP_SYNC doesn't mean we don't have to solve the bigger
>>> issue of making existing applications work safely.
>> I want to add one more thing to this discussion, just for the sake of
>> clarity.  When I talk about existing applications and pmem, I mean
>> applications that already know how to detect and recover from torn
>> sectors.  Any application that assumes hardware does not tear sectors
>> should be run on a file system layered on top of the btt.
> Which turns off DAX, and hence makes this a moot discussion because
> mmap is then buffered through the page cache and hence applications
> *must use msync/fsync* to provide data integrity. Which also makes
> them safe to use with DAX if we have a working fsync.
>
> Keep in mind that existing storage technologies tear fileystem data
> writes, too, because user data writes are filesystem block sized and
> not atomic at the device level (i.e.  typical is 512 byte sector, 4k
> filesystem block size, so there are 7 points in a single write where
> a tear can occur on a crash).
Is that really true? Storage to date is on the PCIE/SATA etc IO chain. 
The locks and application crash scenarios when traversing down this 
chain are such that the device will not have its DMA programmed until 
the whole 4K etc page is flushed to memory, pinned for DMA, etc. Then 
the DMA to the device is kicked off. If power crashes during the DMA, 
either we have devices which are supercapped or battery backed to flush 
their write caches and or have firmware which will abort the damaged 
results of the torn DMA on the devices internal meta-data recovery when 
power is restored. (The hardware/firmware on an HDD has been way more 
complex than the simple mind model might lead one to expect for years). 
All of this wrapped inside filesystem transaction semantics.

This is a crucial difference for "storage class memory" on the DRAM bus. 
The NVDIMMs cannot be DMA masters and instead passively receive 
cache-line writes. A "buffered DIMM" as alluded to in the pmem.io Device 
Writers Guide might have intelligence on the DIIMM to detect, map and 
recover tearing via the Block Window Aperture driver interface but on a 
PMEM interface cannot do so. Hence btt on the host with full 
transparency to manage the memory on the NVDIMM is required for the PMEM 
driver. Given this it doesn't make sense to try and put it on the device 
for the BW driver either.

In both cases, btt is not indirecting the buffer (as for a DMA master IO 
type device) but is simply using the same pmem api primitives to manage 
its own meta data about the filesystem writes to detect and recover from 
tears after the event. In what sense is DAX disabled for this?

So I think (please correct me if I'm wrong) but actually the 
hardware/firmware guys have been fixing the torn sector problem for the 
last 30 years and the "storage on the memory channel" has reintroduced 
the problem. So to use as SSD analogy, you fix this problem with the 
FTL, and as we've seen with recent software defined flash and 
openchannel approaches, you can either have the FTL on the device or on 
the host. Absence of the bus master DMA on a DIMM (even with the BW 
aperture software) makes a device based solution problematic so the host 
solution a la btt is required for both PMEM and BW.

>
> IOWs existing storage already has the capability of tearing user
> data on crash and has been doing so for a least they last 30 years.
> Hence I really don't see any fundamental difference here with
> pmem+DAX - the only difference is that the tear granuarlity is
> smaller (CPU cacheline rather than sector).
>
> Cheers,
>
> Dave.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>