Re: Testing devices for discard support properly

[Ric Wheeler <ricwheeler@gmail.com>]

On 5/8/19 11:55 PM, Martin K. Petersen wrote:
> 
> Dave,
> 
>> Only when told to do PUNCH_HOLE|NO_HIDE_STALE which means "we don't
>> care what the device does" as this fallcoate command provides no
>> guarantees for the data returned by subsequent reads. It is,
>> esssentially, a get out of gaol free mechanism for indeterminate
>> device capabilities.
> 
> Correct. But the point of discard is to be a lightweight mechanism to
> convey to the device that a block range is no longer in use. Nothing
> more, nothing less.
> 
> Not everybody wants the device to spend resources handling unwritten
> extents. I understand the importance of that use case for XFS but other
> users really just need deallocate semantics.
> 
>> People used to make that assertion about filesystems, too. It took
>> linux filesystem developers years to realise that unwritten extents
>> are actually very simple and require very little extra code and no
>> extra space in metadata to implement. If you are already tracking
>> allocated blocks/space, then you're 99% of the way to efficient
>> management of logically zeroed disk space.
> 
> I don't disagree. But since "discard performance" checkmark appears to
> be absent from every product requirements document known to man, very
> little energy has been devoted to ensuring that discard operations can
> coexist with read/write I/O without impeding the performance.
> 
> I'm not saying it's impossible. Just that so far it hasn't been a
> priority. Even large volume customers have been unable to compel their
> suppliers to produce a device that doesn't suffer one way or the other.
> 
> On the SSD device side, vendors typically try to strike a suitable
> balance between what's handled by the FTL and what's handled by
> over-provisioning.
> 
>>> 2. Our expectation for the allocating REQ_ZEROOUT (FL_ZERO_RANGE), which
>>>     gets translated into NVMe WRITE ZEROES, SCSI WRITE SAME, is that the
>>>     command executes in O(n) but that it is faster -- or at least not
>>>     worse -- than doing a regular WRITE to the same block range.
>>
>> You're missing the important requirement of fallocate(ZERO_RANGE):
>> that the space is also allocated and ENOSPC will never be returned
>> for subsequent writes to that range. i.e. it is allocated but
>> "unwritten" space that contains zeros.
> 
> That's what I implied when comparing it to a WRITE.
> 
>>> 3. Our expectation for the deallocating REQ_ZEROOUT (FL_PUNCH_HOLE),
>>>     which gets translated into ATA DSM TRIM w/ whitelist, NVMe WRITE
>>>     ZEROES w/ DEAC, SCSI WRITE SAME w/ UNMAP, is that the command will
>>>     execute in O(1) for any portion of the block range described by the
>>
>> FL_PUNCH_HOLE has no O(1) requirement - it has a "all possible space
>> must be freed" requirement. The larger the range, to longer it will
>> take.
> 
> OK, so maybe my O() notation lacked a media access moniker. What I meant
> to convey was that no media writes take place for the properly aligned
> multiple of the internal granularity. The FTL update takes however long
> it takes, but the only potential media accesses would be the head and
> tail pieces. For some types of devices, these might be handled in
> translation tables. But for others, zeroing blocks on the media is the
> only way to do it.
> 
>> That's expected, and exaclty what filesystems do for sub-block punch
>> and zeroing ranges.
> 
> Yep.
> 
>> What I'm saying is that we should be pushing standards to ensure (3)
>> is correctly standardise, certified and implemented because that is
>> what the "Linux OS" requires from future hardware.
> 
> That's well-defined for both NVMe and SCSI.
> 
> However, I do not agree that a deallocate operation has to imply
> zeroing. I think there are valid use cases for pure deallocate.
> 
> In an ideal world the performance difference between (1) and (3) would
> be negligible and make this distinction moot. However, we have to
> support devices that have a wide variety of media and hardware
> characteristics. So I don't see pure deallocate going away. Doesn't mean
> that I am not pushing vendors to handle (3) because I think it is very
> important. And why we defined WRITE ZEROES in the first place.
> 

All of this makes sense to me.

I think that we can get value out of measuring how close various devices 
come to realizing the above assumptions.  Clearly, file systems (as 
Chris mentioned) do have to adapt to varying device performance issues, 
but I think today the variation can be orders of magnitude for large 
(whole device) discards and that it not something that is easy to 
tolerate....

ric