Re: Testing devices for discard support properly

[Chris Mason <clm@fb.com>]

On 7 May 2019, at 16:09, Bryan Gurney wrote:

> I found an example in my trace of the "two bands of latency" behavior.
> Consider these three segments of trace data during the writes:
>

[ ... ]

> There's an average latency of 14 milliseconds for these 128 kilobyte
> writes.  At 0.218288794 seconds, we can see a sudden appearance of 1.7
> millisecond latency times, much lower than the average.
>
> Then we see an alternation of 1.7 millisecond completions and 14
> millisecond completions, with these two "latency groups" increasing,
> up to about 14 milliseconds and 25 milliseconds at 0.241287187 seconds
> into the trace.
>
> At 0.317351888 seconds, we see the pattern start again, with a sudden
> appearance of 1.89 millisecond latency write completions, among 14.7
> millisecond latency write completions.
>
> If you graph it, it looks like a "triangle wave" pulse, with a
> duration of about 23 milliseconds, that repeats after about 100
> milliseconds.  In a way, it's like a "heartbeat".  This wouldn't be as
> easy to detect with a simple "average" or "percentile" reading.
>
> This was during a simple sequential write at a queue depth of 32, but
> what happens with a write after a discard in the same region of
> sectors?  This behavior could change, depending on different drive
> models, and/or drive controller algorithms.
>

I think these are all really interesting, and definitely support the 
idea of a series of tests we do to make sure a drive implements discard 
in the general ways that we expect.

But with that said, I think a more important discussion as filesystem 
developers is how we protect the rest of the filesystem from high 
latencies caused by discards.  For reads and writes, we've been doing 
this for a long time.  IO schedulers have all kinds of checks and 
balances for REQ_META or REQ_SYNC, and we throttle dirty pages and 
readahead and dance around request batching etc etc.

But for discards, we just open the floodgates and hope it works out.  At 
some point we're going to have to figure out how to queue and throttle 
discards as well as we do reads/writes.  That's kind of tricky because 
the FS needs to coordinate when we're allowed to discard something and 
needs to know when the discard is done, and we all have different 
schemes for keeping track.

-chris