Re: [Report] requests are submitted to hardware in reverse order from nvme/virtio-blk queue_rqs()

[Bart Van Assche <bvanassche@acm.org>]

On 1/26/24 6:10 AM, Christoph Hellwig wrote:
> On Wed, Jan 24, 2024 at 07:59:54PM +0800, Ming Lei wrote:
>> Hello,
>>
>> Requests are added to plug list in reverse order, and both virtio-blk
>> and nvme retrieves request from plug list in order, so finally requests
>> are submitted to hardware in reverse order via nvme_queue_rqs() or
>> virtio_queue_rqs, see:
> 
>> May this reorder be one problem for virtio-blk and nvme-pci?
> 
> It it isn't really a problem for the drivers, but de-serializing
> I/O patterns tends to be not good.  I know at least a couple cases
> where this would hurt:
> 
>   - SSDs with sequential write detection
>   - zoned SSDs with zoned append, as this now turns a sequential
>     user write pattern into one that is fairly random
>   - HDDs much prefer real sequential I/O, although until nvme HDDs
>     go beyong the prototype stage that's probably not hitting this
>     case yet
> 
> So yes, we should fix this.

(replying to an email from January)

For my patch series that supports pipelining for zoned writes, I need
the submission order to be preserved. Jens mentioned two possible
solutions:
- Either keep the approach that requests on plug->mq_list are in reverse
   order and reverse the request order just before submitting requests.
- Or change plug->mq_list into a doubly linked list.

The second approach seems the most interesting to me. I'm concerned that
with the first approach it will be difficult to preserve the request
order if a subset of the requests on plug->mq_list are submitted, e.g.
because a queue full condition is encountered by 
blk_mq_dispatch_plug_list().

Thanks,

Bart.