Re: raid0 vs io_uring

[Jens Axboe <axboe@kernel.dk>]

On 11/15/21 1:05 AM, Avi Kivity wrote:
> On 11/14/21 20:23, Jens Axboe wrote:
>> On 11/14/21 10:07 AM, Avi Kivity wrote:
>>> Running a trivial randread, direct=1 fio workload against a RAID-0
>>> composed of some nvme devices, I see this pattern:
>>>
>>>
>>>                fio-7066  [009]  1800.209865: function: io_submit_sqes
>>>                fio-7066  [009]  1800.209866: function:
>>> rcu_read_unlock_strict
>>>                fio-7066  [009]  1800.209866: function:
>>> io_submit_sqe
>>>                fio-7066  [009]  1800.209866: function:
>>> io_init_req
>>>                fio-7066  [009]  1800.209866:
>>> function:                      io_file_get
>>>                fio-7066  [009]  1800.209866:
>>> function:                         fget_many
>>>                fio-7066  [009]  1800.209866:
>>> function:                            __fget_files
>>>                fio-7066  [009]  1800.209867:
>>> function:                               rcu_read_unlock_strict
>>>                fio-7066  [009]  1800.209867: function:
>>> io_req_prep
>>>                fio-7066  [009]  1800.209867:
>>> function:                      io_prep_rw
>>>                fio-7066  [009]  1800.209867: function:
>>> io_queue_sqe
>>>                fio-7066  [009]  1800.209867:
>>> function:                      io_req_defer
>>>                fio-7066  [009]  1800.209867:
>>> function:                      __io_queue_sqe
>>>                fio-7066  [009]  1800.209868:
>>> function:                         io_issue_sqe
>>>                fio-7066  [009]  1800.209868:
>>> function:                            io_read
>>>                fio-7066  [009]  1800.209868:
>>> function:                               io_import_iovec
>>>                fio-7066  [009]  1800.209868:
>>> function:                               __io_file_supports_async
>>>                fio-7066  [009]  1800.209868:
>>> function:                                  I_BDEV
>>>                fio-7066  [009]  1800.209868:
>>> function:                               __kmalloc
>>>                fio-7066  [009]  1800.209868:
>>> function:                                  kmalloc_slab
>>>                fio-7066  [009]  1800.209868: function: __cond_resched
>>>                fio-7066  [009]  1800.209868: function:
>>> rcu_all_qs
>>>                fio-7066  [009]  1800.209869: function: should_failslab
>>>                fio-7066  [009]  1800.209869:
>>> function:                               io_req_map_rw
>>>                fio-7066  [009]  1800.209869:
>>> function:                         io_arm_poll_handler
>>>                fio-7066  [009]  1800.209869:
>>> function:                         io_queue_async_work
>>>                fio-7066  [009]  1800.209869:
>>> function:                            io_prep_async_link
>>>                fio-7066  [009]  1800.209869:
>>> function:                               io_prep_async_work
>>>                fio-7066  [009]  1800.209870:
>>> function:                            io_wq_enqueue
>>>                fio-7066  [009]  1800.209870:
>>> function:                               io_wqe_enqueue
>>>                fio-7066  [009]  1800.209870:
>>> function:                                  _raw_spin_lock_irqsave
>>>                fio-7066  [009]  1800.209870: function:
>>> _raw_spin_unlock_irqrestore
>>>
>>>
>>>
>>>   From which I deduce that __io_file_supports_async() (today named
>>> __io_file_supports_nowait) returns false, and therefore every io_uring
>>> operation is bounced to a workqueue with the resulting great loss in
>>> performance.
>>>
>>>
>>> However, I also see NOWAIT is part of the default set of flags:
>>>
>>>
>>> #define QUEUE_FLAG_MQ_DEFAULT   ((1 << QUEUE_FLAG_IO_STAT) |            \
>>>                                    (1 << QUEUE_FLAG_SAME_COMP) |          \
>>>                                    (1 << QUEUE_FLAG_NOWAIT))
>>>
>>> and I don't see that md touches it (I do see that dm plays with it).
>>>
>>>
>>> So, what's the story? does md not support NOWAIT? If so, that's a huge
>>> blow to io_uring with md. If it does, are there any clues about why I
>>> see requests bouncing to a workqueue?
>> That is indeed the story, dm supports it but md doesn't just yet.
> 
> 
> Ah, so I missed md clearing the default flags somewhere.
> 
> 
> This is a false negative from io_uring's point of view, yes? An md on 
> nvme would be essentially nowait in normal operation, it just doesn't 
> know it. aio on the same device would not block on the same workload.

There are still conditions where it can block, it just didn't in your
test case.

-- 
Jens Axboe