Re: [RFC 2/2] io_uring: mark REQ_NOWAIT for a non-mq queue as unspported

[Jens Axboe <axboe@kernel.dk>]

On 5/28/20 12:35 PM, Jeff Moyer wrote:
> Bijan Mottahedeh <bijan.mottahedeh@oracle.com> writes:
> 
>> Mark a REQ_NOWAIT request for a non-mq queue as unspported instead of
>> retryable since otherwise the io_uring layer will keep resubmitting
>> the request.
> 
> Getting back to this...
> 
> Jens, right now (using your io_uring-5.7 or linus' tree) fio's
> t/io_uring will never get io completions when run against a file on a
> file system that is backed by lvm.  The system will have one workqueue
> per sqe submitted, all spinning, eating up CPU time.
> 
> # ./t/io_uring /mnt/test/poo 
> Added file /mnt/test/poo
> sq_ring ptr = 0x0x7fbed40ae000
> sqes ptr    = 0x0x7fbed40ac000
> cq_ring ptr = 0x0x7fbed40aa000
> polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256
> submitter=3851
> IOPS=128, IOS/call=6/0, inflight=128 (128)
> IOPS=0, IOS/call=0/0, inflight=128 (128)
> IOPS=0, IOS/call=0/0, inflight=128 (128)
> IOPS=0, IOS/call=0/0, inflight=128 (128)
> IOPS=0, IOS/call=0/0, inflight=128 (128)
> IOPS=0, IOS/call=0/0, inflight=128 (128)
> ...
> 
> # ps auxw | grep io_wqe
> root      3849 80.1  0.0      0     0 ?        R    14:32   0:40 [io_wqe_worker-0]
> root      3850  0.0  0.0      0     0 ?        S    14:32   0:00 [io_wqe_worker-0]
> root      3853 72.8  0.0      0     0 ?        R    14:32   0:36 [io_wqe_worker-0]
> root      3854 81.4  0.0      0     0 ?        R    14:32   0:40 [io_wqe_worker-1]
> root      3855 74.8  0.0      0     0 ?        R    14:32   0:37 [io_wqe_worker-0]
> root      3856 74.8  0.0      0     0 ?        R    14:32   0:37 [io_wqe_worker-1]
> ...
> 
> # ps auxw | grep io_wqe | grep -v grep | wc -l
> 129
> 
> With this patch applied, the test program will exit without doing I/O
> (which I don't think is the right behavior either, right?):
> 
> # t/io_uring /mnt/test/poo
> Added file /mnt/test/poo
> sq_ring ptr = 0x0x7fdb98f00000
> sqes ptr    = 0x0x7fdb98efe000
> cq_ring ptr = 0x0x7fdb98efc000
> polled=1, fixedbufs=1, buffered=0 QD=128, sq_ring=128, cq_ring=256
> submitter=33233
> io: unexpected ret=-95
> Your filesystem/driver/kernel doesn't support polled IO
> IOPS=128, IOS/call=32/0, inflight=128 (127)
> 
> /mnt/test is an xfs file system on top of a linear LVM volume on an nvme
> device (with 8 poll queues configured).

poll won't work over dm, so that looks correct. What happens if you edit
it and disable poll? Would be curious to see both buffered = 0 and
buffered = 1 runs with that.

I'll try this here too.

-- 
Jens Axboe