Re: [PATCH 4/4] io_uring: mark opcodes that always need io-wq punt

[Jens Axboe <axboe@kernel.dk>]

On 4/24/23 8:50?PM, Ming Lei wrote:
> On Mon, Apr 24, 2023 at 08:18:02PM -0600, Jens Axboe wrote:
>> On 4/24/23 8:13?PM, Ming Lei wrote:
>>> On Mon, Apr 24, 2023 at 08:08:09PM -0600, Jens Axboe wrote:
>>>> On 4/24/23 6:57?PM, Ming Lei wrote:
>>>>> On Mon, Apr 24, 2023 at 09:24:33AM -0600, Jens Axboe wrote:
>>>>>> On 4/24/23 1:30?AM, Ming Lei wrote:
>>>>>>> On Thu, Apr 20, 2023 at 12:31:35PM -0600, Jens Axboe wrote:
>>>>>>>> Add an opdef bit for them, and set it for the opcodes where we always
>>>>>>>> need io-wq punt. With that done, exclude them from the file_can_poll()
>>>>>>>> check in terms of whether or not we need to punt them if any of the
>>>>>>>> NO_OFFLOAD flags are set.
>>>>>>>>
>>>>>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>>>>>>> ---
>>>>>>>>  io_uring/io_uring.c |  2 +-
>>>>>>>>  io_uring/opdef.c    | 22 ++++++++++++++++++++--
>>>>>>>>  io_uring/opdef.h    |  2 ++
>>>>>>>>  3 files changed, 23 insertions(+), 3 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
>>>>>>>> index fee3e461e149..420cfd35ebc6 100644
>>>>>>>> --- a/io_uring/io_uring.c
>>>>>>>> +++ b/io_uring/io_uring.c
>>>>>>>> @@ -1948,7 +1948,7 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
>>>>>>>>  		return -EBADF;
>>>>>>>>  
>>>>>>>>  	if (issue_flags & IO_URING_F_NO_OFFLOAD &&
>>>>>>>> -	    (!req->file || !file_can_poll(req->file)))
>>>>>>>> +	    (!req->file || !file_can_poll(req->file) || def->always_iowq))
>>>>>>>>  		issue_flags &= ~IO_URING_F_NONBLOCK;
>>>>>>>
>>>>>>> I guess the check should be !def->always_iowq?
>>>>>>
>>>>>> How so? Nobody that takes pollable files should/is setting
>>>>>> ->always_iowq. If we can poll the file, we should not force inline
>>>>>> submission. Basically the ones setting ->always_iowq always do -EAGAIN
>>>>>> returns if nonblock == true.
>>>>>
>>>>> I meant IO_URING_F_NONBLOCK is cleared here for  ->always_iowq, and
>>>>> these OPs won't return -EAGAIN, then run in the current task context
>>>>> directly.
>>>>
>>>> Right, of IO_URING_F_NO_OFFLOAD is set, which is entirely the point of
>>>> it :-)
>>>
>>> But ->always_iowq isn't actually _always_ since fallocate/fsync/... are
>>> not punted to iowq in case of IO_URING_F_NO_OFFLOAD, looks the naming of
>>> ->always_iowq is a bit confusing?
>>
>> Yeah naming isn't that great, I can see how that's bit confusing. I'll
>> be happy to take suggestions on what would make it clearer.
> 
> Except for the naming, I am also wondering why these ->always_iowq OPs
> aren't punted to iowq in case of IO_URING_F_NO_OFFLOAD, given it
> shouldn't improve performance by doing so because these OPs are supposed
> to be slow and always slept, not like others(buffered writes, ...),
> can you provide one hint about not offloading these OPs? Or is it just that
> NO_OFFLOAD needs to not offload every OPs?

The whole point of NO_OFFLOAD is that items that would normally be
passed to io-wq are just run inline. This provides a way to reap the
benefits of batched submissions and syscall reductions. Some opcodes
will just never be async, and io-wq offloads are not very fast. Some of
them can eventually be migrated to async support, if the underlying
mechanics support it.

You'll note that none of the ->always_iowq opcodes are pollable. If
NO_OFFLOAD is setup, it's pointless NOT to issue them with NONBLOCK
cleared, as you'd just get -EAGAIN and then need to call them again with
NONBLOCK cleared from the same context.

For naming, maybe ->always_iowq is better as ->no_nonblock or something
like that. Or perhaps get rid of the double negation and just call it
->blocking, or maybe ->no_async_support to make it clearer?

> Or can we rename IORING_SETUP_NO_OFFLOAD as IORING_SETUP_SUBMIT_MAY_WAIT
> and still punt ->always_iowq OPs to iowq?

I think NO_OFFLOAD better explains that we'll never offload to io-wq. I
would've called it NO_IOWQ, but I don't think that's understandable to
users in the same way. The problem is that the user does need some
knowledge of how ios are issued and completed in io_uring to fully grok
what it does, which I'll put in the man pages.

-- 
Jens Axboe