Re: [PATCH 5/9] io_uring: support SQE group

Pavel Begunkov
2024-05-02T14:27:47Z

On 4/30/24 16:00, Ming Lei wrote:
> On Tue, Apr 30, 2024 at 01:27:10PM +0100, Pavel Begunkov wrote:
...
>>>> And what does it achieve? The infra has matured since early days,
>>>> it saves user-kernel transitions at best but not context switching
>>>> overhead, and not even that if you do wait(1) and happen to catch
>>>> middle CQEs. And it disables LAZY_WAKE, so CQ side batching with
>>>> timers and what not is effectively useless with links.
>>>
>>> Not only the context switch, it supports 1:N or N:M dependency which
>>
>> I completely missed, how N:M is supported? That starting to sound
>> terrifying.
>
> N:M is actually from Kevin's idea.
>
> sqe group can be made to be more flexible by:
>
> Inside the group, all SQEs are submitted in parallel, so there isn't any
> dependency among SQEs in one group.
>
> The 1st SQE is group leader, and the other SQEs are group member. The whole
> group share single IOSQE_IO_LINK and IOSQE_IO_DRAIN from group leader, and
> the two flags can't be set for group members.
>
> When the group is in one link chain, this group isn't submitted until
> the previous SQE or group is completed. And the following SQE or group
> can't be started if this group isn't completed.
>
> When IOSQE_IO_DRAIN is set for group leader, all requests in this group
> and previous requests submitted are drained. Given IOSQE_IO_DRAIN can
> be set for group leader only, we respect IO_DRAIN for SQE group by
> always completing group leader as the last on in the group.
>
> SQE group provides flexible way to support N:M dependency, such as:
>
> - group A is chained with group B together by IOSQE_IO_LINK
> - group A has N SQEs
> - group B has M SQEs
>
> then M SQEs in group B depend on N SQEs in group A.
>
>
>>
>>> is missing in io_uring, but also makes async application easier to write by
>>> saving extra context switches, which just adds extra intermediate states for
>>> application.
>>
>> You're still executing requests (i.e. ->issue) primarily from the
>> submitter task context, they would still fly back to the task and
>> wake it up. You may save something by completing all of them
>> together via that refcounting, but you might just as well try to
>> batch CQ, which is a more generic issue. It's not clear what
>> context switches you save then.
>
> Wrt. the above N:M example, one io_uring_enter() is enough, and
> it can't be done in single context switch without sqe group, please
> see the liburing test code:

Do you mean doing all that in a single system call? The main
performance problem for io_uring is waiting, i.e. schedule()ing
the task out and in, that's what I meant by context switching.

--
Pavel Begunkov