io_uring link semantics (was [PATCH 5/6] io_uring: add support for futex wake and wait)

[Gabriel Krisman Bertazi <krisman@suse.de>]

Jens Axboe <axboe@kernel.dk> writes:

> On 6/12/23 5:00?PM, Gabriel Krisman Bertazi wrote:

>>>> Even with an asynchronous model, it might make sense to halt execution
>>>> of further queued operations until futex completes.  I think
>>>> IOSQE_IO_DRAIN is a barrier only against the submission part, so it
>>>> wouldn't hep.  Is there a way to ensure this ordering?
>>>
>>> You'd use link for that - link whatever depends on the wake to the futex
>>> wait. Or just queue it up once you reap the wait completion, when that
>>> is posted because we got woken.
>> 
>> The challenge of linked requests, in my opinion, is that once a link
>> chain starts, everything needs to be link together, and a single error
>> fails everything, which is ok when operations are related, but
>> not so much when doing IO to different files from the same ring.
>
> Not quite sure if you're misunderstanding links, or just have a
> different use case in mind. You can certainly have several independent
> chains of links.

I might be. Or my use case might be bogus. Please, correct me if either
is the case.

My understanding is that a link is a sequence of commands all carrying
the IOSQE_IO_LINK flag.  io_uring guarantees the ordering within the
link and, if a previous command fails, the rest of the link chain is
aborted.

But, if I have independent commands to submit in between, i.e. on a
different fd, I might want an intermediary operation to not be dependent
on the rest of the link without breaking the chain.  Most of the time I
know ahead of time the entire chain, and I can batch the operations
together.  But, I think it might be a problem specifically for some
unbounded commands like FUTEX_WAIT and recv.  I want a specific
operation to depend on a recv, but I might not be able to specify ahead
of time all of the dependent operations. I'd need to wait for a recv
command to complete and only then issue the dependency, to guarantee
ordering, or I make sure that everything I put on the ring in the
meantime is part of one big link submitted sequentially.

A related issue/example that comes to mind would be two recvs/sends
against the same socket.  When doing a syscall, I know the first recv
will return ahead of the second because it is, well, synchronous.  On
io_uring, I think it must be a link.  I might be recv'ing a huge stream
from the network, and I can't tell if the packet is done on a single
recv.  I could have to issue a second recv but I either make it linked
ahead of time, or I need to wait for the first recv to complete, to only
then submit the second one.  The problem is the ordering of recvs; from
my understanding of the code, I cannot assure the first recv will
complete before the second, without a link.

Sorry if I'm wrong and there are ways around it, but it is a struggling
points for me at the moment with using io_uring.

-- 
Gabriel Krisman Bertazi