RE: RFC - Tap io_uring PMD

[Konstantin Ananyev <konstantin.ananyev@huawei.com>]

> > > > > > Probably the hardest part of using io_uring is figuring out how
> > to
> > > > > > collect
> > > > > > completions. The simplest way would be to handle all
> > completions rx
> > > > and
> > > > > > tx
> > > > > > in the rx_burst function.
> > > > >
> > > > > Please don't mix RX and TX, unless explicitly requested by the
> > > > application through the recently introduced "mbuf recycle" feature.
> > > >
> > > > The issue is Rx and Tx share a single fd and ioring for completion
> > is
> > > > per fd.
> > > > The implementation for ioring came from the storage side so
> > initially
> > > > it was for fixing
> > > > the broken Linux AIO support.
> > > >
> > > > Some other devices only have single interrupt or ring shared with
> > rx/tx
> > > > so not unique.
> > > > Virtio, netvsc, and some NIC's.
> > > >
> > > > The problem is that if Tx completes descriptors then there needs to
> > be
> > > > locking
> > > > to prevent Rx thread and Tx thread overlapping. And a spin lock is
> > a
> > > > performance buzz kill.
> > >
> > > Brainstorming a bit here...
> > > What if the new TAP io_uring PMD is designed to use two io_urings per
> > port, one for RX and another one for TX on the same TAP interface?
> > > This requires that a TAP interface can be referenced via two file
> > descriptors (one fd for the RX io_uring and another fd for the TX
> > io_uring), e.g. by using dup() to create the additional file
> > descriptor. I don't know if this is possible, and if it works with
> > io_uring.
> >
> > There a couple of problems with multiple fd's.
> >   - multiple fds pointing to same internal tap queue are not going to
> > get completed separately.
> >   - when multi-proc is supported, limit of 253 fd's in Unix domain IPC
> > comes into play
> >   - tap does not support tx only fd for queues. If fd is queue of tap,
> > receive fan out will go to it.
> >
> > If DPDK was more flexible, harvesting of completion could be done via
> > another thread but that is not general enough
> > to work transparently with all applications.  Existing TAP device plays
> > with SIGIO, but signals are slower.
> 
> I have now read up a bit about io_uring, so here are some thoughts and ideas...
> 
> To avoid locking, there should only be one writer of io_uring Submission Queue Events (SQE), and only one reader of io_uring
> Completion Queue Events (CQE) per TAP interface.
> 
> From what I understand, the TAP io_uring PMD only supports one RX queue per port and one TX queue per port (i.e. per TAP
> interface). We can take advantage of this:
> 
> We can use rte_tx() as the Submission Queue writer and rte_rx() as the Completion Queue reader.
> 
> The PMD must have two internal rte_rings for respectively RX refill and TX completion events.
> 
> rte_rx() does the following:
> Read the Completion Queue;
> If RX CQE, pass the data to the next RX MBUF, convert the RX CQE to an RX Refill SQE and enqueue it in the RX Refill rte_ring;
> If TX CQE, enqueue it in the TX Completion rte_ring;
> Repeat until nb_pkts RX CQEs have been received, or no more CQE's are available. (This complies with the rte_rx() API, which says
> that less than nb_pkts is only returned if no more packets are available for receiving.)
> 
> rte_tx() does the following:
> Pass the data from the TX MBUFs to io_uring TX SQEs, using the TX CQEs in the TX Completion rte_ring, and write them to the io_uring
> Submission Queue.
> Dequeue any RX Refill SQEs from the RX Refill rte_ring and write them to the io_uring Submission Queue.
> 
> This means that the application must call both rte_rx() and rte_tx(); but it would be allowed to call rte_tx() with zero MBUFs.
> 
> The internal rte_rings are Single-Producer, Single-Consumer, and large enough to hold all TX+RX descriptors.
> 
> 
> Alternatively, we can let rte_rx() do all the work and use an rte_ring in the opposite direction...
> 
> The PMD must have two internal rte_rings, one for TX MBUFs and one for TX CQEs. (The latter can be a stack, or any other type of
> container.)
> 
> rte_tx() only does the following:
> Enqueue the TX MBUFs to the TX MBUF rte_ring.
> 
> rte_rx() does the following:
> Dequeue any TX MBUFs from the TX MBUF rte_ring, convert them to TX SQEs, using the TX CQEs in the TX Completion rte_ring, and
> write them to the io_uring Submission Queue.
> Read the Completion Queue;
> If TX CQE, enqueue it in the TX Completion rte_ring;
> If RX CQE, pass the data to the next RX MBUF, convert the RX CQE to an RX Refill SQE and write it to the io_uring Submission Queue;
> Repeat until nb_pkts RX CQEs have been received, or no more CQE's are available. (This complies with the rte_rx() API, which says
> that less than nb_pkts is only returned if no more packets are available for receiving.)
> 
> With the second design, the PMD can support multiple TX queues by using a Multi-Producer rte_ring for the TX MBUFs.
> But it postpones all transmits until rte_rx() is called, so I don't really like it.
> 
> Of the two designs, the first feels more natural to me.
> And if some application absolutely needs multiple TX queues, it can implement a Multi-Producer, Single-Consumer rte_ring as an
> intermediate step in front of the PMD's single TX queue.

And why we can't simply have 2 io_uring(s): one for RX ops, second for TX ops?