graph: dispatch mode with libevent for control plane

["Robin Jarry" <rjarry@redhat.com>]

Hi folks,

I had a look at the graph dispatch mode and wondered if it could be 
adapted to be used with event loops for "slow path" processing.

The use case I have in mind is having lcores dedicated for control plane 
operations which are not latency sensitive. It would be nice to allow 
data plane workers to send packets to these lcores while keeping the 
packets in the graph.

The problem I have with this solution for now, is that all CPUs 
processing the graph with rte_graph_walk_mcore_dispatch() are supposed 
to do busy polling. This is not compatible with select/epoll based event 
loops.

Would it be possible to have a way for busy-polling threads to signal 
non-busy-polling threads that there are packets waiting for them in the 
graph ring? In grout, we implemented something outside of the graph 
using pthread_cond_signal().

https://github.com/DPDK/grout/blob/main/modules/infra/datapath/control_output.c
https://github.com/DPDK/grout/blob/main/modules/infra/control/control_output.c

Unless I am mistaken, this does not perform any syscall and does not 
require acquiring any lock. Only the "slow" thread needs to acquire 
locks and wait for the data path threads to call pthread_cond_signal().

This solution can only be used for busy-polling -> epoll-based 
communication, for the other direction epoll-based -> busy-polling there 
would not need any specific signaling required. Only posting the packets 
to the graph ring would be enough.

What do you think? How would you integrate this into rte_graph? What 
would the API look like in your opinion?

Thanks!

-- 
Robin