Preemption in rings

Preemption in rings

[Carlos Franco]

Hello

I have read the documentation and the answer of Olivier about a question to clarify the preemption restriction in the rings.

He wrote:

- a pthread doing multi-producers enqueues on a given ring must not  be preempted by another pthread doing a multi-producer enqueue on the same ring.

- a pthread doing multi-consumers dequeues on a given ring must not be preempted by another pthread doing a multi-consumer dequeue on  the same ring.

I suppose this is a something to be taken into account with threads not created by the DPDK rte, because the threads of DPDK are attached to a core and will never be scheduled into the cores of other DPDK threads, no?

I mean, lets only take into account the enqueuing. if there is a ring R where ONLY the DPDK threads enqueue packets, even if they are interrupted (IO access, sleep, mutex... ), and other non DPDK threads are scheduled into their cores, as long as only DPDK threads enqueue in the ring it is safe, no?

If not, would prohibit the access to the DPDK used cores (isocpus) work?

Thanks

Carlos

Re: Preemption in rings

[Olivier MATZ]

Hi Carlos,

On 01/18/2014 02:10 PM, Carlos Franco wrote:
 > - a pthread doing multi-producers enqueues on a given ring must
 >   not be preempted by another pthread doing a multi-producer
 >   enqueue on the same ring.
 >
 > - a pthread doing multi-consumers dequeues on a given ring must
 >   not be preempted by another pthread doing a multi-consumer
 >   dequeue on the same ring.
 >
 > I suppose this is a something to be taken into account with
 > threads not created by the DPDK rte, because the threads of DPDK
 > are attached to a core and will never be scheduled into the cores
 > of other DPDK threads, no?

Yes.

 > I mean, lets only take into account the enqueuing. if there is a
 > ring R where ONLY the DPDK threads enqueue packets, even if they
 > are interrupted (IO access, sleep, mutex... ), and other non DPDK
 > threads are scheduled into their cores, as long as only DPDK
 > threads enqueue in the ring it is safe, no?

It's safe, but it may cause performance issue. For instance, if a
DPDK pthread is interrupted by the kernel during a critical period
of an enqueue, all other DPDK pthreads enqueueing on this ring will
block.

Of course, this could also happen with a rte_spinlock().

 > If not, would prohibit the access to the DPDK used
 > cores (isocpus) work?

Yes, using "isolcpu" is a good idea to avoid the problem described
above.

 From my experience, using a simple "taskset" to force the other
applications to run on different cores than the DPDK is enough
to avoid significant performance issues. Indeed, the only problem
would be related to kernel work (interruptions and kernel threads),
and they shouldn't last long enough to fill the queues.

Regards,
Olivier