Question on the Ring Library

Question on the Ring Library

[Sambath Kumar Balasubramanian]

Hi,

  The ring library seems to be an excellent IPC. But looking at one use
case where the fast path code posts events to event thread for example, the
event thread will spend some cycles polling the ring rather than waiting
for the event. One approach could be a fast path code basically posts the
event in the ring as is today and there is a background thread that polls
the queues and wakes up the event threads. This is similar to Linux
SOFTIRQs.The event threads are asynchronous. Is this a fair model to avoid
extra polling CPU cycles by the event threads? Is there any other
alternatives in dpdk?

Regards,
Sambath

Re: Question on the Ring Library

[François-Frédéric Ozog]

Hi

On a 10Gbps link, there is a new packet every 650ns on average on each
direction. So handling latency is extremely important.

Traditional "fast" userland mutexes involves system call and scheduling
costs (look at the kernel code: it is "hairy"). I measured difference
between mutex controlled fifo and DPDK controlled fifo on a Xeon E5-2680v2,
1867MHz RAM: for times the performance... I consider the cost of a mutex
lock is something close to 400ns on average (well I don't say that it always
costs that, but the delays can be not predictable and extremely high - I saw
a few ms even if you use isolcpus and precise affinities-). So if you plan
to do 10Gbps stuff, I guess polling is the only way to go.

François-Frédéric


> -----Message d'origine-----
> De : dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] De la part de Sambath Kumar
> Balasubramanian
> Envoyé : mercredi 4 décembre 2013 12:47
> À : dev-VfR2kkLFssw@public.gmane.org
> Objet : [dpdk-dev] Question on the Ring Library
> 
> Hi,
> 
>   The ring library seems to be an excellent IPC. But looking at one use
> case where the fast path code posts events to event thread for example,
the
> event thread will spend some cycles polling the ring rather than waiting
> for the event. One approach could be a fast path code basically posts the
> event in the ring as is today and there is a background thread that polls
> the queues and wakes up the event threads. This is similar to Linux
> SOFTIRQs.The event threads are asynchronous. Is this a fair model to avoid
> extra polling CPU cycles by the event threads? Is there any other
> alternatives in dpdk?
> 
> Regards,
> Sambath

Re: Question on the Ring Library

[Sambath Kumar Balasubramanian]

Thanks François-Frédéric. That puts real good perspective on the cost for
the vent assuming each packet in the fast will result
in an event. If event rate is orders of magnitude less than the packet
rate, then I guess we can still achieve 10G since the "extra cost" will be
in the event thread and not in the fast path thread.

Regards,
Sambath

Re: Question on the Ring Library

[François-Frédéric Ozog]

You are welcome !

 

Even if you insert packets in batch into a fifo, the mutex is still
unpredictable. If one pthread_lock costs 1ms, you are going to lose packets,
regardless of the number of RSS queues and ring sizes… Batching comes with
another issue: need to flush a batch after a certain timeout if you do not
have packets. Userland timer resolution is such that you are introducing
port to port latency quite a lot: DPDK native, you can manage to get a 2µs
latency (including PCI express), while with batch it may be 1ms… You may
find smarter algorithms but still you will be an order of magnitude higher
than DPDK.

 

Hope this helps.

 

François-Frédéric

 

De : Sambath Kumar Balasubramanian
[mailto:sambath.balasubramanian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org] 
Envoyé : mercredi 4 décembre 2013 15:01
À : François-Frédéric Ozog
Cc : dev-VfR2kkLFssw@public.gmane.org
Objet : Re: [dpdk-dev] Question on the Ring Library

 

Thanks François-Frédéric. That puts real good perspective on the cost for
the vent assuming each packet in the fast will result

in an event. If event rate is orders of magnitude less than the packet rate,
then I guess we can still achieve 10G since the "extra cost" will be in the
event thread and not in the fast path thread.

 

Regards,

Sambath

Re: Question on the Ring Library

[Sambath Kumar Balasubramanian]

Thanks François-Frédéric. Trying to embark on a small prototype and see the
results. Thanks for the timing data. Really helpful.

Regards,
Sambath


On Wed, Dec 4, 2013 at 12:02 PM, François-Frédéric Ozog <ff-SpBG2i6TTxU@public.gmane.org> wrote:

> You are welcome !
>
>
>
> Even if you insert packets in batch into a fifo, the mutex is still
> unpredictable. If one pthread_lock costs 1ms, you are going to lose
> packets, regardless of the number of RSS queues and ring sizes… Batching
> comes with another issue: need to flush a batch after a certain timeout if
> you do not have packets. Userland timer resolution is such that you are
> introducing port to port latency quite a lot: DPDK native, you can manage
> to get a 2µs latency (including PCI express), while with batch it may be
> 1ms… You may find smarter algorithms but still you will be an order of
> magnitude higher than DPDK.
>
>
>
> Hope this helps.
>
>
>
> François-Frédéric
>
>
>
> *De :* Sambath Kumar Balasubramanian [mailto:
> sambath.balasubramanian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]
> *Envoyé :* mercredi 4 décembre 2013 15:01
> *À :* François-Frédéric Ozog
> *Cc :* dev-VfR2kkLFssw@public.gmane.org
> *Objet :* Re: [dpdk-dev] Question on the Ring Library
>
>
>
> Thanks François-Frédéric. That puts real good perspective on the cost for
> the vent assuming each packet in the fast will result
>
> in an event. If event rate is orders of magnitude less than the packet
> rate, then I guess we can still achieve 10G since the "extra cost" will be
> in the event thread and not in the fast path thread.
>
>
>
> Regards,
>
> Sambath
>

Re: Question on the Ring Library

[Stephen Hemminger]

On Wed, 4 Dec 2013 03:46:36 -0800
Sambath Kumar Balasubramanian <sambath.balasubramanian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Hi,
> 
>   The ring library seems to be an excellent IPC. But looking at one use
> case where the fast path code posts events to event thread for example, the
> event thread will spend some cycles polling the ring rather than waiting
> for the event. One approach could be a fast path code basically posts the
> event in the ring as is today and there is a background thread that polls
> the queues and wakes up the event threads. This is similar to Linux
> SOFTIRQs.The event threads are asynchronous. Is this a fair model to avoid
> extra polling CPU cycles by the event threads? Is there any other
> alternatives in dpdk?
> 
> Regards,
> Sambath

I have in several cases combined RTE ring with use of eventfd + poll to
get wakeup

Re: Question on the Ring Library

[Sambath Kumar Balasubramanian]

Thanks Stephen. I was going to do prototype something similar (not doing
the wakeup inline but using a background thread)
and is it a worthwhile effort to move this as a feature of the RTE ring or
is it best left at the application level.


On Wed, Dec 4, 2013 at 1:25 PM, Stephen Hemminger <
stephen-OTpzqLSitTUnbdJkjeBofR2eb7JE58TQ@public.gmane.org> wrote:

> On Wed, 4 Dec 2013 03:46:36 -0800
> Sambath Kumar Balasubramanian <sambath.balasubramanian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> > Hi,
> >
> >   The ring library seems to be an excellent IPC. But looking at one use
> > case where the fast path code posts events to event thread for example,
> the
> > event thread will spend some cycles polling the ring rather than waiting
> > for the event. One approach could be a fast path code basically posts the
> > event in the ring as is today and there is a background thread that polls
> > the queues and wakes up the event threads. This is similar to Linux
> > SOFTIRQs.The event threads are asynchronous. Is this a fair model to
> avoid
> > extra polling CPU cycles by the event threads? Is there any other
> > alternatives in dpdk?
> >
> > Regards,
> > Sambath
>
> I have in several cases combined RTE ring with use of eventfd + poll to
> get wakeup
>

Re: Question on the Ring Library

[Stephen Hemminger]

On Wed, 4 Dec 2013 13:47:10 -0800
Sambath Kumar Balasubramanian <sambath.balasubramanian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Thanks Stephen. I was going to do prototype something similar (not doing
> the wakeup inline but using a background thread)
> and is it a worthwhile effort to move this as a feature of the RTE ring or
> is it best left at the application level.
> 

I prefer that RTE ring stays as simple as possible.
If you need a higher level abstraction, built it yourself

Re: Question on the Ring Library

[Sambath Kumar Balasubramanian]

Understood. Thanks for getting back.

Regards,
Sambath


On Wed, Dec 4, 2013 at 1:51 PM, Stephen Hemminger <
stephen-OTpzqLSitTUnbdJkjeBofR2eb7JE58TQ@public.gmane.org> wrote:

> On Wed, 4 Dec 2013 13:47:10 -0800
> Sambath Kumar Balasubramanian <sambath.balasubramanian-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> > Thanks Stephen. I was going to do prototype something similar (not doing
> > the wakeup inline but using a background thread)
> > and is it a worthwhile effort to move this as a feature of the RTE ring
> or
> > is it best left at the application level.
> >
>
> I prefer that RTE ring stays as simple as possible.
> If you need a higher level abstraction, built it yourself
>