Decoupling DPDK from EAL

Decoupling DPDK from EAL

[Jason Vassbender]

Hello,

I am trying to integrate DPDK into an existing application in order to
improve packet processing latency, but it is proving rather difficult
because of DPDK's dependency on EAL's thread management and bootstrap
mechanism. Our application already has its own framework for managing
threads and their affinities/priorities, IPC, timers and its own bootstrap
mechanism (not necessarily via command line arguments), we wish to integrate
DPDK as an alternative network back-end, but it wants to to take over our entire
way of doing things.

Are there any plans to decouple DPDK's core functionality away from
EAL so that it can be more easily integrated into existing
applications?

-Jason

Re: Decoupling DPDK from EAL

[François-Frédéric Ozog]

Hi,

I just completed such a consulting mission for a customer. They were using
libpcap as the network back end and the most challenging hurdle was to
transform a single threaded capture architecture to a multi-threaded one
with DPDK. The other key take away, is that DPDK capture helps to get only
20% of the 20 times performance boost I managed to achieve: most of the
latency is due to "application" and other internal communication mechanisms.

So I agree that DPDK is not light, but I think most of the power of DPDK
comes from EAL thread management and "IPC"... 

Having said all that, I may have missed a critical point, so, what is the
specific major hurdle you see in the integration?

François-Frédéric


> -----Message d'origine-----
> De : dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] De la part de Jason Vassbender
> Envoyé : mardi 3 décembre 2013 22:51
> À : dev-VfR2kkLFssw@public.gmane.org
> Objet : [dpdk-dev] Decoupling DPDK from EAL
> 
> Hello,
> 
> I am trying to integrate DPDK into an existing application in order to
> improve packet processing latency, but it is proving rather difficult
> because of DPDK's dependency on EAL's thread management and bootstrap
> mechanism. Our application already has its own framework for managing
> threads and their affinities/priorities, IPC, timers and its own bootstrap
> mechanism (not necessarily via command line arguments), we wish to
> integrate DPDK as an alternative network back-end, but it wants to to take
> over our entire way of doing things.
> 
> Are there any plans to decouple DPDK's core functionality away from EAL so
> that it can be more easily integrated into existing applications?
> 
> -Jason

Re: Decoupling DPDK from EAL

[Jason Vassbender]

Hey,

I guess the main hurdle is that we already have our own multi-threaded
architecture and ways to control thread startup/shutdown, priorities
and affinities and they are all balanced very delicately (our
application is latency sensitive, runs on rt_preempt, boots with
isolcpus, etc). In addition, we are already using the command line to
initialize some of our things, and part of the configuration for the
application does not even come from the command line, but from eg.
XML configuration file over the network. So ideally what I would have
preferred is that EAL initialization could be done by other means (for
example a simple initialization function with a dictionary as to be more
flexible) and thread creation/shutdown could be left to the application
if it so desires, provided it meets the execution conditions expected
by DPDK.

Essentially, at its current state, DPDK offers a complete solution to
your problem including the entire surrounding framework. But for most
big applications they already have their own frameworks in place and
integrating DPDK becomes harder than it should be. So if DPDK
were to be decoupled from EAL, made more modular, and some of the
functions optionally left to applications to provide if they already have
the facilities for them would make integration much easier and more
flexible.

-Jason

On Wed, Dec 4, 2013 at 10:27 AM, François-Frédéric Ozog <ff-SpBG2i6TTxU@public.gmane.org> wrote:
> Hi,
>
> I just completed such a consulting mission for a customer. They were using
> libpcap as the network back end and the most challenging hurdle was to
> transform a single threaded capture architecture to a multi-threaded one
> with DPDK. The other key take away, is that DPDK capture helps to get only
> 20% of the 20 times performance boost I managed to achieve: most of the
> latency is due to "application" and other internal communication mechanisms.
>
> So I agree that DPDK is not light, but I think most of the power of DPDK
> comes from EAL thread management and "IPC"...
>
> Having said all that, I may have missed a critical point, so, what is the
> specific major hurdle you see in the integration?
>
> François-Frédéric
>
>
>> -----Message d'origine-----
>> De : dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] De la part de Jason Vassbender
>> Envoyé : mardi 3 décembre 2013 22:51
>> À : dev-VfR2kkLFssw@public.gmane.org
>> Objet : [dpdk-dev] Decoupling DPDK from EAL
>>
>> Hello,
>>
>> I am trying to integrate DPDK into an existing application in order to
>> improve packet processing latency, but it is proving rather difficult
>> because of DPDK's dependency on EAL's thread management and bootstrap
>> mechanism. Our application already has its own framework for managing
>> threads and their affinities/priorities, IPC, timers and its own bootstrap
>> mechanism (not necessarily via command line arguments), we wish to
>> integrate DPDK as an alternative network back-end, but it wants to to take
>> over our entire way of doing things.
>>
>> Are there any plans to decouple DPDK's core functionality away from EAL so
>> that it can be more easily integrated into existing applications?
>>
>> -Jason
>

Re: Decoupling DPDK from EAL

[François-Frédéric Ozog]

Hi,

I think I get the picture. DPDK is not really flexible at memory allocation
(nor the Linux kernel which requires boot parameters for 1GB huge pages)...
Let's assume that "static" memory configuration is acceptable. 

Is the thread model integration issue related to the fact we set affinity
ATFER thread creation? (I agree this is complex, but you can still create an
adaptation layer in your thread API to accommodate DPDK thread model).

François-Frédéric


> -----Message d'origine-----
> De : Jason Vassbender [mailto:jason.vassbender-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]
> Envoyé : mercredi 4 décembre 2013 10:25
> À : François-Frédéric Ozog
> Cc : dev-VfR2kkLFssw@public.gmane.org
> Objet : Re: [dpdk-dev] Decoupling DPDK from EAL
> 
> Hey,
> 
> I guess the main hurdle is that we already have our own multi-threaded
> architecture and ways to control thread startup/shutdown, priorities and
> affinities and they are all balanced very delicately (our application is
> latency sensitive, runs on rt_preempt, boots with isolcpus, etc). In
> addition, we are already using the command line to initialize some of our
> things, and part of the configuration for the application does not even
> come from the command line, but from eg.
> XML configuration file over the network. So ideally what I would have
> preferred is that EAL initialization could be done by other means (for
> example a simple initialization function with a dictionary as to be more
> flexible) and thread creation/shutdown could be left to the application if
> it so desires, provided it meets the execution conditions expected by
DPDK.
> 
> Essentially, at its current state, DPDK offers a complete solution to your
> problem including the entire surrounding framework. But for most big
> applications they already have their own frameworks in place and
> integrating DPDK becomes harder than it should be. So if DPDK were to be
> decoupled from EAL, made more modular, and some of the functions
optionally
> left to applications to provide if they already have the facilities for
> them would make integration much easier and more flexible.
> 
> -Jason
> 
> On Wed, Dec 4, 2013 at 10:27 AM, François-Frédéric Ozog <ff-SpBG2i6TTxU@public.gmane.org>
> wrote:
> > Hi,
> >
> > I just completed such a consulting mission for a customer. They were
> > using libpcap as the network back end and the most challenging hurdle
> > was to transform a single threaded capture architecture to a
> > multi-threaded one with DPDK. The other key take away, is that DPDK
> > capture helps to get only 20% of the 20 times performance boost I
> > managed to achieve: most of the latency is due to "application" and
other
> internal communication mechanisms.
> >
> > So I agree that DPDK is not light, but I think most of the power of
> > DPDK comes from EAL thread management and "IPC"...
> >
> > Having said all that, I may have missed a critical point, so, what is
> > the specific major hurdle you see in the integration?
> >
> > François-Frédéric
> >
> >
> >> -----Message d'origine-----
> >> De : dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] De la part de Jason Vassbender
> >> Envoyé : mardi 3 décembre 2013 22:51 À : dev-VfR2kkLFssw@public.gmane.org Objet :
> >> [dpdk-dev] Decoupling DPDK from EAL
> >>
> >> Hello,
> >>
> >> I am trying to integrate DPDK into an existing application in order
> >> to improve packet processing latency, but it is proving rather
> >> difficult because of DPDK's dependency on EAL's thread management and
> >> bootstrap mechanism. Our application already has its own framework
> >> for managing threads and their affinities/priorities, IPC, timers and
> >> its own bootstrap mechanism (not necessarily via command line
> >> arguments), we wish to integrate DPDK as an alternative network
> >> back-end, but it wants to to take over our entire way of doing things.
> >>
> >> Are there any plans to decouple DPDK's core functionality away from
> >> EAL so that it can be more easily integrated into existing
applications?
> >>
> >> -Jason
> >

Re: Decoupling DPDK from EAL

[Jason Vassbender]

Hi,

The memory allocation is not an issue since that is contained entirely
within DPDK itself and does not leak outside, i.e. all DPDK data
structures are managed with DPDK memory management functions and
that's valid and OK.

The thread model integration issue is because EAL creates its own
threads and sets its own affinities/priorities. My current plan is
indeed to wrap the DPDK thread stuff with my our own threading
facilities, so that when the DPDK thread starts up and calls my
function i will set up our own threading meta-data structures and set
the proper affinity/priority, then call our processing loop that
expects our meta-data to be in place.

As for the command line arguments; I plan to synthesize the command
line parameters that EAL expects, and then provide them to it in the
form of argc/argv so that i can do late-initialization via e.g.
external configuration data that I receive over the network after the
application is already running.

-Jason

On Wed, Dec 4, 2013 at 3:37 PM, François-Frédéric Ozog <ff-SpBG2i6TTxU@public.gmane.org> wrote:
> Hi,
>
> I think I get the picture. DPDK is not really flexible at memory allocation
> (nor the Linux kernel which requires boot parameters for 1GB huge pages)...
> Let's assume that "static" memory configuration is acceptable.
>
> Is the thread model integration issue related to the fact we set affinity
> ATFER thread creation? (I agree this is complex, but you can still create an
> adaptation layer in your thread API to accommodate DPDK thread model).
>
> François-Frédéric
>
>
>> -----Message d'origine-----
>> De : Jason Vassbender [mailto:jason.vassbender-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]
>> Envoyé : mercredi 4 décembre 2013 10:25
>> À : François-Frédéric Ozog
>> Cc : dev-VfR2kkLFssw@public.gmane.org
>> Objet : Re: [dpdk-dev] Decoupling DPDK from EAL
>>
>> Hey,
>>
>> I guess the main hurdle is that we already have our own multi-threaded
>> architecture and ways to control thread startup/shutdown, priorities and
>> affinities and they are all balanced very delicately (our application is
>> latency sensitive, runs on rt_preempt, boots with isolcpus, etc). In
>> addition, we are already using the command line to initialize some of our
>> things, and part of the configuration for the application does not even
>> come from the command line, but from eg.
>> XML configuration file over the network. So ideally what I would have
>> preferred is that EAL initialization could be done by other means (for
>> example a simple initialization function with a dictionary as to be more
>> flexible) and thread creation/shutdown could be left to the application if
>> it so desires, provided it meets the execution conditions expected by
> DPDK.
>>
>> Essentially, at its current state, DPDK offers a complete solution to your
>> problem including the entire surrounding framework. But for most big
>> applications they already have their own frameworks in place and
>> integrating DPDK becomes harder than it should be. So if DPDK were to be
>> decoupled from EAL, made more modular, and some of the functions
> optionally
>> left to applications to provide if they already have the facilities for
>> them would make integration much easier and more flexible.
>>
>> -Jason
>>
>> On Wed, Dec 4, 2013 at 10:27 AM, François-Frédéric Ozog <ff@ozog.com>
>> wrote:
>> > Hi,
>> >
>> > I just completed such a consulting mission for a customer. They were
>> > using libpcap as the network back end and the most challenging hurdle
>> > was to transform a single threaded capture architecture to a
>> > multi-threaded one with DPDK. The other key take away, is that DPDK
>> > capture helps to get only 20% of the 20 times performance boost I
>> > managed to achieve: most of the latency is due to "application" and
> other
>> internal communication mechanisms.
>> >
>> > So I agree that DPDK is not light, but I think most of the power of
>> > DPDK comes from EAL thread management and "IPC"...
>> >
>> > Having said all that, I may have missed a critical point, so, what is
>> > the specific major hurdle you see in the integration?
>> >
>> > François-Frédéric
>> >
>> >
>> >> -----Message d'origine-----
>> >> De : dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] De la part de Jason Vassbender
>> >> Envoyé : mardi 3 décembre 2013 22:51 À : dev-VfR2kkLFssw@public.gmane.org Objet :
>> >> [dpdk-dev] Decoupling DPDK from EAL
>> >>
>> >> Hello,
>> >>
>> >> I am trying to integrate DPDK into an existing application in order
>> >> to improve packet processing latency, but it is proving rather
>> >> difficult because of DPDK's dependency on EAL's thread management and
>> >> bootstrap mechanism. Our application already has its own framework
>> >> for managing threads and their affinities/priorities, IPC, timers and
>> >> its own bootstrap mechanism (not necessarily via command line
>> >> arguments), we wish to integrate DPDK as an alternative network
>> >> back-end, but it wants to to take over our entire way of doing things.
>> >>
>> >> Are there any plans to decouple DPDK's core functionality away from
>> >> EAL so that it can be more easily integrated into existing
> applications?
>> >>
>> >> -Jason
>> >
>