Re: DPDK and forked processes

[Thomas Monjalon <thomas@monjalon.net>]

27/07/2018 15:46, Eads, Gage:
> As this discussion has broad implications for DPDK, is it a good candidate for a techboard meeting topic? 

We can discuss it in techboard, but usually we prefer discussing topics
whose resolution is not clear.
In this case, I think everybody agree with Anatoly, isn't it?


> > -----Original Message-----
> > From: Burakov, Anatoly
> > Sent: Monday, July 16, 2018 10:09 AM
> > To: Eads, Gage <gage.eads@intel.com>; dev@dpdk.org
> > Subject: Re: DPDK and forked processes
> > 
> > On 16-Jul-18 4:00 PM, Eads, Gage wrote:
> > > Hi all,
> > >
> > > Does DPDK support forking secondary processes after executing
> > > rte_eal_init()? The l2fwd_fork example and at least one application
> > > (OpenEM: https://sourceforge.net/projects/eventmachine/) use this
> > > model, and they do so by fixing up the EAL internals (e.g. manually
> > > changing process_type from primary to secondary) at the start of the
> > > child process. This feels like a hack, and I can’t find any
> > > documentation describing this model.
> > >
> > > Moreover, this approach doesn’t appear to be compatible with recent
> > > EAL changes. For instance, the multi-process communication creates a
> > > couple handler threads (“rte_mp_handle” and “rte_mp_async”) during EAL
> > > initialization. The child processes won’t inherit these threads, and
> > > so won’t be able to participate in multi-process comms. This means the
> > > reworked memory subsystem and upcoming device hotplug support
> > > (http://mails.dpdk.org/archives/dev/2018-July/107704.html) won’t work
> > > with this fork-after-init model.
> > >
> > > This is just one example – there may be other features/subsystems that
> > > won’t work. As far as I can tell there is no official stance (though
> > > the l2fwd_fork example implies it’s supported, IMO); I think either
> > > DPDK should either drop the example and not support this model, or
> > > support it and either document its limitations or resolve them. This
> > > model could be an interesting way to run multi-process DPDK on an
> > > ASLR-enabled system, but supporting this wouldn’t be trivial.
> > >
> > > Thanks,
> > >
> > > Gage
> > >
> > 
> > I think it's a very bad idea to use such a model in recent versions of DPDK. As you
> > have correctly pointed out, IPC will not work in such a scenario, and given how
> > our memory subsystem relies on IPC, this is a recipe for memory corruption and
> > divergent memory maps (since technically both initial and forked processes
> > believe they are primary).
> > 
> > Even hacking rte_config to make DPDK think it's a secondary process will not
> > work, because the initialization has already completed, but all of the threads
> > (IPC, interrupt, etc.) are gone and correct IPC socket was not created, which
> > means the process becomes invisible to the primary for all intents and purposes.
> > 
> > We _could_ introduce some kind of "official DPDK fork" function that would fork
> > the process and then restart interrupt, IPC etc. stuff on an already running
> > instance of DPDK, but that seems like a workaround for a problem that shouldn't
> > exist in the first place, because such usage is fundamentally incompatible with
> > DPDK as it stands now.
> > 
> > --
> > Thanks,
> > Anatoly
>