Re: [PATCH 0/3] Defer lcore variables allocation

[Stephen Hemminger <stephen@networkplumber.org>]

On Fri, 06 Dec 2024 16:55:30 +0100
Thomas Monjalon <thomas@monjalon.net> wrote:

> 06/12/2024 12:01, Mattias Rönnblom:
> > On 2024-12-05 18:57, David Marchand wrote:
> > In retrospect, maybe the offset between lcore variable instances could 
> > have been encoded into the handle, and thus one could use 
> > different-sized offset for different variables.  
> 
> Yes it would allow to allocate a minimum size,
> instead of having a default which is also a maximum limit size of an object.
> 
> It is not too late to change the behavior as the API is experimental.
> 
> > > The general question on whether lcore variables in constructor should
> > > be forbidden, is left to a later discussion.  
> > 
> > That discussion could be extended to cover the question if RTE_INIT() 
> > type constructors should be used at all. Intuitively, it seems better if 
> > all DPDK initialization, or at least all EAL init, happens at the time 
> > of rte_eal_init(), in some ordered/organized fashion.  
> 
> Yes we may avoid constructors and instead have callbacks called in rte_eal_init().
> In order to not break the RTE_INIT API, we could define some new macros
> for registering such rte_eal_init callbacks.
> 
> 

My intuition is that the OVS problem with using mlockall() is caused because when
malloc is used, the malloc code will pre-allocate a new arena (memory area) for use.
If the malloc takes before the mlockall() it will then be pinned even if not used.
If the malloc takes place later, perhaps that arena is coming from unpinned area.
Many more details on glibc malloc here:
   https://sourceware.org/glibc/wiki/MallocInternals

Using blunt tool like mlockall() will have unintended side effects.

The issue with constructors, is they look good when they are simple, statless,
and only a few of them. But they get to be a undebuggable mess when the constructor
does complex stuff; has dependencies; and there are lots of them.

As a refinement, maybe having a way to register callback to be called in parallel
after EAL has started threads.  But some things like random() need to be available
early in startup.