Re: EAL feature request

[Stephen Hemminger <stephen@networkplumber.org>]

On Thu, 17 Sep 2026 14:48:24 +0200
Morten Brørup <mb@smartsharesystems.com> wrote:

> Bruce, David,
> 
> I don't know if this would be useful or not, but here goes...
> 
> Situation today:
> 
> DPDK has 3 types of threads with a valid lcore_id:
> 1. EAL threads, 
> 2. Service threads,
> 3. Non-EAL threads.
> 
> EAL threads are controlled by DPDK, and associated with physical cores at startup using the --lcores=<lcore_id>@<cpu_ids> parameter, and their OS cpuset/affinity etc. is managed by DPDK.
> 
> Service threads are also managed by DPDK.
> 
> Non-EAL threads are application created threads, which are simply assigned a valid lcore_id, but their OS cpuset/affinity etc. remains managed by the application.
> 
> Furthermore, macros such as RTE_LCORE_FOREACH_WORKER() only iterate over EAL threads, and rte_lcore_is_enabled() is only true for EAL threads.
> AFAIU, this means that DPDK only considers EAL threads as fastpath ("WORKER") threads.
> (The main thread is also an EAL thread, although it is usually not a fastpath thread. But its OS cpuset/affinity etc. is managed by DPDK, which is good.)
> 
> 
> Feature request:
> 
> Functions to create and destroy EAL threads at runtime, so applications like Grout could dynamically create/destroy threads that:
> 1. DPDK considers EAL (i.e. WORKER) threads,
> 2. Have their OS cpuset/affinity etc. managed by DPDK.
> 
> Such functions would probably need to take <lcore_id> and <cpu_ids> parameters, for dynamically updating the list of DPDK managed lcores and CPUs at runtime.
> 
> Maybe the ability to dynamically manipulate the <lcore_id>@<cpu_ids> mapping could be a separate set of APIs, so rte_thread_create_eal() would not require those parameters. (Don't know, just brainstorming.)
> 
> 
> Reason for asking:
> 
> Because DPDK doesn't provide this feature (hot-plug/remove of EAL threads), Grout has to take on the task of performing its own thread management (OS cpuset/affinity etc.), register its fastpath threads as Non-EAL threads, and cannot use RTE_LCORE_FOREACH_WORKER()/rte_lcore_is_enabled() for them, but considers Non-EAL as fastpath threads.
> 
> 
> Generally it would be logical for an application to register its control plane threads as Non-EAL threads to get the benefits that come with a valid lcore_id. But how then can DPDK discriminate between control plane and fastpath threads? Only the application knows.
> 
> It would be nice if DPDK's perception of which threads are fastpath or control plane was aligned with the application's perception.
> 
> 
> NB:
> For completeness, I should mention threads with no lcore_id, created via rte_thread_create_control(), and the OS cpuset/affinity etc. managed by the application.
> 
> 
> Venlig hilsen / Kind regards,
> -Morten Brørup
> 


After thinking about this a little more, it looks like the best design is:
   - have fixed pool of cpu's reserved and isolated
   - in that pool make sure and skip cpu 0 and its sibling shared core
   - use a script to not only isolate cpu from scheduler but also IRQ mapping, RCU, etc
   - start DPDK with that full pool of isolated cpu's
   - keep main lcore around for telemetry, management, alarms, etc.
   - launch worker lcore's on demand; start with one and if load gets bigger than use eal to launch more.
   - if demand wains, the worker lcore should return from the thread routine and go back to idle.

The key concepts:
   - keep worker cpu's isolated
   - idle workers sit in OS asleep on mutex so can reduce power/load

The hard parts:
   - deciding when to ramp up/down active worker count is hard heuristic to get right
   - reprogramming RSS and flows when workers are added is non trivial
   - stray packets during worker changes
   - synchronization of worker addition and removal