EAL feature request

EAL feature request

[Morten Brørup]

Bruce, David,

I don't know if this would be useful or not, but here goes...

Situation today:

DPDK has 3 types of threads with a valid lcore_id:
1. EAL threads, 
2. Service threads,
3. Non-EAL threads.

EAL threads are controlled by DPDK, and associated with physical cores at startup using the --lcores=<lcore_id>@<cpu_ids> parameter, and their OS cpuset/affinity etc. is managed by DPDK.

Service threads are also managed by DPDK.

Non-EAL threads are application created threads, which are simply assigned a valid lcore_id, but their OS cpuset/affinity etc. remains managed by the application.

Furthermore, macros such as RTE_LCORE_FOREACH_WORKER() only iterate over EAL threads, and rte_lcore_is_enabled() is only true for EAL threads.
AFAIU, this means that DPDK only considers EAL threads as fastpath ("WORKER") threads.
(The main thread is also an EAL thread, although it is usually not a fastpath thread. But its OS cpuset/affinity etc. is managed by DPDK, which is good.)


Feature request:

Functions to create and destroy EAL threads at runtime, so applications like Grout could dynamically create/destroy threads that:
1. DPDK considers EAL (i.e. WORKER) threads,
2. Have their OS cpuset/affinity etc. managed by DPDK.

Such functions would probably need to take <lcore_id> and <cpu_ids> parameters, for dynamically updating the list of DPDK managed lcores and CPUs at runtime.

Maybe the ability to dynamically manipulate the <lcore_id>@<cpu_ids> mapping could be a separate set of APIs, so rte_thread_create_eal() would not require those parameters. (Don't know, just brainstorming.)


Reason for asking:

Because DPDK doesn't provide this feature (hot-plug/remove of EAL threads), Grout has to take on the task of performing its own thread management (OS cpuset/affinity etc.), register its fastpath threads as Non-EAL threads, and cannot use RTE_LCORE_FOREACH_WORKER()/rte_lcore_is_enabled() for them, but considers Non-EAL as fastpath threads.


Generally it would be logical for an application to register its control plane threads as Non-EAL threads to get the benefits that come with a valid lcore_id. But how then can DPDK discriminate between control plane and fastpath threads? Only the application knows.

It would be nice if DPDK's perception of which threads are fastpath or control plane was aligned with the application's perception.


NB:
For completeness, I should mention threads with no lcore_id, created via rte_thread_create_control(), and the OS cpuset/affinity etc. managed by the application.


Venlig hilsen / Kind regards,
-Morten Brørup

Re: EAL feature request

[Bruce Richardson]

On Thu, Sep 17, 2026 at 02:48:24PM +0200, Morten Brørup wrote:
> Bruce, David,
> 
> I don't know if this would be useful or not, but here goes...
> 
> Situation today:
> 
> DPDK has 3 types of threads with a valid lcore_id:
> 1. EAL threads, 
> 2. Service threads,
> 3. Non-EAL threads.
> 
> EAL threads are controlled by DPDK, and associated with physical cores at startup using the --lcores=<lcore_id>@<cpu_ids> parameter, and their OS cpuset/affinity etc. is managed by DPDK.
> 
> Service threads are also managed by DPDK.
> 
> Non-EAL threads are application created threads, which are simply assigned a valid lcore_id, but their OS cpuset/affinity etc. remains managed by the application.
> 
> Furthermore, macros such as RTE_LCORE_FOREACH_WORKER() only iterate over EAL threads, and rte_lcore_is_enabled() is only true for EAL threads.
> AFAIU, this means that DPDK only considers EAL threads as fastpath ("WORKER") threads.
> (The main thread is also an EAL thread, although it is usually not a fastpath thread. But its OS cpuset/affinity etc. is managed by DPDK, which is good.)
> 
> 
> Feature request:
> 
> Functions to create and destroy EAL threads at runtime, so applications like Grout could dynamically create/destroy threads that:
> 1. DPDK considers EAL (i.e. WORKER) threads,
> 2. Have their OS cpuset/affinity etc. managed by DPDK.
> 
> Such functions would probably need to take <lcore_id> and <cpu_ids> parameters, for dynamically updating the list of DPDK managed lcores and CPUs at runtime.
> 
> Maybe the ability to dynamically manipulate the <lcore_id>@<cpu_ids> mapping could be a separate set of APIs, so rte_thread_create_eal() would not require those parameters. (Don't know, just brainstorming.)
> 
> 
> Reason for asking:
> 
> Because DPDK doesn't provide this feature (hot-plug/remove of EAL threads), Grout has to take on the task of performing its own thread management (OS cpuset/affinity etc.), register its fastpath threads as Non-EAL threads, and cannot use RTE_LCORE_FOREACH_WORKER()/rte_lcore_is_enabled() for them, but considers Non-EAL as fastpath threads.
> 
> 
> Generally it would be logical for an application to register its control plane threads as Non-EAL threads to get the benefits that come with a valid lcore_id. But how then can DPDK discriminate between control plane and fastpath threads? Only the application knows.
> 
> It would be nice if DPDK's perception of which threads are fastpath or control plane was aligned with the application's perception.
> 

I've thought a bit about this in the past, and come to the conclusion that,
as currently designed, DPDK doesn't support spawning thread instances, but
it instead supports the idea of spawning work on already-existing threads.
I believe the original intention was that the maximum number of threads
would always exist when DPDK was running, but that they would be stopped
and started by the app by assigning them work - the threads being asleep in
the kernel in the meantime.

While it would be possible to make that model work for most use cases, I do
agree that your suggestion is a reasonable one, and probably is easier for
apps to work with.

/Bruce

Re: EAL feature request

[Stephen Hemminger]

On Thu, 17 Sep 2026 14:01:40 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> On Thu, Sep 17, 2026 at 02:48:24PM +0200, Morten Brørup wrote:
> > Bruce, David,
> > 
> > I don't know if this would be useful or not, but here goes...
> > 
> > Situation today:
> > 
> > DPDK has 3 types of threads with a valid lcore_id:
> > 1. EAL threads, 
> > 2. Service threads,
> > 3. Non-EAL threads.
> > 
> > EAL threads are controlled by DPDK, and associated with physical cores at startup using the --lcores=<lcore_id>@<cpu_ids> parameter, and their OS cpuset/affinity etc. is managed by DPDK.
> > 
> > Service threads are also managed by DPDK.
> > 
> > Non-EAL threads are application created threads, which are simply assigned a valid lcore_id, but their OS cpuset/affinity etc. remains managed by the application.
> > 
> > Furthermore, macros such as RTE_LCORE_FOREACH_WORKER() only iterate over EAL threads, and rte_lcore_is_enabled() is only true for EAL threads.
> > AFAIU, this means that DPDK only considers EAL threads as fastpath ("WORKER") threads.
> > (The main thread is also an EAL thread, although it is usually not a fastpath thread. But its OS cpuset/affinity etc. is managed by DPDK, which is good.)
> > 
> > 
> > Feature request:
> > 
> > Functions to create and destroy EAL threads at runtime, so applications like Grout could dynamically create/destroy threads that:
> > 1. DPDK considers EAL (i.e. WORKER) threads,
> > 2. Have their OS cpuset/affinity etc. managed by DPDK.
> > 
> > Such functions would probably need to take <lcore_id> and <cpu_ids> parameters, for dynamically updating the list of DPDK managed lcores and CPUs at runtime.
> > 
> > Maybe the ability to dynamically manipulate the <lcore_id>@<cpu_ids> mapping could be a separate set of APIs, so rte_thread_create_eal() would not require those parameters. (Don't know, just brainstorming.)
> > 
> > 
> > Reason for asking:
> > 
> > Because DPDK doesn't provide this feature (hot-plug/remove of EAL threads), Grout has to take on the task of performing its own thread management (OS cpuset/affinity etc.), register its fastpath threads as Non-EAL threads, and cannot use RTE_LCORE_FOREACH_WORKER()/rte_lcore_is_enabled() for them, but considers Non-EAL as fastpath threads.
> > 
> > 
> > Generally it would be logical for an application to register its control plane threads as Non-EAL threads to get the benefits that come with a valid lcore_id. But how then can DPDK discriminate between control plane and fastpath threads? Only the application knows.
> > 
> > It would be nice if DPDK's perception of which threads are fastpath or control plane was aligned with the application's perception.
> >   
> 
> I've thought a bit about this in the past, and come to the conclusion that,
> as currently designed, DPDK doesn't support spawning thread instances, but
> it instead supports the idea of spawning work on already-existing threads.
> I believe the original intention was that the maximum number of threads
> would always exist when DPDK was running, but that they would be stopped
> and started by the app by assigning them work - the threads being asleep in
> the kernel in the meantime.
> 
> While it would be possible to make that model work for most use cases, I do
> agree that your suggestion is a reasonable one, and probably is easier for
> apps to work with.
> 
> /Bruce

Also doing spawning of new threads doesn't work well with isolated cpus and/or cgroups.
You really want isolated CPU's with DPDK's polling model and that set is not
easily controllable. You need to use cgroups but also need to force a bunch
of other things like kthreads, RCU callbacks, and IRQs away from that CPU.

Re: EAL feature request

[Stephen Hemminger]

On Thu, 17 Sep 2026 14:48:24 +0200
Morten Brørup <mb@smartsharesystems.com> wrote:

> Bruce, David,
> 
> I don't know if this would be useful or not, but here goes...
> 
> Situation today:
> 
> DPDK has 3 types of threads with a valid lcore_id:
> 1. EAL threads, 
> 2. Service threads,
> 3. Non-EAL threads.
> 
> EAL threads are controlled by DPDK, and associated with physical cores at startup using the --lcores=<lcore_id>@<cpu_ids> parameter, and their OS cpuset/affinity etc. is managed by DPDK.
> 
> Service threads are also managed by DPDK.
> 
> Non-EAL threads are application created threads, which are simply assigned a valid lcore_id, but their OS cpuset/affinity etc. remains managed by the application.
> 
> Furthermore, macros such as RTE_LCORE_FOREACH_WORKER() only iterate over EAL threads, and rte_lcore_is_enabled() is only true for EAL threads.
> AFAIU, this means that DPDK only considers EAL threads as fastpath ("WORKER") threads.
> (The main thread is also an EAL thread, although it is usually not a fastpath thread. But its OS cpuset/affinity etc. is managed by DPDK, which is good.)
> 
> 
> Feature request:
> 
> Functions to create and destroy EAL threads at runtime, so applications like Grout could dynamically create/destroy threads that:
> 1. DPDK considers EAL (i.e. WORKER) threads,
> 2. Have their OS cpuset/affinity etc. managed by DPDK.
> 
> Such functions would probably need to take <lcore_id> and <cpu_ids> parameters, for dynamically updating the list of DPDK managed lcores and CPUs at runtime.
> 
> Maybe the ability to dynamically manipulate the <lcore_id>@<cpu_ids> mapping could be a separate set of APIs, so rte_thread_create_eal() would not require those parameters. (Don't know, just brainstorming.)
> 
> 
> Reason for asking:
> 
> Because DPDK doesn't provide this feature (hot-plug/remove of EAL threads), Grout has to take on the task of performing its own thread management (OS cpuset/affinity etc.), register its fastpath threads as Non-EAL threads, and cannot use RTE_LCORE_FOREACH_WORKER()/rte_lcore_is_enabled() for them, but considers Non-EAL as fastpath threads.
> 
> 
> Generally it would be logical for an application to register its control plane threads as Non-EAL threads to get the benefits that come with a valid lcore_id. But how then can DPDK discriminate between control plane and fastpath threads? Only the application knows.
> 
> It would be nice if DPDK's perception of which threads are fastpath or control plane was aligned with the application's perception.
> 
> 
> NB:
> For completeness, I should mention threads with no lcore_id, created via rte_thread_create_control(), and the OS cpuset/affinity etc. managed by the application.
> 
> 
> Venlig hilsen / Kind regards,
> -Morten Brørup
> 


After thinking about this a little more, it looks like the best design is:
   - have fixed pool of cpu's reserved and isolated
   - in that pool make sure and skip cpu 0 and its sibling shared core
   - use a script to not only isolate cpu from scheduler but also IRQ mapping, RCU, etc
   - start DPDK with that full pool of isolated cpu's
   - keep main lcore around for telemetry, management, alarms, etc.
   - launch worker lcore's on demand; start with one and if load gets bigger than use eal to launch more.
   - if demand wains, the worker lcore should return from the thread routine and go back to idle.

The key concepts:
   - keep worker cpu's isolated
   - idle workers sit in OS asleep on mutex so can reduce power/load

The hard parts:
   - deciding when to ramp up/down active worker count is hard heuristic to get right
   - reprogramming RSS and flows when workers are added is non trivial
   - stray packets during worker changes
   - synchronization of worker addition and removal