* [PATCH v5 1/4] eal/linux: filter uevent Netlink in kernel
2026-08-19 19:08 ` [PATCH v5 0/4] eal: uevent overrun mitigation Stephen Hemminger
@ 2026-08-19 19:08 ` Stephen Hemminger
2026-08-20 1:38 ` [EXTERNAL] " Long Li
2026-08-19 19:08 ` [PATCH v5 2/4] eal/linux: increase uevent socket buffer Stephen Hemminger
` (3 subsequent siblings)
4 siblings, 1 reply; 15+ messages in thread
From: Stephen Hemminger @ 2026-08-19 19:08 UTC (permalink / raw)
To: dev
Cc: Long Li, Stephen Hemminger, stable, Randy Tice, Thomas Monjalon,
Jianfeng Tan, Jeff Guo
The EAL uevent only needs/wants messages from the kernel.
Better to let kernel side do the filtering to avoid any overrun
issues in DPDK interrupt thread.
There is no exposed API definition here. The convention
is kernel is 1 and libudev uses 2.
Since kernel is now filtering can remove step in uevent parsing.
Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
Cc: stable@dpdk.org
Reported-by: Randy Tice <rtice@cisco.com>
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
.mailmap | 1 +
lib/eal/linux/eal_dev.c | 8 ++++----
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/.mailmap b/.mailmap
index fcb3d1bb3f..9b7713b6df 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1380,6 +1380,7 @@ Ralf Hoffmann <ralf.hoffmann@allegro-packets.com>
Rami Rosen <ramirose@gmail.com> <rami.rosen@intel.com>
Rami Rosen <ramirose@gmail.com> <roszenrami@gmail.com>
Randy Schacher <stuart.schacher@broadcom.com>
+Randy Tice <rtice@cisco.com>
Rani Sharoni <ranish@nvidia.com>
Ranjit Menon <ranjit.menon@intel.com>
Rasesh Mody <rmody@marvell.com> <rasesh.mody@cavium.com>
diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index ec408649d0..666967667f 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -29,6 +29,9 @@ static bool hotplug_handle;
#define EAL_UEV_MSG_LEN 4096
#define EAL_UEV_MSG_ELEM_LEN 128
+/* Listen only to messages from kernel (not libudev) */
+#define EAL_UEV_GROUP_KERNEL 1
+
/*
* spinlock for device hot-unplug failure handling. If it try to access bus or
* device, such as handle sigbus on bus or handle memory failure for device
@@ -116,7 +119,7 @@ dev_uev_socket_fd_create(void)
memset(&addr, 0, sizeof(addr));
addr.nl_family = AF_NETLINK;
addr.nl_pid = 0;
- addr.nl_groups = 0xffffffff;
+ addr.nl_groups = EAL_UEV_GROUP_KERNEL;
ret = bind(fd, (struct sockaddr *) &addr, sizeof(addr));
if (ret < 0) {
@@ -164,9 +167,6 @@ dev_uev_parse(const char *buf, struct rte_dev_event *event, int length)
* check device uevent from kernel side, no need to check
* uevent from udev.
*/
- if (!strncmp(buf, "libudev", 7)) {
- return -1;
- }
if (!strncmp(buf, "ACTION=", 7)) {
buf += 7;
i += 7;
--
2.53.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* RE: [EXTERNAL] [PATCH v5 1/4] eal/linux: filter uevent Netlink in kernel
2026-08-19 19:08 ` [PATCH v5 1/4] eal/linux: filter uevent Netlink in kernel Stephen Hemminger
@ 2026-08-20 1:38 ` Long Li
0 siblings, 0 replies; 15+ messages in thread
From: Long Li @ 2026-08-20 1:38 UTC (permalink / raw)
To: Stephen Hemminger, dev@dpdk.org
Cc: stable@dpdk.org, Randy Tice, Thomas Monjalon, Jianfeng Tan,
Jeff Guo
> The EAL uevent only needs/wants messages from the kernel.
> Better to let kernel side do the filtering to avoid any overrun issues in DPDK
> interrupt thread.
>
> There is no exposed API definition here. The convention is kernel is 1 and
> libudev uses 2.
>
> Since kernel is now filtering can remove step in uevent parsing.
>
> Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
> Cc: stable@dpdk.org
>
> Reported-by: Randy Tice <rtice@cisco.com>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by: Long Li <longli@microsoft.com>
> ---
> .mailmap | 1 +
> lib/eal/linux/eal_dev.c | 8 ++++----
> 2 files changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/.mailmap b/.mailmap
> index fcb3d1bb3f..9b7713b6df 100644
> --- a/.mailmap
> +++ b/.mailmap
> @@ -1380,6 +1380,7 @@ Ralf Hoffmann <ralf.hoffmann@allegro-
> packets.com>
> Rami Rosen <ramirose@gmail.com> <rami.rosen@intel.com> Rami Rosen
> <ramirose@gmail.com> <roszenrami@gmail.com> Randy Schacher
> <stuart.schacher@broadcom.com>
> +Randy Tice <rtice@cisco.com>
> Rani Sharoni <ranish@nvidia.com>
> Ranjit Menon <ranjit.menon@intel.com>
> Rasesh Mody <rmody@marvell.com> <rasesh.mody@cavium.com> diff --git
> a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c index
> ec408649d0..666967667f 100644
> --- a/lib/eal/linux/eal_dev.c
> +++ b/lib/eal/linux/eal_dev.c
> @@ -29,6 +29,9 @@ static bool hotplug_handle; #define EAL_UEV_MSG_LEN
> 4096 #define EAL_UEV_MSG_ELEM_LEN 128
>
> +/* Listen only to messages from kernel (not libudev) */ #define
> +EAL_UEV_GROUP_KERNEL 1
> +
> /*
> * spinlock for device hot-unplug failure handling. If it try to access bus or
> * device, such as handle sigbus on bus or handle memory failure for device
> @@ -116,7 +119,7 @@ dev_uev_socket_fd_create(void)
> memset(&addr, 0, sizeof(addr));
> addr.nl_family = AF_NETLINK;
> addr.nl_pid = 0;
> - addr.nl_groups = 0xffffffff;
> + addr.nl_groups = EAL_UEV_GROUP_KERNEL;
>
> ret = bind(fd, (struct sockaddr *) &addr, sizeof(addr));
> if (ret < 0) {
> @@ -164,9 +167,6 @@ dev_uev_parse(const char *buf, struct rte_dev_event
> *event, int length)
> * check device uevent from kernel side, no need to check
> * uevent from udev.
> */
> - if (!strncmp(buf, "libudev", 7)) {
> - return -1;
> - }
> if (!strncmp(buf, "ACTION=", 7)) {
> buf += 7;
> i += 7;
> --
> 2.53.0
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 2/4] eal/linux: increase uevent socket buffer
2026-08-19 19:08 ` [PATCH v5 0/4] eal: uevent overrun mitigation Stephen Hemminger
2026-08-19 19:08 ` [PATCH v5 1/4] eal/linux: filter uevent Netlink in kernel Stephen Hemminger
@ 2026-08-19 19:08 ` Stephen Hemminger
2026-08-20 1:39 ` [EXTERNAL] " Long Li
2026-08-19 19:08 ` [PATCH v5 3/4] eal/linux: report uevent socket overrun Stephen Hemminger
` (2 subsequent siblings)
4 siblings, 1 reply; 15+ messages in thread
From: Stephen Hemminger @ 2026-08-19 19:08 UTC (permalink / raw)
To: dev; +Cc: Long Li, Stephen Hemminger, stable, Randy Tice, Jianfeng Tan,
Jeff Guo
The kernel netlink socket into DPDK can get overrun if DPDK
interrupt thread is not keeping up. To reduce the possibility
increase the socket receive buffer to 4M.
Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
Cc: stable@dpdk.org
Reported-by: Randy Tice <rtice@cisco.com>
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
lib/eal/linux/eal_dev.c | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index 666967667f..fb5c8bf070 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -32,6 +32,9 @@ static bool hotplug_handle;
/* Listen only to messages from kernel (not libudev) */
#define EAL_UEV_GROUP_KERNEL 1
+/* Large uevent buffer */
+#define EAL_UEV_MSG_RCVBUF (4 * 1024 * 1024)
+
/*
* spinlock for device hot-unplug failure handling. If it try to access bus or
* device, such as handle sigbus on bus or handle memory failure for device
@@ -103,6 +106,29 @@ static int cmp_dev_name(const struct rte_device *dev,
return strcmp(dev->name, name);
}
+/*
+ * To avoid losing uevents increase the netlink receive buffer size,
+ * and override the kernel clamp value if necessary.
+ */
+static int
+dev_uev_set_rcvbuf(int fd, int n)
+{
+ int ret, val;
+ socklen_t len = sizeof(val);
+
+ ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &n, sizeof(n));
+ if (ret < 0)
+ return ret;
+
+ /* kernel may have clamped our request */
+ ret = getsockopt(fd, SOL_SOCKET, SO_RCVBUF, &val, &len);
+ if (ret >= 0 && len == sizeof(val) && val == n * 2)
+ return 0; /* request worked */
+
+ /* try again to override kernel restriction */
+ return setsockopt(fd, SOL_SOCKET, SO_RCVBUFFORCE, &n, sizeof(n));
+}
+
static int
dev_uev_socket_fd_create(void)
{
@@ -127,6 +153,9 @@ dev_uev_socket_fd_create(void)
goto err;
}
+ if (dev_uev_set_rcvbuf(fd, EAL_UEV_MSG_RCVBUF) < 0)
+ EAL_LOG(NOTICE, "Failed to set rcvbuf.");
+
if (rte_intr_fd_set(intr_handle, fd))
goto err;
--
2.53.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* RE: [EXTERNAL] [PATCH v5 2/4] eal/linux: increase uevent socket buffer
2026-08-19 19:08 ` [PATCH v5 2/4] eal/linux: increase uevent socket buffer Stephen Hemminger
@ 2026-08-20 1:39 ` Long Li
0 siblings, 0 replies; 15+ messages in thread
From: Long Li @ 2026-08-20 1:39 UTC (permalink / raw)
To: Stephen Hemminger, dev@dpdk.org
Cc: stable@dpdk.org, Randy Tice, Jianfeng Tan, Jeff Guo
> The kernel netlink socket into DPDK can get overrun if DPDK interrupt thread is
> not keeping up. To reduce the possibility increase the socket receive buffer to
> 4M.
>
> Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
> Cc: stable@dpdk.org
>
> Reported-by: Randy Tice <rtice@cisco.com>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by: Long Li <longli@microsoft.com>
> ---
> lib/eal/linux/eal_dev.c | 29 +++++++++++++++++++++++++++++
> 1 file changed, 29 insertions(+)
>
> diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c index
> 666967667f..fb5c8bf070 100644
> --- a/lib/eal/linux/eal_dev.c
> +++ b/lib/eal/linux/eal_dev.c
> @@ -32,6 +32,9 @@ static bool hotplug_handle;
> /* Listen only to messages from kernel (not libudev) */ #define
> EAL_UEV_GROUP_KERNEL 1
>
> +/* Large uevent buffer */
> +#define EAL_UEV_MSG_RCVBUF (4 * 1024 * 1024)
> +
> /*
> * spinlock for device hot-unplug failure handling. If it try to access bus or
> * device, such as handle sigbus on bus or handle memory failure for device
> @@ -103,6 +106,29 @@ static int cmp_dev_name(const struct rte_device
> *dev,
> return strcmp(dev->name, name);
> }
>
> +/*
> + * To avoid losing uevents increase the netlink receive buffer size,
> + * and override the kernel clamp value if necessary.
> + */
> +static int
> +dev_uev_set_rcvbuf(int fd, int n)
> +{
> + int ret, val;
> + socklen_t len = sizeof(val);
> +
> + ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &n, sizeof(n));
> + if (ret < 0)
> + return ret;
> +
> + /* kernel may have clamped our request */
> + ret = getsockopt(fd, SOL_SOCKET, SO_RCVBUF, &val, &len);
> + if (ret >= 0 && len == sizeof(val) && val == n * 2)
> + return 0; /* request worked */
> +
> + /* try again to override kernel restriction */
> + return setsockopt(fd, SOL_SOCKET, SO_RCVBUFFORCE, &n, sizeof(n)); }
> +
> static int
> dev_uev_socket_fd_create(void)
> {
> @@ -127,6 +153,9 @@ dev_uev_socket_fd_create(void)
> goto err;
> }
>
> + if (dev_uev_set_rcvbuf(fd, EAL_UEV_MSG_RCVBUF) < 0)
> + EAL_LOG(NOTICE, "Failed to set rcvbuf.");
> +
> if (rte_intr_fd_set(intr_handle, fd))
> goto err;
>
> --
> 2.53.0
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 3/4] eal/linux: report uevent socket overrun
2026-08-19 19:08 ` [PATCH v5 0/4] eal: uevent overrun mitigation Stephen Hemminger
2026-08-19 19:08 ` [PATCH v5 1/4] eal/linux: filter uevent Netlink in kernel Stephen Hemminger
2026-08-19 19:08 ` [PATCH v5 2/4] eal/linux: increase uevent socket buffer Stephen Hemminger
@ 2026-08-19 19:08 ` Stephen Hemminger
2026-08-20 1:41 ` [EXTERNAL] " Long Li
2026-08-19 19:08 ` [PATCH v5 4/4] doc: add warning about control threads Stephen Hemminger
2026-08-20 0:27 ` [EXTERNAL] [PATCH v5 0/4] eal: uevent overrun mitigation Long Li
4 siblings, 1 reply; 15+ messages in thread
From: Stephen Hemminger @ 2026-08-19 19:08 UTC (permalink / raw)
To: dev; +Cc: Long Li, Stephen Hemminger, stable, Randy Tice, Jianfeng Tan,
Jeff Guo
Make uevent handler handle case where kernel netlink receive
buffer is overrun. If this happens the DPDK device state
will potentially be out of sync, but better not to give up.
The code already handles the case of kernel sending a zero
length message; it just ignores it.
Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
Cc: stable@dpdk.org
Reported-by: Randy Tice <rtice@cisco.com>
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
lib/eal/linux/eal_dev.c | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index fb5c8bf070..b535122ff6 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -270,11 +270,20 @@ dev_uev_handler(__rte_unused void *param)
ret = recv(rte_intr_fd_get(intr_handle), buf, EAL_UEV_MSG_LEN,
MSG_DONTWAIT);
- if (ret < 0 && errno == EAGAIN)
- return;
- else if (ret <= 0) {
- /* connection is closed or broken, can not up again. */
- EAL_LOG(ERR, "uevent socket connection is broken.");
+ if (ret < 0) {
+ /* transient error */
+ if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
+ return;
+
+ /* kernel netlink messages lost */
+ if (errno == ENOBUFS) {
+ EAL_LOG(NOTICE, "kernel receive buffer overrun");
+ return;
+ }
+
+ EAL_LOG(ERR, "unexpected error on uevent recv: %s",
+ strerror(errno));
+
rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
return;
}
--
2.53.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* RE: [EXTERNAL] [PATCH v5 3/4] eal/linux: report uevent socket overrun
2026-08-19 19:08 ` [PATCH v5 3/4] eal/linux: report uevent socket overrun Stephen Hemminger
@ 2026-08-20 1:41 ` Long Li
0 siblings, 0 replies; 15+ messages in thread
From: Long Li @ 2026-08-20 1:41 UTC (permalink / raw)
To: Stephen Hemminger, dev@dpdk.org
Cc: stable@dpdk.org, Randy Tice, Jianfeng Tan, Jeff Guo
>
> Make uevent handler handle case where kernel netlink receive buffer is
> overrun. If this happens the DPDK device state will potentially be out of sync,
> but better not to give up.
>
> The code already handles the case of kernel sending a zero length message; it
> just ignores it.
>
> Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
> Cc: stable@dpdk.org
>
> Reported-by: Randy Tice <rtice@cisco.com>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by: Long Li <longli@microsoft.com>
> ---
> lib/eal/linux/eal_dev.c | 19 ++++++++++++++-----
> 1 file changed, 14 insertions(+), 5 deletions(-)
>
> diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c index
> fb5c8bf070..b535122ff6 100644
> --- a/lib/eal/linux/eal_dev.c
> +++ b/lib/eal/linux/eal_dev.c
> @@ -270,11 +270,20 @@ dev_uev_handler(__rte_unused void *param)
>
> ret = recv(rte_intr_fd_get(intr_handle), buf, EAL_UEV_MSG_LEN,
> MSG_DONTWAIT);
> - if (ret < 0 && errno == EAGAIN)
> - return;
> - else if (ret <= 0) {
> - /* connection is closed or broken, can not up again. */
> - EAL_LOG(ERR, "uevent socket connection is broken.");
> + if (ret < 0) {
> + /* transient error */
> + if (errno == EAGAIN || errno == EWOULDBLOCK || errno ==
> EINTR)
> + return;
> +
> + /* kernel netlink messages lost */
> + if (errno == ENOBUFS) {
> + EAL_LOG(NOTICE, "kernel receive buffer overrun");
> + return;
> + }
> +
> + EAL_LOG(ERR, "unexpected error on uevent recv: %s",
> + strerror(errno));
> +
> rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
> return;
> }
> --
> 2.53.0
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v5 4/4] doc: add warning about control threads
2026-08-19 19:08 ` [PATCH v5 0/4] eal: uevent overrun mitigation Stephen Hemminger
` (2 preceding siblings ...)
2026-08-19 19:08 ` [PATCH v5 3/4] eal/linux: report uevent socket overrun Stephen Hemminger
@ 2026-08-19 19:08 ` Stephen Hemminger
2026-08-20 0:27 ` [EXTERNAL] [PATCH v5 0/4] eal: uevent overrun mitigation Long Li
4 siblings, 0 replies; 15+ messages in thread
From: Stephen Hemminger @ 2026-08-19 19:08 UTC (permalink / raw)
To: dev; +Cc: Long Li, Stephen Hemminger, Anatoly Burakov
It is not obvious enough that control threads must run for
DPDK to work correctly. Add a caveat to EAL documentation.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
doc/guides/prog_guide/env_abstraction_layer.rst | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 0e1d044190..f36f9956c6 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -819,6 +819,20 @@ controlled with tools like taskset (Linux) or cpuset (FreeBSD),
- with affinity restricted to 2-3, the Control Threads will end up on
CPU 2 (main lcore, which is the default when no CPU is available).
+DPDK uses control threads internally and those threads need to be able to run.
+If all available CPUs are used as dataplane lcores,
+control threads fall back to the main lcore and compete with a busy polling loop.
+Ensure that at least a part of one CPU that is available for handling control events.
+
+The effects of control thread starvation are not always obvious,
+and include delayed alarms, missed device and hotplug events, unresponsive telemetry,
+and multi-process requests timing out so that secondary processes fail to start.
+
+.. warning::
+ On Linux, if DPDK lcore threads run under a real-time scheduling policy
+ such as ``SCHED_FIFO`` or ``SCHED_RR`` on the same CPU as a control thread,
+ then kernel events will be missed.
+
.. _eal_known_issue_label:
Known Issues
--
2.53.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* RE: [EXTERNAL] [PATCH v5 0/4] eal: uevent overrun mitigation
2026-08-19 19:08 ` [PATCH v5 0/4] eal: uevent overrun mitigation Stephen Hemminger
` (3 preceding siblings ...)
2026-08-19 19:08 ` [PATCH v5 4/4] doc: add warning about control threads Stephen Hemminger
@ 2026-08-20 0:27 ` Long Li
4 siblings, 0 replies; 15+ messages in thread
From: Long Li @ 2026-08-20 0:27 UTC (permalink / raw)
To: Stephen Hemminger, dev@dpdk.org
> This set of patches is an extension of the proposed patch to handle netlink
> uevent socket overrun reported by Randy Tice.
> Mostly impacts users of Azure VF events (MLX5 and MANA).
>
> DPDK uses netlink uevent to track hotplug events reported by the kernel. Since
> this is a one-way non-blocking channel there is no fool proof way to prevent
> missing events if DPDK interrupt thread does not keep up.
>
> What this series does is:
> - use netlink group to only receive events from kernel
> - increase socket buffer to have more headroom
> - report the problem more clearly in log
> - document requirement for control threads
I tested the patch series for MANA. The test passed.
>
> Stephen Hemminger (4):
> eal/linux: filter uevent Netlink in kernel
> eal/linux: increase uevent socket buffer
> eal/linux: report uevent socket overrun
> doc: add warning about control threads
>
> .mailmap | 1 +
> .../prog_guide/env_abstraction_layer.rst | 14 +++++
> lib/eal/linux/eal_dev.c | 56 ++++++++++++++++---
> 3 files changed, 62 insertions(+), 9 deletions(-)
>
> --
> 2.53.0
^ permalink raw reply [flat|nested] 15+ messages in thread