[PATCH] eal/linux: harden uevent recv error handling

[PATCH] eal/linux: harden uevent recv error handling

[Randy Tice]

The Linux uevent handler is harded for non-blocking receive behavior

Signed-off-by: Randy Tice <rtice@cisco.com>
---
 lib/eal/linux/eal_dev.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index ec408649d0..29a889800a 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -241,9 +241,14 @@ dev_uev_handler(__rte_unused void *param)
 
 	ret = recv(rte_intr_fd_get(intr_handle), buf, EAL_UEV_MSG_LEN,
 		   MSG_DONTWAIT);
-	if (ret < 0 && errno == EAGAIN)
+	if (ret < 0 &&
+		(errno == EAGAIN || errno == EWOULDBLOCK))
 		return;
-	else if (ret <= 0) {
+	else if (ret < 0 &&
+		 (errno == ENOBUFS || errno == ENOMEM)) {
+		EAL_LOG(ERR, "unexpected error on uvent recv: %d", errno);
+		return;
+	} else if (ret <= 0) {
 		/* connection is closed or broken, can not up again. */
 		EAL_LOG(ERR, "uevent socket connection is broken.");
 		rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
-- 
2.51.0

Re: [PATCH] eal/linux: harden uevent recv error handling

[Stephen Hemminger]

On Thu, 13 Aug 2026 16:16:56 -0400
Randy Tice <rtice@cisco.com> wrote:

> The Linux uevent handler is harded for non-blocking receive behavior
> 
> Signed-off-by: Randy Tice <rtice@cisco.com>
> ---

Good idea, but the code needs to unregister the uevent fd on error.
Also, fix spelling error, decode error message, and handle odd
case where recv() got interrupted.

Something like (untested):

diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index ec408649d0..1985d5e122 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -241,11 +241,17 @@ dev_uev_handler(__rte_unused void *param)
 
 	ret = recv(rte_intr_fd_get(intr_handle), buf, EAL_UEV_MSG_LEN,
 		   MSG_DONTWAIT);
-	if (ret < 0 && errno == EAGAIN)
-		return;
-	else if (ret <= 0) {
-		/* connection is closed or broken, can not up again. */
-		EAL_LOG(ERR, "uevent socket connection is broken.");
+	if (ret <= 0) {
+		if (ret < 0) {
+			/* non blocking or interrupted */
+			if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
+				return;
+			EAL_ERR(ERR, "unexpected error on uevent: %s",
+				strerror(errno));
+		} else {
+			/* zero length recv is end of file */
+			EAL_LOG(ERR, "uevent socket connection is broken.");
+		}
 		rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
 		return;
 	}

[PATCH v2] eal/linux: harden uevent recv error handling

[Randy Tice]

The Linux uevent handler is harded for non-blocking receive behavior
and transient ENOBUF issue during hot plug/unplug testing with MANA.

Signed-off-by: Randy Tice <rtice@cisco.com>
---

v2:
Addressed review comments regarding spelling and missing code comments.
Limited the transient memory issue to just ENOBUF which was the only
original error observed.
---
 lib/eal/linux/eal_dev.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index ec408649d0..f1d2bc3d34 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -241,9 +241,16 @@ dev_uev_handler(__rte_unused void *param)
 
 	ret = recv(rte_intr_fd_get(intr_handle), buf, EAL_UEV_MSG_LEN,
 		   MSG_DONTWAIT);
-	if (ret < 0 && errno == EAGAIN)
+	if (ret < 0 &&
+		(errno == EAGAIN || errno == EWOULDBLOCK || EINTR)) {
+		/* non-blocking or interrupted */
 		return;
-	else if (ret <= 0) {
+	} else if (ret < 0 && (errno == ENOBUFS)) {
+		/* non-fatal transient memory condition */
+		EAL_LOG(ERR, "unexpected error on uevent recv: %s",
+			strerror(errno));
+		return;
+	} else if (ret <= 0) {
 		/* connection is closed or broken, can not up again. */
 		EAL_LOG(ERR, "uevent socket connection is broken.");
 		rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
-- 
2.51.0

[PATCH v3] eal/linux: harden uevent recv error handling

[Randy Tice]

The Linux uevent handler is harded for non-blocking receive behavior
and transient ENOBUF issue during hot plug/unplug testing with MANA.

Signed-off-by: Randy Tice <rtice@cisco.com>
---

v3:
Fixed build issue with bitwise operator vs test against errno.

v2:
Addressed review comments regarding spelling and missing code comments.
Limited the transient memory issue to just ENOBUF which was the only
original error observed.
---
 lib/eal/linux/eal_dev.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index ec408649d0..f1d2bc3d34 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -241,9 +241,16 @@ dev_uev_handler(__rte_unused void *param)
 
 	ret = recv(rte_intr_fd_get(intr_handle), buf, EAL_UEV_MSG_LEN,
 		   MSG_DONTWAIT);
-	if (ret < 0 && errno == EAGAIN)
+	if (ret < 0 &&
+		(errno == EAGAIN || errno == EWOULDBLOCK || EINTR)) {
+		/* non-blocking or interrupted */
 		return;
-	else if (ret <= 0) {
+	} else if (ret < 0 && (errno == ENOBUFS)) {
+		/* non-fatal transient memory condition */
+		EAL_LOG(ERR, "unexpected error on uevent recv: %s",
+			strerror(errno));
+		return;
+	} else if (ret <= 0) {
 		/* connection is closed or broken, can not up again. */
 		EAL_LOG(ERR, "uevent socket connection is broken.");
 		rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
-- 
2.51.0

[PATCH v4] eal/linux: harden uevent recv error handling

[Randy Tice]

The Linux uevent handler is harded for non-blocking receive behavior
and transient ENOBUF issue during hot plug/unplug testing with MANA.

Signed-off-by: Randy Tice <rtice@cisco.com>
---

v4:
Wrong patch submitted

v3:
Fixed build issue with bitwise operator vs test against errno.

v2:
Addressed review comments regarding spelling and missing code comments.
Limited the transient memory issue to just ENOBUF which was the only
original error observed.
---
 lib/eal/linux/eal_dev.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index ec408649d0..5b3e9dea9f 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -241,9 +241,16 @@ dev_uev_handler(__rte_unused void *param)
 
 	ret = recv(rte_intr_fd_get(intr_handle), buf, EAL_UEV_MSG_LEN,
 		   MSG_DONTWAIT);
-	if (ret < 0 && errno == EAGAIN)
+	if (ret < 0 &&
+		(errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)) {
+		/* non-blocking or interrupted */
 		return;
-	else if (ret <= 0) {
+	} else if (ret < 0 && (errno == ENOBUFS)) {
+		/* non-fatal transient memory condition */
+		EAL_LOG(ERR, "unexpected error on uevent recv: %s",
+			strerror(errno));
+		return;
+	} else if (ret <= 0) {
 		/* connection is closed or broken, can not up again. */
 		EAL_LOG(ERR, "uevent socket connection is broken.");
 		rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
-- 
2.51.0

Re: [PATCH v4] eal/linux: harden uevent recv error handling

[Stephen Hemminger]

On Mon, 17 Aug 2026 16:16:02 -0400
Randy Tice <rtice@cisco.com> wrote:

> -	if (ret < 0 && errno == EAGAIN)
> +	if (ret < 0 &&
> +		(errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)) {
> +		/* non-blocking or interrupted */
>  		return;
> -	else if (ret <= 0) {
> +	} else if (ret < 0 && (errno == ENOBUFS)) {
> +		/* non-fatal transient memory condition */
> +		EAL_LOG(ERR, "unexpected error on uevent recv: %s",
> +			strerror(errno));
> +		return;
> +	} else if (ret <= 0) {
>  		/* connection is closed or broken, can not up again. */
>  		EAL_LOG(ERR, "uevent socket connection is broken.");
>  		rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
> -- 
Since there are multiple error conditions it reads better with single (ret < 0) if followed
by looking at errno.

PS: if you are getting ENOBUFS, the root cause is not having big enough socket receive
buffer and/or starving out the core handling control operations.

If you looks a uevent the internal state of devices is corrupted. Probably need
to figure out how to schedule some form of rescanning.

[PATCH v5 0/4] eal: uevent overrun mitigation

[Stephen Hemminger]

This set of patches is an extension of the proposed patch
to handle netlink uevent socket overrun reported by Randy Tice.
Mostly impacts users of Azure VF events (MLX5 and MANA).

DPDK uses netlink uevent to track hotplug events reported
by the kernel. Since this is a one-way non-blocking channel
there is no fool proof way to prevent missing events if DPDK
interrupt thread does not keep up.

What this series does is:
  - use netlink group to only receive events from kernel
  - increase socket buffer to have more headroom
  - report the problem more clearly in log
  - document requirement for control threads

Stephen Hemminger (4):
  eal/linux: filter uevent Netlink in kernel
  eal/linux: increase uevent socket buffer
  eal/linux: report uevent socket overrun
  doc: add warning about control threads

 .mailmap                                      |  1 +
 .../prog_guide/env_abstraction_layer.rst      | 14 +++++
 lib/eal/linux/eal_dev.c                       | 56 ++++++++++++++++---
 3 files changed, 62 insertions(+), 9 deletions(-)

-- 
2.53.0

[PATCH v5 1/4] eal/linux: filter uevent Netlink in kernel

[Stephen Hemminger]

The EAL uevent only needs/wants messages from the kernel.
Better to let kernel side do the filtering to avoid any overrun
issues in DPDK interrupt thread.

There is no exposed API definition here. The convention
is kernel is 1 and libudev uses 2.

Since kernel is now filtering can remove step in uevent parsing.

Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
Cc: stable@dpdk.org

Reported-by: Randy Tice <rtice@cisco.com>
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 .mailmap                | 1 +
 lib/eal/linux/eal_dev.c | 8 ++++----
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/.mailmap b/.mailmap
index fcb3d1bb3f..9b7713b6df 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1380,6 +1380,7 @@ Ralf Hoffmann <ralf.hoffmann@allegro-packets.com>
 Rami Rosen <ramirose@gmail.com> <rami.rosen@intel.com>
 Rami Rosen <ramirose@gmail.com> <roszenrami@gmail.com>
 Randy Schacher <stuart.schacher@broadcom.com>
+Randy Tice <rtice@cisco.com>
 Rani Sharoni <ranish@nvidia.com>
 Ranjit Menon <ranjit.menon@intel.com>
 Rasesh Mody <rmody@marvell.com> <rasesh.mody@cavium.com>
diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index ec408649d0..666967667f 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -29,6 +29,9 @@ static bool hotplug_handle;
 #define EAL_UEV_MSG_LEN 4096
 #define EAL_UEV_MSG_ELEM_LEN 128
 
+/* Listen only to messages from kernel (not libudev) */
+#define EAL_UEV_GROUP_KERNEL 1
+
 /*
  * spinlock for device hot-unplug failure handling. If it try to access bus or
  * device, such as handle sigbus on bus or handle memory failure for device
@@ -116,7 +119,7 @@ dev_uev_socket_fd_create(void)
 	memset(&addr, 0, sizeof(addr));
 	addr.nl_family = AF_NETLINK;
 	addr.nl_pid = 0;
-	addr.nl_groups = 0xffffffff;
+	addr.nl_groups = EAL_UEV_GROUP_KERNEL;
 
 	ret = bind(fd, (struct sockaddr *) &addr, sizeof(addr));
 	if (ret < 0) {
@@ -164,9 +167,6 @@ dev_uev_parse(const char *buf, struct rte_dev_event *event, int length)
 		 * check device uevent from kernel side, no need to check
 		 * uevent from udev.
 		 */
-		if (!strncmp(buf, "libudev", 7)) {
-			return -1;
-		}
 		if (!strncmp(buf, "ACTION=", 7)) {
 			buf += 7;
 			i += 7;
-- 
2.53.0

RE: [EXTERNAL] [PATCH v5 1/4] eal/linux: filter uevent Netlink in kernel

[Long Li]

> The EAL uevent only needs/wants messages from the kernel.
> Better to let kernel side do the filtering to avoid any overrun issues in DPDK
> interrupt thread.
> 
> There is no exposed API definition here. The convention is kernel is 1 and
> libudev uses 2.
> 
> Since kernel is now filtering can remove step in uevent parsing.
> 
> Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
> Cc: stable@dpdk.org
> 
> Reported-by: Randy Tice <rtice@cisco.com>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>

Reviewed-by: Long Li <longli@microsoft.com>




> ---
>  .mailmap                | 1 +
>  lib/eal/linux/eal_dev.c | 8 ++++----
>  2 files changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/.mailmap b/.mailmap
> index fcb3d1bb3f..9b7713b6df 100644
> --- a/.mailmap
> +++ b/.mailmap
> @@ -1380,6 +1380,7 @@ Ralf Hoffmann <ralf.hoffmann@allegro-
> packets.com>
>  Rami Rosen <ramirose@gmail.com> <rami.rosen@intel.com>  Rami Rosen
> <ramirose@gmail.com> <roszenrami@gmail.com>  Randy Schacher
> <stuart.schacher@broadcom.com>
> +Randy Tice <rtice@cisco.com>
>  Rani Sharoni <ranish@nvidia.com>
>  Ranjit Menon <ranjit.menon@intel.com>
>  Rasesh Mody <rmody@marvell.com> <rasesh.mody@cavium.com> diff --git
> a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c index
> ec408649d0..666967667f 100644
> --- a/lib/eal/linux/eal_dev.c
> +++ b/lib/eal/linux/eal_dev.c
> @@ -29,6 +29,9 @@ static bool hotplug_handle;  #define EAL_UEV_MSG_LEN
> 4096  #define EAL_UEV_MSG_ELEM_LEN 128
> 
> +/* Listen only to messages from kernel (not libudev) */ #define
> +EAL_UEV_GROUP_KERNEL 1
> +
>  /*
>   * spinlock for device hot-unplug failure handling. If it try to access bus or
>   * device, such as handle sigbus on bus or handle memory failure for device
> @@ -116,7 +119,7 @@ dev_uev_socket_fd_create(void)
>  	memset(&addr, 0, sizeof(addr));
>  	addr.nl_family = AF_NETLINK;
>  	addr.nl_pid = 0;
> -	addr.nl_groups = 0xffffffff;
> +	addr.nl_groups = EAL_UEV_GROUP_KERNEL;
> 
>  	ret = bind(fd, (struct sockaddr *) &addr, sizeof(addr));
>  	if (ret < 0) {
> @@ -164,9 +167,6 @@ dev_uev_parse(const char *buf, struct rte_dev_event
> *event, int length)
>  		 * check device uevent from kernel side, no need to check
>  		 * uevent from udev.
>  		 */
> -		if (!strncmp(buf, "libudev", 7)) {
> -			return -1;
> -		}
>  		if (!strncmp(buf, "ACTION=", 7)) {
>  			buf += 7;
>  			i += 7;
> --
> 2.53.0

[PATCH v5 2/4] eal/linux: increase uevent socket buffer

[Stephen Hemminger]

The kernel netlink socket into DPDK can get overrun if DPDK
interrupt thread is not keeping up. To reduce the possibility
increase the socket receive buffer to 4M.

Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
Cc: stable@dpdk.org

Reported-by: Randy Tice <rtice@cisco.com>
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 lib/eal/linux/eal_dev.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index 666967667f..fb5c8bf070 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -32,6 +32,9 @@ static bool hotplug_handle;
 /* Listen only to messages from kernel (not libudev) */
 #define EAL_UEV_GROUP_KERNEL 1
 
+/* Large uevent buffer */
+#define EAL_UEV_MSG_RCVBUF (4 * 1024 * 1024)
+
 /*
  * spinlock for device hot-unplug failure handling. If it try to access bus or
  * device, such as handle sigbus on bus or handle memory failure for device
@@ -103,6 +106,29 @@ static int cmp_dev_name(const struct rte_device *dev,
 	return strcmp(dev->name, name);
 }
 
+/*
+ * To avoid losing uevents increase the netlink receive buffer size,
+ * and override the kernel clamp value if necessary.
+ */
+static int
+dev_uev_set_rcvbuf(int fd, int n)
+{
+	int ret, val;
+	socklen_t len = sizeof(val);
+
+	ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &n, sizeof(n));
+	if (ret < 0)
+		return ret;
+
+	/* kernel may have clamped our request */
+	ret = getsockopt(fd, SOL_SOCKET, SO_RCVBUF, &val, &len);
+	if (ret >= 0 && len == sizeof(val) && val == n * 2)
+		return 0; /* request worked */
+
+	/* try again to override kernel restriction */
+	return setsockopt(fd, SOL_SOCKET, SO_RCVBUFFORCE, &n, sizeof(n));
+}
+
 static int
 dev_uev_socket_fd_create(void)
 {
@@ -127,6 +153,9 @@ dev_uev_socket_fd_create(void)
 		goto err;
 	}
 
+	if (dev_uev_set_rcvbuf(fd, EAL_UEV_MSG_RCVBUF) < 0)
+		EAL_LOG(NOTICE, "Failed to set rcvbuf.");
+
 	if (rte_intr_fd_set(intr_handle, fd))
 		goto err;
 
-- 
2.53.0

RE: [EXTERNAL] [PATCH v5 2/4] eal/linux: increase uevent socket buffer

[Long Li]

> The kernel netlink socket into DPDK can get overrun if DPDK interrupt thread is
> not keeping up. To reduce the possibility increase the socket receive buffer to
> 4M.
> 
> Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
> Cc: stable@dpdk.org
> 
> Reported-by: Randy Tice <rtice@cisco.com>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>

Reviewed-by: Long Li <longli@microsoft.com>


> ---
>  lib/eal/linux/eal_dev.c | 29 +++++++++++++++++++++++++++++
>  1 file changed, 29 insertions(+)
> 
> diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c index
> 666967667f..fb5c8bf070 100644
> --- a/lib/eal/linux/eal_dev.c
> +++ b/lib/eal/linux/eal_dev.c
> @@ -32,6 +32,9 @@ static bool hotplug_handle;
>  /* Listen only to messages from kernel (not libudev) */  #define
> EAL_UEV_GROUP_KERNEL 1
> 
> +/* Large uevent buffer */
> +#define EAL_UEV_MSG_RCVBUF (4 * 1024 * 1024)
> +
>  /*
>   * spinlock for device hot-unplug failure handling. If it try to access bus or
>   * device, such as handle sigbus on bus or handle memory failure for device
> @@ -103,6 +106,29 @@ static int cmp_dev_name(const struct rte_device
> *dev,
>  	return strcmp(dev->name, name);
>  }
> 
> +/*
> + * To avoid losing uevents increase the netlink receive buffer size,
> + * and override the kernel clamp value if necessary.
> + */
> +static int
> +dev_uev_set_rcvbuf(int fd, int n)
> +{
> +	int ret, val;
> +	socklen_t len = sizeof(val);
> +
> +	ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &n, sizeof(n));
> +	if (ret < 0)
> +		return ret;
> +
> +	/* kernel may have clamped our request */
> +	ret = getsockopt(fd, SOL_SOCKET, SO_RCVBUF, &val, &len);
> +	if (ret >= 0 && len == sizeof(val) && val == n * 2)
> +		return 0; /* request worked */
> +
> +	/* try again to override kernel restriction */
> +	return setsockopt(fd, SOL_SOCKET, SO_RCVBUFFORCE, &n, sizeof(n)); }
> +
>  static int
>  dev_uev_socket_fd_create(void)
>  {
> @@ -127,6 +153,9 @@ dev_uev_socket_fd_create(void)
>  		goto err;
>  	}
> 
> +	if (dev_uev_set_rcvbuf(fd, EAL_UEV_MSG_RCVBUF) < 0)
> +		EAL_LOG(NOTICE, "Failed to set rcvbuf.");
> +
>  	if (rte_intr_fd_set(intr_handle, fd))
>  		goto err;
> 
> --
> 2.53.0

[PATCH v5 3/4] eal/linux: report uevent socket overrun

[Stephen Hemminger]

Make uevent handler handle case where kernel netlink receive
buffer is overrun. If this happens the DPDK device state
will potentially be out of sync, but better not to give up.

The code already handles the case of kernel sending a zero
length message; it just ignores it.

Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
Cc: stable@dpdk.org

Reported-by: Randy Tice <rtice@cisco.com>
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 lib/eal/linux/eal_dev.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c
index fb5c8bf070..b535122ff6 100644
--- a/lib/eal/linux/eal_dev.c
+++ b/lib/eal/linux/eal_dev.c
@@ -270,11 +270,20 @@ dev_uev_handler(__rte_unused void *param)
 
 	ret = recv(rte_intr_fd_get(intr_handle), buf, EAL_UEV_MSG_LEN,
 		   MSG_DONTWAIT);
-	if (ret < 0 && errno == EAGAIN)
-		return;
-	else if (ret <= 0) {
-		/* connection is closed or broken, can not up again. */
-		EAL_LOG(ERR, "uevent socket connection is broken.");
+	if (ret < 0) {
+		/* transient error */
+		if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)
+			return;
+
+		/* kernel netlink messages lost */
+		if (errno == ENOBUFS) {
+			EAL_LOG(NOTICE, "kernel receive buffer overrun");
+			return;
+		}
+
+		EAL_LOG(ERR, "unexpected error on uevent recv: %s",
+			strerror(errno));
+
 		rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
 		return;
 	}
-- 
2.53.0

RE: [EXTERNAL] [PATCH v5 3/4] eal/linux: report uevent socket overrun

[Long Li]

> 
> Make uevent handler handle case where kernel netlink receive buffer is
> overrun. If this happens the DPDK device state will potentially be out of sync,
> but better not to give up.
> 
> The code already handles the case of kernel sending a zero length message; it
> just ignores it.
> 
> Fixes: 0d0f478d0483 ("eal/linux: add uevent parse and process")
> Cc: stable@dpdk.org
> 
> Reported-by: Randy Tice <rtice@cisco.com>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>

Reviewed-by: Long Li <longli@microsoft.com>


> ---
>  lib/eal/linux/eal_dev.c | 19 ++++++++++++++-----
>  1 file changed, 14 insertions(+), 5 deletions(-)
> 
> diff --git a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c index
> fb5c8bf070..b535122ff6 100644
> --- a/lib/eal/linux/eal_dev.c
> +++ b/lib/eal/linux/eal_dev.c
> @@ -270,11 +270,20 @@ dev_uev_handler(__rte_unused void *param)
> 
>  	ret = recv(rte_intr_fd_get(intr_handle), buf, EAL_UEV_MSG_LEN,
>  		   MSG_DONTWAIT);
> -	if (ret < 0 && errno == EAGAIN)
> -		return;
> -	else if (ret <= 0) {
> -		/* connection is closed or broken, can not up again. */
> -		EAL_LOG(ERR, "uevent socket connection is broken.");
> +	if (ret < 0) {
> +		/* transient error */
> +		if (errno == EAGAIN || errno == EWOULDBLOCK || errno ==
> EINTR)
> +			return;
> +
> +		/* kernel netlink messages lost */
> +		if (errno == ENOBUFS) {
> +			EAL_LOG(NOTICE, "kernel receive buffer overrun");
> +			return;
> +		}
> +
> +		EAL_LOG(ERR, "unexpected error on uevent recv: %s",
> +			strerror(errno));
> +
>  		rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
>  		return;
>  	}
> --
> 2.53.0

[PATCH v5 4/4] doc: add warning about control threads

[Stephen Hemminger]

It is not obvious enough that control threads must run for
DPDK to work correctly. Add a caveat to EAL documentation.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 doc/guides/prog_guide/env_abstraction_layer.rst | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 0e1d044190..f36f9956c6 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -819,6 +819,20 @@ controlled with tools like taskset (Linux) or cpuset (FreeBSD),
 - with affinity restricted to 2-3, the Control Threads will end up on
   CPU 2 (main lcore, which is the default when no CPU is available).
 
+DPDK uses control threads internally and those threads need to be able to run.
+If all available CPUs are used as dataplane lcores,
+control threads fall back to the main lcore and compete with a busy polling loop.
+Ensure that at least a part of one CPU that is available for handling control events.
+
+The effects of control thread starvation are not always obvious,
+and include delayed alarms, missed device and hotplug events, unresponsive telemetry,
+and multi-process requests timing out so that secondary processes fail to start.
+
+.. warning::
+   On Linux, if DPDK lcore threads run under a real-time scheduling policy
+   such as ``SCHED_FIFO`` or ``SCHED_RR`` on the same CPU as a control thread,
+   then kernel events will be missed.
+
 .. _eal_known_issue_label:
 
 Known Issues
-- 
2.53.0

RE: [EXTERNAL] [PATCH v5 0/4] eal: uevent overrun mitigation

[Long Li]

> This set of patches is an extension of the proposed patch to handle netlink
> uevent socket overrun reported by Randy Tice.
> Mostly impacts users of Azure VF events (MLX5 and MANA).
> 
> DPDK uses netlink uevent to track hotplug events reported by the kernel. Since
> this is a one-way non-blocking channel there is no fool proof way to prevent
> missing events if DPDK interrupt thread does not keep up.
> 
> What this series does is:
>   - use netlink group to only receive events from kernel
>   - increase socket buffer to have more headroom
>   - report the problem more clearly in log
>   - document requirement for control threads

I tested the patch series for MANA. The test passed.

> 
> Stephen Hemminger (4):
>   eal/linux: filter uevent Netlink in kernel
>   eal/linux: increase uevent socket buffer
>   eal/linux: report uevent socket overrun
>   doc: add warning about control threads
> 
>  .mailmap                                      |  1 +
>  .../prog_guide/env_abstraction_layer.rst      | 14 +++++
>  lib/eal/linux/eal_dev.c                       | 56 ++++++++++++++++---
>  3 files changed, 62 insertions(+), 9 deletions(-)
> 
> --
> 2.53.0