[PATCH v2] tools: hv: ignore ENOBUFS and ENOMEM in the KVP daemon

[PATCH v2] tools: hv: ignore ENOBUFS and ENOMEM in the KVP daemon

[Dexuan Cui]

Under high memory pressure and very high KVP R/W test pressure, the netlink
recvfrom() may transiently return ENOBUFS to the daemon -- we found this
during a 2-week stress test.

We'd better not terminate the daemon on the failure, because a typical KVP
user will re-try the R/W and hopefully it will succeed next time.

We can also ignore the errors on sending.

Cc: Vitaly Kuznetsov <vkuznets@redhat.com>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Signed-off-by: Dexuan Cui <decui@microsoft.com>
---

v2: I also ignore the errors on sending, as Vitaly suggested.

 tools/hv/hv_kvp_daemon.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
index 22b0764..6a6432a 100644
--- a/tools/hv/hv_kvp_daemon.c
+++ b/tools/hv/hv_kvp_daemon.c
@@ -1559,8 +1559,15 @@ int main(int argc, char *argv[])
 				addr_p, &addr_l);
 
 		if (len < 0) {
+			int saved_errno = errno;
 			syslog(LOG_ERR, "recvfrom failed; pid:%u error:%d %s",
 					addr.nl_pid, errno, strerror(errno));
+
+			if (saved_errno == ENOBUFS) {
+				syslog(LOG_ERR, "receive error: ignored");
+				continue;
+			}
+
 			close(fd);
 			return -1;
 		}
@@ -1763,8 +1770,15 @@ kvp_done:
 
 		len = netlink_send(fd, incoming_cn_msg);
 		if (len < 0) {
+			int saved_errno = errno;
 			syslog(LOG_ERR, "net_link send failed; error: %d %s", errno,
 					strerror(errno));
+
+			if (saved_errno == ENOMEM || saved_errno == ENOBUFS) {
+				syslog(LOG_ERR, "send error: ignored");
+				continue;
+			}
+
 			exit(EXIT_FAILURE);
 		}
 	}
-- 
1.9.1

Re: [PATCH v2] tools: hv: ignore ENOBUFS and ENOMEM in the KVP daemon

[Vitaly Kuznetsov]

Dexuan Cui <decui@microsoft.com> writes:

> Under high memory pressure and very high KVP R/W test pressure, the netlink
> recvfrom() may transiently return ENOBUFS to the daemon -- we found this
> during a 2-week stress test.
>
> We'd better not terminate the daemon on the failure, because a typical KVP
> user will re-try the R/W and hopefully it will succeed next time.
>
> We can also ignore the errors on sending.
>
> Cc: Vitaly Kuznetsov <vkuznets@redhat.com>
> Cc: K. Y. Srinivasan <kys@microsoft.com>
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> ---
>
> v2: I also ignore the errors on sending, as Vitaly suggested.

Thanks,

Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>

>
>  tools/hv/hv_kvp_daemon.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
> index 22b0764..6a6432a 100644
> --- a/tools/hv/hv_kvp_daemon.c
> +++ b/tools/hv/hv_kvp_daemon.c
> @@ -1559,8 +1559,15 @@ int main(int argc, char *argv[])
>  				addr_p, &addr_l);
>
>  		if (len < 0) {
> +			int saved_errno = errno;
>  			syslog(LOG_ERR, "recvfrom failed; pid:%u error:%d %s",
>  					addr.nl_pid, errno, strerror(errno));
> +
> +			if (saved_errno == ENOBUFS) {
> +				syslog(LOG_ERR, "receive error: ignored");
> +				continue;
> +			}
> +
>  			close(fd);
>  			return -1;
>  		}
> @@ -1763,8 +1770,15 @@ kvp_done:
>
>  		len = netlink_send(fd, incoming_cn_msg);
>  		if (len < 0) {
> +			int saved_errno = errno;
>  			syslog(LOG_ERR, "net_link send failed; error: %d %s", errno,
>  					strerror(errno));
> +
> +			if (saved_errno == ENOMEM || saved_errno == ENOBUFS) {
> +				syslog(LOG_ERR, "send error: ignored");
> +				continue;
> +			}
> +
>  			exit(EXIT_FAILURE);
>  		}
>  	}

-- 
  Vitaly