[PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[wangyunjian]

From: Yunjian Wang <wangyunjian@huawei.com>

Currently the driver doesn't drop a packet which can't be sent by tun
(e.g bad packet). In this case, the driver will always process the
same packet lead to the tx queue stuck.

To fix this issue:
1. in the case of persistent failure (e.g bad packet), the driver
   can skip this descriptor by ignoring the error.
2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
   the driver schedules the worker to try again.

Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")
Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
Acked-by: Willem de Bruijn <willemb@google.com>
---
 drivers/vhost/net.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index c8784dfafdd7..01558fb2c552 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock)
 				msg.msg_flags &= ~MSG_MORE;
 		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
-		if (unlikely(err < 0)) {
+		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS)) {
 			vhost_discard_vq_desc(vq, 1);
 			vhost_net_enable_vq(net, vq);
 			break;
 		}
-		if (err != len)
+		if (err >= 0 && err != len)
 			pr_debug("Truncated TX packet: len %d != %zd\n",
 				 err, len);
 done:
@@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 			msg.msg_flags &= ~MSG_MORE;
 		}
 
-		/* TODO: Check specific error and bomb out unless ENOBUFS? */
 		err = sock->ops->sendmsg(sock, &msg, len);
 		if (unlikely(err < 0)) {
 			if (zcopy_used) {
@@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
 				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
 					% UIO_MAXIOV;
 			}
-			vhost_discard_vq_desc(vq, 1);
-			vhost_net_enable_vq(net, vq);
-			break;
+			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {
+				vhost_discard_vq_desc(vq, 1);
+				vhost_net_enable_vq(net, vq);
+				break;
+			}
 		}
-		if (err != len)
+		if (err >= 0 && err != len)
 			pr_debug("Truncated TX packet: "
 				 " len %d != %zd\n", err, len);
 		if (!zcopy_used)
-- 
2.23.0

Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[Michael S. Tsirkin]

On Fri, Dec 25, 2020 at 03:24:33PM +0800, wangyunjian wrote:
> From: Yunjian Wang <wangyunjian@huawei.com>
> 
> Currently the driver doesn't drop a packet which can't be sent by tun
> (e.g bad packet). In this case, the driver will always process the
> same packet lead to the tx queue stuck.
> 
> To fix this issue:
> 1. in the case of persistent failure (e.g bad packet), the driver
>    can skip this descriptor by ignoring the error.
> 2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
>    the driver schedules the worker to try again.
> 
> Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")
> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> Acked-by: Willem de Bruijn <willemb@google.com>
> ---
>  drivers/vhost/net.c | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index c8784dfafdd7..01558fb2c552 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock)
>  				msg.msg_flags &= ~MSG_MORE;
>  		}
>  
> -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
>  		err = sock->ops->sendmsg(sock, &msg, len);
> -		if (unlikely(err < 0)) {
> +		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS)) {
>  			vhost_discard_vq_desc(vq, 1);
>  			vhost_net_enable_vq(net, vq);
>  			break;
>  		}


Hmm, there's the case of link being temporarily down (e.g. for
reconfigure), which IIRC returns EIO.

It's also probably a good idea to keep unlikely(err < 0) around
and then just regular err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS
since that first test can be done faster.


> -		if (err != len)
> +		if (err >= 0 && err != len)
>  			pr_debug("Truncated TX packet: len %d != %zd\n",
>  				 err, len);
>  done:
> @@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
>  			msg.msg_flags &= ~MSG_MORE;
>  		}
>  
> -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
>  		err = sock->ops->sendmsg(sock, &msg, len);
>  		if (unlikely(err < 0)) {
>  			if (zcopy_used) {
> @@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
>  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
>  					% UIO_MAXIOV;
>  			}
> -			vhost_discard_vq_desc(vq, 1);
> -			vhost_net_enable_vq(net, vq);
> -			break;
> +			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {


same here

> +				vhost_discard_vq_desc(vq, 1);
> +				vhost_net_enable_vq(net, vq);
> +				break;
> +			}
>  		}
> -		if (err != len)
> +		if (err >= 0 && err != len)
>  			pr_debug("Truncated TX packet: "
>  				 " len %d != %zd\n", err, len);
>  		if (!zcopy_used)
> -- 
> 2.23.0

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[Michael S. Tsirkin]

On Fri, Dec 25, 2020 at 03:24:33PM +0800, wangyunjian wrote:
> From: Yunjian Wang <wangyunjian@huawei.com>
> 
> Currently the driver doesn't drop a packet which can't be sent by tun
> (e.g bad packet). In this case, the driver will always process the
> same packet lead to the tx queue stuck.
> 
> To fix this issue:
> 1. in the case of persistent failure (e.g bad packet), the driver
>    can skip this descriptor by ignoring the error.
> 2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
>    the driver schedules the worker to try again.
> 
> Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")
> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> Acked-by: Willem de Bruijn <willemb@google.com>
> ---
>  drivers/vhost/net.c | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index c8784dfafdd7..01558fb2c552 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock)
>  				msg.msg_flags &= ~MSG_MORE;
>  		}
>  
> -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
>  		err = sock->ops->sendmsg(sock, &msg, len);
> -		if (unlikely(err < 0)) {
> +		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS)) {
>  			vhost_discard_vq_desc(vq, 1);
>  			vhost_net_enable_vq(net, vq);
>  			break;
>  		}


Hmm, there's the case of link being temporarily down (e.g. for
reconfigure), which IIRC returns EIO.

It's also probably a good idea to keep unlikely(err < 0) around
and then just regular err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS
since that first test can be done faster.


> -		if (err != len)
> +		if (err >= 0 && err != len)
>  			pr_debug("Truncated TX packet: len %d != %zd\n",
>  				 err, len);
>  done:
> @@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
>  			msg.msg_flags &= ~MSG_MORE;
>  		}
>  
> -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
>  		err = sock->ops->sendmsg(sock, &msg, len);
>  		if (unlikely(err < 0)) {
>  			if (zcopy_used) {
> @@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
>  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
>  					% UIO_MAXIOV;
>  			}
> -			vhost_discard_vq_desc(vq, 1);
> -			vhost_net_enable_vq(net, vq);
> -			break;
> +			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {


same here

> +				vhost_discard_vq_desc(vq, 1);
> +				vhost_net_enable_vq(net, vq);
> +				break;
> +			}
>  		}
> -		if (err != len)
> +		if (err >= 0 && err != len)
>  			pr_debug("Truncated TX packet: "
>  				 " len %d != %zd\n", err, len);
>  		if (!zcopy_used)
> -- 
> 2.23.0

Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[Michael S. Tsirkin]

On Fri, Dec 25, 2020 at 03:24:33PM +0800, wangyunjian wrote:
> From: Yunjian Wang <wangyunjian@huawei.com>
> 
> Currently the driver doesn't drop a packet which can't be sent by tun
> (e.g bad packet). In this case, the driver will always process the
> same packet lead to the tx queue stuck.

So not making progress on a bad packet has some advantages,
e.g. this is easier to debug.
When is it important to drop the packet and continue?


> To fix this issue:
> 1. in the case of persistent failure (e.g bad packet), the driver
>    can skip this descriptor by ignoring the error.
> 2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
>    the driver schedules the worker to try again.
> 
> Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")

I'd just drop this tag, looks more like a feature than a bug ...


> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> Acked-by: Willem de Bruijn <willemb@google.com>
> ---
>  drivers/vhost/net.c | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index c8784dfafdd7..01558fb2c552 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock)
>  				msg.msg_flags &= ~MSG_MORE;
>  		}
>  
> -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
>  		err = sock->ops->sendmsg(sock, &msg, len);
> -		if (unlikely(err < 0)) {
> +		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS)) {
>  			vhost_discard_vq_desc(vq, 1);
>  			vhost_net_enable_vq(net, vq);
>  			break;
>  		}
> -		if (err != len)
> +		if (err >= 0 && err != len)
>  			pr_debug("Truncated TX packet: len %d != %zd\n",
>  				 err, len);
>  done:
> @@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
>  			msg.msg_flags &= ~MSG_MORE;
>  		}
>  
> -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
>  		err = sock->ops->sendmsg(sock, &msg, len);
>  		if (unlikely(err < 0)) {
>  			if (zcopy_used) {
> @@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
>  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
>  					% UIO_MAXIOV;
>  			}
> -			vhost_discard_vq_desc(vq, 1);
> -			vhost_net_enable_vq(net, vq);
> -			break;
> +			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {
> +				vhost_discard_vq_desc(vq, 1);
> +				vhost_net_enable_vq(net, vq);
> +				break;
> +			}
>  		}
> -		if (err != len)
> +		if (err >= 0 && err != len)
>  			pr_debug("Truncated TX packet: "
>  				 " len %d != %zd\n", err, len);
>  		if (!zcopy_used)
> -- 
> 2.23.0

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[Michael S. Tsirkin]

On Fri, Dec 25, 2020 at 03:24:33PM +0800, wangyunjian wrote:
> From: Yunjian Wang <wangyunjian@huawei.com>
> 
> Currently the driver doesn't drop a packet which can't be sent by tun
> (e.g bad packet). In this case, the driver will always process the
> same packet lead to the tx queue stuck.

So not making progress on a bad packet has some advantages,
e.g. this is easier to debug.
When is it important to drop the packet and continue?


> To fix this issue:
> 1. in the case of persistent failure (e.g bad packet), the driver
>    can skip this descriptor by ignoring the error.
> 2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
>    the driver schedules the worker to try again.
> 
> Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")

I'd just drop this tag, looks more like a feature than a bug ...


> Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> Acked-by: Willem de Bruijn <willemb@google.com>
> ---
>  drivers/vhost/net.c | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index c8784dfafdd7..01558fb2c552 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net, struct socket *sock)
>  				msg.msg_flags &= ~MSG_MORE;
>  		}
>  
> -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
>  		err = sock->ops->sendmsg(sock, &msg, len);
> -		if (unlikely(err < 0)) {
> +		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS)) {
>  			vhost_discard_vq_desc(vq, 1);
>  			vhost_net_enable_vq(net, vq);
>  			break;
>  		}
> -		if (err != len)
> +		if (err >= 0 && err != len)
>  			pr_debug("Truncated TX packet: len %d != %zd\n",
>  				 err, len);
>  done:
> @@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
>  			msg.msg_flags &= ~MSG_MORE;
>  		}
>  
> -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
>  		err = sock->ops->sendmsg(sock, &msg, len);
>  		if (unlikely(err < 0)) {
>  			if (zcopy_used) {
> @@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock)
>  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
>  					% UIO_MAXIOV;
>  			}
> -			vhost_discard_vq_desc(vq, 1);
> -			vhost_net_enable_vq(net, vq);
> -			break;
> +			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {
> +				vhost_discard_vq_desc(vq, 1);
> +				vhost_net_enable_vq(net, vq);
> +				break;
> +			}
>  		}
> -		if (err != len)
> +		if (err >= 0 && err != len)
>  			pr_debug("Truncated TX packet: "
>  				 " len %d != %zd\n", err, len);
>  		if (!zcopy_used)
> -- 
> 2.23.0

RE: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[wangyunjian]

> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst@redhat.com]
> Sent: Sunday, December 27, 2020 7:21 PM
> To: wangyunjian <wangyunjian@huawei.com>
> Cc: netdev@vger.kernel.org; jasowang@redhat.com;
> willemdebruijn.kernel@gmail.com; virtualization@lists.linux-foundation.org;
> Lilijun (Jerry) <jerry.lilijun@huawei.com>; chenchanghu
> <chenchanghu@huawei.com>; xudingke <xudingke@huawei.com>; huangbin (J)
> <brian.huangbin@huawei.com>
> Subject: Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg
> fails
> 
> On Fri, Dec 25, 2020 at 03:24:33PM +0800, wangyunjian wrote:
> > From: Yunjian Wang <wangyunjian@huawei.com>
> >
> > Currently the driver doesn't drop a packet which can't be sent by tun
> > (e.g bad packet). In this case, the driver will always process the
> > same packet lead to the tx queue stuck.
> 
> So not making progress on a bad packet has some advantages, e.g. this is
> easier to debug.
> When is it important to drop the packet and continue?

In the case, the VM will not be able to send packets persistently. Services of VM
are affected.

Thanks

> 
> 
> > To fix this issue:
> > 1. in the case of persistent failure (e.g bad packet), the driver
> >    can skip this descriptor by ignoring the error.
> > 2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
> >    the driver schedules the worker to try again.
> >
> > Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")
> 
> I'd just drop this tag, looks more like a feature than a bug ...

OK

> 
> 
> > Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> > Acked-by: Willem de Bruijn <willemb@google.com>
> > ---
> >  drivers/vhost/net.c | 16 ++++++++--------
> >  1 file changed, 8 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index
> > c8784dfafdd7..01558fb2c552 100644
> > --- a/drivers/vhost/net.c
> > +++ b/drivers/vhost/net.c
> > @@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net,
> struct socket *sock)
> >  				msg.msg_flags &= ~MSG_MORE;
> >  		}
> >
> > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> >  		err = sock->ops->sendmsg(sock, &msg, len);
> > -		if (unlikely(err < 0)) {
> > +		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS))
> > +{
> >  			vhost_discard_vq_desc(vq, 1);
> >  			vhost_net_enable_vq(net, vq);
> >  			break;
> >  		}
> > -		if (err != len)
> > +		if (err >= 0 && err != len)
> >  			pr_debug("Truncated TX packet: len %d != %zd\n",
> >  				 err, len);
> >  done:
> > @@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net
> *net, struct socket *sock)
> >  			msg.msg_flags &= ~MSG_MORE;
> >  		}
> >
> > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> >  		err = sock->ops->sendmsg(sock, &msg, len);
> >  		if (unlikely(err < 0)) {
> >  			if (zcopy_used) {
> > @@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net
> *net, struct socket *sock)
> >  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
> >  					% UIO_MAXIOV;
> >  			}
> > -			vhost_discard_vq_desc(vq, 1);
> > -			vhost_net_enable_vq(net, vq);
> > -			break;
> > +			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {
> > +				vhost_discard_vq_desc(vq, 1);
> > +				vhost_net_enable_vq(net, vq);
> > +				break;
> > +			}
> >  		}
> > -		if (err != len)
> > +		if (err >= 0 && err != len)
> >  			pr_debug("Truncated TX packet: "
> >  				 " len %d != %zd\n", err, len);
> >  		if (!zcopy_used)
> > --
> > 2.23.0

Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[Michael S. Tsirkin]

On Mon, Dec 28, 2020 at 11:55:36AM +0000, wangyunjian wrote:
> > -----Original Message-----
> > From: Michael S. Tsirkin [mailto:mst@redhat.com]
> > Sent: Sunday, December 27, 2020 7:21 PM
> > To: wangyunjian <wangyunjian@huawei.com>
> > Cc: netdev@vger.kernel.org; jasowang@redhat.com;
> > willemdebruijn.kernel@gmail.com; virtualization@lists.linux-foundation.org;
> > Lilijun (Jerry) <jerry.lilijun@huawei.com>; chenchanghu
> > <chenchanghu@huawei.com>; xudingke <xudingke@huawei.com>; huangbin (J)
> > <brian.huangbin@huawei.com>
> > Subject: Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg
> > fails
> > 
> > On Fri, Dec 25, 2020 at 03:24:33PM +0800, wangyunjian wrote:
> > > From: Yunjian Wang <wangyunjian@huawei.com>
> > >
> > > Currently the driver doesn't drop a packet which can't be sent by tun
> > > (e.g bad packet). In this case, the driver will always process the
> > > same packet lead to the tx queue stuck.
> > 
> > So not making progress on a bad packet has some advantages, e.g. this is
> > easier to debug.
> > When is it important to drop the packet and continue?
> 
> In the case, the VM will not be able to send packets persistently. Services of VM
> are affected.
> 
> Thanks


Well VM can always harm itself right? Just halt the CPU, services will
be affected ;)


> > 
> > 
> > > To fix this issue:
> > > 1. in the case of persistent failure (e.g bad packet), the driver
> > >    can skip this descriptor by ignoring the error.
> > > 2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
> > >    the driver schedules the worker to try again.
> > >
> > > Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")
> > 
> > I'd just drop this tag, looks more like a feature than a bug ...
> 
> OK
> 
> > 
> > 
> > > Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> > > Acked-by: Willem de Bruijn <willemb@google.com>
> > > ---
> > >  drivers/vhost/net.c | 16 ++++++++--------
> > >  1 file changed, 8 insertions(+), 8 deletions(-)
> > >
> > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index
> > > c8784dfafdd7..01558fb2c552 100644
> > > --- a/drivers/vhost/net.c
> > > +++ b/drivers/vhost/net.c
> > > @@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net,
> > struct socket *sock)
> > >  				msg.msg_flags &= ~MSG_MORE;
> > >  		}
> > >
> > > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> > >  		err = sock->ops->sendmsg(sock, &msg, len);
> > > -		if (unlikely(err < 0)) {
> > > +		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS))
> > > +{
> > >  			vhost_discard_vq_desc(vq, 1);
> > >  			vhost_net_enable_vq(net, vq);
> > >  			break;
> > >  		}
> > > -		if (err != len)
> > > +		if (err >= 0 && err != len)
> > >  			pr_debug("Truncated TX packet: len %d != %zd\n",
> > >  				 err, len);
> > >  done:
> > > @@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net
> > *net, struct socket *sock)
> > >  			msg.msg_flags &= ~MSG_MORE;
> > >  		}
> > >
> > > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> > >  		err = sock->ops->sendmsg(sock, &msg, len);
> > >  		if (unlikely(err < 0)) {
> > >  			if (zcopy_used) {
> > > @@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net
> > *net, struct socket *sock)
> > >  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
> > >  					% UIO_MAXIOV;
> > >  			}
> > > -			vhost_discard_vq_desc(vq, 1);
> > > -			vhost_net_enable_vq(net, vq);
> > > -			break;
> > > +			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {
> > > +				vhost_discard_vq_desc(vq, 1);
> > > +				vhost_net_enable_vq(net, vq);
> > > +				break;
> > > +			}
> > >  		}
> > > -		if (err != len)
> > > +		if (err >= 0 && err != len)
> > >  			pr_debug("Truncated TX packet: "
> > >  				 " len %d != %zd\n", err, len);
> > >  		if (!zcopy_used)
> > > --
> > > 2.23.0

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[Michael S. Tsirkin]

On Mon, Dec 28, 2020 at 11:55:36AM +0000, wangyunjian wrote:
> > -----Original Message-----
> > From: Michael S. Tsirkin [mailto:mst@redhat.com]
> > Sent: Sunday, December 27, 2020 7:21 PM
> > To: wangyunjian <wangyunjian@huawei.com>
> > Cc: netdev@vger.kernel.org; jasowang@redhat.com;
> > willemdebruijn.kernel@gmail.com; virtualization@lists.linux-foundation.org;
> > Lilijun (Jerry) <jerry.lilijun@huawei.com>; chenchanghu
> > <chenchanghu@huawei.com>; xudingke <xudingke@huawei.com>; huangbin (J)
> > <brian.huangbin@huawei.com>
> > Subject: Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg
> > fails
> > 
> > On Fri, Dec 25, 2020 at 03:24:33PM +0800, wangyunjian wrote:
> > > From: Yunjian Wang <wangyunjian@huawei.com>
> > >
> > > Currently the driver doesn't drop a packet which can't be sent by tun
> > > (e.g bad packet). In this case, the driver will always process the
> > > same packet lead to the tx queue stuck.
> > 
> > So not making progress on a bad packet has some advantages, e.g. this is
> > easier to debug.
> > When is it important to drop the packet and continue?
> 
> In the case, the VM will not be able to send packets persistently. Services of VM
> are affected.
> 
> Thanks


Well VM can always harm itself right? Just halt the CPU, services will
be affected ;)


> > 
> > 
> > > To fix this issue:
> > > 1. in the case of persistent failure (e.g bad packet), the driver
> > >    can skip this descriptor by ignoring the error.
> > > 2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
> > >    the driver schedules the worker to try again.
> > >
> > > Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")
> > 
> > I'd just drop this tag, looks more like a feature than a bug ...
> 
> OK
> 
> > 
> > 
> > > Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> > > Acked-by: Willem de Bruijn <willemb@google.com>
> > > ---
> > >  drivers/vhost/net.c | 16 ++++++++--------
> > >  1 file changed, 8 insertions(+), 8 deletions(-)
> > >
> > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index
> > > c8784dfafdd7..01558fb2c552 100644
> > > --- a/drivers/vhost/net.c
> > > +++ b/drivers/vhost/net.c
> > > @@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net,
> > struct socket *sock)
> > >  				msg.msg_flags &= ~MSG_MORE;
> > >  		}
> > >
> > > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> > >  		err = sock->ops->sendmsg(sock, &msg, len);
> > > -		if (unlikely(err < 0)) {
> > > +		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS))
> > > +{
> > >  			vhost_discard_vq_desc(vq, 1);
> > >  			vhost_net_enable_vq(net, vq);
> > >  			break;
> > >  		}
> > > -		if (err != len)
> > > +		if (err >= 0 && err != len)
> > >  			pr_debug("Truncated TX packet: len %d != %zd\n",
> > >  				 err, len);
> > >  done:
> > > @@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net
> > *net, struct socket *sock)
> > >  			msg.msg_flags &= ~MSG_MORE;
> > >  		}
> > >
> > > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> > >  		err = sock->ops->sendmsg(sock, &msg, len);
> > >  		if (unlikely(err < 0)) {
> > >  			if (zcopy_used) {
> > > @@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net
> > *net, struct socket *sock)
> > >  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
> > >  					% UIO_MAXIOV;
> > >  			}
> > > -			vhost_discard_vq_desc(vq, 1);
> > > -			vhost_net_enable_vq(net, vq);
> > > -			break;
> > > +			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {
> > > +				vhost_discard_vq_desc(vq, 1);
> > > +				vhost_net_enable_vq(net, vq);
> > > +				break;
> > > +			}
> > >  		}
> > > -		if (err != len)
> > > +		if (err >= 0 && err != len)
> > >  			pr_debug("Truncated TX packet: "
> > >  				 " len %d != %zd\n", err, len);
> > >  		if (!zcopy_used)
> > > --
> > > 2.23.0

RE: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[wangyunjian]

> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst@redhat.com]
> Sent: Sunday, December 27, 2020 7:21 PM
> To: wangyunjian <wangyunjian@huawei.com>
> Cc: netdev@vger.kernel.org; jasowang@redhat.com;
> willemdebruijn.kernel@gmail.com; virtualization@lists.linux-foundation.org;
> Lilijun (Jerry) <jerry.lilijun@huawei.com>; chenchanghu
> <chenchanghu@huawei.com>; xudingke <xudingke@huawei.com>; huangbin (J)
> <brian.huangbin@huawei.com>
> Subject: Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg
> fails
> 
> On Fri, Dec 25, 2020 at 03:24:33PM +0800, wangyunjian wrote:
> > From: Yunjian Wang <wangyunjian@huawei.com>
> >
> > Currently the driver doesn't drop a packet which can't be sent by tun
> > (e.g bad packet). In this case, the driver will always process the
> > same packet lead to the tx queue stuck.
> 
> So not making progress on a bad packet has some advantages, e.g. this is
> easier to debug.
> When is it important to drop the packet and continue?
> 
> 
> > To fix this issue:
> > 1. in the case of persistent failure (e.g bad packet), the driver
> >    can skip this descriptor by ignoring the error.
> > 2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
> >    the driver schedules the worker to try again.
> >
> > Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")
> 
> I'd just drop this tag, looks more like a feature than a bug ...

Do these two patches need to be sent separately?

Thanks

> 
> 
> > Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> > Acked-by: Willem de Bruijn <willemb@google.com>
> > ---
> >  drivers/vhost/net.c | 16 ++++++++--------
> >  1 file changed, 8 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index
> > c8784dfafdd7..01558fb2c552 100644
> > --- a/drivers/vhost/net.c
> > +++ b/drivers/vhost/net.c
> > @@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net,
> struct socket *sock)
> >  				msg.msg_flags &= ~MSG_MORE;
> >  		}
> >
> > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> >  		err = sock->ops->sendmsg(sock, &msg, len);
> > -		if (unlikely(err < 0)) {
> > +		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS))
> > +{
> >  			vhost_discard_vq_desc(vq, 1);
> >  			vhost_net_enable_vq(net, vq);
> >  			break;
> >  		}
> > -		if (err != len)
> > +		if (err >= 0 && err != len)
> >  			pr_debug("Truncated TX packet: len %d != %zd\n",
> >  				 err, len);
> >  done:
> > @@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net
> *net, struct socket *sock)
> >  			msg.msg_flags &= ~MSG_MORE;
> >  		}
> >
> > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> >  		err = sock->ops->sendmsg(sock, &msg, len);
> >  		if (unlikely(err < 0)) {
> >  			if (zcopy_used) {
> > @@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net
> *net, struct socket *sock)
> >  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
> >  					% UIO_MAXIOV;
> >  			}
> > -			vhost_discard_vq_desc(vq, 1);
> > -			vhost_net_enable_vq(net, vq);
> > -			break;
> > +			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {
> > +				vhost_discard_vq_desc(vq, 1);
> > +				vhost_net_enable_vq(net, vq);
> > +				break;
> > +			}
> >  		}
> > -		if (err != len)
> > +		if (err >= 0 && err != len)
> >  			pr_debug("Truncated TX packet: "
> >  				 " len %d != %zd\n", err, len);
> >  		if (!zcopy_used)
> > --
> > 2.23.0

Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[Michael S. Tsirkin]

On Mon, Dec 28, 2020 at 01:27:42PM +0000, wangyunjian wrote:
> > -----Original Message-----
> > From: Michael S. Tsirkin [mailto:mst@redhat.com]
> > Sent: Sunday, December 27, 2020 7:21 PM
> > To: wangyunjian <wangyunjian@huawei.com>
> > Cc: netdev@vger.kernel.org; jasowang@redhat.com;
> > willemdebruijn.kernel@gmail.com; virtualization@lists.linux-foundation.org;
> > Lilijun (Jerry) <jerry.lilijun@huawei.com>; chenchanghu
> > <chenchanghu@huawei.com>; xudingke <xudingke@huawei.com>; huangbin (J)
> > <brian.huangbin@huawei.com>
> > Subject: Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg
> > fails
> > 
> > On Fri, Dec 25, 2020 at 03:24:33PM +0800, wangyunjian wrote:
> > > From: Yunjian Wang <wangyunjian@huawei.com>
> > >
> > > Currently the driver doesn't drop a packet which can't be sent by tun
> > > (e.g bad packet). In this case, the driver will always process the
> > > same packet lead to the tx queue stuck.
> > 
> > So not making progress on a bad packet has some advantages, e.g. this is
> > easier to debug.
> > When is it important to drop the packet and continue?
> > 
> > 
> > > To fix this issue:
> > > 1. in the case of persistent failure (e.g bad packet), the driver
> > >    can skip this descriptor by ignoring the error.
> > > 2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
> > >    the driver schedules the worker to try again.
> > >
> > > Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")
> > 
> > I'd just drop this tag, looks more like a feature than a bug ...
> 
> Do these two patches need to be sent separately?
> 
> Thanks

Makes sense to me.

> > 
> > 
> > > Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> > > Acked-by: Willem de Bruijn <willemb@google.com>
> > > ---
> > >  drivers/vhost/net.c | 16 ++++++++--------
> > >  1 file changed, 8 insertions(+), 8 deletions(-)
> > >
> > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index
> > > c8784dfafdd7..01558fb2c552 100644
> > > --- a/drivers/vhost/net.c
> > > +++ b/drivers/vhost/net.c
> > > @@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net,
> > struct socket *sock)
> > >  				msg.msg_flags &= ~MSG_MORE;
> > >  		}
> > >
> > > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> > >  		err = sock->ops->sendmsg(sock, &msg, len);
> > > -		if (unlikely(err < 0)) {
> > > +		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS))
> > > +{
> > >  			vhost_discard_vq_desc(vq, 1);
> > >  			vhost_net_enable_vq(net, vq);
> > >  			break;
> > >  		}
> > > -		if (err != len)
> > > +		if (err >= 0 && err != len)
> > >  			pr_debug("Truncated TX packet: len %d != %zd\n",
> > >  				 err, len);
> > >  done:
> > > @@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net
> > *net, struct socket *sock)
> > >  			msg.msg_flags &= ~MSG_MORE;
> > >  		}
> > >
> > > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> > >  		err = sock->ops->sendmsg(sock, &msg, len);
> > >  		if (unlikely(err < 0)) {
> > >  			if (zcopy_used) {
> > > @@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net
> > *net, struct socket *sock)
> > >  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
> > >  					% UIO_MAXIOV;
> > >  			}
> > > -			vhost_discard_vq_desc(vq, 1);
> > > -			vhost_net_enable_vq(net, vq);
> > > -			break;
> > > +			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {
> > > +				vhost_discard_vq_desc(vq, 1);
> > > +				vhost_net_enable_vq(net, vq);
> > > +				break;
> > > +			}
> > >  		}
> > > -		if (err != len)
> > > +		if (err >= 0 && err != len)
> > >  			pr_debug("Truncated TX packet: "
> > >  				 " len %d != %zd\n", err, len);
> > >  		if (!zcopy_used)
> > > --
> > > 2.23.0

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg fails

[Michael S. Tsirkin]

On Mon, Dec 28, 2020 at 01:27:42PM +0000, wangyunjian wrote:
> > -----Original Message-----
> > From: Michael S. Tsirkin [mailto:mst@redhat.com]
> > Sent: Sunday, December 27, 2020 7:21 PM
> > To: wangyunjian <wangyunjian@huawei.com>
> > Cc: netdev@vger.kernel.org; jasowang@redhat.com;
> > willemdebruijn.kernel@gmail.com; virtualization@lists.linux-foundation.org;
> > Lilijun (Jerry) <jerry.lilijun@huawei.com>; chenchanghu
> > <chenchanghu@huawei.com>; xudingke <xudingke@huawei.com>; huangbin (J)
> > <brian.huangbin@huawei.com>
> > Subject: Re: [PATCH net v5 2/2] vhost_net: fix tx queue stuck when sendmsg
> > fails
> > 
> > On Fri, Dec 25, 2020 at 03:24:33PM +0800, wangyunjian wrote:
> > > From: Yunjian Wang <wangyunjian@huawei.com>
> > >
> > > Currently the driver doesn't drop a packet which can't be sent by tun
> > > (e.g bad packet). In this case, the driver will always process the
> > > same packet lead to the tx queue stuck.
> > 
> > So not making progress on a bad packet has some advantages, e.g. this is
> > easier to debug.
> > When is it important to drop the packet and continue?
> > 
> > 
> > > To fix this issue:
> > > 1. in the case of persistent failure (e.g bad packet), the driver
> > >    can skip this descriptor by ignoring the error.
> > > 2. in the case of transient failure (e.g -ENOBUFS, -EAGAIN and -ENOMEM),
> > >    the driver schedules the worker to try again.
> > >
> > > Fixes: 3a4d5c94e959 ("vhost_net: a kernel-level virtio server")
> > 
> > I'd just drop this tag, looks more like a feature than a bug ...
> 
> Do these two patches need to be sent separately?
> 
> Thanks

Makes sense to me.

> > 
> > 
> > > Signed-off-by: Yunjian Wang <wangyunjian@huawei.com>
> > > Acked-by: Willem de Bruijn <willemb@google.com>
> > > ---
> > >  drivers/vhost/net.c | 16 ++++++++--------
> > >  1 file changed, 8 insertions(+), 8 deletions(-)
> > >
> > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index
> > > c8784dfafdd7..01558fb2c552 100644
> > > --- a/drivers/vhost/net.c
> > > +++ b/drivers/vhost/net.c
> > > @@ -827,14 +827,13 @@ static void handle_tx_copy(struct vhost_net *net,
> > struct socket *sock)
> > >  				msg.msg_flags &= ~MSG_MORE;
> > >  		}
> > >
> > > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> > >  		err = sock->ops->sendmsg(sock, &msg, len);
> > > -		if (unlikely(err < 0)) {
> > > +		if (unlikely(err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS))
> > > +{
> > >  			vhost_discard_vq_desc(vq, 1);
> > >  			vhost_net_enable_vq(net, vq);
> > >  			break;
> > >  		}
> > > -		if (err != len)
> > > +		if (err >= 0 && err != len)
> > >  			pr_debug("Truncated TX packet: len %d != %zd\n",
> > >  				 err, len);
> > >  done:
> > > @@ -922,7 +921,6 @@ static void handle_tx_zerocopy(struct vhost_net
> > *net, struct socket *sock)
> > >  			msg.msg_flags &= ~MSG_MORE;
> > >  		}
> > >
> > > -		/* TODO: Check specific error and bomb out unless ENOBUFS? */
> > >  		err = sock->ops->sendmsg(sock, &msg, len);
> > >  		if (unlikely(err < 0)) {
> > >  			if (zcopy_used) {
> > > @@ -931,11 +929,13 @@ static void handle_tx_zerocopy(struct vhost_net
> > *net, struct socket *sock)
> > >  				nvq->upend_idx = ((unsigned)nvq->upend_idx - 1)
> > >  					% UIO_MAXIOV;
> > >  			}
> > > -			vhost_discard_vq_desc(vq, 1);
> > > -			vhost_net_enable_vq(net, vq);
> > > -			break;
> > > +			if (err == -EAGAIN || err == -ENOMEM || err == -ENOBUFS) {
> > > +				vhost_discard_vq_desc(vq, 1);
> > > +				vhost_net_enable_vq(net, vq);
> > > +				break;
> > > +			}
> > >  		}
> > > -		if (err != len)
> > > +		if (err >= 0 && err != len)
> > >  			pr_debug("Truncated TX packet: "
> > >  				 " len %d != %zd\n", err, len);
> > >  		if (!zcopy_used)
> > > --
> > > 2.23.0