[PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full

[PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full

[Sridhar Samudrala]

When running guest to remote host TCP stream test using vhost-net
via tap/macvtap, i am seeing network transmit hangs. This happens
when handle_tx() returns because of the socket send queue full 
condition.
This patch fixes this by restarting tx poll when hitting this
condition.

Signed-off-by: Sridhar Samudrala <sri@us.ibm.com>

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index 91a324c..82d4bbe 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net)
 	if (!sock)
 		return;
 
-	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
-	if (wmem >= sock->sk->sk_sndbuf)
-		return;
-
 	use_mm(net->dev.mm);
 	mutex_lock(&vq->mutex);
+
+	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
+	if (wmem >= sock->sk->sk_sndbuf) {
+		tx_poll_start(net, sock);
+		set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
+		goto unlock;
+	}
+
 	vhost_disable_notify(vq);
 
 	if (wmem < sock->sk->sk_sndbuf * 2)
@@ -178,6 +182,7 @@ static void handle_tx(struct vhost_net *net)
 		}
 	}
 
+unlock:
 	mutex_unlock(&vq->mutex);
 	unuse_mm(net->dev.mm);
 }

Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full

[Michael S. Tsirkin]

On Thu, Feb 18, 2010 at 12:59:11PM -0800, Sridhar Samudrala wrote:
> When running guest to remote host TCP stream test using vhost-net
> via tap/macvtap, i am seeing network transmit hangs. This happens
> when handle_tx() returns because of the socket send queue full 
> condition.
> This patch fixes this by restarting tx poll when hitting this
> condition.


Thanks! I would like to better understand what happens exactly.
Some questions below:

> 
> Signed-off-by: Sridhar Samudrala <sri@us.ibm.com>
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index 91a324c..82d4bbe 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net)
>  	if (!sock)
>  		return;
>  
> -	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> -	if (wmem >= sock->sk->sk_sndbuf)
> -		return;
> -

The disadvantage here is that a spurious wakeup
when queue is still full becomes more expensive.

>  	use_mm(net->dev.mm);
>  	mutex_lock(&vq->mutex);
> +
> +	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> +	if (wmem >= sock->sk->sk_sndbuf) {
> +		tx_poll_start(net, sock);

Hmm. We already do
                       if (wmem >= sock->sk->sk_sndbuf * 3 / 4) {
                                tx_poll_start(net, sock);
                                set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
                                break;
                        }
why does not this code trigger here?


> +		set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
              
Isn't the bit already set? If not, why?

> +		goto unlock;
> +	}
> +
>  	vhost_disable_notify(vq);
>  
>  	if (wmem < sock->sk->sk_sndbuf * 2)
> @@ -178,6 +182,7 @@ static void handle_tx(struct vhost_net *net)
>  		}
>  	}
>  
> +unlock:
>  	mutex_unlock(&vq->mutex);
>  	unuse_mm(net->dev.mm);
>  }
> 

Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full

[Sridhar Samudrala]

On Fri, 2010-02-19 at 00:30 +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 18, 2010 at 12:59:11PM -0800, Sridhar Samudrala wrote:
> > When running guest to remote host TCP stream test using vhost-net
> > via tap/macvtap, i am seeing network transmit hangs. This happens
> > when handle_tx() returns because of the socket send queue full 
> > condition.
> > This patch fixes this by restarting tx poll when hitting this
> > condition.
> 
> 
> Thanks! I would like to better understand what happens exactly.
> Some questions below:
> 
> > 
> > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com>
> > 
> > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> > index 91a324c..82d4bbe 100644
> > --- a/drivers/vhost/net.c
> > +++ b/drivers/vhost/net.c
> > @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net)
> >  	if (!sock)
> >  		return;
> >  
> > -	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> > -	if (wmem >= sock->sk->sk_sndbuf)
> > -		return;
> > -
> 
> The disadvantage here is that a spurious wakeup
> when queue is still full becomes more expensive.
> 
> >  	use_mm(net->dev.mm);
> >  	mutex_lock(&vq->mutex);
> > +
> > +	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> > +	if (wmem >= sock->sk->sk_sndbuf) {
> > +		tx_poll_start(net, sock);
> 
> Hmm. We already do
>                        if (wmem >= sock->sk->sk_sndbuf * 3 / 4) {
>                                 tx_poll_start(net, sock);
>                                 set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
>                                 break;
>                         }
> why does not this code trigger here?

This check is done only when the ring is empty(head == vq->num).
But we are breaking out of the loop here.
                if (unlikely(total_len >= VHOST_NET_WEIGHT)) {
                        vhost_poll_queue(&vq->poll);
                        break;
                }

I guess tx_poll_start() is missing here. The following patch fixes
the hang and may be a better fix.

Signed-off-by: Sridhar Samudrala <sri@us.ibm.com>

diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index 4c89283..fe9d296 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -172,6 +172,7 @@ static void handle_tx(struct vhost_net *net)
 		vhost_add_used_and_signal(&net->dev, vq, head, 0);
 		total_len += len;
 		if (unlikely(total_len >= VHOST_NET_WEIGHT)) {
+			tx_poll_start(net, sock);
 			vhost_poll_queue(&vq->poll);
 			break;
 		}

Thanks
Sridhar

Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full

[Michael S. Tsirkin]

On Thu, Feb 18, 2010 at 06:00:07PM -0800, Sridhar Samudrala wrote:
> On Fri, 2010-02-19 at 00:30 +0200, Michael S. Tsirkin wrote:
> > On Thu, Feb 18, 2010 at 12:59:11PM -0800, Sridhar Samudrala wrote:
> > > When running guest to remote host TCP stream test using vhost-net
> > > via tap/macvtap, i am seeing network transmit hangs. This happens
> > > when handle_tx() returns because of the socket send queue full 
> > > condition.
> > > This patch fixes this by restarting tx poll when hitting this
> > > condition.
> > 
> > 
> > Thanks! I would like to better understand what happens exactly.
> > Some questions below:
> > 
> > > 
> > > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com>
> > > 
> > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> > > index 91a324c..82d4bbe 100644
> > > --- a/drivers/vhost/net.c
> > > +++ b/drivers/vhost/net.c
> > > @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net)
> > >  	if (!sock)
> > >  		return;
> > >  
> > > -	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> > > -	if (wmem >= sock->sk->sk_sndbuf)
> > > -		return;
> > > -
> > 
> > The disadvantage here is that a spurious wakeup
> > when queue is still full becomes more expensive.
> > 
> > >  	use_mm(net->dev.mm);
> > >  	mutex_lock(&vq->mutex);
> > > +
> > > +	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> > > +	if (wmem >= sock->sk->sk_sndbuf) {
> > > +		tx_poll_start(net, sock);
> > 
> > Hmm. We already do
> >                        if (wmem >= sock->sk->sk_sndbuf * 3 / 4) {
> >                                 tx_poll_start(net, sock);
> >                                 set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
> >                                 break;
> >                         }
> > why does not this code trigger here?
> 
> This check is done only when the ring is empty(head == vq->num).
> But we are breaking out of the loop here.
>                 if (unlikely(total_len >= VHOST_NET_WEIGHT)) {
>                         vhost_poll_queue(&vq->poll);
>                         break;
>                 }
> 
> I guess tx_poll_start() is missing here. The following patch fixes
> the hang and may be a better fix.
> 
> Signed-off-by: Sridhar Samudrala <sri@us.ibm.com>
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index 4c89283..fe9d296 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -172,6 +172,7 @@ static void handle_tx(struct vhost_net *net)
>  		vhost_add_used_and_signal(&net->dev, vq, head, 0);
>  		total_len += len;
>  		if (unlikely(total_len >= VHOST_NET_WEIGHT)) {
> +			tx_poll_start(net, sock);
>  			vhost_poll_queue(&vq->poll);
>  			break;
>  		}
> 
> Thanks
> Sridhar


Hmm, this happens when
we have polled a lot of packets, and want to
give another vq a chance to poll.
Looks like a strange place to add it.

-- 
MST

Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full

[Sridhar Samudrala]

On Fri, 2010-02-19 at 16:42 +0200, Michael S. Tsirkin wrote:

> > > Hmm. We already do
> > >                        if (wmem >= sock->sk->sk_sndbuf * 3 / 4) {
> > >                                 tx_poll_start(net, sock);
> > >                                 set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
> > >                                 break;
> > >                         }
> > > why does not this code trigger here?
> > 
> > This check is done only when the ring is empty(head == vq->num).
> > But we are breaking out of the loop here.
> >                 if (unlikely(total_len >= VHOST_NET_WEIGHT)) {
> >                         vhost_poll_queue(&vq->poll);
> >                         break;
> >                 }
> > 
> > I guess tx_poll_start() is missing here. The following patch fixes
> > the hang and may be a better fix.
> > 
> > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com>
> > 
> > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> > index 4c89283..fe9d296 100644
> > --- a/drivers/vhost/net.c
> > +++ b/drivers/vhost/net.c
> > @@ -172,6 +172,7 @@ static void handle_tx(struct vhost_net *net)
> >  		vhost_add_used_and_signal(&net->dev, vq, head, 0);
> >  		total_len += len;
> >  		if (unlikely(total_len >= VHOST_NET_WEIGHT)) {
> > +			tx_poll_start(net, sock);
> >  			vhost_poll_queue(&vq->poll);
> >  			break;
> >  		}
> > 
> > Thanks
> > Sridhar
> 
> 
> Hmm, this happens when
> we have polled a lot of packets, and want to
> give another vq a chance to poll.
> Looks like a strange place to add it.

I am also seeing sendmsg() calls failing with EAGAIN. Could be a bug in
handling this error. The check for sendq full is done outside the for
loop. It is possible that we can run out of sendq space within the for
loop. Should we check for wmem within the for loop?

Thanks
Sridhar

Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full

[Michael S. Tsirkin]

On Thu, Feb 18, 2010 at 12:59:11PM -0800, Sridhar Samudrala wrote:
> When running guest to remote host TCP stream test using vhost-net
> via tap/macvtap, i am seeing network transmit hangs. This happens
> when handle_tx() returns because of the socket send queue full 
> condition.
> This patch fixes this by restarting tx poll when hitting this
> condition.
> 
> Signed-off-by: Sridhar Samudrala <sri@us.ibm.com>
> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index 91a324c..82d4bbe 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net)
>  	if (!sock)
>  		return;
>  
> -	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> -	if (wmem >= sock->sk->sk_sndbuf)
> -		return;
> -
>  	use_mm(net->dev.mm);
>  	mutex_lock(&vq->mutex);
> +
> +	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> +	if (wmem >= sock->sk->sk_sndbuf) {
> +		tx_poll_start(net, sock);
> +		set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
> +		goto unlock;
> +	}
> +
>  	vhost_disable_notify(vq);
>  
>  	if (wmem < sock->sk->sk_sndbuf * 2)
> @@ -178,6 +182,7 @@ static void handle_tx(struct vhost_net *net)
>  		}
>  	}
>  
> +unlock:
>  	mutex_unlock(&vq->mutex);
>  	unuse_mm(net->dev.mm);
>  }


It might be better to avoid use_mm when ring is full.
Does the following fix the tx hang for you?


diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index 4c89283..f5f6efe 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -113,8 +113,12 @@ static void handle_tx(struct vhost_net *net)
 		return;
 
 	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
-	if (wmem >= sock->sk->sk_sndbuf)
-		return;
+	if (wmem >= sock->sk->sk_sndbuf) {
+		mutex_lock(&vq->mutex);
+		tx_poll_start(net, sock);
+		mutex_unlock(&vq->mutex);
+                return;
+	}
 
 	use_mm(net->dev.mm);
 	mutex_lock(&vq->mutex);

Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full

[Sridhar Samudrala]

On Tue, 2010-02-23 at 12:24 +0200, Michael S. Tsirkin wrote:
> On Thu, Feb 18, 2010 at 12:59:11PM -0800, Sridhar Samudrala wrote:
> > When running guest to remote host TCP stream test using vhost-net
> > via tap/macvtap, i am seeing network transmit hangs. This happens
> > when handle_tx() returns because of the socket send queue full 
> > condition.
> > This patch fixes this by restarting tx poll when hitting this
> > condition.
> > 
> > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com>
> > 
> > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> > index 91a324c..82d4bbe 100644
> > --- a/drivers/vhost/net.c
> > +++ b/drivers/vhost/net.c
> > @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net)
> >  	if (!sock)
> >  		return;
> >  
> > -	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> > -	if (wmem >= sock->sk->sk_sndbuf)
> > -		return;
> > -
> >  	use_mm(net->dev.mm);
> >  	mutex_lock(&vq->mutex);
> > +
> > +	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> > +	if (wmem >= sock->sk->sk_sndbuf) {
> > +		tx_poll_start(net, sock);
> > +		set_bit(SOCK_ASYNC_NOSPACE, &sock->flags);
> > +		goto unlock;
> > +	}
> > +
> >  	vhost_disable_notify(vq);
> >  
> >  	if (wmem < sock->sk->sk_sndbuf * 2)
> > @@ -178,6 +182,7 @@ static void handle_tx(struct vhost_net *net)
> >  		}
> >  	}
> >  
> > +unlock:
> >  	mutex_unlock(&vq->mutex);
> >  	unuse_mm(net->dev.mm);
> >  }
> 
> 
> It might be better to avoid use_mm when ring is full.
> Does the following fix the tx hang for you?

Yes. this fixes the tx hang.

Thanks
Sridhar

> 
> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
> index 4c89283..f5f6efe 100644
> --- a/drivers/vhost/net.c
> +++ b/drivers/vhost/net.c
> @@ -113,8 +113,12 @@ static void handle_tx(struct vhost_net *net)
>  		return;
> 
>  	wmem = atomic_read(&sock->sk->sk_wmem_alloc);
> -	if (wmem >= sock->sk->sk_sndbuf)
> -		return;
> +	if (wmem >= sock->sk->sk_sndbuf) {
> +		mutex_lock(&vq->mutex);
> +		tx_poll_start(net, sock);
> +		mutex_unlock(&vq->mutex);
> +                return;
> +	}
> 
>  	use_mm(net->dev.mm);
>  	mutex_lock(&vq->mutex);
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full

[Shirley Ma]

This also fixes tx hung issue between guest and guest communication
through vhost.

Thanks
Shirley