* [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full @ 2010-02-18 20:59 Sridhar Samudrala 2010-02-18 22:30 ` Michael S. Tsirkin 2010-02-23 10:24 ` Michael S. Tsirkin 0 siblings, 2 replies; 8+ messages in thread From: Sridhar Samudrala @ 2010-02-18 20:59 UTC (permalink / raw) To: Michael S. Tsirkin, David Miller; +Cc: netdev When running guest to remote host TCP stream test using vhost-net via tap/macvtap, i am seeing network transmit hangs. This happens when handle_tx() returns because of the socket send queue full condition. This patch fixes this by restarting tx poll when hitting this condition. Signed-off-by: Sridhar Samudrala <sri@us.ibm.com> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 91a324c..82d4bbe 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net) if (!sock) return; - wmem = atomic_read(&sock->sk->sk_wmem_alloc); - if (wmem >= sock->sk->sk_sndbuf) - return; - use_mm(net->dev.mm); mutex_lock(&vq->mutex); + + wmem = atomic_read(&sock->sk->sk_wmem_alloc); + if (wmem >= sock->sk->sk_sndbuf) { + tx_poll_start(net, sock); + set_bit(SOCK_ASYNC_NOSPACE, &sock->flags); + goto unlock; + } + vhost_disable_notify(vq); if (wmem < sock->sk->sk_sndbuf * 2) @@ -178,6 +182,7 @@ static void handle_tx(struct vhost_net *net) } } +unlock: mutex_unlock(&vq->mutex); unuse_mm(net->dev.mm); } ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full 2010-02-18 20:59 [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full Sridhar Samudrala @ 2010-02-18 22:30 ` Michael S. Tsirkin 2010-02-19 2:00 ` Sridhar Samudrala 2010-02-23 10:24 ` Michael S. Tsirkin 1 sibling, 1 reply; 8+ messages in thread From: Michael S. Tsirkin @ 2010-02-18 22:30 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: David Miller, netdev On Thu, Feb 18, 2010 at 12:59:11PM -0800, Sridhar Samudrala wrote: > When running guest to remote host TCP stream test using vhost-net > via tap/macvtap, i am seeing network transmit hangs. This happens > when handle_tx() returns because of the socket send queue full > condition. > This patch fixes this by restarting tx poll when hitting this > condition. Thanks! I would like to better understand what happens exactly. Some questions below: > > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com> > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c > index 91a324c..82d4bbe 100644 > --- a/drivers/vhost/net.c > +++ b/drivers/vhost/net.c > @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net) > if (!sock) > return; > > - wmem = atomic_read(&sock->sk->sk_wmem_alloc); > - if (wmem >= sock->sk->sk_sndbuf) > - return; > - The disadvantage here is that a spurious wakeup when queue is still full becomes more expensive. > use_mm(net->dev.mm); > mutex_lock(&vq->mutex); > + > + wmem = atomic_read(&sock->sk->sk_wmem_alloc); > + if (wmem >= sock->sk->sk_sndbuf) { > + tx_poll_start(net, sock); Hmm. We already do if (wmem >= sock->sk->sk_sndbuf * 3 / 4) { tx_poll_start(net, sock); set_bit(SOCK_ASYNC_NOSPACE, &sock->flags); break; } why does not this code trigger here? > + set_bit(SOCK_ASYNC_NOSPACE, &sock->flags); Isn't the bit already set? If not, why? > + goto unlock; > + } > + > vhost_disable_notify(vq); > > if (wmem < sock->sk->sk_sndbuf * 2) > @@ -178,6 +182,7 @@ static void handle_tx(struct vhost_net *net) > } > } > > +unlock: > mutex_unlock(&vq->mutex); > unuse_mm(net->dev.mm); > } > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full 2010-02-18 22:30 ` Michael S. Tsirkin @ 2010-02-19 2:00 ` Sridhar Samudrala 2010-02-19 14:42 ` Michael S. Tsirkin 0 siblings, 1 reply; 8+ messages in thread From: Sridhar Samudrala @ 2010-02-19 2:00 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: David Miller, netdev On Fri, 2010-02-19 at 00:30 +0200, Michael S. Tsirkin wrote: > On Thu, Feb 18, 2010 at 12:59:11PM -0800, Sridhar Samudrala wrote: > > When running guest to remote host TCP stream test using vhost-net > > via tap/macvtap, i am seeing network transmit hangs. This happens > > when handle_tx() returns because of the socket send queue full > > condition. > > This patch fixes this by restarting tx poll when hitting this > > condition. > > > Thanks! I would like to better understand what happens exactly. > Some questions below: > > > > > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com> > > > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c > > index 91a324c..82d4bbe 100644 > > --- a/drivers/vhost/net.c > > +++ b/drivers/vhost/net.c > > @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net) > > if (!sock) > > return; > > > > - wmem = atomic_read(&sock->sk->sk_wmem_alloc); > > - if (wmem >= sock->sk->sk_sndbuf) > > - return; > > - > > The disadvantage here is that a spurious wakeup > when queue is still full becomes more expensive. > > > use_mm(net->dev.mm); > > mutex_lock(&vq->mutex); > > + > > + wmem = atomic_read(&sock->sk->sk_wmem_alloc); > > + if (wmem >= sock->sk->sk_sndbuf) { > > + tx_poll_start(net, sock); > > Hmm. We already do > if (wmem >= sock->sk->sk_sndbuf * 3 / 4) { > tx_poll_start(net, sock); > set_bit(SOCK_ASYNC_NOSPACE, &sock->flags); > break; > } > why does not this code trigger here? This check is done only when the ring is empty(head == vq->num). But we are breaking out of the loop here. if (unlikely(total_len >= VHOST_NET_WEIGHT)) { vhost_poll_queue(&vq->poll); break; } I guess tx_poll_start() is missing here. The following patch fixes the hang and may be a better fix. Signed-off-by: Sridhar Samudrala <sri@us.ibm.com> diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 4c89283..fe9d296 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -172,6 +172,7 @@ static void handle_tx(struct vhost_net *net) vhost_add_used_and_signal(&net->dev, vq, head, 0); total_len += len; if (unlikely(total_len >= VHOST_NET_WEIGHT)) { + tx_poll_start(net, sock); vhost_poll_queue(&vq->poll); break; } Thanks Sridhar ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full 2010-02-19 2:00 ` Sridhar Samudrala @ 2010-02-19 14:42 ` Michael S. Tsirkin 2010-02-19 21:19 ` Sridhar Samudrala 0 siblings, 1 reply; 8+ messages in thread From: Michael S. Tsirkin @ 2010-02-19 14:42 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: David Miller, netdev On Thu, Feb 18, 2010 at 06:00:07PM -0800, Sridhar Samudrala wrote: > On Fri, 2010-02-19 at 00:30 +0200, Michael S. Tsirkin wrote: > > On Thu, Feb 18, 2010 at 12:59:11PM -0800, Sridhar Samudrala wrote: > > > When running guest to remote host TCP stream test using vhost-net > > > via tap/macvtap, i am seeing network transmit hangs. This happens > > > when handle_tx() returns because of the socket send queue full > > > condition. > > > This patch fixes this by restarting tx poll when hitting this > > > condition. > > > > > > Thanks! I would like to better understand what happens exactly. > > Some questions below: > > > > > > > > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com> > > > > > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c > > > index 91a324c..82d4bbe 100644 > > > --- a/drivers/vhost/net.c > > > +++ b/drivers/vhost/net.c > > > @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net) > > > if (!sock) > > > return; > > > > > > - wmem = atomic_read(&sock->sk->sk_wmem_alloc); > > > - if (wmem >= sock->sk->sk_sndbuf) > > > - return; > > > - > > > > The disadvantage here is that a spurious wakeup > > when queue is still full becomes more expensive. > > > > > use_mm(net->dev.mm); > > > mutex_lock(&vq->mutex); > > > + > > > + wmem = atomic_read(&sock->sk->sk_wmem_alloc); > > > + if (wmem >= sock->sk->sk_sndbuf) { > > > + tx_poll_start(net, sock); > > > > Hmm. We already do > > if (wmem >= sock->sk->sk_sndbuf * 3 / 4) { > > tx_poll_start(net, sock); > > set_bit(SOCK_ASYNC_NOSPACE, &sock->flags); > > break; > > } > > why does not this code trigger here? > > This check is done only when the ring is empty(head == vq->num). > But we are breaking out of the loop here. > if (unlikely(total_len >= VHOST_NET_WEIGHT)) { > vhost_poll_queue(&vq->poll); > break; > } > > I guess tx_poll_start() is missing here. The following patch fixes > the hang and may be a better fix. > > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com> > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c > index 4c89283..fe9d296 100644 > --- a/drivers/vhost/net.c > +++ b/drivers/vhost/net.c > @@ -172,6 +172,7 @@ static void handle_tx(struct vhost_net *net) > vhost_add_used_and_signal(&net->dev, vq, head, 0); > total_len += len; > if (unlikely(total_len >= VHOST_NET_WEIGHT)) { > + tx_poll_start(net, sock); > vhost_poll_queue(&vq->poll); > break; > } > > Thanks > Sridhar Hmm, this happens when we have polled a lot of packets, and want to give another vq a chance to poll. Looks like a strange place to add it. -- MST ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full 2010-02-19 14:42 ` Michael S. Tsirkin @ 2010-02-19 21:19 ` Sridhar Samudrala 0 siblings, 0 replies; 8+ messages in thread From: Sridhar Samudrala @ 2010-02-19 21:19 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: David Miller, netdev On Fri, 2010-02-19 at 16:42 +0200, Michael S. Tsirkin wrote: > > > Hmm. We already do > > > if (wmem >= sock->sk->sk_sndbuf * 3 / 4) { > > > tx_poll_start(net, sock); > > > set_bit(SOCK_ASYNC_NOSPACE, &sock->flags); > > > break; > > > } > > > why does not this code trigger here? > > > > This check is done only when the ring is empty(head == vq->num). > > But we are breaking out of the loop here. > > if (unlikely(total_len >= VHOST_NET_WEIGHT)) { > > vhost_poll_queue(&vq->poll); > > break; > > } > > > > I guess tx_poll_start() is missing here. The following patch fixes > > the hang and may be a better fix. > > > > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com> > > > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c > > index 4c89283..fe9d296 100644 > > --- a/drivers/vhost/net.c > > +++ b/drivers/vhost/net.c > > @@ -172,6 +172,7 @@ static void handle_tx(struct vhost_net *net) > > vhost_add_used_and_signal(&net->dev, vq, head, 0); > > total_len += len; > > if (unlikely(total_len >= VHOST_NET_WEIGHT)) { > > + tx_poll_start(net, sock); > > vhost_poll_queue(&vq->poll); > > break; > > } > > > > Thanks > > Sridhar > > > Hmm, this happens when > we have polled a lot of packets, and want to > give another vq a chance to poll. > Looks like a strange place to add it. I am also seeing sendmsg() calls failing with EAGAIN. Could be a bug in handling this error. The check for sendq full is done outside the for loop. It is possible that we can run out of sendq space within the for loop. Should we check for wmem within the for loop? Thanks Sridhar ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full 2010-02-18 20:59 [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full Sridhar Samudrala 2010-02-18 22:30 ` Michael S. Tsirkin @ 2010-02-23 10:24 ` Michael S. Tsirkin 2010-02-23 17:31 ` Sridhar Samudrala 1 sibling, 1 reply; 8+ messages in thread From: Michael S. Tsirkin @ 2010-02-23 10:24 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: David Miller, netdev On Thu, Feb 18, 2010 at 12:59:11PM -0800, Sridhar Samudrala wrote: > When running guest to remote host TCP stream test using vhost-net > via tap/macvtap, i am seeing network transmit hangs. This happens > when handle_tx() returns because of the socket send queue full > condition. > This patch fixes this by restarting tx poll when hitting this > condition. > > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com> > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c > index 91a324c..82d4bbe 100644 > --- a/drivers/vhost/net.c > +++ b/drivers/vhost/net.c > @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net) > if (!sock) > return; > > - wmem = atomic_read(&sock->sk->sk_wmem_alloc); > - if (wmem >= sock->sk->sk_sndbuf) > - return; > - > use_mm(net->dev.mm); > mutex_lock(&vq->mutex); > + > + wmem = atomic_read(&sock->sk->sk_wmem_alloc); > + if (wmem >= sock->sk->sk_sndbuf) { > + tx_poll_start(net, sock); > + set_bit(SOCK_ASYNC_NOSPACE, &sock->flags); > + goto unlock; > + } > + > vhost_disable_notify(vq); > > if (wmem < sock->sk->sk_sndbuf * 2) > @@ -178,6 +182,7 @@ static void handle_tx(struct vhost_net *net) > } > } > > +unlock: > mutex_unlock(&vq->mutex); > unuse_mm(net->dev.mm); > } It might be better to avoid use_mm when ring is full. Does the following fix the tx hang for you? diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 4c89283..f5f6efe 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -113,8 +113,12 @@ static void handle_tx(struct vhost_net *net) return; wmem = atomic_read(&sock->sk->sk_wmem_alloc); - if (wmem >= sock->sk->sk_sndbuf) - return; + if (wmem >= sock->sk->sk_sndbuf) { + mutex_lock(&vq->mutex); + tx_poll_start(net, sock); + mutex_unlock(&vq->mutex); + return; + } use_mm(net->dev.mm); mutex_lock(&vq->mutex); ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full 2010-02-23 10:24 ` Michael S. Tsirkin @ 2010-02-23 17:31 ` Sridhar Samudrala 2010-02-26 3:09 ` Shirley Ma 0 siblings, 1 reply; 8+ messages in thread From: Sridhar Samudrala @ 2010-02-23 17:31 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: David Miller, netdev On Tue, 2010-02-23 at 12:24 +0200, Michael S. Tsirkin wrote: > On Thu, Feb 18, 2010 at 12:59:11PM -0800, Sridhar Samudrala wrote: > > When running guest to remote host TCP stream test using vhost-net > > via tap/macvtap, i am seeing network transmit hangs. This happens > > when handle_tx() returns because of the socket send queue full > > condition. > > This patch fixes this by restarting tx poll when hitting this > > condition. > > > > Signed-off-by: Sridhar Samudrala <sri@us.ibm.com> > > > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c > > index 91a324c..82d4bbe 100644 > > --- a/drivers/vhost/net.c > > +++ b/drivers/vhost/net.c > > @@ -113,12 +113,16 @@ static void handle_tx(struct vhost_net *net) > > if (!sock) > > return; > > > > - wmem = atomic_read(&sock->sk->sk_wmem_alloc); > > - if (wmem >= sock->sk->sk_sndbuf) > > - return; > > - > > use_mm(net->dev.mm); > > mutex_lock(&vq->mutex); > > + > > + wmem = atomic_read(&sock->sk->sk_wmem_alloc); > > + if (wmem >= sock->sk->sk_sndbuf) { > > + tx_poll_start(net, sock); > > + set_bit(SOCK_ASYNC_NOSPACE, &sock->flags); > > + goto unlock; > > + } > > + > > vhost_disable_notify(vq); > > > > if (wmem < sock->sk->sk_sndbuf * 2) > > @@ -178,6 +182,7 @@ static void handle_tx(struct vhost_net *net) > > } > > } > > > > +unlock: > > mutex_unlock(&vq->mutex); > > unuse_mm(net->dev.mm); > > } > > > It might be better to avoid use_mm when ring is full. > Does the following fix the tx hang for you? Yes. this fixes the tx hang. Thanks Sridhar > > diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c > index 4c89283..f5f6efe 100644 > --- a/drivers/vhost/net.c > +++ b/drivers/vhost/net.c > @@ -113,8 +113,12 @@ static void handle_tx(struct vhost_net *net) > return; > > wmem = atomic_read(&sock->sk->sk_wmem_alloc); > - if (wmem >= sock->sk->sk_sndbuf) > - return; > + if (wmem >= sock->sk->sk_sndbuf) { > + mutex_lock(&vq->mutex); > + tx_poll_start(net, sock); > + mutex_unlock(&vq->mutex); > + return; > + } > > use_mm(net->dev.mm); > mutex_lock(&vq->mutex); > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full 2010-02-23 17:31 ` Sridhar Samudrala @ 2010-02-26 3:09 ` Shirley Ma 0 siblings, 0 replies; 8+ messages in thread From: Shirley Ma @ 2010-02-26 3:09 UTC (permalink / raw) To: Sridhar Samudrala; +Cc: Michael S. Tsirkin, David Miller, netdev This also fixes tx hung issue between guest and guest communication through vhost. Thanks Shirley ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2010-03-01 18:20 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-02-18 20:59 [PATCH net-next-2.6] vhost: Restart tx poll when socket send queue is full Sridhar Samudrala 2010-02-18 22:30 ` Michael S. Tsirkin 2010-02-19 2:00 ` Sridhar Samudrala 2010-02-19 14:42 ` Michael S. Tsirkin 2010-02-19 21:19 ` Sridhar Samudrala 2010-02-23 10:24 ` Michael S. Tsirkin 2010-02-23 17:31 ` Sridhar Samudrala 2010-02-26 3:09 ` Shirley Ma
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).