From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: vhost-net patches Date: Wed, 28 Oct 2009 17:39:00 +0200 Message-ID: <20091028153859.GA28926@redhat.com> References: <20091023110438.GA20229@redhat.com> <1256310168.4443.2.camel@localhost.localdomain> <1256310765.4443.4.camel@localhost.localdomain> <1256315020.4443.12.camel@localhost.localdomain> <20091026200513.GA26623@redhat.com> <1256592889.10142.8.camel@localhost.localdomain> <20091027064302.GB26914@redhat.com> <1256654819.4753.6.camel@localhost.localdomain> <20091027152753.GA4622@redhat.com> <1256661378.6745.2.camel@localhost.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Sridhar Samudrala , Shirley Ma , David Stevens , kvm@vger.kernel.org, sri@linux.vnet.ibm.com, mashirle@linux.vnet.ibm.com To: Shirley Ma Return-path: Received: from mx1.redhat.com ([209.132.183.28]:32259 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754388AbZJ1PlU (ORCPT ); Wed, 28 Oct 2009 11:41:20 -0400 Content-Disposition: inline In-Reply-To: <1256661378.6745.2.camel@localhost.localdomain> Sender: kvm-owner@vger.kernel.org List-ID: On Tue, Oct 27, 2009 at 09:36:18AM -0700, Shirley Ma wrote: > Hello Michael, > > On Tue, 2009-10-27 at 17:27 +0200, Michael S. Tsirkin wrote: > > Possibly GFP_ATOMIC allocations in vring_add_indirect are failing? > > Is there a chance you are tight on guest memory for some reason? > > with vhost, virtio does currently consume a bit more memory than > > with userspace backend. > > I did see memory leak on host every time after exiting guest. I don't > know where. Do you see it? > > Anyway after I reboot host and restart guest with large memory > allocation, I do see performance improves to 3xxxMb/s, and occasionally > reaches 40xxMb/s. But "queue full" still exists, I can avoid the problem > by increasing send queue size from qemu. Here's another hack to try. It will break raw sockets, but just as a test: diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index a140dad..e830b30 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -106,22 +106,41 @@ static void handle_tx(struct vhost_net *net) .msg_flags = MSG_DONTWAIT, }; size_t len, total_len = 0; - int err; + int err, wmem; size_t hdr_size; struct socket *sock = rcu_dereference(vq->private_data); - if (!sock || !sock_writeable(sock->sk)) + if (!sock) + return; + + wmem = atomic_read(&sock->sk->sk_wmem_alloc); + if (wmem >= sock->sk->sk_sndbuf) return; use_mm(net->dev.mm); mutex_lock(&vq->mutex); - tx_poll_stop(net); + vhost_no_notify(vq); + + if (wmem >= sock->sk->sk_sndbuf * 3 / 4) { + //tx_poll_start(net); + } else { + //tx_poll_stop(net); + } hdr_size = vq->hdr_size; for (;;) { head = vhost_get_vq_desc(&net->dev, vq, vq->iov, &out, &in); /* Nothing new? Wait for eventfd to tell us they refilled. */ - if (head == vq->num) + if (head == vq->num) { + wmem = atomic_read(&sock->sk->sk_wmem_alloc); + if (wmem >= sock->sk->sk_sndbuf * 3 / 4) { + set_bit(SOCK_ASYNC_NOSPACE, &sock->flags); + break; + } + if (vhost_notify(vq)) { + continue; + } break; + } if (in) { vq_err(vq, "Unexpected descriptor format for TX: " "out %d, int %d\n", out, in); diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index 30708c6..67bfc08 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -775,7 +775,7 @@ void vhost_no_notify(struct vhost_virtqueue *vq) int vhost_init(void) { - vhost_workqueue = create_workqueue("vhost"); + vhost_workqueue = create_singlethread_workqueue("vhost"); if (!vhost_workqueue) return -ENOMEM; return 0;