From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sasha Levin Subject: Re: [PATCH] virtio-ring: Use threshold for switching to indirect descriptors Date: Sun, 04 Dec 2011 20:23:59 +0200 Message-ID: <1323023039.3256.7.camel@lappy> References: <4ED4F30F.8000603@redhat.com> <1322669511.3985.8.camel@lappy> <87wrahrp0u.fsf@rustcorp.com.au> <20111201075847.GA5479@redhat.com> <1322726977.3259.3.camel@lappy> <20111201102640.GB8822@redhat.com> <87zkfbre9x.fsf@rustcorp.com.au> <1322913028.3782.4.camel@lappy> <4EDB5EF0.2010909@redhat.com> <1323000831.4205.4.camel@lappy> <20111204162221.GB22501@redhat.com> <1323020088.3256.3.camel@lappy> <4EDBAFC5.2010405@redhat.com> <1323020374.3256.5.camel@lappy> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Cc: markmc@redhat.com, kvm@vger.kernel.org, "Michael S. Tsirkin" , linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org To: Avi Kivity Return-path: In-Reply-To: <1323020374.3256.5.camel@lappy> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org List-Id: kvm.vger.kernel.org On Sun, 2011-12-04 at 19:39 +0200, Sasha Levin wrote: > On Sun, 2011-12-04 at 19:37 +0200, Avi Kivity wrote: > > On 12/04/2011 07:34 PM, Sasha Levin wrote: > > > > > > > > I'm confused. didn't you see a bigger benefit for guest->host by > > > > switching indirect off? > > > > > > The 5% improvement is over the 'regular' indirect on, not over indirect > > > off. Sorry for the confusion there. > > > > > > I suggested this change regardless of the outcome of indirect descriptor > > > threshold discussion, since it would help anyways. > > > > For net, this makes sense. For block, it reduces the effective queue > > depth, so it's not a trivial change. It probably makes sense there too, > > though. > > It doesn't have to be limited at that number, anything above that can go > through the regular kmalloc() path. Something like the following patch: diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index c7a2c20..3166ca0 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -82,6 +82,7 @@ struct vring_virtqueue /* Host supports indirect buffers */ bool indirect; + struct kmem_cache *indirect_cache; /* Host publishes avail event idx */ bool event; @@ -110,6 +111,9 @@ struct vring_virtqueue #define to_vvq(_vq) container_of(_vq, struct vring_virtqueue, vq) +static unsigned int ind_alloc_thresh = 0; +module_param(ind_alloc_thresh, uint, S_IRUGO); + /* Set up an indirect table of descriptors and add it to the queue. */ static int vring_add_indirect(struct vring_virtqueue *vq, struct scatterlist sg[], @@ -121,7 +125,10 @@ static int vring_add_indirect(struct vring_virtqueue *vq, unsigned head; int i; - desc = kmalloc((out + in) * sizeof(struct vring_desc), gfp); + if ((out + in) <= ind_alloc_thresh) + desc = kmem_cache_alloc(vq->indirect_cache, gfp); + else + desc = kmalloc((out + in) * sizeof(struct vring_desc), gfp); if (!desc) return -ENOMEM; @@ -479,6 +486,9 @@ struct virtqueue *vring_new_virtqueue(unsigned int num, vq->broken = false; vq->last_used_idx = 0; vq->num_added = 0; + if (ind_alloc_thresh) + vq->indirect_cache = KMEM_CACHE(vring_desc[ind_alloc_thresh], 0); list_add_tail(&vq->vq.list, &vdev->vqs); #ifdef DEBUG vq->in_use = false; -- Sasha.