From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: virtio-net mq vq initialization Date: Sun, 14 Apr 2013 13:01:56 +0300 Message-ID: <20130414100156.GB2548@redhat.com> References: <1365698186-27355-1-git-send-email-will.deacon@arm.com> <516716B8.9090408@oracle.com> <874nfc9e1r.fsf@rustcorp.com.au> <5169CCDD.4070205@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Rusty Russell , Will Deacon , kvm@vger.kernel.org, penberg@kernel.org, marc.zyngier@arm.com To: Sasha Levin Return-path: Received: from mx1.redhat.com ([209.132.183.28]:26002 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750992Ab3DNKCK (ORCPT ); Sun, 14 Apr 2013 06:02:10 -0400 Content-Disposition: inline In-Reply-To: <5169CCDD.4070205@oracle.com> Sender: kvm-owner@vger.kernel.org List-ID: On Sat, Apr 13, 2013 at 05:23:41PM -0400, Sasha Levin wrote: > On 04/12/2013 07:36 AM, Rusty Russell wrote: > > Sasha Levin writes: > >> On 04/11/2013 12:36 PM, Will Deacon wrote: > >>> Hello folks, > >>> > >>> Here's the latest round of ARM fixes and updates for kvmtool. Most of > >>> this is confined to the arm/ subdirectory, with the exception of a fix > >>> to the virtio-mmio vq definitions due to the multi-queue work from > >>> Sasha. I'm not terribly happy about that code though, since it seriously > >>> increases the memory footprint of the guest. > >>> > >>> Without multi-queue, we can boot Debian Wheezy to a prompt in 38MB. With > >>> the new changes, that increases to 170MB! Any chance we can try and tackle > >>> this regression please? I keep getting bitten by the OOM killer :( > >> > >> (cc Rusty, MST) > >> > >> The spec defines the operation of a virtio-net device with regards to multiple > >> queues as follows: > >> > >> """ > >> Device Initialization > >> > >> 1. The initialization routine should identify the receive and transmission > >> virtqueues, up to N+1 of each kind. If VIRTIO_NET_F_MQ feature > >> bit is negotiated, N=max_virtqueue_pairs-1, otherwise identify N=0. > >> > >> [...] > >> > >> 5. Only receiveq0, transmitq0 and controlq are used by default. To use more > >> queues driver must negotiate the VIRTIO_NET_F_MQ feature; initialize > >> up to max_virtqueue_pairs of each of transmit and receive queues; execute_ > >> VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command specifying > >> the number of the transmit and receive queues that is going to be > >> used and wait until the device consumes the controlq buffer and acks this > >> command. > >> """ > >> > >> And kvmtool follows that to the letter: It will initialize the maximum amount of > >> queues it can support during initialization and will start using them only when > >> the device tells it it should use them. > >> > >> As Will has stated, this causes a memory issue since all the data structures that hold > >> all possible queues get initialized regardless of whether we actually need them or not, > >> which is quite troublesome for systems with small RAM. > >> > >> > >> Rusty, MST, would you be open to a spec and code change that would initialize the > >> RX/TX vqs on demand instead of on device initialization? Or is there an easier way > >> to work around this issue? > > > > I'm confused. kvmtool is using too much memory, or the guest? If > > kvmtool, the Device Initialization section above applies to the driver, > > not the device. If the guest, well, the language says "UP TO N+1". You > > want a small guest, don't use them all. Or any... > > > > What am I missing? > > It's in the guest - sorry. I was only trying to say that kvmtool doesn't do anything > odd with regards to initializing virtio-net. > > The thing is that there should be a difference between just allowing a larger number > of queues and actually using them (i.e. enabling them with ethtool). Right now I see > the kernel lose 130MB just by having kvmtool offer 8 queue pairs, without actually > using those queues. > > Yes, we can make it configurable in kvmtool (and I will make it so so the arm folks > could continue working with tiny guests) but does it make sense that you have to do > this configuration in *2* places? First in the hypervisor and then inside the guest? > > Memory usage should ideally depend on whether you are actually using multiple queues, > not on whether you just allow using those queues. > > > Thanks, > Sasha 8 queues eat up 130MB? Most of the memory is likely for the buffers? I think we could easily allocate these lazily as queues are enabled, without protocol changes. It's harder to clean them as there's no way to reset a specific queue, but maybe that' good enough for your purposes? -- MST