virtio-net mq vq initialization (was: [PATCH 0/5] Usual batch of random ARM fixes for kvmtool)

[Sasha Levin <sasha.levin@oracle.com>]

On 04/11/2013 12:36 PM, Will Deacon wrote:
> Hello folks,
> 
> Here's the latest round of ARM fixes and updates for kvmtool. Most of
> this is confined to the arm/ subdirectory, with the exception of a fix
> to the virtio-mmio vq definitions due to the multi-queue work from
> Sasha. I'm not terribly happy about that code though, since it seriously
> increases the memory footprint of the guest.
> 
> Without multi-queue, we can boot Debian Wheezy to a prompt in 38MB. With
> the new changes, that increases to 170MB! Any chance we can try and tackle
> this regression please? I keep getting bitten by the OOM killer :(

(cc Rusty, MST)

The spec defines the operation of a virtio-net device with regards to multiple
queues as follows:

"""
Device Initialization

	1. The initialization routine should identify the receive and transmission
virtqueues, up to N+1 of each kind. If VIRTIO_NET_F_MQ feature
bit is negotiated, N=max_virtqueue_pairs-1, otherwise identify N=0.

	[...]

	5. Only receiveq0, transmitq0 and controlq are used by default. To use more
queues driver must negotiate the VIRTIO_NET_F_MQ feature; initialize
up to max_virtqueue_pairs of each of transmit and receive queues; execute_
VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command specifying
the number of the transmit and receive queues that is going to be
used and wait until the device consumes the controlq buffer and acks this
command.
"""

And kvmtool follows that to the letter: It will initialize the maximum amount of
queues it can support during initialization and will start using them only when
the device tells it it should use them.

As Will has stated, this causes a memory issue since all the data structures that hold
all possible queues get initialized regardless of whether we actually need them or not,
which is quite troublesome for systems with small RAM.


Rusty, MST, would you be open to a spec and code change that would initialize the
RX/TX vqs on demand instead of on device initialization? Or is there an easier way
to work around this issue?


Thanks,
Sasha