From mboxrd@z Thu Jan  1 00:00:00 1970
From: Arjan Koers <0h61vkll2ly8@xutrox.com>
Subject: vhost-net guest memory usage
Date: Mon, 27 Dec 2010 21:48:19 +0100
Message-ID: <4D18FB93.8000105@xutrox.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
To: kvm@vger.kernel.org
Return-path: <kvm-owner@vger.kernel.org>
Received: from smtp-out1.tiscali.nl ([195.241.79.176]:43146 "EHLO
	smtp-out1.tiscali.nl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1751132Ab0L0VIm (ORCPT <rfc822;kvm@vger.kernel.org>);
	Mon, 27 Dec 2010 16:08:42 -0500
Received: from [82.169.46.210] (helo=plutonium.lan)
	by smtp-out1.tiscali.nl with esmtp (Exim)
	(envelope-from <0h61vkll2ly8@xutrox.com>)
	id 1PXJzR-0001lZ-1w
	for kvm@vger.kernel.org; Mon, 27 Dec 2010 21:48:21 +0100
Received: from [IPv6:2001:610:620:0:8551:4bbd:4077:f18d] (unknown [IPv6:2001:610:620:0:8551:4bbd:4077:f18d])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by plutonium.lan (Postfix) with ESMTPS id 6C26847C01
	for <kvm@vger.kernel.org>; Mon, 27 Dec 2010 20:48:18 +0000 (UTC)
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

When running my guests with 'vhost=on', a lot less guest memory is
available (about 18.5 MB less for each networking device).

I searched the mailing list archive and found a comment by Michael S.
Tsirkin:
"with vhost, virtio does currently consume a bit more memory than with
userspace backend"

Is 18.5 MB per device the expected amount, and can I change some
setting to reduce it?


Here's the 'free' output of a guest with 7 tap devices (qemu-kvm 0.13.0,
linux 2.6.36.2 host and guest):

Started the guest with 'vhost=off':
	             total       used       free     shared    buffers     cached
	Mem:        253564      39260     214304          0       1732      15124
	-/+ buffers/cache:      22404     231160
	Swap:       409620          0     409620

Same guest, but with 'vhost=on':
	             total       used       free     shared    buffers     cached
	Mem:        253564     169092      84472          0       1744      15120
	-/+ buffers/cache:     152228     101336
	Swap:       409620          0     409620

The command that started the guest:
/opt/kvm/bin/qemu-system-x86_64 \
  -nographic \
  -daemonize \
  -cpu host \
  -M pc-0.13 \
  -m 256 \
  -mem-path /hugepages \
  -smp 2 \
  -name gw \
  -uuid d9d6332d-10ba-4cfa-925b-4e4d14f7cd86 \
  -boot c \
  -drive file=/opt/kvm/var/img/gw.raw,if=virtio,index=0,boot=on,cache=none \
  -device virtio-net-pci,netdev=virtio.0,mac=fe:60:20:3a:8d:37 \
  -netdev tap,id=virtio.0,ifname=gw-lan,script=no,downscript=no,vhost=on \
  -device virtio-net-pci,netdev=virtio.1,mac=06:c8:4d:85:47:45 \
  -netdev tap,id=virtio.1,ifname=gw-dmz,script=no,downscript=no,vhost=on \
  -device virtio-net-pci,netdev=virtio.2,mac=e2:57:e2:6b:67:7c \
  -netdev tap,id=virtio.2,ifname=gw-inet,script=no,downscript=no,vhost=on \
  -device virtio-net-pci,netdev=virtio.3,mac=72:47:f5:4d:59:01 \
  -netdev tap,id=virtio.3,ifname=gw-lanv,script=no,downscript=no,vhost=on \
  -device virtio-net-pci,netdev=virtio.4,mac=02:de:c4:7b:71:6e \
  -netdev tap,id=virtio.4,ifname=gw-dmzv,script=no,downscript=no,vhost=on \
  -device virtio-net-pci,netdev=virtio.5,mac=fe:6a:ac:4e:16:f3 \
  -netdev tap,id=virtio.5,ifname=gw-lanvpn,script=no,downscript=no,vhost=on \
  -device virtio-net-pci,netdev=virtio.6,mac=7e:33:75:7a:ec:cd \
  -netdev tap,id=virtio.6,ifname=gw-dmzvpn,script=no,downscript=no,vhost=on \
  -serial unix:/opt/kvm/var/run/gw-serial.socket,server,nowait \
  -monitor unix:/opt/kvm/var/run/gw-monitor.socket,server,nowait \
  -parallel none \
  -vga cirrus \
  -runas kvm-gw