* [PATCH 0/3] vhost: fix use_mm usage
From: Michael S. Tsirkin @ 2009-12-20 17:16 UTC (permalink / raw)
To: Rusty Russell, virtualization, linux-kernel; +Cc: Al Viro
This patchset fixes an issue, pointed out by Al Viro, in how vhost net
calls use_mm, Works fine for me, and it's pretty straightforward. If no
one sees any issues with it, this will be a good candidate for 2.6.33
(assuming vhost net itself gets pulled :( ).
Michael S. Tsirkin (3):
vhost: prevent modification of an active ring
vhost: add access_ok checks
vhost: make default mapping empty by default
drivers/vhost/net.c | 23 ++++++--
drivers/vhost/vhost.c | 153 ++++++++++++++++++++++++++++++++++++++++++++-----
drivers/vhost/vhost.h | 2 +
3 files changed, 157 insertions(+), 21 deletions(-)
--
MST
^ permalink raw reply
* [PULL rexmit] virtio & lguest
From: Rusty Russell @ 2009-12-18 2:09 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Michael S. Tsirkin, linux-kernel, virtualization
Mostly the new experimental vhost driver.
The following changes since commit b8a7f3cd7e8212e5c572178ff3b5a514861036a5:
Linus Torvalds (1):
Merge branch 'master' of git://git.kernel.org/.../viro/vfs-2.6
are available in the git repository at:
ssh://master.kernel.org/pub/scm/linux/kernel/git/rusty/linux-2.6-for-linus.git virtio-lguest
Adam Litke (2):
virtio: Add memory statistics reporting to the balloon driver (V4)
virtio: Fix scheduling while atomic in virtio_balloon stats
Michael S. Tsirkin (4):
tun: export underlying socket
mm: export use_mm/unuse_mm to modules
vhost_net: a kernel-level virtio server
vhost: add missing architectures
Rusty Russell (1):
lguest: remove unneeded zlib.h include in example launcher
Documentation/lguest/lguest.c | 1 -
MAINTAINERS | 9 +
arch/ia64/kvm/Kconfig | 1 +
arch/powerpc/kvm/Kconfig | 1 +
arch/s390/kvm/Kconfig | 1 +
arch/x86/kvm/Kconfig | 1 +
drivers/Makefile | 1 +
drivers/net/tun.c | 101 ++++-
drivers/vhost/Kconfig | 11 +
drivers/vhost/Makefile | 2 +
drivers/vhost/net.c | 648 ++++++++++++++++++++++++++
drivers/vhost/vhost.c | 968 +++++++++++++++++++++++++++++++++++++++
drivers/vhost/vhost.h | 159 +++++++
drivers/virtio/virtio_balloon.c | 108 ++++-
include/linux/Kbuild | 1 +
include/linux/if_tun.h | 14 +
include/linux/miscdevice.h | 1 +
include/linux/vhost.h | 130 ++++++
include/linux/virtio_balloon.h | 15 +
mm/mmu_context.c | 3 +
20 files changed, 2148 insertions(+), 28 deletions(-)
create mode 100644 drivers/vhost/Kconfig
create mode 100644 drivers/vhost/Makefile
create mode 100644 drivers/vhost/net.c
create mode 100644 drivers/vhost/vhost.c
create mode 100644 drivers/vhost/vhost.h
create mode 100644 include/linux/vhost.h
commit 740773dda21f343074235c63dc5bb83fa69887d4
Author: Michael S. Tsirkin <mst@redhat.com>
Date: Wed Nov 4 17:55:02 2009 +0200
tun: export underlying socket
Tun device looks similar to a packet socket
in that both pass complete frames from/to userspace.
This patch fills in enough fields in the socket underlying tun driver
to support sendmsg/recvmsg operations, and message flags
MSG_TRUNC and MSG_DONTWAIT, and exports access to this socket
to modules. Regular read/write behaviour is unchanged.
This way, code using raw sockets to inject packets
into a physical device, can support injecting
packets into host network stack almost without modification.
First user of this interface will be vhost virtualization
accelerator.
Signed-off-by: "Michael S. Tsirkin" <mst@redhat.com>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
Acked-by: "David S. Miller" <davem@davemloft.net>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
drivers/net/tun.c | 101 +++++++++++++++++++++++++++++++++++++++---------
include/linux/if_tun.h | 14 +++++++
2 files changed, 96 insertions(+), 19 deletions(-)
commit 3f451096d762f71defd8cd5cd821b0e8aa57edf3
Author: Michael S. Tsirkin <mst@redhat.com>
Date: Wed Nov 4 17:55:38 2009 +0200
mm: export use_mm/unuse_mm to modules
vhost net module wants to do copy to/from user from a kernel thread,
which needs use_mm. Export it to modules.
Acked-by: Andrea Arcangeli <aarcange@redhat.com>
Signed-off-by: "Michael S. Tsirkin" <mst@redhat.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
mm/mmu_context.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)
commit 2c1566dc10c2dcacfbc386cae49d027b8e9e87df
Author: Michael S. Tsirkin <mst@redhat.com>
Date: Mon Nov 9 19:22:30 2009 +0200
vhost_net: a kernel-level virtio server
What it is: vhost net is a character device that can be used to reduce
the number of system calls involved in virtio networking.
Existing virtio net code is used in the guest without modification.
There's similarity with vringfd, with some differences and reduced scope
- uses eventfd for signalling
- structures can be moved around in memory at any time (good for
migration, bug work-arounds in userspace)
- write logging is supported (good for migration)
- support memory table and not just an offset (needed for kvm)
common virtio related code has been put in a separate file vhost.c and
can be made into a separate module if/when more backends appear. I used
Rusty's lguest.c as the source for developing this part : this supplied
me with witty comments I wouldn't be able to write myself.
What it is not: vhost net is not a bus, and not a generic new system
call. No assumptions are made on how guest performs hypercalls.
Userspace hypervisors are supported as well as kvm.
How it works: Basically, we connect virtio frontend (configured by
userspace) to a backend. The backend could be a network device, or a tap
device. Backend is also configured by userspace, including vlan/mac
etc.
Status: This works for me, and I haven't see any crashes.
Compared to userspace, people reported improved latency (as I save up to
4 system calls per packet), as well as better bandwidth and CPU
utilization.
Features that I plan to look at in the future:
- mergeable buffers
- zero copy
- scalability tuning: figure out the best threading model to use
Note on RCU usage (this is also documented in vhost.h, near
private_pointer which is the value protected by this variant of RCU):
what is happening is that the rcu_dereference() is being used in a
workqueue item. The role of rcu_read_lock() is taken on by the start of
execution of the workqueue item, of rcu_read_unlock() by the end of
execution of the workqueue item, and of synchronize_rcu() by
flush_workqueue()/flush_work(). In the future we might need to apply
some gcc attribute or sparse annotation to the function passed to
INIT_WORK(). Paul's ack below is for this RCU usage.
(Includes fixes by Alan Cox <alan@linux.intel.com>)
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Signed-off-by: "Michael S. Tsirkin" <mst@redhat.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
MAINTAINERS | 9 +
arch/x86/kvm/Kconfig | 1 +
drivers/Makefile | 1 +
drivers/vhost/Kconfig | 11 +
drivers/vhost/Makefile | 2 +
drivers/vhost/net.c | 648 +++++++++++++++++++++++++++++
drivers/vhost/vhost.c | 968 ++++++++++++++++++++++++++++++++++++++++++++
drivers/vhost/vhost.h | 159 ++++++++
include/linux/Kbuild | 1 +
include/linux/miscdevice.h | 1 +
include/linux/vhost.h | 130 ++++++
11 files changed, 1931 insertions(+), 0 deletions(-)
commit f085e4f06bf04d45cb7498e9cb14c2e002dd1c31
Author: Michael S. Tsirkin <mst@redhat.com>
Date: Thu Dec 17 15:01:46 2009 +0200
vhost: add missing architectures
vhost is completely portable, but Kconfig include was missing for all
architectures besides x86, so it did not appear in the menu. Add the
relevant Kconfig includes to all architectures that support
virtualization.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
arch/ia64/kvm/Kconfig | 1 +
arch/powerpc/kvm/Kconfig | 1 +
arch/s390/kvm/Kconfig | 1 +
3 files changed, 3 insertions(+), 0 deletions(-)
commit a8945c9bf89df5deaba16c2a65f4cf18060299c2
Author: Adam Litke <agl@us.ibm.com>
Date: Mon Nov 30 10:14:15 2009 -0600
virtio: Add memory statistics reporting to the balloon driver (V4)
Changes since V3:
- Do not do endian conversions as they will be done in the host
- Report stats that reference a quantity of memory in bytes
- Minor coding style updates
Changes since V2:
- Increase stat field size to 64 bits
- Report all sizes in kb (not pages)
- Drop anon_pages stat and fix endianness conversion
Changes since V1:
- Use a virtqueue instead of the device config space
When using ballooning to manage overcommitted memory on a host, a system for
guests to communicate their memory usage to the host can provide information
that will minimize the impact of ballooning on the guests. The current method
employs a daemon running in each guest that communicates memory statistics to a
host daemon at a specified time interval. The host daemon aggregates this
information and inflates and/or deflates balloons according to the level of
host memory pressure. This approach is effective but overly complex since a
daemon must be installed inside each guest and coordinated to communicate with
the host. A simpler approach is to collect memory statistics in the virtio
balloon driver and communicate them directly to the hypervisor.
This patch enables the guest-side support by adding stats collection and
reporting to the virtio balloon driver.
Signed-off-by: Adam Litke <agl@us.ibm.com>
Cc: Anthony Liguori <anthony@codemonkey.ws>
Cc: virtualization@lists.linux-foundation.org
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> (minor fixes)
drivers/virtio/virtio_balloon.c | 94 +++++++++++++++++++++++++++++++++++---
include/linux/virtio_balloon.h | 15 ++++++
2 files changed, 101 insertions(+), 8 deletions(-)
commit 417c5a603344ef72958f6813393690a2bdca030f
Author: Adam Litke <agl@us.ibm.com>
Date: Thu Dec 10 16:35:15 2009 -0600
virtio: Fix scheduling while atomic in virtio_balloon stats
This is a fix for my earlier patch: "virtio: Add memory statistics reporting to
the balloon driver (V4)".
I discovered that all_vm_events() can sleep and therefore stats collection
cannot be done in interrupt context. One solution is to handle the interrupt
by noting that stats need to be collected and waking the existing vballoon
kthread which will complete the work via stats_handle_request(). Rusty, is
this a saner way of doing business?
There is one issue that I would like a broader opinion on. In stats_request, I
update vb->need_stats_update and then wake up the kthread. The kthread uses
vb->need_stats_update as a condition variable. Do I need a memory barrier
between the update and wake_up to ensure that my kthread sees the correct
value? My testing suggests that it is not needed but I would like some
confirmation from the experts.
Signed-off-by: Adam Litke <agl@us.ibm.com>
To: Rusty Russell <rusty@rustcorp.com.au>
Cc: Anthony Liguori <aliguori@linux.vnet.ibm.com>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
drivers/virtio/virtio_balloon.c | 22 ++++++++++++++++++----
1 files changed, 18 insertions(+), 4 deletions(-)
commit c7ff121447eaeb96fc722016db9dce4cfa69d4fa
Author: Rusty Russell <rusty@rustcorp.com.au>
Date: Fri Dec 18 12:36:51 2009 -0600
lguest: remove unneeded zlib.h include in example launcher
Two years ago 5bbf89fc2608 removed the horrible bzImage unpacking code.
Now it's time to remove the unneeded zlib.h include, too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Documentation/lguest/lguest.c | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
^ permalink raw reply
* [PATCH-cc-fixed] vhost: add missing architectures
From: Michael S. Tsirkin @ 2009-12-17 13:01 UTC (permalink / raw)
To: rusty, virtualization
Cc: Fenghua Yu, Tony Luck, linux-ia64, kvm, Michael S. Tsirkin,
kvm-ia64, linux-s390, linux-kernel, kvm-ppc, Avi Kivity
vhost is completely portable, but Kconfig include was missing for all
architectures besides x86, so it did not appear in the menu. Add the
relevant Kconfig includes to all architectures that support
virtualization.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
Some IBM addresses seem to be bouncing. Reposting with these removed,
please send replies to this shorter list to avoid bounces. Sorry about
the noise.
Hi Rusty,
please apply the following trivial fixup patch for 2.6.33.
Thanks!
arch/ia64/kvm/Kconfig | 1 +
arch/powerpc/kvm/Kconfig | 1 +
arch/s390/kvm/Kconfig | 1 +
3 files changed, 3 insertions(+), 0 deletions(-)
diff --git a/arch/ia64/kvm/Kconfig b/arch/ia64/kvm/Kconfig
index ef3e7be..01c7579 100644
--- a/arch/ia64/kvm/Kconfig
+++ b/arch/ia64/kvm/Kconfig
@@ -47,6 +47,7 @@ config KVM_INTEL
Provides support for KVM on Itanium 2 processors equipped with the VT
extensions.
+source drivers/vhost/Kconfig
source drivers/virtio/Kconfig
endif # VIRTUALIZATION
diff --git a/arch/powerpc/kvm/Kconfig b/arch/powerpc/kvm/Kconfig
index c299268..a1b4c5d 100644
--- a/arch/powerpc/kvm/Kconfig
+++ b/arch/powerpc/kvm/Kconfig
@@ -58,6 +58,7 @@ config KVM_E500
If unsure, say N.
+source drivers/vhost/Kconfig
source drivers/virtio/Kconfig
endif # VIRTUALIZATION
diff --git a/arch/s390/kvm/Kconfig b/arch/s390/kvm/Kconfig
index bf164fc..6be6aea 100644
--- a/arch/s390/kvm/Kconfig
+++ b/arch/s390/kvm/Kconfig
@@ -36,6 +36,7 @@ config KVM
# OK, it's a little counter-intuitive to do this, but it puts it neatly under
# the virtualization menu.
+source drivers/vhost/Kconfig
source drivers/virtio/Kconfig
endif # VIRTUALIZATION
--
1.6.6.rc1.43.gf55cc
^ permalink raw reply related
* [PATCH] vhost: add missing architectures
From: Michael S. Tsirkin @ 2009-12-17 12:50 UTC (permalink / raw)
To: rusty, virtualization
Cc: Fenghua Yu, Tony Luck, linux-ia64, Hollis Blanchard,
Michael S. Tsirkin, kvm-ia64, linux-s390, Heiko Carstens,
linux-kernel, kvm-ppc, Christian Ehrhardt, Avi Kivity, kvm,
Martin Schwidefsky, linux390, Carsten Otte, Christian Borntraeger
vhost is completely portable, but Kconfig include was missing for all
architectures besides x86, so it did not appear in the menu. Add the
relevant Kconfig includes to all architectures that support
virtualization.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
Hi Rusty,
please apply the following trivial fixup patch for 2.6.33.
Thanks!
arch/ia64/kvm/Kconfig | 1 +
arch/powerpc/kvm/Kconfig | 1 +
arch/s390/kvm/Kconfig | 1 +
3 files changed, 3 insertions(+), 0 deletions(-)
diff --git a/arch/ia64/kvm/Kconfig b/arch/ia64/kvm/Kconfig
index ef3e7be..01c7579 100644
--- a/arch/ia64/kvm/Kconfig
+++ b/arch/ia64/kvm/Kconfig
@@ -47,6 +47,7 @@ config KVM_INTEL
Provides support for KVM on Itanium 2 processors equipped with the VT
extensions.
+source drivers/vhost/Kconfig
source drivers/virtio/Kconfig
endif # VIRTUALIZATION
diff --git a/arch/powerpc/kvm/Kconfig b/arch/powerpc/kvm/Kconfig
index c299268..a1b4c5d 100644
--- a/arch/powerpc/kvm/Kconfig
+++ b/arch/powerpc/kvm/Kconfig
@@ -58,6 +58,7 @@ config KVM_E500
If unsure, say N.
+source drivers/vhost/Kconfig
source drivers/virtio/Kconfig
endif # VIRTUALIZATION
diff --git a/arch/s390/kvm/Kconfig b/arch/s390/kvm/Kconfig
index bf164fc..6be6aea 100644
--- a/arch/s390/kvm/Kconfig
+++ b/arch/s390/kvm/Kconfig
@@ -36,6 +36,7 @@ config KVM
# OK, it's a little counter-intuitive to do this, but it puts it neatly under
# the virtualization menu.
+source drivers/vhost/Kconfig
source drivers/virtio/Kconfig
endif # VIRTUALIZATION
--
1.6.6.rc1.43.gf55cc
^ permalink raw reply related
* RE: Guest bridge setup variations
From: Leonid Grossman @ 2009-12-17 6:18 UTC (permalink / raw)
To: Arnd Bergmann, virtualization; +Cc: qemu-devel
In-Reply-To: <200912161515.32820.arnd@arndb.de>
> -----Original Message-----
> From: Arnd Bergmann [mailto:arnd@arndb.de]
> Sent: Wednesday, December 16, 2009 6:16 AM
> To: virtualization@lists.linux-foundation.org
> Cc: Leonid Grossman; qemu-devel@nongnu.org
> Subject: Re: Guest bridge setup variations
>
> On Wednesday 16 December 2009, Leonid Grossman wrote:
> > > > 3. Doing the bridging in the NIC using macvlan in passthrough
> > > > mode. This lowers the CPU utilization further compared to 2,
> > > > at the expense of limiting throughput by the performance of
> > > > the PCIe interconnect to the adapter. Whether or not this
> > > > is a win is workload dependent.
> >
> > This is certainly true today for pci-e 1.1 and 2.0 devices, but
> > as NICs move to pci-e 3.0 (while remaining almost exclusively dual
> port
> > 10GbE for a long while),
> > EVB internal bandwidth will significantly exceed external bandwidth.
> > So, #3 can become a win for most inter-guest workloads.
>
> Right, it's also hardware dependent, but it usually comes down
> to whether it's cheaper to spend CPU cycles or to spend IO bandwidth.
>
> I would be surprised if all future machines with PCIe 3.0 suddenly
have
> a huge surplus of bandwidth but no CPU to keep up with that.
>
> > > > Access controls now happen
> > > > in the NIC. Currently, this is not supported yet, due to lack of
> > > > device drivers, but it will be an important scenario in the
> future
> > > > according to some people.
> >
> > Actually, x3100 10GbE drivers support this today via sysfs interface
> to
> > the host driver
> > that can choose to control VEB tables (and therefore MAC addresses,
> vlan
> > memberships, etc. for all passthru interfaces behind the VEB).
>
> Ok, I didn't know about that.
>
> > OF course a more generic vendor-independent interface will be
> important
> > in the future.
>
> Right. I hope we can come up with something soon. I'll have a look at
> what your driver does and see if that can be abstracted in some way.
Sounds good, please let us know if looking at the code/documentation
will suffice or you need a couple cards to go along with the code.
> I expect that if we can find an interface between the kernel and
device
> driver for two or three NIC implementations that it will be good
enough
> to adapt to everyone else as well.
The interface will likely evolve along with EVB standards and other
developments, but
initial implementation can be pretty basic (and vendor-independent).
Early IOV NIC deployments can benefit from an interface that sets couple
VF parameters missing in "legacy" NIC interface - things like bandwidth
limit and list of MAC addresses (since setting a NIC in promisc mode
doesn't work well for VEB, it is currently forced to learn the addresses
it is configured for).
The interface can also include querying IOV NIC capabilities like number
of VFs, support for VEB and/or VEPA mode, etc as well as getting VF
stats and MAC/VLAN tables - all in all, it is not a long list.
>
> Arnd
^ permalink raw reply
* Re: Guest bridge setup variations
From: Arnd Bergmann @ 2009-12-16 14:15 UTC (permalink / raw)
To: virtualization; +Cc: Leonid Grossman, qemu-devel
In-Reply-To: <78C9135A3D2ECE4B8162EBDCE82CAD7705FDEEDA@nekter>
On Wednesday 16 December 2009, Leonid Grossman wrote:
> > > 3. Doing the bridging in the NIC using macvlan in passthrough
> > > mode. This lowers the CPU utilization further compared to 2,
> > > at the expense of limiting throughput by the performance of
> > > the PCIe interconnect to the adapter. Whether or not this
> > > is a win is workload dependent.
>
> This is certainly true today for pci-e 1.1 and 2.0 devices, but
> as NICs move to pci-e 3.0 (while remaining almost exclusively dual port
> 10GbE for a long while),
> EVB internal bandwidth will significantly exceed external bandwidth.
> So, #3 can become a win for most inter-guest workloads.
Right, it's also hardware dependent, but it usually comes down
to whether it's cheaper to spend CPU cycles or to spend IO bandwidth.
I would be surprised if all future machines with PCIe 3.0 suddenly have
a huge surplus of bandwidth but no CPU to keep up with that.
> > > Access controls now happen
> > > in the NIC. Currently, this is not supported yet, due to lack of
> > > device drivers, but it will be an important scenario in the future
> > > according to some people.
>
> Actually, x3100 10GbE drivers support this today via sysfs interface to
> the host driver
> that can choose to control VEB tables (and therefore MAC addresses, vlan
> memberships, etc. for all passthru interfaces behind the VEB).
Ok, I didn't know about that.
> OF course a more generic vendor-independent interface will be important
> in the future.
Right. I hope we can come up with something soon. I'll have a look at
what your driver does and see if that can be abstracted in some way.
I expect that if we can find an interface between the kernel and device
driver for two or three NIC implementations that it will be good enough
to adapt to everyone else as well.
Arnd
^ permalink raw reply
* RE: Guest bridge setup variations
From: Leonid Grossman @ 2009-12-16 0:55 UTC (permalink / raw)
To: virtualization; +Cc: qemu-devel
In-Reply-To: <78C9135A3D2ECE4B8162EBDCE82CAD7705FDEDCF@nekter>
> > -----Original Message-----
> > From: virtualization-bounces@lists.linux-foundation.org
> > [mailto:virtualization-bounces@lists.linux-foundation.org] On Behalf
> Of
> > Arnd Bergmann
> > Sent: Tuesday, December 08, 2009 8:08 AM
> > To: virtualization@lists.linux-foundation.org
> > Cc: qemu-devel@nongnu.org
> > Subject: Guest bridge setup variations
> >
> > As promised, here is my small writeup on which setups I feel
> > are important in the long run for server-type guests. This
> > does not cover -net user, which is really for desktop kinds
> > of applications where you do not want to connect into the
> > guest from another IP address.
> >
> > I can see four separate setups that we may or may not want to
> > support, the main difference being how the forwarding between
> > guests happens:
> >
> > 1. The current setup, with a bridge and tun/tap devices on ports
> > of the bridge. This is what Gerhard's work on access controls is
> > focused on and the only option where the hypervisor actually
> > is in full control of the traffic between guests. CPU utilization
> should
> > be highest this way, and network management can be a burden,
> > because the controls are done through a Linux, libvirt and/or
> Director
> > specific interface.
> >
> > 2. Using macvlan as a bridging mechanism, replacing the bridge
> > and tun/tap entirely. This should offer the best performance on
> > inter-guest communication, both in terms of throughput and
> > CPU utilization, but offer no access control for this traffic at
all.
> > Performance of guest-external traffic should be slightly better
> > than bridge/tap.
> >
> > 3. Doing the bridging in the NIC using macvlan in passthrough
> > mode. This lowers the CPU utilization further compared to 2,
> > at the expense of limiting throughput by the performance of
> > the PCIe interconnect to the adapter. Whether or not this
> > is a win is workload dependent.
This is certainly true today for pci-e 1.1 and 2.0 devices, but
as NICs move to pci-e 3.0 (while remaining almost exclusively dual port
10GbE for a long while),
EVB internal bandwidth will significantly exceed external bandwidth.
So, #3 can become a win for most inter-guest workloads.
> > Access controls now happen
> > in the NIC. Currently, this is not supported yet, due to lack of
> > device drivers, but it will be an important scenario in the future
> > according to some people.
Actually, x3100 10GbE drivers support this today via sysfs interface to
the host driver
that can choose to control VEB tables (and therefore MAC addresses, vlan
memberships, etc. for all passthru interfaces behind the VEB).
OF course a more generic vendor-independent interface will be important
in the future.
> >
> > 4. Using macvlan for actual VEPA on the outbound interface.
> > This is mostly interesting because it makes the network access
> > controls visible in an external switch that is already managed.
> > CPU utilization and guest-external throughput should be
> > identical to 3, but inter-guest latency can only be worse because
> > all frames go through the external switch.
> >
> > In case 2 through 4, we have the choice between macvtap and
> > the raw packet interface for connecting macvlan to qemu.
> > Raw sockets are better tested right now, while macvtap has
> > better permission management (i.e. it does not require
> > CAP_NET_ADMIN). Neither one is upstream though at the
> > moment. The raw driver only requires qemu patches, while
> > macvtap requires both a new kernel driver and a trivial change
> > in qemu.
> >
> > In all four cases, vhost-net could be used to move the workload
> > from user space into the kernel, which may be an advantage.
> > The decision for or against vhost-net is entirely independent of
> > the other decisions.
> >
> > Arnd
> > _______________________________________________
> > Virtualization mailing list
> > Virtualization@lists.linux-foundation.org
> > https://lists.linux-foundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH 12/28] virtio: console: Buffer data that comes in from the host
From: Rusty Russell @ 2009-12-15 22:55 UTC (permalink / raw)
To: Amit Shah; +Cc: Shirley Ma, Michael S. Tsirkin, virtualization
In-Reply-To: <20091215104519.GA26975@amit-x200.redhat.com>
On Tue, 15 Dec 2009 09:15:19 pm Amit Shah wrote:
> On (Tue) Dec 08 2009 [11:46:10], Rusty Russell wrote:
> > On Fri, 4 Dec 2009 09:32:46 pm Amit Shah wrote:
> > > On (Thu) Dec 03 2009 [09:13:25], Amit Shah wrote:
> > > > On (Thu) Dec 03 2009 [09:24:23], Rusty Russell wrote:
> > > > > On Wed, 2 Dec 2009 07:54:06 pm Amit Shah wrote:
> > > > > > On (Wed) Dec 02 2009 [14:14:20], Rusty Russell wrote:
> > > > > > > On Sat, 28 Nov 2009 05:20:35 pm Amit Shah wrote:
> > > > > > > > The console could be flooded with data from the host; handle
> > > > > > > > this situation by buffering the data.
> > > > > > >
> > > > > > > All this complexity makes me really wonder if we should just
> > > > > > > have the host say the max # ports it will ever use, and just do this
> > > > > > > really dumbly. Yes, it's a limitation, but it'd be much simpler.
> > > > > >
> > > > > > As in make sure the max nr ports is less than 255 and have per-port vqs?
> > > > > > And then the buffering will be done inside the vqs themselves?
> > > > >
> > > > > Well < 128 (two vqs per port). The config would say (with a feature bit)
> > > > > how many vq pairs there are.
> > > >
> > > > Sure. This was how the previous versions behaved as well.
> > >
> > > I forgot one detail:
> > >
> > > http://www.mail-archive.com/virtualization@lists.linux-foundation.org/msg06079.html
> > >
> > > Some API changes are needed to pre-declare the number of vqs and the
> > > selectively enable them as ports get added.
> >
> > Couldn't we make it that all vqs *exist*, they're just unused unless the
> > bitmap (or whatever) indicates?
>
> Yes, but the current interface makes that a bit difficult: find_vqs
> needs the entire array for the callbacks. So if instead of find_vqs, we
> could have two functions,
>
> ret = init_vqs(vdev, nr_vqs);
> for (i = 0; i < nr_vqs; i += 2)
> enable_vqs(vdev, i, 2, callbacks, names);
>
> this would be simplified and we can also then enabling and disabling vqs
> as ports get hot plugged / unplugged.
No, I was thinking we find_vqs them all. We just don't put any buffers in
the unused ones.
Cheers,
Rusty.
^ permalink raw reply
* Re: [PATCH 12/28] virtio: console: Buffer data that comes in from the host
From: Amit Shah @ 2009-12-15 10:45 UTC (permalink / raw)
To: Rusty Russell; +Cc: Shirley Ma, Michael S. Tsirkin, virtualization
In-Reply-To: <200912081146.11158.rusty@rustcorp.com.au>
On (Tue) Dec 08 2009 [11:46:10], Rusty Russell wrote:
> On Fri, 4 Dec 2009 09:32:46 pm Amit Shah wrote:
> > On (Thu) Dec 03 2009 [09:13:25], Amit Shah wrote:
> > > On (Thu) Dec 03 2009 [09:24:23], Rusty Russell wrote:
> > > > On Wed, 2 Dec 2009 07:54:06 pm Amit Shah wrote:
> > > > > On (Wed) Dec 02 2009 [14:14:20], Rusty Russell wrote:
> > > > > > On Sat, 28 Nov 2009 05:20:35 pm Amit Shah wrote:
> > > > > > > The console could be flooded with data from the host; handle
> > > > > > > this situation by buffering the data.
> > > > > >
> > > > > > All this complexity makes me really wonder if we should just
> > > > > > have the host say the max # ports it will ever use, and just do this
> > > > > > really dumbly. Yes, it's a limitation, but it'd be much simpler.
> > > > >
> > > > > As in make sure the max nr ports is less than 255 and have per-port vqs?
> > > > > And then the buffering will be done inside the vqs themselves?
> > > >
> > > > Well < 128 (two vqs per port). The config would say (with a feature bit)
> > > > how many vq pairs there are.
> > >
> > > Sure. This was how the previous versions behaved as well.
> >
> > I forgot one detail:
> >
> > http://www.mail-archive.com/virtualization@lists.linux-foundation.org/msg06079.html
> >
> > Some API changes are needed to pre-declare the number of vqs and the
> > selectively enable them as ports get added.
>
> Couldn't we make it that all vqs *exist*, they're just unused unless the
> bitmap (or whatever) indicates?
Yes, but the current interface makes that a bit difficult: find_vqs
needs the entire array for the callbacks. So if instead of find_vqs, we
could have two functions,
ret = init_vqs(vdev, nr_vqs);
for (i = 0; i < nr_vqs; i += 2)
enable_vqs(vdev, i, 2, callbacks, names);
this would be simplified and we can also then enabling and disabling vqs
as ports get hot plugged / unplugged.
Amit
^ permalink raw reply
* Re: [RFC] macvlan: add tap device backend
From: Arnd Bergmann @ 2009-12-14 15:40 UTC (permalink / raw)
To: Patrick McHardy
Cc: Herbert Xu, Michael S. Tsirkin, Fischer, Anna, netdev, bridge,
linux-kernel, virtualization, Anthony Liguori,
Edge Virtual Bridging, Stephen Hemminger, davem
In-Reply-To: <4B26353A.9020005@trash.net>
On Monday 14 December 2009, Patrick McHardy wrote:
> > + classdev = device_create(macvtap_class, &dev->dev, devt,
> > + dev, "tap%d", dev->ifindex);
> > + if (IS_ERR(classdev)) {
> > + err = PTR_ERR(classdev);
> > + macvtap_del_queues(dev);
> > + macvlan_dellink(dev, NULL);
>
> I think this may cause a double free since macvlan_dellink() will
> free the device and rtnl_newlink() will free it again on error.
Ah, right. I've changed this part a few times now, but I think introduced
the same bug again in the current version. I'll send out what I have now
if you want to take another look.
Arnd
^ permalink raw reply
* Re: [RFC] macvlan: add tap device backend
From: Patrick McHardy @ 2009-12-14 12:53 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Herbert Xu, Michael S. Tsirkin, Fischer, Anna, netdev, bridge,
linux-kernel, virtualization, Anthony Liguori,
Edge Virtual Bridging, Stephen Hemminger, davem
In-Reply-To: <1259862720-28432-3-git-send-email-arnd@arndb.de>
Arnd Bergmann wrote:
> +static int macvtap_newlink(struct net *src_net,
> + struct net_device *dev,
> + struct nlattr *tb[],
> + struct nlattr *data[])
> +{
> + struct device *classdev;
> + dev_t devt;
> + int err;
> +
> + err = macvlan_common_newlink(src_net, dev, tb, data,
> + macvtap_receive, macvtap_forward);
> + if (err)
> + goto out;
> +
> + devt = MKDEV(MAJOR(macvtap_major), dev->ifindex);
> +
> + classdev = device_create(macvtap_class, &dev->dev, devt,
> + dev, "tap%d", dev->ifindex);
> + if (IS_ERR(classdev)) {
> + err = PTR_ERR(classdev);
> + macvtap_del_queues(dev);
> + macvlan_dellink(dev, NULL);
I think this may cause a double free since macvlan_dellink() will
free the device and rtnl_newlink() will free it again on error.
> + }
> +
> +out:
> + return err;
> +}
> +
> +static void macvtap_dellink(struct net_device *dev,
> + struct list_head *head)
> +{
> + device_destroy(macvtap_class,
> + MKDEV(MAJOR(macvtap_major), dev->ifindex));
> +
> + macvtap_del_queues(dev);
> + macvlan_dellink(dev, head);
> +}
> +
> +static struct rtnl_link_ops macvtap_link_ops __read_mostly = {
> + .kind = "macvtap",
> + .newlink = macvtap_newlink,
> + .dellink = macvtap_dellink,
> +};
^ permalink raw reply
* Re: [RFC] macvlan: add tap device backend
From: David Miller @ 2009-12-14 12:38 UTC (permalink / raw)
To: arnd
Cc: herbert, mst, anna.fischer, netdev, bridge, linux-kernel,
virtualization, anthony, evb, shemminger
In-Reply-To: <200912141300.37063.arnd@arndb.de>
From: Arnd Bergmann <arnd@arndb.de>
Date: Mon, 14 Dec 2009 13:00:36 +0100
> c) prepare a combined patch for net-next.git, or
This is probably fine.
I'll be taking patches into net-next-2.6 right after Linus
releases 2.6.33-rc1.
^ permalink raw reply
* Re: [Qemu-devel] Guest bridge setup variations
From: Arnd Bergmann @ 2009-12-14 12:09 UTC (permalink / raw)
To: virtualization; +Cc: qemu-devel, Anthony Liguori
In-Reply-To: <4B218A10.6000904@codemonkey.ws>
On Friday 11 December 2009, Anthony Liguori wrote:
> Arnd Bergmann wrote:
> >> 3) given an fd, treat a vhost-style interface
> >
> > This could mean two things, not sure which one you mean. Either the
> > file descriptor could be the vhost file descriptor, or the socket or tap file
> > descriptor from above, with qemu operating on the vhost interface itself.
> >
> > Either option has its advantages, but I guess we should only implement
> > one of the two to keep it simple.
> >
>
> I was thinking the socket/tap descriptor.
ok.
> > Right. I wonder if this helper should integrate with netcf as an abstraction,
> > or if we should rather do something generic. It may also be a good idea
> > to let the helper decide which of the three options you listed to use
> > and pass that back to qemu unless the user overrides it. The decision
> > probably needs to be host specific, depending e.g. on the availability
> > and version of tools (brctl, iproute, maybe tunctl, ...), the respective
> > kernel modules (vhost, macvlan, bridge, tun, ...) and policy (VEPA, vlan,
> > ebtables). Ideally the approach should be generic enough to work on
> > other platforms (BSD, Solaris, Windows, ...).
>
> For helpers, I think I'd like to stick with what we currently support,
> and then allow for a robust way for there to be independent projects
> that implement their own helpers. For instance, I would love it if
> netcf had a qemu network "plugin" helper.
Moving netcf specific qemu helpers into the netcf project sounds good.
I'm not sure what you mean with 'stick to what we currently support',
do you mean with helpers that ship with qemu itself? That sounds
reasonable, though I'd obviously like to make sure they also work
with macvtap, which is currently not supported unless you pass an
open file descriptor into -net tap,fd.
> There's just too much in the networking space all wrapped up in layers
> of policy decisions. I think it's time to move it out of qemu.
Yes.
Arnd
^ permalink raw reply
* Re: [RFC] macvlan: add tap device backend
From: Arnd Bergmann @ 2009-12-14 12:00 UTC (permalink / raw)
To: Herbert Xu
Cc: Edge Virtual Bridging, Michael S. Tsirkin, Fischer, Anna, netdev,
bridge, linux-kernel, David S. Miller, Anthony Liguori,
virtualization, Stephen Hemminger
In-Reply-To: <20091212084112.GA8948@gondor.apana.org.au>
On Saturday 12 December 2009, Herbert Xu wrote:
> On Thu, Dec 03, 2009 at 06:52:00PM +0100, Arnd Bergmann wrote:
> > This is a second prototype of a new interface into the network
> > stack, to eventually replace tun/tap and the bridge driver
> > in certain virtual machine setups. The code has changed
> > significantly, but the goals still remain.
>
> The idea looks pretty sound to me.
Thanks!
> > I'm posting this mainly to solicit feedback. Not sure how
> > chances for integration into 2.6.33 are, given that the merge
> > window is opening already.
>
> It's way too late for 2.6.33. But the earlier this becomes ready
> then the earlier it can go into linux-next.
Right. Since the code is basically working (I have two known bugs
I'm still working on) and people want to test it, what is the
best path? I could
a) keep my git tree for myself,
b) have it added to linux-next,
c) prepare a combined patch for net-next.git, or
d) have it added to drivers/staging
Arnd
^ permalink raw reply
* Re: [RFC] macvlan: add tap device backend
From: Herbert Xu @ 2009-12-12 8:41 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Edge Virtual Bridging, Michael S. Tsirkin, Fischer, Anna, netdev,
bridge, linux-kernel, David S. Miller, Anthony Liguori,
virtualization, Stephen Hemminger
In-Reply-To: <1259862720-28432-3-git-send-email-arnd@arndb.de>
On Thu, Dec 03, 2009 at 06:52:00PM +0100, Arnd Bergmann wrote:
> This is a second prototype of a new interface into the network
> stack, to eventually replace tun/tap and the bridge driver
> in certain virtual machine setups. The code has changed
> significantly, but the goals still remain.
The idea looks pretty sound to me.
> I'm posting this mainly to solicit feedback. Not sure how
> chances for integration into 2.6.33 are, given that the merge
> window is opening already.
It's way too late for 2.6.33. But the earlier this becomes ready
then the earlier it can go into linux-next.
Thanks,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCHv9 3/3] vhost_net: a kernel-level virtio server
From: Shirley Ma @ 2009-12-11 15:52 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Xin, Xiaohui, Eric Dumazet, kvm@vger.kernel.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org, linux-mm@kvack.org,
s.hetze@linux-ag.com, hpa@zytor.com, Daniel Walker, mingo@elte.hu,
akpm@linux-foundation.org
In-Reply-To: <20091122103511.GB13644@redhat.com>
On Sun, 2009-11-22 at 12:35 +0200, Michael S. Tsirkin wrote:
> These results where sent by Shirley Ma (Cc'd).
> I think they were with tap, host-to-guest/guest-to-host
Yes, you are right.
Shirley
^ permalink raw reply
* Re: [Qemu-devel] Guest bridge setup variations
From: Anthony Liguori @ 2009-12-10 23:53 UTC (permalink / raw)
To: Arnd Bergmann; +Cc: qemu-devel, virtualization
In-Reply-To: <200912101019.58555.arnd@arndb.de>
Arnd Bergmann wrote:
>> 3) given an fd, treat a vhost-style interface
>>
>
> This could mean two things, not sure which one you mean. Either the
> file descriptor could be the vhost file descriptor, or the socket or tap file
> descriptor from above, with qemu operating on the vhost interface itself.
>
> Either option has its advantages, but I guess we should only implement
> one of the two to keep it simple.
>
I was thinking the socket/tap descriptor.
>> I believe we should continue supporting the mechanisms we support
>> today. However, for people that invoke qemu directly from the command
>> line, I believe we should provide a mechanism like the tap helper that
>> can be used to call out to a separate program to create these initial
>> file descriptors. We'll have to think about how we can make this
>> integrate well so that the syntax isn't clumsy.
>>
>
> Right. I wonder if this helper should integrate with netcf as an abstraction,
> or if we should rather do something generic. It may also be a good idea
> to let the helper decide which of the three options you listed to use
> and pass that back to qemu unless the user overrides it. The decision
> probably needs to be host specific, depending e.g. on the availability
> and version of tools (brctl, iproute, maybe tunctl, ...), the respective
> kernel modules (vhost, macvlan, bridge, tun, ...) and policy (VEPA, vlan,
> ebtables). Ideally the approach should be generic enough to work on
> other platforms (BSD, Solaris, Windows, ...).
>
For helpers, I think I'd like to stick with what we currently support,
and then allow for a robust way for there to be independent projects
that implement their own helpers. For instance, I would love it if
netcf had a qemu network "plugin" helper.
There's just too much in the networking space all wrapped up in layers
of policy decisions. I think it's time to move it out of qemu.
> One thing I realized the last time we discussed the helper approach is
> that qemu should not need to know or care about the arguments passed
> to the helper, otherwise you get all the complexity back in qemu that
> you're trying to avoid. Maybe for 0.13 we can convert -net socket and
> -net tap to just pass all their options to the helper and move that code
> out of qemu, along with introducting the new syntax.
>
Yes, I was thinking the same thing. New syntax may need exploring.
> Another unrelated issue that I think needs to be addressed in a
> network code cleanup is adding better support for multi-queue
> transmit and receive. I've prepared macvtap for that by letting you
> open the chardev multiple times to get one queue per guest CPU,
> but that needs to be supported by qemu and virtio-net as well
> to actually parallelize network operation. Ideally, two guest CPUs
> should be able to transmit and receive on separate queues of the
> adapter without ever having to access any shared resources.
>
Multiqueue adds another dimension but I think your approach is pretty
much right on the money. Have multiple fds for each queue and we would
support a mechanism with helpers to receive multiple fds as appropriate.
Regards,
Anthony Liguori
^ permalink raw reply
* Re: [Qemu-devel] Re: Guest bridge setup variations
From: Alexander Graf @ 2009-12-10 20:37 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Fischer, Anna, qemu-devel,
virtualization@lists.linux-foundation.org
In-Reply-To: <200912102020.26724.arnd@arndb.de>
On 10.12.2009, at 21:20, Arnd Bergmann wrote:
> On Thursday 10 December 2009 19:14:28 Alexander Graf wrote:
>>> This is something I also have been thinking about, but it is not what
>>> I was referring to above. I think it would be good to keep the three
>>> cases (macvlan, VMDq, SR-IOV) as similar as possible from the user
>>> perspective, so using macvlan as an infrastructure for all of them
>>> sounds reasonable to me.
>>
>> Oh, so you'd basically do -net vt-d,if=eth0 and the rest would
>> automatically work? That's a pretty slick idea!
>
> I was only referring to how they get set up under the covers, e.g.
> creating the virtual device, configuring the MAC address etc, not
> the qemu side, but that would probably make sense as well.
>
> Or even better, qemu should probably not even know the difference
> between macvlan and VT-d. In both cases, it would open a macvtap
> file, but for VT-d adapters, the macvlan infrastructure can
> use hardware support, much in the way that VLAN tagging gets
> offloaded automatically to the hardware.
Well, vt-d means we use PCI passthrough. But it probably makes sense to have a -net bridge,if=eth0 that automatically uses whatever is around (pci passthrough, macvtap, anthony's bridge script, etc.). Of course we should leverage vmdq for macvtap whenever available :-).
Alex
^ permalink raw reply
* Re: [Qemu-devel] Re: Guest bridge setup variations
From: Arnd Bergmann @ 2009-12-10 20:20 UTC (permalink / raw)
To: qemu-devel; +Cc: Fischer, Anna, virtualization@lists.linux-foundation.org
In-Reply-To: <784CAF36-3F28-4FFD-82A2-0A4E54EDB831@suse.de>
On Thursday 10 December 2009 19:14:28 Alexander Graf wrote:
> > This is something I also have been thinking about, but it is not what
> > I was referring to above. I think it would be good to keep the three
> > cases (macvlan, VMDq, SR-IOV) as similar as possible from the user
> > perspective, so using macvlan as an infrastructure for all of them
> > sounds reasonable to me.
>
> Oh, so you'd basically do -net vt-d,if=eth0 and the rest would
> automatically work? That's a pretty slick idea!
I was only referring to how they get set up under the covers, e.g.
creating the virtual device, configuring the MAC address etc, not
the qemu side, but that would probably make sense as well.
Or even better, qemu should probably not even know the difference
between macvlan and VT-d. In both cases, it would open a macvtap
file, but for VT-d adapters, the macvlan infrastructure can
use hardware support, much in the way that VLAN tagging gets
offloaded automatically to the hardware.
Arnd <><
^ permalink raw reply
* Re: Guest bridge setup variations
From: Alexander Graf @ 2009-12-10 19:14 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Fischer, Anna, qemu-devel@nongnu.org,
virtualization@lists.linux-foundation.org
In-Reply-To: <200912101518.36128.arnd@arndb.de>
On 10.12.2009, at 15:18, Arnd Bergmann wrote:
> On Thursday 10 December 2009, Fischer, Anna wrote:
>>>
>>> 3. Doing the bridging in the NIC using macvlan in passthrough
>>> mode. This lowers the CPU utilization further compared to 2,
>>> at the expense of limiting throughput by the performance of
>>> the PCIe interconnect to the adapter. Whether or not this
>>> is a win is workload dependent. Access controls now happen
>>> in the NIC. Currently, this is not supported yet, due to lack of
>>> device drivers, but it will be an important scenario in the future
>>> according to some people.
>>
>> Can you differentiate this option from typical PCI pass-through mode?
>> It is not clear to me where macvlan sits in a setup where the NIC does
>> bridging.
>
> In this setup (hypothetical so far, the code doesn't exist yet), we use
> the configuration logic of macvlan, but not the forwarding. This also
> doesn't do PCI pass-through but instead gives all the logical interfaces
> to the host, using only the bridging and traffic separation capabilities
> of the NIC, but not the PCI-separation.
>
> Intel calls this mode VMDq, as opposed to SR-IOV, which implies
> the assignment of the adapter to a guest.
>
> It was confusing of me to call it passthrough above, sorry for that.
>
>> Typically, in a PCI pass-through configuration, all configuration goes
>> through the physical function device driver (and all data goes directly
>> to the NIC). Are you suggesting to use macvlan as a common
>> configuration layer that then configures the underlying NIC?
>> I could see some benefit in such a model, though I am not certain I
>> understand you correctly.
>
> This is something I also have been thinking about, but it is not what
> I was referring to above. I think it would be good to keep the three
> cases (macvlan, VMDq, SR-IOV) as similar as possible from the user
> perspective, so using macvlan as an infrastructure for all of them
> sounds reasonable to me.
Oh, so you'd basically do -net vt-d,if=eth0 and the rest would automatically work? That's a pretty slick idea!
Alex
^ permalink raw reply
* Re: Guest bridge setup variations
From: Arnd Bergmann @ 2009-12-10 14:18 UTC (permalink / raw)
To: Fischer, Anna
Cc: qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org
In-Reply-To: <0199E0D51A61344794750DC57738F58E6D7083F314@GVW1118EXC.americas.hpqcorp.net>
On Thursday 10 December 2009, Fischer, Anna wrote:
> >
> > 3. Doing the bridging in the NIC using macvlan in passthrough
> > mode. This lowers the CPU utilization further compared to 2,
> > at the expense of limiting throughput by the performance of
> > the PCIe interconnect to the adapter. Whether or not this
> > is a win is workload dependent. Access controls now happen
> > in the NIC. Currently, this is not supported yet, due to lack of
> > device drivers, but it will be an important scenario in the future
> > according to some people.
>
> Can you differentiate this option from typical PCI pass-through mode?
> It is not clear to me where macvlan sits in a setup where the NIC does
> bridging.
In this setup (hypothetical so far, the code doesn't exist yet), we use
the configuration logic of macvlan, but not the forwarding. This also
doesn't do PCI pass-through but instead gives all the logical interfaces
to the host, using only the bridging and traffic separation capabilities
of the NIC, but not the PCI-separation.
Intel calls this mode VMDq, as opposed to SR-IOV, which implies
the assignment of the adapter to a guest.
It was confusing of me to call it passthrough above, sorry for that.
> Typically, in a PCI pass-through configuration, all configuration goes
> through the physical function device driver (and all data goes directly
> to the NIC). Are you suggesting to use macvlan as a common
> configuration layer that then configures the underlying NIC?
> I could see some benefit in such a model, though I am not certain I
> understand you correctly.
This is something I also have been thinking about, but it is not what
I was referring to above. I think it would be good to keep the three
cases (macvlan, VMDq, SR-IOV) as similar as possible from the user
perspective, so using macvlan as an infrastructure for all of them
sounds reasonable to me.
The difference between VMDq and SR-IOV in that case would be
that the former uses a virtio-net driver in the guest and a hardware
driver in the host, while the latter uses a hardware driver in the guest
only. The data flow on these two would be identical though, while
in the classic macvlan the data forwarding decisions are made in
the host kernel.
Arnd
^ permalink raw reply
* RE: Guest bridge setup variations
From: Fischer, Anna @ 2009-12-10 12:26 UTC (permalink / raw)
To: Arnd Bergmann
Cc: qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org
In-Reply-To: <200912081707.42660.arnd@arndb.de>
> Subject: Guest bridge setup variations
>
> As promised, here is my small writeup on which setups I feel
> are important in the long run for server-type guests. This
> does not cover -net user, which is really for desktop kinds
> of applications where you do not want to connect into the
> guest from another IP address.
>
> I can see four separate setups that we may or may not want to
> support, the main difference being how the forwarding between
> guests happens:
>
> 1. The current setup, with a bridge and tun/tap devices on ports
> of the bridge. This is what Gerhard's work on access controls is
> focused on and the only option where the hypervisor actually
> is in full control of the traffic between guests. CPU utilization should
> be highest this way, and network management can be a burden,
> because the controls are done through a Linux, libvirt and/or Director
> specific interface.
>
> 2. Using macvlan as a bridging mechanism, replacing the bridge
> and tun/tap entirely. This should offer the best performance on
> inter-guest communication, both in terms of throughput and
> CPU utilization, but offer no access control for this traffic at all.
> Performance of guest-external traffic should be slightly better
> than bridge/tap.
>
> 3. Doing the bridging in the NIC using macvlan in passthrough
> mode. This lowers the CPU utilization further compared to 2,
> at the expense of limiting throughput by the performance of
> the PCIe interconnect to the adapter. Whether or not this
> is a win is workload dependent. Access controls now happen
> in the NIC. Currently, this is not supported yet, due to lack of
> device drivers, but it will be an important scenario in the future
> according to some people.
Can you differentiate this option from typical PCI pass-through mode? It is not clear to me where macvlan sits in a setup where the NIC does bridging.
Typically, in a PCI pass-through configuration, all configuration goes through the physical function device driver (and all data goes directly to the NIC). Are you suggesting to use macvlan as a common configuration layer that then configures the underlying NIC? I could see some benefit in such a model, though I am not certain I understand you correctly.
Thanks,
Anna
^ permalink raw reply
* Re: [Qemu-devel] Guest bridge setup variations
From: Arnd Bergmann @ 2009-12-10 9:19 UTC (permalink / raw)
To: Anthony Liguori; +Cc: qemu-devel, virtualization
In-Reply-To: <4B1FFC30.40904@codemonkey.ws>
On Wednesday 09 December 2009, Anthony Liguori wrote:
> While we go over all of these things one thing is becoming clear to me.
> We need to get qemu out of the network configuration business. There's
> too much going on here.
Agreed.
> What I'd like to see is the following interfaces supported:
>
> 1) given an fd, make socket calls to send packets. Could be used with a
> raw socket, a multicast or tcp socket.
> 2) given an fd, use tap-style read/write calls to send packets*
yes.
> 3) given an fd, treat a vhost-style interface
This could mean two things, not sure which one you mean. Either the
file descriptor could be the vhost file descriptor, or the socket or tap file
descriptor from above, with qemu operating on the vhost interface itself.
Either option has its advantages, but I guess we should only implement
one of the two to keep it simple.
> I believe we should continue supporting the mechanisms we support
> today. However, for people that invoke qemu directly from the command
> line, I believe we should provide a mechanism like the tap helper that
> can be used to call out to a separate program to create these initial
> file descriptors. We'll have to think about how we can make this
> integrate well so that the syntax isn't clumsy.
Right. I wonder if this helper should integrate with netcf as an abstraction,
or if we should rather do something generic. It may also be a good idea
to let the helper decide which of the three options you listed to use
and pass that back to qemu unless the user overrides it. The decision
probably needs to be host specific, depending e.g. on the availability
and version of tools (brctl, iproute, maybe tunctl, ...), the respective
kernel modules (vhost, macvlan, bridge, tun, ...) and policy (VEPA, vlan,
ebtables). Ideally the approach should be generic enough to work on
other platforms (BSD, Solaris, Windows, ...).
One thing I realized the last time we discussed the helper approach is
that qemu should not need to know or care about the arguments passed
to the helper, otherwise you get all the complexity back in qemu that
you're trying to avoid. Maybe for 0.13 we can convert -net socket and
-net tap to just pass all their options to the helper and move that code
out of qemu, along with introducting the new syntax.
Another unrelated issue that I think needs to be addressed in a
network code cleanup is adding better support for multi-queue
transmit and receive. I've prepared macvtap for that by letting you
open the chardev multiple times to get one queue per guest CPU,
but that needs to be supported by qemu and virtio-net as well
to actually parallelize network operation. Ideally, two guest CPUs
should be able to transmit and receive on separate queues of the
adapter without ever having to access any shared resources.
Arnd
^ permalink raw reply
* Re: [Qemu-devel] Guest bridge setup variations
From: Anthony Liguori @ 2009-12-09 19:36 UTC (permalink / raw)
To: Arnd Bergmann; +Cc: qemu-devel, virtualization
In-Reply-To: <200912081707.42660.arnd@arndb.de>
Arnd Bergmann wrote:
> As promised, here is my small writeup on which setups I feel
> are important in the long run for server-type guests. This
> does not cover -net user, which is really for desktop kinds
> of applications where you do not want to connect into the
> guest from another IP address.
>
> I can see four separate setups that we may or may not want to
> support, the main difference being how the forwarding between
> guests happens:
>
> 1. The current setup, with a bridge and tun/tap devices on ports
> of the bridge. This is what Gerhard's work on access controls is
> focused on and the only option where the hypervisor actually
> is in full control of the traffic between guests. CPU utilization should
> be highest this way, and network management can be a burden,
> because the controls are done through a Linux, libvirt and/or Director
> specific interface.
>
Typical bridging.
> 2. Using macvlan as a bridging mechanism, replacing the bridge
> and tun/tap entirely. This should offer the best performance on
> inter-guest communication, both in terms of throughput and
> CPU utilization, but offer no access control for this traffic at all.
> Performance of guest-external traffic should be slightly better
> than bridge/tap.
>
Optimization to typical bridge (no traffic control).
> 3. Doing the bridging in the NIC using macvlan in passthrough
> mode. This lowers the CPU utilization further compared to 2,
> at the expense of limiting throughput by the performance of
> the PCIe interconnect to the adapter. Whether or not this
> is a win is workload dependent. Access controls now happen
> in the NIC. Currently, this is not supported yet, due to lack of
> device drivers, but it will be an important scenario in the future
> according to some people.
>
Optimization to typical bridge (hardware accelerated).
> 4. Using macvlan for actual VEPA on the outbound interface.
> This is mostly interesting because it makes the network access
> controls visible in an external switch that is already managed.
> CPU utilization and guest-external throughput should be
> identical to 3, but inter-guest latency can only be worse because
> all frames go through the external switch.
>
VEPA.
While we go over all of these things one thing is becoming clear to me.
We need to get qemu out of the network configuration business. There's
too much going on here.
What I'd like to see is the following interfaces supported:
1) given an fd, make socket calls to send packets. Could be used with a
raw socket, a multicast or tcp socket.
2) given an fd, use tap-style read/write calls to send packets*
3) given an fd, treat a vhost-style interface
* need to make all tun ioctls optional based on passed in flags
Every backend we have today could be implemented in terms of one of the
above three. They really come down to how the fd is created and setup.
I believe we should continue supporting the mechanisms we support
today. However, for people that invoke qemu directly from the command
line, I believe we should provide a mechanism like the tap helper that
can be used to call out to a separate program to create these initial
file descriptors. We'll have to think about how we can make this
integrate well so that the syntax isn't clumsy.
Regards,
Anthony Liguori
^ permalink raw reply
* Guest bridge setup variations
From: Arnd Bergmann @ 2009-12-08 16:07 UTC (permalink / raw)
To: virtualization; +Cc: qemu-devel
As promised, here is my small writeup on which setups I feel
are important in the long run for server-type guests. This
does not cover -net user, which is really for desktop kinds
of applications where you do not want to connect into the
guest from another IP address.
I can see four separate setups that we may or may not want to
support, the main difference being how the forwarding between
guests happens:
1. The current setup, with a bridge and tun/tap devices on ports
of the bridge. This is what Gerhard's work on access controls is
focused on and the only option where the hypervisor actually
is in full control of the traffic between guests. CPU utilization should
be highest this way, and network management can be a burden,
because the controls are done through a Linux, libvirt and/or Director
specific interface.
2. Using macvlan as a bridging mechanism, replacing the bridge
and tun/tap entirely. This should offer the best performance on
inter-guest communication, both in terms of throughput and
CPU utilization, but offer no access control for this traffic at all.
Performance of guest-external traffic should be slightly better
than bridge/tap.
3. Doing the bridging in the NIC using macvlan in passthrough
mode. This lowers the CPU utilization further compared to 2,
at the expense of limiting throughput by the performance of
the PCIe interconnect to the adapter. Whether or not this
is a win is workload dependent. Access controls now happen
in the NIC. Currently, this is not supported yet, due to lack of
device drivers, but it will be an important scenario in the future
according to some people.
4. Using macvlan for actual VEPA on the outbound interface.
This is mostly interesting because it makes the network access
controls visible in an external switch that is already managed.
CPU utilization and guest-external throughput should be
identical to 3, but inter-guest latency can only be worse because
all frames go through the external switch.
In case 2 through 4, we have the choice between macvtap and
the raw packet interface for connecting macvlan to qemu.
Raw sockets are better tested right now, while macvtap has
better permission management (i.e. it does not require
CAP_NET_ADMIN). Neither one is upstream though at the
moment. The raw driver only requires qemu patches, while
macvtap requires both a new kernel driver and a trivial change
in qemu.
In all four cases, vhost-net could be used to move the workload
from user space into the kernel, which may be an advantage.
The decision for or against vhost-net is entirely independent of
the other decisions.
Arnd
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox