Network throughput limits for local VM <-> VM communication

Network throughput limits for local VM <-> VM communication

[Fischer, Anna]

I am testing network throughput between two guests residing on the same physical machine. I use a bridge to pass packets between those guests and the virtio NIC model. I am wondering why the throughput only goes up to about 970Mbps. Should we not be able to achieve much higher throughput if the packets do not actually go out on the physical wire? What are the limitations for throughput performance under KVM/virtio? I can see that by default the interfaces (the tap devices) have TX queue length set to 500, and I wonder if increasing this would make any difference? Also, are there other things I would need to consider to achieve higher throughput numbers for local guest <-> guest communication? The CPU is not maxed out at all, and shows as being idle for most of the time while the throughput
  does not increase any more.

I run KVM under standard Fedora Core 10 with a Linux kernel 2.6.27.

Thanks,
Anna

Re: Network throughput limits for local VM <-> VM communication

[Mark McLoughlin]

On Tue, 2009-06-09 at 11:06 +0000, Fischer, Anna wrote:

> I am testing network throughput between two guests residing on the
> same physical machine. I use a bridge to pass packets between those
> guests and the virtio NIC model. I am wondering why the throughput
> only goes up to about 970Mbps. Should we not be able to achieve much
> higher throughput if the packets do not actually go out on the
> physical wire? What are the limitations for throughput performance
> under KVM/virtio? I can see that by default the interfaces (the tap
> devices) have TX queue length set to 500, and I wonder if increasing
> this would make any difference? Also, are there other things I would
> need to consider to achieve higher throughput numbers for local guest
> <-> guest communication? The CPU is not maxed out at all, and shows as
> being idle for most of the time while the throughput does not increase
> any more.
> 
> I run KVM under standard Fedora Core 10 with a Linux kernel 2.6.27.

The first thing to check is that GSO is enabled - you can check with
"ethtool -k eth0" in the guests.

Are you starting qemu from the command line or e.g. using libvirt? The
libvirt version in F-10 didn't know how to enable IFF_VNET_HDR on the
tapfd before passing it to qemu.

Really, I'd suggest updating to F-11 before digging further - you'll
have qemu-kvm-0.10.5, linux-2.6.29.4 and libvirt-0.6.2.

Cheers,
Mark.

RE: Network throughput limits for local VM <-> VM communication

[Fischer, Anna]

> Subject: Re: Network throughput limits for local VM <-> VM
> communication
> 
> On Tue, 2009-06-09 at 11:06 +0000, Fischer, Anna wrote:
> 
> > I am testing network throughput between two guests residing on the
> > same physical machine. I use a bridge to pass packets between those
> > guests and the virtio NIC model. I am wondering why the throughput
> > only goes up to about 970Mbps. Should we not be able to achieve much
> > higher throughput if the packets do not actually go out on the
> > physical wire? What are the limitations for throughput performance
> > under KVM/virtio? I can see that by default the interfaces (the tap
> > devices) have TX queue length set to 500, and I wonder if increasing
> > this would make any difference? Also, are there other things I would
> > need to consider to achieve higher throughput numbers for local guest
> > <-> guest communication? The CPU is not maxed out at all, and shows
> as
> > being idle for most of the time while the throughput does not
> increase
> > any more.
> >
> > I run KVM under standard Fedora Core 10 with a Linux kernel 2.6.27.
> 
> The first thing to check is that GSO is enabled - you can check with
> "ethtool -k eth0" in the guests.
> 
> Are you starting qemu from the command line or e.g. using libvirt? The
> libvirt version in F-10 didn't know how to enable IFF_VNET_HDR on the
> tapfd before passing it to qemu.
> 
> Really, I'd suggest updating to F-11 before digging further - you'll
> have qemu-kvm-0.10.5, linux-2.6.29.4 and libvirt-0.6.2.

I use libvirt (virt-manager), but I do run the latest libvirt 0.6.4 compiled from source. Does that make any difference? Upgrading to FC11 would not be ideal for me at this point, so it would be nice to get around that.

GSO was enabled. Switching it off does not make any difference.

Something else I notice - I run 3 KVM guests, one of these being the router (with 2 vNICs), and the other two communicating with each other. So packets pass from one guest to the router and then to the other guest. When I stress this set-up with high-throughput network testing tools, then the network hangs after a few seconds. This happens for all TCP tests. When I do UDP and rate limit to something < 10Mbps then it works fine. However, when I don't do any rate limiting, then it completely breaks. I have tried using virtio and using the emulated QEMU virtual NICs. It does not make a difference. It seems as if there is an overflow somewhere when QEMU/virtio cannot cope with the network load any more, and then the virtual interfaces don't seem to transmit anything anymore. It seems to mostly
  work again when I shut down and start up the interfaces of the router inside of the guest. I use two bridges (and VLANs) that pass packets between sending/receiving guests and the routing guest. The set-up works fine for simple ping and other communication that is low-throughput type traffic.

Cheers,
Anna

Re: Network throughput limits for local VM <-> VM communication

[Arnd Bergmann]

On Tuesday 09 June 2009, Fischer, Anna wrote:

> I have tried using virtio and using the emulated QEMU virtual NICs.
> It does not make a difference. It seems as if there is an overflow somewhere
> when QEMU/virtio cannot cope with the network load any more, and then the
> virtual interfaces don't seem to transmit anything anymore. It seems to
> mostly work again when I shut down and start up the interfaces of the router
> inside of the guest. I use two bridges (and VLANs) that pass packets between
> sending/receiving guests and the routing guest. The set-up works fine for
> simple ping and other communication that is low-throughput type traffic.

Have you tried eliminating VLAN to simplify the setup?

Does it change when the guests communicate over a -net socket interface
with your router instead of the -net tap + bridge in the host?

	Arnd <><

RE: Network throughput limits for local VM <-> VM communication

[Fischer, Anna]

> Subject: Re: Network throughput limits for local VM <-> VM
> communication
> 
> On Tuesday 09 June 2009, Fischer, Anna wrote:
> 
> > I have tried using virtio and using the emulated QEMU virtual NICs.
> > It does not make a difference. It seems as if there is an overflow
> somewhere
> > when QEMU/virtio cannot cope with the network load any more, and then
> the
> > virtual interfaces don't seem to transmit anything anymore. It seems
> to
> > mostly work again when I shut down and start up the interfaces of the
> router
> > inside of the guest. I use two bridges (and VLANs) that pass packets
> between
> > sending/receiving guests and the routing guest. The set-up works fine
> for
> > simple ping and other communication that is low-throughput type
> traffic.
> 
> Have you tried eliminating VLAN to simplify the setup?

No - but there is a relating bug in the tun/tap interface (well, it is not really a bug but simply the way tun/tap works) that will cause packets to be replicated on all the tap interfaces (across all bridges attached to those) if I do not configure VLANs. This will result in a system that is even more overloaded. I had discovered this a while back when running UDP stress tests under 10G.

 
> Does it change when the guests communicate over a -net socket interface
> with your router instead of the -net tap + bridge in the host?

I have not tried this - I need the bridge in the network data path for some testing, so using the -net socket interface would not solve my problem.

However, I have just today managed to get around this bug by using the e1000 QEMU emulated NIC model and this seems to do the trick. Now the throughput is still very low, but that might simply be because my system is too weak. When using the e1000 model instead of rtl8139 or virtio, I do not have any network crashes any more.

I have been looking through the RedHat bug lists to search for some hints today, and it seems as if there are a lot of people seeing the network under KVM break down under heavy load. I think that this is something that needs some further investigation. I can provide more detailed system set-up etc, it should be easy to reproduce this.

Thanks for your help,
Anna

Re: Network throughput limits for local VM <-> VM communication

[Arnd Bergmann]

On Wednesday 10 June 2009, Fischer, Anna wrote:
> > Have you tried eliminating VLAN to simplify the setup?
> 
> No - but there is a relating bug in the tun/tap interface (well, it is not
> really a bug but simply the way tun/tap works) that will cause packets to
> be replicated on all the tap interfaces (across all bridges attached to
> those) if I do not configure VLANs. This will result in a system that is
> even more overloaded. I had discovered this a while back when running
> UDP stress tests under 10G.

Not sure I understand. Do you mean you have all three guests connected
to the same bridge? If you want the router guest to be the only connection,
you should not connect the two bridges anywhere else, so I don't see
how packets can go from one bridge to the other one, except through the
router.

> > Does it change when the guests communicate over a -net socket interface
> > with your router instead of the -net tap + bridge in the host?
> 
> I have not tried this - I need the bridge in the network data path for
> some testing, so using the -net socket interface would not solve my problem.

I did not mean this to solve your problem but to hunt down the bug.
If the problem only exists with the host bridge device, we should look
there, but if it persists, we can probably rule out the tap, bridge and vlan
code in the host as the problem source.
 
> However, I have just today managed to get around this bug by using the
> e1000 QEMU emulated NIC model and this seems to do the trick. Now the
> throughput is still very low, but that might simply be because my system
> is too weak. When using the e1000 model instead of rtl8139 or virtio, 
> I do not have any network crashes any more.

That could either indicate a bug in rtl8139 and virtio, or that the
specific timing of the e1000 model hides this bug.

What happens if only one side uses e1000 while the other still uses
virtio? What about any of the other models?

	Arnd <><

RE: Network throughput limits for local VM <-> VM communication

[Fischer, Anna]

> Subject: Re: Network throughput limits for local VM <-> VM
> communication
> 
> On Wednesday 10 June 2009, Fischer, Anna wrote:
> > > Have you tried eliminating VLAN to simplify the setup?
> >
> > No - but there is a relating bug in the tun/tap interface (well, it
> is not
> > really a bug but simply the way tun/tap works) that will cause
> packets to
> > be replicated on all the tap interfaces (across all bridges attached
> to
> > those) if I do not configure VLANs. This will result in a system that
> is
> > even more overloaded. I had discovered this a while back when running
> > UDP stress tests under 10G.
> 
> Not sure I understand. Do you mean you have all three guests connected
> to the same bridge? If you want the router guest to be the only
> connection,
> you should not connect the two bridges anywhere else, so I don't see
> how packets can go from one bridge to the other one, except through the
> router.

I am using two bridges, and yes, in theory, the router should be the only connection between the two guests. However, without VLANs, the tun interface will pass packets to all tap interfaces. It has to, as it doesn't know to which one the packet has to go to. It does not look at packets, it simply copies buffers from userspace to the tap interface in the kernel. The tap interface then eventually drops the packet, if the MAC address does not match its own. So packets will not actually go across both bridges, because the tap interface that should not receive the packet does drop it. However, it does receive the packet and processes it to some extend which causes some overhead. As I was told by someone at KVM/RedHat, this does not happen when using VLANs as then there will be a direct mapping
  between any tun<->tap device and so no packet replication across multiple tap devices.
 

> > > Does it change when the guests communicate over a -net socket
> interface
> > > with your router instead of the -net tap + bridge in the host?
> >
> > I have not tried this - I need the bridge in the network data path
> for
> > some testing, so using the -net socket interface would not solve my
> problem.
> 
> I did not mean this to solve your problem but to hunt down the bug.
> If the problem only exists with the host bridge device, we should look
> there, but if it persists, we can probably rule out the tap, bridge and
> vlan
> code in the host as the problem source.

Yes, I understand you were trying to help and using the -net socket interface would help to narrow down where the problem could be. I just have not yet managed to set this up, but I might do if I find the time in the next days. I was hoping that other people might have seen the same issues I see, but unfortunately I did not get that many replies/suggestions on this issue from the list at all.
 

> > However, I have just today managed to get around this bug by using
> the
> > e1000 QEMU emulated NIC model and this seems to do the trick. Now the
> > throughput is still very low, but that might simply be because my
> system
> > is too weak. When using the e1000 model instead of rtl8139 or virtio,
> > I do not have any network crashes any more.
> 
> That could either indicate a bug in rtl8139 and virtio, or that the
> specific timing of the e1000 model hides this bug.
> 
> What happens if only one side uses e1000 while the other still uses
> virtio? What about any of the other models?

Good question. I will try this out and post the results.

Cheers,
Anna

Re: Network throughput limits for local VM <-> VM communication

[Avi Kivity]

Fischer, Anna wrote:
> I am using two bridges, and yes, in theory, the router should be the only connection between the two guests. However, without VLANs, the tun interface will pass packets to all tap interfaces. It has to, as it doesn't know to which one the packet has to go to. It does not look at packets, it simply copies buffers from userspace to the tap interface in the kernel. The tap interface then eventually drops the packet, if the MAC address does not match its own. So packets will not actually go across both bridges, because the tap interface that should not receive the packet does drop it. However, it does receive the packet and processes it to some extend which causes some overhead. As I was told by someone at KVM/RedHat, this does not happen when using VLANs as then there will be a direct mappi
 ng between any tun<->tap device and so no packet replication across multiple tap devices.
>   

This only happens if the receiving tap never sends out packets.  If the 
tap interface does send out packets, the bridge will associate their MAC 
address with that interface, and future packets will only be forwarded 
there.

Is this your scenario?

-- 
error compiling committee.c: too many arguments to function

RE: Network throughput limits for local VM <-> VM communication

[Fischer, Anna]

> Subject: Re: Network throughput limits for local VM <-> VM
> communication
> 
> Fischer, Anna wrote:
> > I am using two bridges, and yes, in theory, the router should be the
> only connection between the two guests. However, without VLANs, the tun
> interface will pass packets to all tap interfaces. It has to, as it
> doesn't know to which one the packet has to go to. It does not look at
> packets, it simply copies buffers from userspace to the tap interface
> in the kernel. The tap interface then eventually drops the packet, if
> the MAC address does not match its own. So packets will not actually go
> across both bridges, because the tap interface that should not receive
> the packet does drop it. However, it does receive the packet and
> processes it to some extend which causes some overhead. As I was told
> by someone at KVM/RedHat, this does not happen when using VLANs as then
> there will be a direct mapping between any tun<->tap device and so no
> packet replication across multiple tap devices.
> >
> 
> This only happens if the receiving tap never sends out packets.  If the
> tap interface does send out packets, the bridge will associate their
> MAC
> address with that interface, and future packets will only be forwarded
> there.
> 
> Is this your scenario?

Not sure I understand. As far as I can see the packets are replicated on the tun/tap interface before they actually enter the bridge. So this is not about the bridge learning MAC addresses and flooding frames to unknown destinations. So I think this is different.

Anna

Re: Network throughput limits for local VM <-> VM communication

[Avi Kivity]

Fischer, Anna wrote:
> Not sure I understand. As far as I can see the packets are replicated on the tun/tap interface before they actually enter the bridge. So this is not about the bridge learning MAC addresses and flooding frames to unknown destinations. So I think this is different.
>   

Okay.

You said:

> However, without VLANs, the tun
> interface will pass packets to all tap interfaces. It has to, as it
> doesn't know to which one the packet has to go to. 

Well, it shouldn't.  The tun interface should pass the packets to just 
one tap interface.

Can you post the qemu command line you're using?  There's a gotcha there 
that can result in what you're seeing.


-- 
error compiling committee.c: too many arguments to function

RE: Network throughput limits for local VM <-> VM communication

[Fischer, Anna]

> Subject: Re: Network throughput limits for local VM <-> VM
> communication
> 
> Fischer, Anna wrote:
> > Not sure I understand. As far as I can see the packets are replicated
> on the tun/tap interface before they actually enter the bridge. So this
> is not about the bridge learning MAC addresses and flooding frames to
> unknown destinations. So I think this is different.
> >
> 
> Okay.
> 
> You said:
> 
> > However, without VLANs, the tun
> > interface will pass packets to all tap interfaces. It has to, as it
> > doesn't know to which one the packet has to go to.
> 
> Well, it shouldn't.  The tun interface should pass the packets to just
> one tap interface.
> 
> Can you post the qemu command line you're using?  There's a gotcha
> there
> that can result in what you're seeing.

Sorry for the late reply on this issue. The command line I am using looks roughly like this:

/usr/bin/qemu-system-x86_64 -m 1024 -smp 2 -name FC10-2 -uuid b811b278-fae2-a3cc-d51d-8f5b078b2477 -boot c -drive file=,if=ide,media=cdrom,index=2 -drive file=/var/lib/libvirt/images/FC10-2.img,if=virtio,index=0,boot=on -net nic,macaddr=54:52:00:11:ae:79,model=e1000 -net tap net nic,macaddr=54:52:00:11:ae:78,model=e1000 -net tap  -serial pty -parallel none -usb -vnc 127.0.0.1:2 -k en-gb -soundhw es1370

This is my "routing VM" that has two network interfaces and routes packets between two subnets. It has one interface plugged into bridge virbr0 and the other interface is plugged into virbr1:

brctl show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.8ac1d18c63ec       no              vnet0
                                                        vnet1
virbr1          8000.2ebfcbb9ed70       no              vnet2
                                                        vnet3

If I use the e1000 virtual NIC model, I see performance drop significantly compared to using virtio_net. However, with virtio_net I have the network stalling after a few seconds of high-throughput traffic (as I mentioned in my previous post). Just to reiterate my scenario: I run three guests on the same physical machine, one guest is my routing VM that is routing IP network traffic between the other two guests.

I am also wondering about the fact that I do not seem to get CPU utilization maxed out in this case while throughput does not go any higher. I do not understand what is stopping KVM from using more CPU for guest I/O processing? There is nothing else running on my machine. I have analyzed the amount of CPU that each KVM thread is using, and I can see that the thread that is running the VCPU of the routing VM which is processing interrupts of the e1000 virtual network card is using the highest amount of CPU. Is there any way that I can optimize my network set-up? Maybe some specific configuration of the e1000 driver within the guest? Are there any known issues with this?

I also see very difference CPU utilization and network throughput figures when pinning threads to CPU cores using taskset. At one point I managed to double the throughput, but I could not reproduce that setup for some reason. What are the major issues that I would need to pay attention to when pinning threads to cores in order to optimize my specific set-up so that I can achieve better network I/O performance?

Thanks for your help.

Anna

Re: Network throughput limits for local VM <-> VM communication

[Avi Kivity]

On 06/17/2009 10:36 AM, Fischer, Anna wrote:
>
> /usr/bin/qemu-system-x86_64 -m 1024 -smp 2 -name FC10-2 -uuid b811b278-fae2-a3cc-d51d-8f5b078b2477 -boot c -drive file=,if=ide,media=cdrom,index=2 -drive file=/var/lib/libvirt/images/FC10-2.img,if=virtio,index=0,boot=on -net nic,macaddr=54:52:00:11:ae:79,model=e1000 -net tap net nic,macaddr=54:52:00:11:ae:78,model=e1000 -net tap  -serial pty -parallel none -usb -vnc 127.0.0.1:2 -k en-gb -soundhw es1370
>
>    

Okay, like I suspected, qemu has a trap here and you walked into it.  
The -net option plugs the device you specify into a virtual hub.  The 
command line you provided plugs the two virtual NICs and the two tap 
devices into one virtual hub, so any packet received from any of the 
four clients will be propagated to the other three.

To get this to work right, specify the vlan= parameter which says which 
virtual hub a component is plugged into.  Note this has nothing to do 
with 802.blah vlans.

So your command line should look like

    qemu ... -net nic,...,vlan=0 -net tap,...,vlan=0 -net nic,...,vlan=1 
-net tap,...,vlan=1

This will give you two virtual hubs, each bridging a virtual nic to a 
tap device.


> This is my "routing VM" that has two network interfaces and routes packets between two subnets. It has one interface plugged into bridge virbr0 and the other interface is plugged into virbr1:
>
> brctl show
> bridge name     bridge id               STP enabled     interfaces
> virbr0          8000.8ac1d18c63ec       no              vnet0
>                                                          vnet1
> virbr1          8000.2ebfcbb9ed70       no              vnet2
>                                                          vnet3
>    

Please redo the tests with qemu vlans but without 802.blah vlans, so we 
see what happens without packet duplication.

> If I use the e1000 virtual NIC model, I see performance drop significantly compared to using virtio_net. However, with virtio_net I have the network stalling after a few seconds of high-throughput traffic (as I mentioned in my previous post). Just to reiterate my scenario: I run three guests on the same physical machine, one guest is my routing VM that is routing IP network traffic between the other two guests.
>
> I am also wondering about the fact that I do not seem to get CPU utilization maxed out in this case while throughput does not go any higher. I do not understand what is stopping KVM from using more CPU for guest I/O processing? There is nothing else running on my machine. I have analyzed the amount of CPU that each KVM thread is using, and I can see that the thread that is running the VCPU of the routing VM which is processing interrupts of the e1000 virtual network card is using the highest amount of CPU. Is there any way that I can optimize my network set-up? Maybe some specific configuration of the e1000 driver within the guest? Are there any known issues with this?
>    

There are known issues with lack of flow control while sending packets 
out of a guest.  If the guest runs tcp that tends to correct for it, but 
if you run a lower level protocol that doesn't have its own flow 
control, the guest may spend a lot of cpu generating packets that are 
eventually dropped.  We are working on fixing this.

> I also see very difference CPU utilization and network throughput figures when pinning threads to CPU cores using taskset. At one point I managed to double the throughput, but I could not reproduce that setup for some reason. What are the major issues that I would need to pay attention to when pinning threads to cores in order to optimize my specific set-up so that I can achieve better network I/O performance?
>    

It's black magic, unfortunately.  But please retry with the fixed 
configuration and we'll continue from there.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

RE: Network throughput limits for local VM <-> VM communication

[Fischer, Anna]

> Subject: Re: Network throughput limits for local VM <-> VM
> communication
> 
> On 06/17/2009 10:36 AM, Fischer, Anna wrote:
> >
> > /usr/bin/qemu-system-x86_64 -m 1024 -smp 2 -name FC10-2 -uuid
> b811b278-fae2-a3cc-d51d-8f5b078b2477 -boot c -drive
> file=,if=ide,media=cdrom,index=2 -drive
> file=/var/lib/libvirt/images/FC10-2.img,if=virtio,index=0,boot=on -net
> nic,macaddr=54:52:00:11:ae:79,model=e1000 -net tap net
> nic,macaddr=54:52:00:11:ae:78,model=e1000 -net tap  -serial pty -
> parallel none -usb -vnc 127.0.0.1:2 -k en-gb -soundhw es1370
> >
> >
> 
> Okay, like I suspected, qemu has a trap here and you walked into it.
> The -net option plugs the device you specify into a virtual hub.  The
> command line you provided plugs the two virtual NICs and the two tap
> devices into one virtual hub, so any packet received from any of the
> four clients will be propagated to the other three.
> 
> To get this to work right, specify the vlan= parameter which says which
> virtual hub a component is plugged into.  Note this has nothing to do
> with 802.blah vlans.
> 
> So your command line should look like
> 
>     qemu ... -net nic,...,vlan=0 -net tap,...,vlan=0 -net
> nic,...,vlan=1
> -net tap,...,vlan=1
> 
> This will give you two virtual hubs, each bridging a virtual nic to a
> tap device.
> 
> > This is my "routing VM" that has two network interfaces and routes
> packets between two subnets. It has one interface plugged into bridge
> virbr0 and the other interface is plugged into virbr1:
> >
> > brctl show
> > bridge name     bridge id               STP enabled     interfaces
> > virbr0          8000.8ac1d18c63ec       no              vnet0
> >                                                          vnet1
> > virbr1          8000.2ebfcbb9ed70       no              vnet2
> >                                                          vnet3
> >
> 
> Please redo the tests with qemu vlans but without 802.blah vlans, so we
> see what happens without packet duplication.

Avi, thanks for your quick reply. I do use the vlan= parameter now, and yes, I do not see packet duplication any more, so everything you said is right and I do understand now why I was seeing packets on both bridges before. So this has nothing to do with tun/tap then but just with the way QEMU "virtual hubs" work. I didn't know about any details on that before.

Even with vlan= enabled, I am still having the same issues with weird CPU utilization and low throughput that I have described below.
 
> > If I use the e1000 virtual NIC model, I see performance drop
> significantly compared to using virtio_net. However, with virtio_net I
> have the network stalling after a few seconds of high-throughput
> traffic (as I mentioned in my previous post). Just to reiterate my
> scenario: I run three guests on the same physical machine, one guest is
> my routing VM that is routing IP network traffic between the other two
> guests.
> >
> > I am also wondering about the fact that I do not seem to get CPU
> utilization maxed out in this case while throughput does not go any
> higher. I do not understand what is stopping KVM from using more CPU
> for guest I/O processing? There is nothing else running on my machine.
> I have analyzed the amount of CPU that each KVM thread is using, and I
> can see that the thread that is running the VCPU of the routing VM
> which is processing interrupts of the e1000 virtual network card is
> using the highest amount of CPU. Is there any way that I can optimize
> my network set-up? Maybe some specific configuration of the e1000
> driver within the guest? Are there any known issues with this?
> >
> 
> There are known issues with lack of flow control while sending packets
> out of a guest.  If the guest runs tcp that tends to correct for it,
> but
> if you run a lower level protocol that doesn't have its own flow
> control, the guest may spend a lot of cpu generating packets that are
> eventually dropped.  We are working on fixing this.

For the tests I run now (with vlan= enabled) I am actually using both TCP and UDP, and I see the problem with virtio_net for both protocols. What I am wondering about though is that I do not seem to have any problems if I communicate directly between the two guests (if I plug then into the same bridge and put them onto the same networks), so why do I only see the problem of stalling network communication when there is a routing VM in the network path? Is this just because the system is even more overloaded in that case? Or could this be an issue related to a dual NIC configuration or the fact that I run multiple bridges on the same physical machine?

When you say "We are working on fixing this." - which code parts are you working on? Is this in the QEMU network I/O processing code or is this virtio_net related?

> > I also see very difference CPU utilization and network throughput
> figures when pinning threads to CPU cores using taskset. At one point I
> managed to double the throughput, but I could not reproduce that setup
> for some reason. What are the major issues that I would need to pay
> attention to when pinning threads to cores in order to optimize my
> specific set-up so that I can achieve better network I/O performance?
> >
> 
> It's black magic, unfortunately.  But please retry with the fixed
> configuration and we'll continue from there.

Retry with "the fixed configuration"? You mean setting the vlan= parameter? I have already used the vlan= parameter for the latest tests, and so the CPU utilization issues I am talking about are happening with that configuration.

Thanks,
Anna

Re: Network throughput limits for local VM <-> VM communication

[Avi Kivity]

On 06/17/2009 11:12 AM, Fischer, Anna wrote:
>
> For the tests I run now (with vlan= enabled) I am actually using both TCP and UDP, and I see the problem with virtio_net for both protocols. What I am wondering about though is that I do not seem to have any problems if I communicate directly between the two guests (if I plug then into the same bridge and put them onto the same networks), so why do I only see the problem of stalling network communication when there is a routing VM in the network path? Is this just because the system is even more overloaded in that case? Or could this be an issue related to a dual NIC configuration or the fact that I run multiple bridges on the same physical machine?
>    

My guess is that somewhere there's a queue that's shorter that the 
virtio queue, or its usable size fluctuates (because it is shared with 
something else).  So TCP flow control doesn't work, and UDP doesn't have 
a chance.

> When you say "We are working on fixing this." - which code parts are you working on? Is this in the QEMU network I/O processing code or is this virtio_net related?
>    

tap. virtio, qemu, maybe more.  It's a difficult problem.

> Retry with "the fixed configuration"? You mean setting the vlan= parameter? I have already used the vlan= parameter for the latest tests, and so the CPU utilization issues I am talking about are happening with that configuration.
>    

Yeah.

Can you compare total data sent and received as seen by the guests?  
That would confirm that packets being dropped causes the slowdown.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

RE: Network throughput limits for local VM <-> VM communication

[Fischer, Anna]

> Subject: Re: Network throughput limits for local VM <-> VM
> communication
> 
> On 06/17/2009 11:12 AM, Fischer, Anna wrote:
> >
> > For the tests I run now (with vlan= enabled) I am actually using both
> TCP and UDP, and I see the problem with virtio_net for both protocols.
> What I am wondering about though is that I do not seem to have any
> problems if I communicate directly between the two guests (if I plug
> then into the same bridge and put them onto the same networks), so why
> do I only see the problem of stalling network communication when there
> is a routing VM in the network path? Is this just because the system is
> even more overloaded in that case? Or could this be an issue related to
> a dual NIC configuration or the fact that I run multiple bridges on the
> same physical machine?
> >
> 
> My guess is that somewhere there's a queue that's shorter that the
> virtio queue, or its usable size fluctuates (because it is shared with
> something else).  So TCP flow control doesn't work, and UDP doesn't
> have
> a chance.
> 
> > When you say "We are working on fixing this." - which code parts are
> you working on? Is this in the QEMU network I/O processing code or is
> this virtio_net related?
> >
> 
> tap. virtio, qemu, maybe more.  It's a difficult problem.
> 
> > Retry with "the fixed configuration"? You mean setting the vlan=
> parameter? I have already used the vlan= parameter for the latest
> tests, and so the CPU utilization issues I am talking about are
> happening with that configuration.
> >
> 
> Yeah.
> 
> Can you compare total data sent and received as seen by the guests?
> That would confirm that packets being dropped causes the slowdown.

Yes, I will check on that and report back. 

It still does not answer my question on why I only see low CPU utilization numbers with the e1000 virtual device model. There is no network stalling or packet drops or any other obvious issues when running with that model, but I am still seeing low CPU utilization numbers. What is preventing KVM here to use more of the host CPU capacity when the host is not doing anything else but run virtual machines? Is there any way that I can get higher CPU utilization out of KVM?

Thanks,
Anna

Re: Network throughput limits for local VM <-> VM communication

[Avi Kivity]

On 06/17/2009 06:41 PM, Fischer, Anna wrote:
> It still does not answer my question on why I only see low CPU utilization numbers with the e1000 virtual device model. There is no network stalling or packet drops or any other obvious issues when running with that model, but I am still seeing low CPU utilization numbers. What is preventing KVM here to use more of the host CPU capacity when the host is not doing anything else but run virtual machines? Is there any way that I can get higher CPU utilization out of KVM?
>    

A missing wakeup might cause this, but I really have no idea how we can 
trace the cause.

-- 
error compiling committee.c: too many arguments to function