Virtio network performance problem

Virtio network performance problem

[Adrian Schmitz]

I'm having a problem with virtio networking. I mentioned this in a
previous thread, but I'm starting a new one since Dor solved the problem
I reported in the original thread. This appears to be a new problem.

Iperf shows throughput of ~120 Mb/s between a Windows Server 2003 x64
guest and the br0 interface on the host. Also, a ping between the same
two interfaces looks pretty bad:

Pinging 10.10.10.128 with 32 bytes of data:
Reply from 10.10.10.128: bytes=32 time=-389ms TTL=64
Reply from 10.10.10.128: bytes=32 time<1ms TTL=64
Reply from 10.10.10.128: bytes=32 time=392ms TTL=64
Reply from 10.10.10.128: bytes=32 time=392ms TTL=64
Reply from 10.10.10.128: bytes=32 time=377ms TTL=64
Reply from 10.10.10.128: bytes=32 time=392ms TTL=64
Reply from 10.10.10.128: bytes=32 time<1ms TTL=64
Reply from 10.10.10.128: bytes=32 time=401ms TTL=64
Reply from 10.10.10.128: bytes=32 time=1ms TTL=64

Using e1000 driver on this guest, iperf shows 320+ Mb/s throughput, and
latency looks like it should.

Host kernel:			2.6.18
KVM Version:			79 (modules and tools built and
installed from source)
Guest OS:   			Win2k3 x64 with all service packs and
patches
Guest virtio drivers:		Version 3 (received from Dor Laor
yesterday.. these fixed the guest OS crashes I was having with version
2)

Any suggestions are greatly appreciated. If there's additional info I
can provide, please let me know.

Re: Virtio network performance problem

[Chris Wedgwood]

On Wed, Dec 03, 2008 at 02:11:45PM -0500, Adrian Schmitz wrote:

> Iperf shows throughput of ~120 Mb/s between a Windows Server 2003 x64
> guest and the br0 interface on the host. Also, a ping between the same
> two interfaces looks pretty bad:
>
> Pinging 10.10.10.128 with 32 bytes of data:
> Reply from 10.10.10.128: bytes=32 time=-389ms TTL=64


TSC instability?  Is this an SMP guest?

Re: Virtio network performance problem

[Chris Wedgwood]

On Wed, Dec 03, 2008 at 11:20:08AM -0800, Chris Wedgwood wrote:

> TSC instability?  Is this an SMP guest?

Actually, worded badly.

A UP guest on an SMP host with unsync'ed TSCs would show this too
obviously.  As would changes in the TSC speed.  The first I guess we
can avoid by pinning that latter I'm not so sure about.

RE: Virtio network performance problem

[Adrian Schmitz]

> On Wed, Dec 03, 2008 at 11:20:08AM -0800, Chris Wedgwood wrote:
> 
> > TSC instability?  Is this an SMP guest?
> 
> Actually, worded badly.
> 
> A UP guest on an SMP host with unsync'ed TSCs would show this too
> obviously.  As would changes in the TSC speed.  The first I guess we
> can avoid by pinning that latter I'm not so sure about.

I don't know enough to tell whether there's a TSC problem or not, to be
honest. The host has 2X AMD quad-core CPUs (let me know if you want the
/proc/cpuinfo). The guest has 2 VCPUs. 

Re: Virtio network performance problem

[Anthony Liguori]

Adrian Schmitz wrote:
> I'm having a problem with virtio networking. I mentioned this in a
> previous thread, but I'm starting a new one since Dor solved the problem
> I reported in the original thread. This appears to be a new problem.
>   

On an older kernel, you'll likely see packet corruption with virtio-net 
because of an uninitialized variable.  Please try the patch I just 
posted (Fix uninitialized offset in virtio-net receive_header) and let 
me know if it helps.

Regards,

Anthony Liguori

RE: Virtio network performance problem

[Adrian Schmitz]

> > On Wed, Dec 03, 2008 at 11:20:08AM -0800, Chris Wedgwood wrote:
> >
> > > TSC instability?  Is this an SMP guest?

Ok, I tried pinning the kvm process to two cores (0,2) on a single
socket, but that didn't seem to make any difference for my virtio
network performance. I also tried pinning the process to a single core,
which also didn't seem to have any effect.

Someone on IRC suggested that it sounded like a clocking issue, since
some of my ping times are negative. He suggested trying a different
clock source. I tried it with dynticks, rtc, and unix. None of them seem
better, although all of them seem "different" in terms of patterns in
the ping times. Sorry if this makes it a long post, but I don't know how
to describe it other than to paste an example (below). Not sure if this
indicates that it is clock-related or if it is meaningless.

In any event, I'm not sure where to go from here. Another suggestion
from IRC was that it was due to the age of my host kernel (2.6.18) and
the fact that it doesn't support high-res timers. If I can avoid
replacing the distro kernel, I'd like to, but I'll do what I have to, I
suppose.

With dynticks (these are all with -net user, as I had some trouble with
my tap interface last night while testing this. The results are roughly
the same as when I was using tap before, though):

Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=143ms TTL=255
Reply from 10.0.2.2: bytes=32 time=143ms TTL=255
Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=143ms TTL=255
Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=-139ms TTL=255
Reply from 10.0.2.2: bytes=32 time=-141ms TTL=255
Reply from 10.0.2.2: bytes=32 time=-133ms TTL=255
Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=143ms TTL=255
Reply from 10.0.2.2: bytes=32 time=1ms TTL=255

With rtc:

Reply from 10.0.2.2: bytes=32 time=-224ms TTL=255
Reply from 10.0.2.2: bytes=32 time=-223ms TTL=255
Reply from 10.0.2.2: bytes=32 time=4ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=225ms TTL=255
Reply from 10.0.2.2: bytes=32 time=-223ms TTL=255
Reply from 10.0.2.2: bytes=32 time=-224ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=225ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=225ms TTL=255
Reply from 10.0.2.2: bytes=32 time=225ms TTL=255

With unix:

Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=-190ms TTL=255
Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255
Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
Reply from 10.0.2.2: bytes=32 time=192ms TTL=255

Re: Virtio network performance problem

[Dor Laor]

Adrian Schmitz wrote:
>>> On Wed, Dec 03, 2008 at 11:20:08AM -0800, Chris Wedgwood wrote:
>>>
>>>       
>>>> TSC instability?  Is this an SMP guest?
>>>>         
>
> Ok, I tried pinning the kvm process to two cores (0,2) on a single
> socket, but that didn't seem to make any difference for my virtio
> network performance. I also tried pinning the process to a single core,
> which also didn't seem to have any effect.
>
>   
I think it is an unsync tsc problem.
First, make sure you pin all of the process threads. There is thread per 
vcpu + io thread +more non relevant.
You can do it by adding the taskset before the cmdline.
Second, you said that you use smp guest. So windows also sees unsync tsc.
So, either test with UP guest or learn how to pin windows receiving ISR, 
DPC and the user app.

Well, testing on Intel or newer AMD is another option.
I tested it again now on Intel with UP guest and there is no such a problem.
Hope to test it next week on AMD SMP guest.

Regards,
Dor
> Someone on IRC suggested that it sounded like a clocking issue, since
> some of my ping times are negative. He suggested trying a different
> clock source. I tried it with dynticks, rtc, and unix. None of them seem
> better, although all of them seem "different" in terms of patterns in
> the ping times. Sorry if this makes it a long post, but I don't know how
> to describe it other than to paste an example (below). Not sure if this
> indicates that it is clock-related or if it is meaningless.
>
> In any event, I'm not sure where to go from here. Another suggestion
> from IRC was that it was due to the age of my host kernel (2.6.18) and
> the fact that it doesn't support high-res timers. If I can avoid
> replacing the distro kernel, I'd like to, but I'll do what I have to, I
> suppose.
>
> With dynticks (these are all with -net user, as I had some trouble with
> my tap interface last night while testing this. The results are roughly
> the same as when I was using tap before, though):
>
> Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=143ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=143ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=143ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=-139ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=-141ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=-133ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=143ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
>
> With rtc:
>
> Reply from 10.0.2.2: bytes=32 time=-224ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=-223ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=4ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=225ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=-223ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=-224ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=225ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=225ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=225ms TTL=255
>
> With unix:
>
> Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time<1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=-190ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=1ms TTL=255
> Reply from 10.0.2.2: bytes=32 time=192ms TTL=255
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

RE: Virtio network performance problem

[Adrian Schmitz]

> I think it is an unsync tsc problem.
> First, make sure you pin all of the process threads. There is thread
> per
> vcpu + io thread +more non relevant.
> You can do it by adding the taskset before the cmdline.
> Second, you said that you use smp guest. So windows also sees unsync
> tsc.
> So, either test with UP guest or learn how to pin windows receiving
> ISR,
> DPC and the user app.
> 
> Well, testing on Intel or newer AMD is another option.
> I tested it again now on Intel with UP guest and there is no such a
> problem.
> Hope to test it next week on AMD SMP guest.
> 
> Regards,
> Dor

I made sure to pin all 5 threads from my guest kvm process to two cores
on a single socket, and also tried pinning them all to a single core,
but neither made a difference. I then powered down all the cores (using
the /sys entries) until only two cores on a single socket were left,
which made no improvement. I then powered all cores down except one (and
verified that only the one showed up in /proc/cpuinfo). Naturally the
guest slowed a bit, but the erratic and negative pings from the guest to
the host br0 were still there. If I boot the guest with smp 1 on the kvm
command line, the ping times are fine. It's only when I start the guest
with 2 VCPUs that the ping times are erratic and some pings are
negative.

Just to recap, this is on a brand new Dell Poweredge R805 with dual
quad-core Opteron 2350s, with KVM-79 built against the CentOS-packaged
2.6.18 kernel. The guest in question is Windows Server 2003 x64 SP2 with
2 VCPUs.

I couldn't think of anything else to try, so I loaded up another server,
a new Dell Poweredge 2950 with dual quad-core Xeon E5430s. I installed
and configured kvm-79 following the exact same process I used to
configure it on the AMD host. When I boot the _same_ guest (it's in
iscsi) on the PE2950/Intel host, the ping times are perfect.

I'm interested to know why this is (before I purchase any more AMD
servers) but my priority right now is trying to figure out what I'm
doing wrong that is causing virtio networking to be extremely slow
(usually around 55Mb/s) on every host I've built it on. I'm sure I'm
doing something wrong and I just can't place it. I'll start another
thread for that, though, because it seems to be a different problem. In
the meantime I'll just keep this guest on my Intel servers, I guess.