virtio performance issue

virtio performance issue

[Ben-Ami Yassour]

I am running virtio with the latest KVM code, and see a significant
performance issue.

Ping to the host (or any other close machine) reports a 4ms delay.

In the same setup with an e1000 emulation (just changing model=virtio to
model=e1000 in the KVM command line), ping reports 0.177ms delay.

BTW, initially I saw that the throughput when using netperf is very low,
even from guest to host, even though the CPU utilization is low.

What might be the problem?

Thanks,
Ben

Re: virtio performance issue

[Anthony Liguori]

Ben-Ami Yassour wrote:
> I am running virtio with the latest KVM code, and see a significant
> performance issue.
>
> Ping to the host (or any other close machine) reports a 4ms delay.
>   

What kvm version and what host kernel version?

It's very easy to mistakenly compile qemu without GSO support too.  You 
have to make sure that the 2.6.27 if_tun.h is being included by QEMU.

Regards,

Anthony Liguori

> In the same setup with an e1000 emulation (just changing model=virtio to
> model=e1000 in the KVM command line), ping reports 0.177ms delay.
>
> BTW, initially I saw that the throughput when using netperf is very low,
> even from guest to host, even though the CPU utilization is low.
>
> What might be the problem?
>
> Thanks,
> Ben
>
>   

Re: virtio performance issue

[Bernhard Schmidt]

Ben-Ami Yassour <benami@il.ibm.com> wrote:

Hello Ben,

> I am running virtio with the latest KVM code, and see a significant
> performance issue.
>
> Ping to the host (or any other close machine) reports a 4ms delay.
>
> In the same setup with an e1000 emulation (just changing model=virtio to
> model=e1000 in the KVM command line), ping reports 0.177ms delay.
>
> BTW, initially I saw that the throughput when using netperf is very low,
> even from guest to host, even though the CPU utilization is low.
>
> What might be the problem?

I had exactly the same issue (with an 2.6.26 kernel and kvm-70 though).
In an attempt to make the guest kernel (w/o modules) as small as
possible I had disabled ACPI in its config. Which works, but introduced
the very same 4ms delay you are seeing and made the VM clock go wild
(even worse with KVM_CLOCK). I did not see other speed issues, but I
have to admit I never benchmarked it.

After enabling ACPI the 4ms delay baseline disappeared and the clock in
the guest is now perfectly in sync with the host.

Regards,
Bernhard

Re: virtio performance issue

[Ben-Ami Yassour]

On Tue, 2008-09-16 at 09:16 -0500, Anthony Liguori wrote:
> Ben-Ami Yassour wrote:
> > I am running virtio with the latest KVM code, and see a significant
> > performance issue.
> >
> > Ping to the host (or any other close machine) reports a 4ms delay.
> >   
> 
> What kvm version and what host kernel version?
> 
> It's very easy to mistakenly compile qemu without GSO support too.  You 
> have to make sure that the 2.6.27 if_tun.h is being included by QEMU.

Is there an option to control GSO support? How?

I am using the kernel and userspace that I pulled from the kvm tree
today.

Based on your comment, we checked and the build of the userspace does
not take if_tun.h from the kernel tree, it takes it from the system
include files.
The reason was that the file was not copied as part of the userspace
build.

To fix this we made the following change:
diff --git a/kernel/Makefile b/kernel/Makefile
index 3f5f6da..b81b098 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -53,7 +53,7 @@ T = $(subst -sync,,$@)-tmp
 header-sync:
        rm -rf $T
        rsync -R \
-            "$(LINUX)"/./include/linux/kvm*.h \
+            "$(LINUX)"/./include/linux/*.h \
             "$(LINUX)"/./include/asm-*/kvm*.h \

Even with this change and compiling the userspace with the correct
if_tun.h the results are the same, ping takes 4ms.

What could be the reason?

Thanks,
Ben

Re: virtio performance issue

[Mark McLoughlin]

On Tue, 2008-09-16 at 22:24 +0300, Ben-Ami Yassour wrote:
> On Tue, 2008-09-16 at 09:16 -0500, Anthony Liguori wrote:
> > Ben-Ami Yassour wrote:
> > > I am running virtio with the latest KVM code, and see a significant
> > > performance issue.
> > >
> > > Ping to the host (or any other close machine) reports a 4ms delay.
> > >   
> > 
> > What kvm version and what host kernel version?
> > 
> > It's very easy to mistakenly compile qemu without GSO support too.  You 
> > have to make sure that the 2.6.27 if_tun.h is being included by QEMU.
> 
> Is there an option to control GSO support? How?

GSO support is unconditionally enabled with model=virtio if
kvm-userspace is built with the correct kernel headers, the host kernel
supports tun/tap's IFF_VNET_HDR extension and if the guest supports GSO.

> I am using the kernel and userspace that I pulled from the kvm tree
> today.
> 
> Based on your comment, we checked and the build of the userspace does
> not take if_tun.h from the kernel tree, it takes it from the system
> include files.
> The reason was that the file was not copied as part of the userspace
> build.
> 
> To fix this we made the following change:
> diff --git a/kernel/Makefile b/kernel/Makefile
> index 3f5f6da..b81b098 100644
> --- a/kernel/Makefile
> +++ b/kernel/Makefile
> @@ -53,7 +53,7 @@ T = $(subst -sync,,$@)-tmp
>  header-sync:
>         rm -rf $T
>         rsync -R \
> -            "$(LINUX)"/./include/linux/kvm*.h \
> +            "$(LINUX)"/./include/linux/*.h \
>              "$(LINUX)"/./include/asm-*/kvm*.h \

Ouch, looks like we need a fix like this alright - maybe just copy
if_tun.h and virtio*.h ?

> Even with this change and compiling the userspace with the correct
> if_tun.h the results are the same, ping takes 4ms.

GSO shouldn't affect ping latency - it should only affect throughput.

I'd expect ping latency to be in the range of .15ms and .3ms since we
delay our reply for .15ms currently.

Is this a regression? Have you tried bisecting it?

Cheers,
Mark.

Re: virtio performance issue

[Ben-Ami Yassour]

On Wed, 2008-09-17 at 11:49 +0100, Mark McLoughlin wrote:
> On Tue, 2008-09-16 at 22:24 +0300, Ben-Ami Yassour wrote:
> > On Tue, 2008-09-16 at 09:16 -0500, Anthony Liguori wrote:
> > > Ben-Ami Yassour wrote:
> > > > I am running virtio with the latest KVM code, and see a significant
> > > > performance issue.
> > > >
> > > > Ping to the host (or any other close machine) reports a 4ms delay.
> > > >   
> > > 
> > > What kvm version and what host kernel version?
> > > 
> > > It's very easy to mistakenly compile qemu without GSO support too.  You 
> > > have to make sure that the 2.6.27 if_tun.h is being included by QEMU.
> > 
> > Is there an option to control GSO support? How?
> 
> GSO support is unconditionally enabled with model=virtio if
> kvm-userspace is built with the correct kernel headers, the host kernel
> supports tun/tap's IFF_VNET_HDR extension and if the guest supports GSO.
How can we verify that GSO is actually used?

> GSO shouldn't affect ping latency - it should only affect throughput.
> 
> I'd expect ping latency to be in the range of .15ms and .3ms since we
> delay our reply for .15ms currently.
> 
> Is this a regression? Have you tried bisecting it?
> 

We are not sure yet what it is. We see very high variability in I/O
rates, and have not yet found a combination of version, environment and
parameters that shows good performance reliably. The head of the tree
does show *bad* performance reliably.
We are trying kvm-73 now.

Thanks,
Ben