Virtio network performance poorer than emulated e1000

Virtio network performance poorer than emulated e1000

[Balachandar]

I am resending this email as Freddie didn't use 'reply to all' when
replying to this message. I am also updating to answer Freddie's
questions..

I can see that virtio network performance is poorer than emaulated
e1000 nic. I did some simple ping test and with emulated e1000 the
average rtt is around 600 microsec. With virtio the average rtt is 800
microsec. I am using a tap + bridge configuration. I run kvm as
follows

kvm -m 512 -hda vdisk.img \
-net nic,model=virtio \
-net tap,ifname=tap0,script=qemu-ifup,downscript=no

I am running Debian squeeze distribution with guest and host kernel 2.6.34.

Does anyone else see some results like this or is it only me? Could
changing the distribution help as i am running a testing one?

Actually we are having a custom application which just measures
ping-pong latency. The results that i get with virtio is around 750
micro seconds. The result i get with emulated e1000 is around 250
micro sec. Also i tried to use vhost but the virtio latency remained
the same.

Also i tried the tests with guest and host 2.6.26 kernel. I get better
results for virtio than e1000. I get 550 for e1000 and 500 for virtio.

Actually my application needs as minimum latency as needed. I am ready
to trade-off throughput and cpu utilization. I was very excited when i
saw the vhost-net module with people claiming low latencies. I am
worried that i am missing some performance offered by virtio and
vhost.


Thanks,
Bala

Re: Virtio network performance poorer than emulated e1000

[Sridhar Samudrala]

On Thu, 2010-07-22 at 19:53 -0400, Balachandar wrote:
> I am resending this email as Freddie didn't use 'reply to all' when
> replying to this message. I am also updating to answer Freddie's
> questions..
> 
> I can see that virtio network performance is poorer than emaulated
> e1000 nic. I did some simple ping test and with emulated e1000 the
> average rtt is around 600 microsec. With virtio the average rtt is 800
> microsec. I am using a tap + bridge configuration. I run kvm as
> follows
> 
> kvm -m 512 -hda vdisk.img \
> -net nic,model=virtio \
> -net tap,ifname=tap0,script=qemu-ifup,downscript=no

With newer qemu-kvm, you need to use -netdev tap to enable checksum/gso
offloads. 

Try
-net nic,model=virtio,netdev=tap0
-netdev tap,ifname=tap0,id=tap0,vhost=on,script=qemu-ifup,downscript=no

Thanks
Sridhar
> 
> I am running Debian squeeze distribution with guest and host kernel 2.6.34.
> 
> Does anyone else see some results like this or is it only me? Could
> changing the distribution help as i am running a testing one?
> 
> Actually we are having a custom application which just measures
> ping-pong latency. The results that i get with virtio is around 750
> micro seconds. The result i get with emulated e1000 is around 250
> micro sec. Also i tried to use vhost but the virtio latency remained
> the same.
> 
> Also i tried the tests with guest and host 2.6.26 kernel. I get better
> results for virtio than e1000. I get 550 for e1000 and 500 for virtio.
> 
> Actually my application needs as minimum latency as needed. I am ready
> to trade-off throughput and cpu utilization. I was very excited when i
> saw the vhost-net module with people claiming low latencies. I am
> worried that i am missing some performance offered by virtio and
> vhost.
> 
> 
> Thanks,
> Bala
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Virtio network performance poorer than emulated e1000

[Balachandar]

On Thu, Jul 22, 2010 at 8:18 PM, Sridhar Samudrala
<samudrala.sridhar@gmail.com> wrote:
> On Thu, 2010-07-22 at 19:53 -0400, Balachandar wrote:
>> I am resending this email as Freddie didn't use 'reply to all' when
>> replying to this message. I am also updating to answer Freddie's
>> questions..
>>
>> I can see that virtio network performance is poorer than emaulated
>> e1000 nic. I did some simple ping test and with emulated e1000 the
>> average rtt is around 600 microsec. With virtio the average rtt is 800
>> microsec. I am using a tap + bridge configuration. I run kvm as
>> follows
>>
>> kvm -m 512 -hda vdisk.img \
>> -net nic,model=virtio \
>> -net tap,ifname=tap0,script=qemu-ifup,downscript=no
>
> With newer qemu-kvm, you need to use -netdev tap to enable checksum/gso
> offloads.
>
> Try
> -net nic,model=virtio,netdev=tap0
> -netdev tap,ifname=tap0,id=tap0,vhost=on,script=qemu-ifup,downscript=no
>
> Thanks
> Sridhar
>>

I tried this one also but not much improvement. Actually i get a some
improvement if i disabled the tx timeout timer in the virtio_net.h in
qemu-kvm. I read that with vhost the data flow path differs from
normal operation. So is the timeout applicable when we  use vhost
also? Actually i dont see any improvement with vhost. I am just
wondering what am i missing? The thing that worries me is that
emulated nic offers much greater performance than virtio with vhost.
So i feel i am doing something wrong but unable to find it.



>> I am running Debian squeeze distribution with guest and host kernel 2.6.34.
>>
>> Does anyone else see some results like this or is it only me? Could
>> changing the distribution help as i am running a testing one?
>>
>> Actually we are having a custom application which just measures
>> ping-pong latency. The results that i get with virtio is around 750
>> micro seconds. The result i get with emulated e1000 is around 250
>> micro sec. Also i tried to use vhost but the virtio latency remained
>> the same.
>>
>> Also i tried the tests with guest and host 2.6.26 kernel. I get better
>> results for virtio than e1000. I get 550 for e1000 and 500 for virtio.
>>
>> Actually my application needs as minimum latency as needed. I am ready
>> to trade-off throughput and cpu utilization. I was very excited when i
>> saw the vhost-net module with people claiming low latencies. I am
>> worried that i am missing some performance offered by virtio and
>> vhost.
>>
>>
>> Thanks,
>> Bala
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Virtio network performance poorer than emulated e1000

[Jes Sorensen]

On 07/23/10 03:31, Balachandar wrote:
> On Thu, Jul 22, 2010 at 8:18 PM, Sridhar Samudrala
> I tried this one also but not much improvement. Actually i get a some
> improvement if i disabled the tx timeout timer in the virtio_net.h in
> qemu-kvm. I read that with vhost the data flow path differs from
> normal operation. So is the timeout applicable when we  use vhost
> also? Actually i dont see any improvement with vhost. I am just
> wondering what am i missing? The thing that worries me is that
> emulated nic offers much greater performance than virtio with vhost.
> So i feel i am doing something wrong but unable to find it.

Could you try measuring something meaningful instead of ping please?
netperf or some other benchmark?

Jes

Re: Virtio network performance poorer than emulated e1000

[Balachandar]

On Fri, Jul 23, 2010 at 3:11 AM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
> On 07/23/10 03:31, Balachandar wrote:
>> On Thu, Jul 22, 2010 at 8:18 PM, Sridhar Samudrala
>> I tried this one also but not much improvement. Actually i get a some
>> improvement if i disabled the tx timeout timer in the virtio_net.h in
>> qemu-kvm. I read that with vhost the data flow path differs from
>> normal operation. So is the timeout applicable when we  use vhost
>> also? Actually i dont see any improvement with vhost. I am just
>> wondering what am i missing? The thing that worries me is that
>> emulated nic offers much greater performance than virtio with vhost.
>> So i feel i am doing something wrong but unable to find it.
>
> Could you try measuring something meaningful instead of ping please?
> netperf or some other benchmark?
>
> Jes
>

Actually i got better results when i downloaded qemu-kvm 0.12.4 from
sourceforge and ran it. Now virtio performs better than emaulated
e1000 with our own simple ping-pong latency tests. Previously i used
Debian squeeze kvm package and i got poor results for virtio. I used
vhost-net as described by the kvm
website.http://www.linux-kvm.org/page/VhostNet. This also seem to have
the same problems i faced with Debian kvm package. How are you guyz
using vhost? Is there any other way to use vhost?

Thanks,
Bala

Re: Virtio network performance poorer than emulated e1000

[Jes Sorensen]

On 07/24/10 17:04, Balachandar wrote:
> On Fri, Jul 23, 2010 at 3:11 AM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
> Actually i got better results when i downloaded qemu-kvm 0.12.4 from
> sourceforge and ran it. Now virtio performs better than emaulated
> e1000 with our own simple ping-pong latency tests. Previously i used
> Debian squeeze kvm package and i got poor results for virtio. I used
> vhost-net as described by the kvm
> website.http://www.linux-kvm.org/page/VhostNet. This also seem to have
> the same problems i faced with Debian kvm package. How are you guyz
> using vhost? Is there any other way to use vhost?

Ok thats good to hear. Sounds like the Debian package might be dodgy, or
they just happened to snapshot at a bad time.

For vhost I am not the expert, maybe someone else knows.

Jes

Re: Virtio network performance poorer than emulated e1000

[Michael Tokarev]

26.07.2010 11:26, Jes Sorensen wrote:
> On 07/24/10 17:04, Balachandar wrote:
>> On Fri, Jul 23, 2010 at 3:11 AM, Jes Sorensen<Jes.Sorensen@redhat.com>  wrote:
>> Actually i got better results when i downloaded qemu-kvm 0.12.4 from
>> sourceforge and ran it. Now virtio performs better than emaulated
>> e1000 with our own simple ping-pong latency tests. Previously i used
>> Debian squeeze kvm package and i got poor results for virtio. I used
>> vhost-net as described by the kvm
>> website.http://www.linux-kvm.org/page/VhostNet. This also seem to have
>> the same problems i faced with Debian kvm package. How are you guyz
>> using vhost? Is there any other way to use vhost?
>
> Ok thats good to hear. Sounds like the Debian package might be dodgy, or
> they just happened to snapshot at a bad time.

I wonder which version it is - the "debian package".  Are we talking
about kvm-72 from debian lenny perhaps?

Current version is 0.12.4-1, which is upstream 0.12.4-1 with a few
bugfixes (such as >1Tb block device corruption).  I'm not sure how
that one might be "dogdy" or snapshotted at a bad time.

Thanks!

/mjt

Re: Virtio network performance poorer than emulated e1000

[Balachandar]

On Mon, Jul 26, 2010 at 9:44 AM, Michael Tokarev <mjt@tls.msk.ru> wrote:
> 26.07.2010 11:26, Jes Sorensen wrote:
>>
>> On 07/24/10 17:04, Balachandar wrote:
>>>
>>> On Fri, Jul 23, 2010 at 3:11 AM, Jes Sorensen<Jes.Sorensen@redhat.com>
>>>  wrote:
>>> Actually i got better results when i downloaded qemu-kvm 0.12.4 from
>>> sourceforge and ran it. Now virtio performs better than emaulated
>>> e1000 with our own simple ping-pong latency tests. Previously i used
>>> Debian squeeze kvm package and i got poor results for virtio. I used
>>> vhost-net as described by the kvm
>>> website.http://www.linux-kvm.org/page/VhostNet. This also seem to have
>>> the same problems i faced with Debian kvm package. How are you guyz
>>> using vhost? Is there any other way to use vhost?
>>
>> Ok thats good to hear. Sounds like the Debian package might be dodgy, or
>> they just happened to snapshot at a bad time.
>
> I wonder which version it is - the "debian package".  Are we talking
> about kvm-72 from debian lenny perhaps?
>
> Current version is 0.12.4-1, which is upstream 0.12.4-1 with a few
> bugfixes (such as >1Tb block device corruption).  I'm not sure how
> that one might be "dogdy" or snapshotted at a bad time.
>
> Thanks!
>
> /mjt
>
I am sorry. I made a mistake while testing it. There is no problem
with the Debian package. I get better latency results when i disable
the TX timeout in the virtio-net.c in qemu-kvm. But there is no
improvement over this result when i use vhost.

Thanks,
Bala

Virtio network performance poorer than emulated e1000

[Balachandar]

I can see that virtio network performance is poorer than emaulated
e1000 nic. I did some simple ping test and with emulated e1000 the
average rtt is around 600 microsec. With virtio the average rtt is 800
microsec. I am using a tap + bridge configuration. I run kvm as
follows

kvm -m 512 -hda vdisk.img \
-net nic,model=virtio \
-net tap,ifname=tap0,script=qemu-ifup,downscript=no

I am running Debian squeeze distribution with guest and host kernel 2.6.34.

Does anyone else see some results like this or is it only me? Could
changing the distribution help as i am running a testing one?

Thanks,
Bala

Re: Virtio network performance poorer than emulated e1000

[Freddie Cash]

On Thu, Jul 22, 2010 at 1:13 PM, Balachandar <bala1486@gmail.com> wrote:
> I can see that virtio network performance is poorer than emaulated
> e1000 nic. I did some simple ping test and with emulated e1000 the
> average rtt is around 600 microsec. With virtio the average rtt is 800
> microsec. I am using a tap + bridge configuration. I run kvm as
> follows
>
> kvm -m 512 -hda vdisk.img \
> -net nic,model=virtio \
> -net tap,ifname=tap0,script=qemu-ifup,downscript=no
>
> I am running Debian squeeze distribution with guest and host kernel 2.6.34.
>
> Does anyone else see some results like this or is it only me? Could
> changing the distribution help as i am running a testing one?

Have you done any testing beyond "ping"?  Doing network throughput
tests using ping is equivalent to using dd to benchmark disk drives:
aka pointless.

Do network file transfers.  Use a network benchmarking tool.  Do
something more in-depth than "ping".

-- 
Freddie Cash
fjwcash@gmail.com

Re: Virtio network performance poorer than emulated e1000

[Sebastian Hetze]

I did some benchmarking mainly with tbench and found virtio better
with big packages/maximum throughput and e1000 better with small
packages/latency. vhost-net improves virtio quite a lot and makes
it superior to the emulated devices.

If network performance is a real issue, DMAR/IOMMU gives you direct
access to dedicated network devices with nearly 100% native throughput
and latency.

Best regards,

  Sebastian

On Thu, Jul 22, 2010 at 04:13:29PM -0400, Balachandar wrote:
> I can see that virtio network performance is poorer than emaulated
> e1000 nic. I did some simple ping test and with emulated e1000 the
> average rtt is around 600 microsec. With virtio the average rtt is 800
> microsec. I am using a tap + bridge configuration. I run kvm as
> follows
> 
> kvm -m 512 -hda vdisk.img \
> -net nic,model=virtio \
> -net tap,ifname=tap0,script=qemu-ifup,downscript=no
> 
> I am running Debian squeeze distribution with guest and host kernel 2.6.34.
> 
> Does anyone else see some results like this or is it only me? Could
> changing the distribution help as i am running a testing one?
> 
> Thanks,
> Bala

Re: Virtio network performance poorer than emulated e1000

[Balachandar]

On Fri, Jul 23, 2010 at 9:37 AM, Sebastian Hetze <s.hetze@linux-ag.com> wrote:
> I did some benchmarking mainly with tbench and found virtio better
> with big packages/maximum throughput and e1000 better with small
> packages/latency. vhost-net improves virtio quite a lot and makes
> it superior to the emulated devices.
>

This is what that concerns me whether i am doing anything wrong.
vhost-net doesn't improve my virtio performance at all.. Thats what
makes me think that i am doing someting wrong. Couls you please tell
me the distribution, qemu-kvm version, guest and host kernel version,
no. of cores in cpu and your cpu info?

Actually my application is small packets and need low latency. I
cannot use dedicated network card for pci passthrough. I want to push
the virtio to work better for better latency for small packets. I dont
mind sacrificing throughput.
When i read that vhost-net reduces 4 to 5 times the latency of virtio,
i was excited but it didn't improve my latency at all.. I am
definitely missing something. Does vhost-net improves virtio latency
for small packets?

I tried disabling the tx timeout in the virtio-net in the qemu-kvm. It
did improve the latency a lot. But still it was far less than the
emulated e1000?

Is there any changes that can be made to virtio-net, vhost-net to
improve latency for small packets?

Thanks,
Bala