Poor Virtio PMD TX Performance

Poor Virtio PMD TX Performance

[Zhou, Tianlin]

Hi there,

We tested TX performance of Virtio PMD by DPDK l2fwd, but found even at 60KPPS (720B packet length) TX rate, there is 1/1000 packet dropping rate.
The log shows "No free tx descriptors to transmit" in Virtio PMD.
Increasing TX queues by modifying DPDK l2fwd can decreases packet dropping rate, but can't ensure no packet dropping unless retransmitting packets that can't be sent successfully.
Oppositely, RX rate can be 600KPPS without packet dropping.

Test Env
- Host CPU: 4 cores, 2127.770MHz
- Host Memory: 8G
- Host OS: Linux dw-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
- Guest CPU: 4 cores, 2127.770MHz
- Guest Memory: 4G
- Guest OS: fedora20

Anybody here face the same problem?

-Tianlin

Re: Poor Virtio PMD TX Performance

[Zhou, Tianlin]

Hi Changchun,

Thanks for your response.
Please see my embedded comments.

What kind of vhost in your test? Linux vhost or dpdk user space vhost?
[tzhou] We use Linux vhost.

Do you enable the dump/log in your test? It will decrease perf of vritio.
[tzhou] No, I did not enable dump/log in the performance testing. 
I just opened log to debug why drop packets and then close it for higher performance.


Which version of dpdk codes are you using? The tip codes in dpdk.org?
[tzhou] I used DPDK 2.0.0 and l2fwd in DPDK 2.0.0.
l2fwd start command: ./l2fwd -c 0x2 -n 1 --p 0x1

qemu start command (I used the virtio interface for RX/TX in GUEST): 
sudo kvm -m 4096M -smp 4 -hda /home/geo/yanghe/fedora20.qcow2  -boot d -daemonize -monitor \ telnet::10024,server,nowait,nodelay \ -cpu host \ -device e1000,netdev=public0,mac=00:0c:29:e1:f3:ff -netdev user,id=public0,hostfwd=tcp::11022-:22 \ -device virtio-net-pci,netdev=public1,mac=00:0c:29:e1:ff:ff,ioeventfd=on -netdev tap,id=public1,ifname=tapvm02,script=/home/geo/tzhou/kvm-image/tup.sh,downscript=no,vhost=on

The qemu version?
[tzhou]
root@dw-2:/home/geo/tzhou/kvm-image# kvm -version QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.3), Copyright (c) 2003-2008 Fabrice Bellard

-Tianlin

-----Original Message-----
From: Ouyang, Changchun [mailto:changchun.ouyang@intel.com]
Sent: Monday, June 08, 2015 10:13 AM
To: Zhou, Tianlin
Cc: Ouyang, Changchun
Subject: RE: Poor Virtio PMD TX Performance

Hi tianlin,

What kind of vhost in your test? Linux vhost or dpdk user space vhost?

Do you enable the dump/log in your test? It will decrease perf of vritio.

Which version of dpdk codes are you using? The tip codes in dpdk.org?

The qemu version?

Thanks for inputs
Changchun


> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhou, Tianlin
> Sent: Friday, June 5, 2015 5:23 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Poor Virtio PMD TX Performance
> 
> Hi there,
> 
> We tested TX performance of Virtio PMD by DPDK l2fwd, but found even 
> at 60KPPS (720B packet length) TX rate, there is 1/1000 packet dropping rate.
> The log shows "No free tx descriptors to transmit" in Virtio PMD.
> Increasing TX queues by modifying DPDK l2fwd can decreases packet 
> dropping rate, but can't ensure no packet dropping unless 
> retransmitting packets that can't be sent successfully.
> Oppositely, RX rate can be 600KPPS without packet dropping.
> 
> Test Env
> - Host CPU: 4 cores, 2127.770MHz
> - Host Memory: 8G
> - Host OS: Linux dw-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10
> 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> - Guest CPU: 4 cores, 2127.770MHz
> - Guest Memory: 4G
> - Guest OS: fedora20
> 
> Anybody here face the same problem?
> 
> -Tianlin

Re: Poor Virtio PMD TX Performance

[Jason Wang]

On 06/12/2015 05:35 PM, Zhou, Tianlin wrote:
> Hi Changchun,
>
> Thanks for your response.
> Please see my embedded comments.
>
> What kind of vhost in your test? Linux vhost or dpdk user space vhost?
> [tzhou] We use Linux vhost.
>
> Do you enable the dump/log in your test? It will decrease perf of vritio.
> [tzhou] No, I did not enable dump/log in the performance testing. 
> I just opened log to debug why drop packets and then close it for higher performance.
>
>
> Which version of dpdk codes are you using? The tip codes in dpdk.org?
> [tzhou] I used DPDK 2.0.0 and l2fwd in DPDK 2.0.0.
> l2fwd start command: ./l2fwd -c 0x2 -n 1 --p 0x1
>
> qemu start command (I used the virtio interface for RX/TX in GUEST): 
> sudo kvm -m 4096M -smp 4 -hda /home/geo/yanghe/fedora20.qcow2  -boot d -daemonize -monitor \ telnet::10024,server,nowait,nodelay \ -cpu host \ -device e1000,netdev=public0,mac=00:0c:29:e1:f3:ff -netdev user,id=public0,hostfwd=tcp::11022-:22 \ -device virtio-net-pci,netdev=public1,mac=00:0c:29:e1:ff:ff,ioeventfd=on -netdev tap,id=public1,ifname=tapvm02,script=/home/geo/tzhou/kvm-image/tup.sh,downscript=no,vhost=on

To enable vhost for pmd I believe you need use vhostforce=on here.

>
> The qemu version?
> [tzhou]
> root@dw-2:/home/geo/tzhou/kvm-image# kvm -version QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.3), Copyright (c) 2003-2008 Fabrice Bellard
>
> -Tianlin
>
> -----Original Message-----
> From: Ouyang, Changchun [mailto:changchun.ouyang@intel.com]
> Sent: Monday, June 08, 2015 10:13 AM
> To: Zhou, Tianlin
> Cc: Ouyang, Changchun
> Subject: RE: Poor Virtio PMD TX Performance
>
> Hi tianlin,
>
> What kind of vhost in your test? Linux vhost or dpdk user space vhost?
>
> Do you enable the dump/log in your test? It will decrease perf of vritio.
>
> Which version of dpdk codes are you using? The tip codes in dpdk.org?
>
> The qemu version?
>
> Thanks for inputs
> Changchun
>
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhou, Tianlin
>> Sent: Friday, June 5, 2015 5:23 PM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] Poor Virtio PMD TX Performance
>>
>> Hi there,
>>
>> We tested TX performance of Virtio PMD by DPDK l2fwd, but found even 
>> at 60KPPS (720B packet length) TX rate, there is 1/1000 packet dropping rate.
>> The log shows "No free tx descriptors to transmit" in Virtio PMD.
>> Increasing TX queues by modifying DPDK l2fwd can decreases packet 
>> dropping rate, but can't ensure no packet dropping unless 
>> retransmitting packets that can't be sent successfully.
>> Oppositely, RX rate can be 600KPPS without packet dropping.
>>
>> Test Env
>> - Host CPU: 4 cores, 2127.770MHz
>> - Host Memory: 8G
>> - Host OS: Linux dw-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10
>> 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>> - Guest CPU: 4 cores, 2127.770MHz
>> - Guest Memory: 4G
>> - Guest OS: fedora20
>>
>> Anybody here face the same problem?
>>
>> -Tianlin

Re: Poor Virtio PMD TX Performance

[Zhou, Tianlin]

Hi Jason,

The same results by adding "vhostforce=on"

-device virtio-net-pci,netdev=public1,mac=00:0c:29:e1:ff:ff,ioeventfd=on -netdev tap,id=public1,ifname=tapvm02,script=/home/geo/tzhou/kvm-image/ovs.sh,downscript=/home/geo/tzhou/kvm-image/ovs_down.sh,vhostforce=on,vnet_hdr=on

Or

-device virtio-net-pci,netdev=public1,mac=00:0c:29:e1:ff:ff,ioeventfd=on -netdev tap,id=public1,ifname=tapvm02,script=/home/geo/tzhou/kvm-image/ovs.sh,downscript=/home/geo/tzhou/kvm-image/ovs_down.sh,vhostforce=on,vhost=on,vnet_hdr=on

-----Original Message-----
From: Jason Wang [mailto:jasowang@redhat.com] 
Sent: Friday, June 12, 2015 5:46 PM
To: Zhou, Tianlin; dev@dpdk.org; Ouyang, Changchun
Subject: Re: [dpdk-dev] Poor Virtio PMD TX Performance



On 06/12/2015 05:35 PM, Zhou, Tianlin wrote:
> Hi Changchun,
>
> Thanks for your response.
> Please see my embedded comments.
>
> What kind of vhost in your test? Linux vhost or dpdk user space vhost?
> [tzhou] We use Linux vhost.
>
> Do you enable the dump/log in your test? It will decrease perf of vritio.
> [tzhou] No, I did not enable dump/log in the performance testing. 
> I just opened log to debug why drop packets and then close it for higher performance.
>
>
> Which version of dpdk codes are you using? The tip codes in dpdk.org?
> [tzhou] I used DPDK 2.0.0 and l2fwd in DPDK 2.0.0.
> l2fwd start command: ./l2fwd -c 0x2 -n 1 --p 0x1
>
> qemu start command (I used the virtio interface for RX/TX in GUEST): 
> sudo kvm -m 4096M -smp 4 -hda /home/geo/yanghe/fedora20.qcow2  -boot d 
> -daemonize -monitor \ telnet::10024,server,nowait,nodelay \ -cpu host 
> \ -device e1000,netdev=public0,mac=00:0c:29:e1:f3:ff -netdev 
> user,id=public0,hostfwd=tcp::11022-:22 \ -device 
> virtio-net-pci,netdev=public1,mac=00:0c:29:e1:ff:ff,ioeventfd=on 
> -netdev 
> tap,id=public1,ifname=tapvm02,script=/home/geo/tzhou/kvm-image/tup.sh,
> downscript=no,vhost=on

To enable vhost for pmd I believe you need use vhostforce=on here.

>
> The qemu version?
> [tzhou]
> root@dw-2:/home/geo/tzhou/kvm-image# kvm -version QEMU emulator 
> version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.3), Copyright (c) 2003-2008 
> Fabrice Bellard
>
> -Tianlin
>
> -----Original Message-----
> From: Ouyang, Changchun [mailto:changchun.ouyang@intel.com]
> Sent: Monday, June 08, 2015 10:13 AM
> To: Zhou, Tianlin
> Cc: Ouyang, Changchun
> Subject: RE: Poor Virtio PMD TX Performance
>
> Hi tianlin,
>
> What kind of vhost in your test? Linux vhost or dpdk user space vhost?
>
> Do you enable the dump/log in your test? It will decrease perf of vritio.
>
> Which version of dpdk codes are you using? The tip codes in dpdk.org?
>
> The qemu version?
>
> Thanks for inputs
> Changchun
>
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhou, Tianlin
>> Sent: Friday, June 5, 2015 5:23 PM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] Poor Virtio PMD TX Performance
>>
>> Hi there,
>>
>> We tested TX performance of Virtio PMD by DPDK l2fwd, but found even 
>> at 60KPPS (720B packet length) TX rate, there is 1/1000 packet dropping rate.
>> The log shows "No free tx descriptors to transmit" in Virtio PMD.
>> Increasing TX queues by modifying DPDK l2fwd can decreases packet 
>> dropping rate, but can't ensure no packet dropping unless 
>> retransmitting packets that can't be sent successfully.
>> Oppositely, RX rate can be 600KPPS without packet dropping.
>>
>> Test Env
>> - Host CPU: 4 cores, 2127.770MHz
>> - Host Memory: 8G
>> - Host OS: Linux dw-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10
>> 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>> - Guest CPU: 4 cores, 2127.770MHz
>> - Guest Memory: 4G
>> - Guest OS: fedora20
>>
>> Anybody here face the same problem?
>>
>> -Tianlin

Re: Poor Virtio PMD TX Performance

[Ouyang, Changchun]

Hi Tianlin

Thanks very much for your inputs,
I will investigate this issue and update to you after I have any findings.

Changchun

> -----Original Message-----
> From: Zhou, Tianlin [mailto:tianlin.zhou@tekcomms.com]
> Sent: Friday, June 12, 2015 5:36 PM
> To: dev@dpdk.org; Ouyang, Changchun
> Subject: RE: Poor Virtio PMD TX Performance
> 
> Hi Changchun,
> 
> Thanks for your response.
> Please see my embedded comments.
> 
> What kind of vhost in your test? Linux vhost or dpdk user space vhost?
> [tzhou] We use Linux vhost.
> 
> Do you enable the dump/log in your test? It will decrease perf of vritio.
> [tzhou] No, I did not enable dump/log in the performance testing.
> I just opened log to debug why drop packets and then close it for higher
> performance.
> 
> 
> Which version of dpdk codes are you using? The tip codes in dpdk.org?
> [tzhou] I used DPDK 2.0.0 and l2fwd in DPDK 2.0.0.
> l2fwd start command: ./l2fwd -c 0x2 -n 1 --p 0x1
> 
> qemu start command (I used the virtio interface for RX/TX in GUEST):
> sudo kvm -m 4096M -smp 4 -hda /home/geo/yanghe/fedora20.qcow2  -boot
> d -daemonize -monitor \ telnet::10024,server,nowait,nodelay \ -cpu host \ -
> device e1000,netdev=public0,mac=00:0c:29:e1:f3:ff -netdev
> user,id=public0,hostfwd=tcp::11022-:22 \ -device virtio-net-
> pci,netdev=public1,mac=00:0c:29:e1:ff:ff,ioeventfd=on -netdev
> tap,id=public1,ifname=tapvm02,script=/home/geo/tzhou/kvm-
> image/tup.sh,downscript=no,vhost=on
> 
> The qemu version?
> [tzhou]
> root@dw-2:/home/geo/tzhou/kvm-image# kvm -version QEMU emulator
> version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.3), Copyright (c) 2003-2008 Fabrice
> Bellard
> 
> -Tianlin
> 
> -----Original Message-----
> From: Ouyang, Changchun [mailto:changchun.ouyang@intel.com]
> Sent: Monday, June 08, 2015 10:13 AM
> To: Zhou, Tianlin
> Cc: Ouyang, Changchun
> Subject: RE: Poor Virtio PMD TX Performance
> 
> Hi tianlin,
> 
> What kind of vhost in your test? Linux vhost or dpdk user space vhost?
> 
> Do you enable the dump/log in your test? It will decrease perf of vritio.
> 
> Which version of dpdk codes are you using? The tip codes in dpdk.org?
> 
> The qemu version?
> 
> Thanks for inputs
> Changchun
> 
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhou, Tianlin
> > Sent: Friday, June 5, 2015 5:23 PM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] Poor Virtio PMD TX Performance
> >
> > Hi there,
> >
> > We tested TX performance of Virtio PMD by DPDK l2fwd, but found even
> > at 60KPPS (720B packet length) TX rate, there is 1/1000 packet dropping
> rate.
> > The log shows "No free tx descriptors to transmit" in Virtio PMD.
> > Increasing TX queues by modifying DPDK l2fwd can decreases packet
> > dropping rate, but can't ensure no packet dropping unless
> > retransmitting packets that can't be sent successfully.
> > Oppositely, RX rate can be 600KPPS without packet dropping.
> >
> > Test Env
> > - Host CPU: 4 cores, 2127.770MHz
> > - Host Memory: 8G
> > - Host OS: Linux dw-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10
> > 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> > - Guest CPU: 4 cores, 2127.770MHz
> > - Guest Memory: 4G
> > - Guest OS: fedora20
> >
> > Anybody here face the same problem?
> >
> > -Tianlin