[PATCH] virtio: Avoid virtio_net TX queue over run

[PATCH] virtio: Avoid virtio_net TX queue over run

[Shirley Ma]

This patch addresses small message size performance in a situation the
KVM guest virtio_net TX queue overrun. This patch adds a new API in
virtio_ring for ring capacity check; and remove KVM guest virtio_net TX
queue send completion interrupts completely. The test has shown that 
whenever the queue is overrun, it's much better to drop a few packets
than stopping TX queue and waiting for host to notify the guest to wake
up the TX queue again, the small messages size performance gain for
single TCP_STREAM BW could be up to 200%-300% and better than bare
metal, and no regression has been found in other situation.

Performance data for 10GbE,

KVM guest to local host:
------------------------
Message size	2.6.38-rc8 	2.6.38-rc8+patch
1024		1770.61		4528.37
2048		2702.30		7110.95
4096		5256.84		10104.76
8192		7543.66		10945.93
16K		10500.47	10783.50
64K		13718.62	13640.80

KVM guest to remote host:
--------------------------
Message size	Bare Metal  2.6.38-rc8	2.6.38-rc8+patch
1024		1802.67		2381.41		5599.15	
2048		4317.87		4094.12		9241.86
4096		6266.15		5231.24		9321.87
8192		8409.17		7952.74		9265.45
16K		9351.63		8260.68		8310.29
64K		9347.94		9103.75		9094.38 

Thanks
Shirley

Re: [PATCH] virtio: Avoid virtio_net TX queue over run

[Michael S. Tsirkin]

On Wed, Mar 16, 2011 at 05:09:55PM -0700, Shirley Ma wrote:
> This patch addresses small message size performance in a situation the
> KVM guest virtio_net TX queue overrun. This patch adds a new API in
> virtio_ring for ring capacity check; and remove KVM guest virtio_net TX
> queue send completion interrupts completely. The test has shown that 
> whenever the queue is overrun, it's much better to drop a few packets
> than stopping TX queue and waiting for host to notify the guest to wake
> up the TX queue again, the small messages size performance gain for
> single TCP_STREAM BW could be up to 200%-300% and better than bare
> metal, and no regression has been found in other situation.
> 
> Performance data for 10GbE,
> 
> KVM guest to local host:
> ------------------------
> Message size	2.6.38-rc8 	2.6.38-rc8+patch
> 1024		1770.61		4528.37
> 2048		2702.30		7110.95
> 4096		5256.84		10104.76
> 8192		7543.66		10945.93
> 16K		10500.47	10783.50
> 64K		13718.62	13640.80
> 
> KVM guest to remote host:
> --------------------------
> Message size	Bare Metal  2.6.38-rc8	2.6.38-rc8+patch
> 1024		1802.67		2381.41		5599.15	
> 2048		4317.87		4094.12		9241.86
> 4096		6266.15		5231.24		9321.87
> 8192		8409.17		7952.74		9265.45
> 16K		9351.63		8260.68		8310.29
> 64K		9347.94		9103.75		9094.38 
> 
> Thanks
> Shirley

Could you add CPU utilization data pls? I also wonder what does this do
to UDP?  Won't a lot of packets be dropped?

-- 
MST

Re: [PATCH] virtio: Avoid virtio_net TX queue over run

[Shirley Ma]

On Thu, 2011-03-17 at 07:05 +0200, Michael S. Tsirkin wrote:
> Could you add CPU utilization data pls? I also wonder what does this
> do
> to UDP?  Won't a lot of packets be dropped?

Guest CPU utilization slightly increased a few %.

UDP performance send rate increased 200% - 400%, recv rate increased
from 10% to around 40% w/i recv errors. I will tune the buffers to see
any better.

I am putting all data together and post the results here soon.

Thanks
Shirley

Re: [PATCH] virtio: Avoid virtio_net TX queue over run

[Eric Dumazet]

Le jeudi 17 mars 2011 à 09:50 -0700, Shirley Ma a écrit :
> On Thu, 2011-03-17 at 07:05 +0200, Michael S. Tsirkin wrote:
> > Could you add CPU utilization data pls? I also wonder what does this
> > do
> > to UDP?  Won't a lot of packets be dropped?
> 
> Guest CPU utilization slightly increased a few %.
> 
> UDP performance send rate increased 200% - 400%, recv rate increased
> from 10% to around 40% w/i recv errors. I will tune the buffers to see
> any better.
> 
> I am putting all data together and post the results here soon.

Could you run your tests with only one cpu ?
(offline all cpus but cpu0 on your hypervisor)

Dropping packets is fine, unless consumer cant run ;)

Re: [PATCH] virtio: Avoid virtio_net TX queue over run

[Shirley Ma]

On Thu, 2011-03-17 at 17:58 +0100, Eric Dumazet wrote:
> Could you run your tests with only one cpu ?
> (offline all cpus but cpu0 on your hypervisor)
> 
> Dropping packets is fine, unless consumer cant run ;) 

Sure, will report the data soon.

Thanks
Shirley

Re: [PATCH] virtio: Avoid virtio_net TX queue over run

[Shirley Ma]

On Thu, 2011-03-17 at 10:01 -0700, Shirley Ma wrote:
> On Thu, 2011-03-17 at 17:58 +0100, Eric Dumazet wrote:
> > Could you run your tests with only one cpu ?
> > (offline all cpus but cpu0 on your hypervisor)
> > 
> > Dropping packets is fine, unless consumer cant run ;) 
> 
> Sure, will report the data soon. 

With a single CPU, before/after with this patch the performance has no
difference for both UDP and TCP, and it seems hitting TX overrun, CPU is
100% used.

Thanks
Shirley

Re: [PATCH] virtio: Avoid virtio_net TX queue over run

[Shirley Ma]

On Thu, 2011-03-17 at 11:49 -0700, Shirley Ma wrote:
> With a single CPU, before/after with this patch the performance has no
> difference for both UDP and TCP, and it seems hitting TX overrun, CPU
> is
> 100% used. 

Sorry, should be: it seems never hitting TX overrun.

Shirley