Re: Network performance with small packets - continued

[Tom Lendacky <tahm@linux.vnet.ibm.com>]

On Thursday, March 10, 2011 11:16:11 am Tom Lendacky wrote:
> On Thursday, March 10, 2011 09:34:22 am Michael S. Tsirkin wrote:
> > On Thu, Mar 10, 2011 at 09:23:42AM -0600, Tom Lendacky wrote:
> > > On Thursday, March 10, 2011 12:54:58 am Michael S. Tsirkin wrote:
> > > > On Wed, Mar 09, 2011 at 05:25:11PM -0600, Tom Lendacky wrote:
> > > > > As for which CPU the interrupt gets pinned to, that doesn't matter
> > > > > - see below.
> > > > 
> > > > So what hurts us the most is that the IRQ jumps between the VCPUs?
> > > 
> > > Yes, it appears that allowing the IRQ to run on more than one vCPU
> > > hurts. Without the publish last used index patch, vhost keeps
> > > injecting an irq for every received packet until the guest eventually
> > > turns off notifications.
> > 
> > Are you sure you see that? If yes publish used should help a lot.
> 
> I definitely see that.  I ran lockstat in the guest and saw the contention
> on the lock when the irq was able to run on either vCPU.  Once the irq was
> pinned the contention disappeared.  The publish used index patch should
> eliminate the extra irq injections and then the pinning or use of
> irqbalance shouldn't be required.  I'm getting a kernel oops during boot
> with the publish last used patches that I pulled from the mailing list - I
> had to make some changes in order to get them to apply and compile and
> might not have done the right things.  Can you re-spin that patchset
> against kvm.git?
> 

Here are the results for the publish last used index patch (with the baseline 
provided again for reference).

Here is the KVM baseline (average of six runs):
  Txn Rate: 87,070.34 Txn/Sec, Pkt Rate: 172,992 Pkts/Sec
  Exits: 148,444.58 Exits/Sec
  TxCPU: 2.40%  RxCPU: 99.35%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 5,154/5,222
  Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0

Using the publish last used index w/o irqbalance (average of six runs):
  Txn Rate: 112,180.10 Txn/Sec, Pkt Rate: 222,878.33 Pkts/Sec
  Exits: 96,280.11 Exits/Sec
  TxCPU: 1.14%  RxCPU: 99.33%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 3,400/3,400
  Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0

About a 29% increase over baseline.

Using the publish last used index w/  irqbalance (average of six runs):
  Txn Rate: 110,891.12 Txn/Sec, Pkt Rate: 220,315.67 Pkts/Sec
  Exits: 97,190.68 Exits/Sec
  TxCPU: 1.10%  RxCPU: 99.38%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 7,040/0
  Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0

About a 27% increase over baseline.

Here is data from running without the publish last used index patch but with 
irqbalance running (pinning results were near identical):
  Txn Rate: 107,714.53 Txn/Sec, Pkt Rate: 214,006 Pkts/Sec
  Exits: 121,050.45 Exits/Sec
  TxCPU: 9.61%  RxCPU: 99.45%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 13,975/0
  Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0

The publish last used index patch provides a 3%-4% improvement while reducing 
the exit rate and interrupt rate in the guest as well as reducing the 
transmitting system CPU% quite dramatically.

> > > Because the irq injections end up overlapping we get contention on the
> > > irq_desc_lock_class lock. Here are some results using the "baseline"
> > > setup with irqbalance running.
> > > 
> > >   Txn Rate: 107,714.53 Txn/Sec, Pkt Rate: 214,006 Pkts/Sec
> > >   Exits: 121,050.45 Exits/Sec
> > >   TxCPU: 9.61%  RxCPU: 99.45%
> > >   Virtio1-input  Interrupts/Sec (CPU0/CPU1): 13,975/0
> > >   Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0
> > > 
> > > About a 24% increase over baseline.  Irqbalance essentially pinned the
> > > virtio irq to CPU0 preventing the irq lock contention and resulting in
> > > nice gains.
> > 
> > OK, so we probably want some form of delayed free for TX
> > on top, and that should get us nice results already.
> > 
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html