Re: Network performance with small packets - continued

[Tom Lendacky <tahm@linux.vnet.ibm.com>]

On Wednesday, March 09, 2011 03:56:15 pm Michael S. Tsirkin wrote:
> On Wed, Mar 09, 2011 at 02:11:07PM -0600, Tom Lendacky wrote:
> > Here are the results again with the addition of the interrupt rate that
> > occurred on the guest virtio_net device:
> > 
> > Here is the KVM baseline (average of six runs):
> >   Txn Rate: 87,070.34 Txn/Sec, Pkt Rate: 172,992 Pkts/Sec
> >   Exits: 148,444.58 Exits/Sec
> >   TxCPU: 2.40%  RxCPU: 99.35%
> >   Virtio1-input  Interrupts/Sec (CPU0/CPU1): 5,154/5,222
> >   Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0
> > 
> > About 42% of baremetal.
> > 
> > Delayed freeing of TX buffers (average of six runs):
> >   Txn Rate: 90,886.19 Txn/Sec, Pkt Rate: 180,571 Pkts/Sec
> >   Exits: 142,681.67 Exits/Sec
> >   TxCPU: 2.78%  RxCPU: 99.36%
> >   Virtio1-input  Interrupts/Sec (CPU0/CPU1): 4,796/4,908
> >   Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0
> > 
> > About a 4% increase over baseline and about 44% of baremetal.
> 
> Looks like delayed freeing is a good idea generally.
> Is this my patch? Yours?

These results are for my patch, I haven't had a chance to run your patch yet.

> 
> > Delaying kick_notify (kick every 5 packets -average of six runs):
> >   Txn Rate: 107,106.36 Txn/Sec, Pkt Rate: 212,796 Pkts/Sec
> >   Exits: 102,587.28 Exits/Sec
> >   TxCPU: 3.03%  RxCPU: 99.33%
> >   Virtio1-input  Interrupts/Sec (CPU0/CPU1): 4,200/4,293
> >   Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0
> > 
> > About a 23% increase over baseline and about 52% of baremetal.
> 
> > Delaying kick_notify and pinning virtio1-input to CPU0 (average of six 
runs):
> What exactly moves the interrupt handler between CPUs?
> irqbalancer?  Does it matter which CPU you pin it to?
> If yes, do you have any idea why?

Looking at the guest, irqbalance isn't running and the smp_affinity for the 
irq is set to 3 (both CPUs).  It could be that irqbalance would help in this 
situation since it would probably change the smp_affinity mask to a single CPU 
and remove the irq lock contention (I think the last used index patch would be 
best though since it will avoid the extra irq injections).  I'll kick off a 
run with irqbalance running.

As for which CPU the interrupt gets pinned to, that doesn't matter - see 
below.

> 
> Also, what happens without delaying kick_notify
> but with pinning?

Here are the results of a single "baseline" run with the IRQ pinned to CPU0:

  Txn Rate: 108,212.12 Txn/Sec, Pkt Rate: 214,994 Pkts/Sec
  Exits: 119,310.21 Exits/Sec
  TxCPU: 9.63%  RxCPU: 99.47%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 
  Virtio1-output Interrupts/Sec (CPU0/CPU1):

and CPU1:

  Txn Rate: 108,053.02 Txn/Sec, Pkt Rate: 214,678 Pkts/Sec
  Exits: 119,320.12 Exits/Sec
  TxCPU: 9.64%  RxCPU: 99.42%
  Virtio1-input  Interrupts/Sec (CPU0/CPU1): 13,608/0
  Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/13,830

About a 24% increase over baseline.

> 
> >   Txn Rate: 153,696.59 Txn/Sec, Pkt Rate: 305,358 Pkts/Sec
> >   Exits: 62,603.37 Exits/Sec
> >   TxCPU: 3.73%  RxCPU: 98.52%
> >   Virtio1-input  Interrupts/Sec (CPU0/CPU1): 11,564/0
> >   Virtio1-output Interrupts/Sec (CPU0/CPU1): 0/0
> > 
> > About a 77% increase over baseline and about 74% of baremetal.
> 
> Hmm we get about 20 packets per interrupt on average.
> That's pretty decent. The problem is with exits.
> Let's try something adaptive in the host?