Re: does anyone run guests for more than 5 minutes? (virtio-net perf anomaly)

[Alex Williamson <alex.williamson@hp.com>]

On Fri, 2009-03-06 at 14:17 -0300, Marcelo Tosatti wrote:
> On Fri, Mar 06, 2009 at 08:26:33AM -0700, Alex Williamson wrote:
> > On Thu, 2009-03-05 at 21:25 -0700, Alex Williamson wrote:
> > > On Thu, 2009-03-05 at 21:04 -0300, Marcelo Tosatti wrote:
> > > > On Tue, Mar 03, 2009 at 01:13:41PM -0700, Alex Williamson wrote:
> > > > > Any guesses as to what might be going on?  Can anyone reproduce?  I'm
> > > > > hoping that I'm doing something dumb, but can't figure out what it is.
> > > > > The system is running v2.6.29-rc6-121-g64e7130 in the guest,
> > > > > v2.6.29-rc6-123-gbd7b3b4 on the host, kvm module kvm-84-620-g5bffffc and
> > > > > userspace kvm-84-95-gea1b668.  Thanks,
> > > > 
> > > > Nope. Collect kvm_stat -l before/after the slowdown?
> > > 
> > > Attached.  This shows about 1 minute of data before the slowdown and a
> > > dramatic change starting around the 80th row.  Thanks,
> > 
> > For a bit easier consumption, here's a google spreadsheet and chart of
> > what appear to be the interesting columns:
> > 
> > http://spreadsheets.google.com/pub?key=pdwpc4VwMbjWxyQfvw8AEYg
> > 
> > Alex
> 
> irq_injection goes up significantly. Is it virtio_net generating more
> interrupts? Check the rate of irq's generated by hw/virtio-net.c and
> compare that with rate seen in /proc/interrupts inside the guest?

The rate inside the guest more than doubles when things slow down, ~2k/s
-> ~4.5k/s.  Also interesting, the soft interrupt time reported by top
in the guest goes from ~65% to zero.  I haven't checked what
hw/virtio-net.c is generating.  FWIW, it's not processor specific, I see
the same thing on an AMD based system, as shown in the updated
spreadsheet.  Thanks,

Alex