From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: does anyone run guests for more than 5 minutes? (virtio-net perf anomaly) Date: Fri, 06 Mar 2009 10:25:19 -0700 Message-ID: <1236360319.7276.209.camel@lappy> References: <1236111221.13193.223.camel@bling> <20090306000445.GD8820@amt.cnet> <1236312791.6002.4.camel@bling> <1236353193.7276.204.camel@lappy> <20090306171740.GA14077@amt.cnet> Mime-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: 7bit Cc: kvm-devel To: Marcelo Tosatti Return-path: Received: from g4t0016.houston.hp.com ([15.201.24.19]:22138 "EHLO g4t0016.houston.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752063AbZCFRYd (ORCPT ); Fri, 6 Mar 2009 12:24:33 -0500 In-Reply-To: <20090306171740.GA14077@amt.cnet> Sender: kvm-owner@vger.kernel.org List-ID: On Fri, 2009-03-06 at 14:17 -0300, Marcelo Tosatti wrote: > On Fri, Mar 06, 2009 at 08:26:33AM -0700, Alex Williamson wrote: > > On Thu, 2009-03-05 at 21:25 -0700, Alex Williamson wrote: > > > On Thu, 2009-03-05 at 21:04 -0300, Marcelo Tosatti wrote: > > > > On Tue, Mar 03, 2009 at 01:13:41PM -0700, Alex Williamson wrote: > > > > > Any guesses as to what might be going on? Can anyone reproduce? I'm > > > > > hoping that I'm doing something dumb, but can't figure out what it is. > > > > > The system is running v2.6.29-rc6-121-g64e7130 in the guest, > > > > > v2.6.29-rc6-123-gbd7b3b4 on the host, kvm module kvm-84-620-g5bffffc and > > > > > userspace kvm-84-95-gea1b668. Thanks, > > > > > > > > Nope. Collect kvm_stat -l before/after the slowdown? > > > > > > Attached. This shows about 1 minute of data before the slowdown and a > > > dramatic change starting around the 80th row. Thanks, > > > > For a bit easier consumption, here's a google spreadsheet and chart of > > what appear to be the interesting columns: > > > > http://spreadsheets.google.com/pub?key=pdwpc4VwMbjWxyQfvw8AEYg > > > > Alex > > irq_injection goes up significantly. Is it virtio_net generating more > interrupts? Check the rate of irq's generated by hw/virtio-net.c and > compare that with rate seen in /proc/interrupts inside the guest? The rate inside the guest more than doubles when things slow down, ~2k/s -> ~4.5k/s. Also interesting, the soft interrupt time reported by top in the guest goes from ~65% to zero. I haven't checked what hw/virtio-net.c is generating. FWIW, it's not processor specific, I see the same thing on an AMD based system, as shown in the updated spreadsheet. Thanks, Alex