From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marcelo Tosatti Subject: Re: pinning, tsc and apic Date: Wed, 14 May 2008 20:25:06 -0300 Message-ID: <20080514232506.GA18079@dmt> References: <20080512191923.GU17938@us.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Cc: kvm-devel@lists.sourceforge.net To: Ryan Harper Return-path: Content-Disposition: inline In-Reply-To: <20080512191923.GU17938@us.ibm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: kvm-devel-bounces@lists.sourceforge.net Errors-To: kvm-devel-bounces@lists.sourceforge.net List-Id: kvm.vger.kernel.org On Mon, May 12, 2008 at 02:19:24PM -0500, Ryan Harper wrote: > I've been digging into some of the instability we see when running > larger numbers of guests at the same time. The test I'm currently using > involves launching 64 1vcpu guests on an 8-way AMD box. With the latest > kvm-userspace git and kvm.git + Gerd's kvmclock fixes, I can launch all > 64 of these 1 second apart, and only a handful (1 to 3) end up not > making it up. In dmesg on the host, I get a couple messages: > > [321365.362534] vcpu not ready for apic_round_robin > > and > > [321503.023788] Unsupported delivery mode 7 > > Now, the interesting bit for me was when I used numactl to pin the guest > to a processor, all of the guests come up with no issues at all. As I > looked into it, it means that we're not running any of the vcpu > migration code which on svm is comprised of tsc_offset recalibration and > apic migration, and on vmx, a little more per-vcpu work Hi Ryan, There are two places that attempt to use delivery mode 7: kexec crash and io_apic_64.c::check_timer(). The later will happen if the guest fails to receive PIT IRQ's for 10 ticks. If you're using HZ=1000 thats 10ms. See timer_irq_works(). The in-kernel pit emulation has logic which avoids injecting more than one IRQ during 10ms. Note that the guest 10ms delay is TSC based and uses only the lower 32-bits of the value. It is quite likely that the TSC adjustment results in them increasing more rapidly then they should. So can you try setting KVM_MAX_PIT_INTR_INTERVAL to a lower value? HZ/10 or something. You can confirm this theory by booting the guests with "apic=debug". ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/