From mboxrd@z Thu Jan  1 00:00:00 1970
From: Glauber Costa <gcosta@redhat.com>
Subject: Re: kvm guest loops_per_jiffy miscalibration under host load
Date: Thu, 03 Jul 2008 10:17:05 -0300
Message-ID: <486CD151.8020004@redhat.com>
References: <20080702164021.GA31751@dmt.cnet>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: kvm-devel <kvm@vger.kernel.org>, kraxel@redhat.com,
	chrisw@redhat.com, aliguori@us.ibm.com
To: Marcelo Tosatti <mtosatti@redhat.com>
Return-path: <kvm-owner@vger.kernel.org>
Received: from mx1.redhat.com ([66.187.233.31]:54343 "EHLO mx1.redhat.com"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1754826AbYGCNSP (ORCPT <rfc822;kvm@vger.kernel.org>);
	Thu, 3 Jul 2008 09:18:15 -0400
In-Reply-To: <20080702164021.GA31751@dmt.cnet>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

Marcelo Tosatti wrote:
> Hello,
> 
> I have been discussing with Glauber and Gerd the problem where KVM
> guests miscalibrate loops_per_jiffy if there's sufficient load on the
> host.
> 
> calibrate_delay_direct() failed to get a good estimate for
> loops_per_jiffy.
> Probably due to long platform interrupts. Consider using "lpj=" boot
> option.
> Calibrating delay loop... <3>107.00 BogoMIPS (lpj=214016)
> 
> While this particular host calculates lpj=1597041.
> 
> This means that udelay() can delay for less than what asked for, with
> fatal results such as:
> 
> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the
> 'noapic' kernel parameter
> 
> This bug is easily triggered with a CPU hungry task on nice -20
> running only during guest calibration (so that the timer check code on
> io_apic_{32,64}.c fails to wait long enough for PIT interrupts to fire).
> 
> The problem is that the calibration routines assume a stable relation
> between timer interrupt frequency (PIT at this boot stage) and
> TSC/execution frequency.
> 
> The emulated timer frequency is based on the host system time and
> therefore virtually resistant against heavy load, while the execution
> of these routines on the guest is suspectible to scheduling of the QEMU
> process.
> 
> To fix this in a transparent way (without direct "lpj=" boot parameter
> assignment or a paravirt equivalent), it would be necessary to base the
> emulated timer frequency on guest execution time instead of host system
> time. But this can introduce timekeeping issues (recent Linux guests
> seem to handle lost/late interrupts fine as long as the clocksource is
> reliable) and just sounds scary.
> 
> Possible solutions:
> 
> - Require the admin to preset "lpj=". Nasty, not user friendly.
> - Pass the proper lpj value via a paravirt interface. Won't cover
>   fullvirt guests.
> - Have the management app guarantee a minimum amount of CPU required
> for proper calibration during guest initialization.
I don't like any of these solutions, and won't defend any of "the one". 
So no hard feelings. But I think the "less worse" among them IMHO is the
paravirt one. At least it goes in the general direction of "paravirt if 
you need to scale over xyz".

I think passing lpj is out of question, and giving the cpu resources for 
that time is kind of a kludge.

Or maybe we could put the timer expiration alone in a separate thread, 
with maximum priority (maybe rt priority)? dunno...