Re: kvm guest loops_per_jiffy miscalibration under host load

[Anthony Liguori <aliguori@us.ibm.com>]

Glauber Costa wrote:
> Marcelo Tosatti wrote:
>> Hello,
>>
>> I have been discussing with Glauber and Gerd the problem where KVM
>> guests miscalibrate loops_per_jiffy if there's sufficient load on the
>> host.
>>
>> calibrate_delay_direct() failed to get a good estimate for
>> loops_per_jiffy.
>> Probably due to long platform interrupts. Consider using "lpj=" boot
>> option.
>> Calibrating delay loop... <3>107.00 BogoMIPS (lpj=214016)
>>
>> While this particular host calculates lpj=1597041.
>>
>> This means that udelay() can delay for less than what asked for, with
>> fatal results such as:
>>
>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
>> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the
>> 'noapic' kernel parameter
>>
>> This bug is easily triggered with a CPU hungry task on nice -20
>> running only during guest calibration (so that the timer check code on
>> io_apic_{32,64}.c fails to wait long enough for PIT interrupts to fire).
>>
>> The problem is that the calibration routines assume a stable relation
>> between timer interrupt frequency (PIT at this boot stage) and
>> TSC/execution frequency.
>>
>> The emulated timer frequency is based on the host system time and
>> therefore virtually resistant against heavy load, while the execution
>> of these routines on the guest is suspectible to scheduling of the QEMU
>> process.
>>
>> To fix this in a transparent way (without direct "lpj=" boot parameter
>> assignment or a paravirt equivalent), it would be necessary to base the
>> emulated timer frequency on guest execution time instead of host system
>> time. But this can introduce timekeeping issues (recent Linux guests
>> seem to handle lost/late interrupts fine as long as the clocksource is
>> reliable) and just sounds scary.
>>
>> Possible solutions:
>>
>> - Require the admin to preset "lpj=". Nasty, not user friendly.
>> - Pass the proper lpj value via a paravirt interface. Won't cover
>>   fullvirt guests.
>> - Have the management app guarantee a minimum amount of CPU required
>> for proper calibration during guest initialization.
> I don't like any of these solutions, and won't defend any of "the 
> one". So no hard feelings. But I think the "less worse" among them 
> IMHO is the
> paravirt one. At least it goes in the general direction of "paravirt 
> if you need to scale over xyz".

I agree.  A paravirt solution solves the problem.

> I think passing lpj is out of question, and giving the cpu resources 
> for that time is kind of a kludge.

It's all heuristics unfortunately.

> Or maybe we could put the timer expiration alone in a separate thread, 
> with maximum priority (maybe rt priority)? dunno...

But then if you have high-load because of a lot of guests running, you 
defeat yourself.  Any attempt to guarantee time to a guest will be 
defeated by lots of guests all attempting calibration at the same time.

Regards,

Anthony Liguori