The HPET issue on Linux

The HPET issue on Linux

[Sheng Yang]

Hi Beth

I still found the emulated HPET would result in some boot failure. For 
example, on my 2.6.30, with HPET enabled, the kernel would fail check_timer(), 
especially in timer_irq_works().

The testing of timer_irq_works() is let 10 ticks pass(using mdelay()), and 
want to confirm the clock source with at least 5 ticks advanced in jiffies. 
I've checked that, on my machine, it would mostly get only 4 ticks when HPET 
enabled, then fail the test. On the other hand, if I using PIT, it would get 
more than 10 ticks(maybe understandable if some complementary ticks there). Of 
course, extend the ticks count/mdelay() time can work.

I think it's a major issue of HPET. And it maybe just due to a too long 
userspace path for interrupt injection... If it's true, I think it's not easy 
to deal with it.

-- 
regards
Yang, Sheng

Re: The HPET issue on Linux

[Gleb Natapov]

On Wed, Jan 06, 2010 at 05:48:52PM +0800, Sheng Yang wrote:
> Hi Beth
> 
> I still found the emulated HPET would result in some boot failure. For 
> example, on my 2.6.30, with HPET enabled, the kernel would fail check_timer(), 
> especially in timer_irq_works().
> 
> The testing of timer_irq_works() is let 10 ticks pass(using mdelay()), and 
> want to confirm the clock source with at least 5 ticks advanced in jiffies. 
> I've checked that, on my machine, it would mostly get only 4 ticks when HPET 
> enabled, then fail the test. On the other hand, if I using PIT, it would get 
> more than 10 ticks(maybe understandable if some complementary ticks there). Of 
> course, extend the ticks count/mdelay() time can work.
> 
> I think it's a major issue of HPET. And it maybe just due to a too long 
> userspace path for interrupt injection... If it's true, I think it's not easy 
> to deal with it.
> 
PIT tick are reinjected automatically, HPET should probably do the same
although it may just create another set of problems.

--
			Gleb.

Re: The HPET issue on Linux

[Dor Laor]

On 01/06/2010 12:09 PM, Gleb Natapov wrote:
> On Wed, Jan 06, 2010 at 05:48:52PM +0800, Sheng Yang wrote:
>> Hi Beth
>>
>> I still found the emulated HPET would result in some boot failure. For
>> example, on my 2.6.30, with HPET enabled, the kernel would fail check_timer(),
>> especially in timer_irq_works().
>>
>> The testing of timer_irq_works() is let 10 ticks pass(using mdelay()), and
>> want to confirm the clock source with at least 5 ticks advanced in jiffies.
>> I've checked that, on my machine, it would mostly get only 4 ticks when HPET
>> enabled, then fail the test. On the other hand, if I using PIT, it would get
>> more than 10 ticks(maybe understandable if some complementary ticks there). Of
>> course, extend the ticks count/mdelay() time can work.
>>
>> I think it's a major issue of HPET. And it maybe just due to a too long
>> userspace path for interrupt injection... If it's true, I think it's not easy
>> to deal with it.
>>
> PIT tick are reinjected automatically, HPET should probably do the same
> although it may just create another set of problems.

Older Linux do automatic adjustment for lost ticks so automatic 
reinjection causes time to run too fast. This is why we added the 
-no-kvm-pit-reinject flag...

It took lots of time to pit/rtc to stabilize, in order of seriously 
consider the hpet emulation, lots of testing should be done.

>
> --
> 			Gleb.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The HPET issue on Linux

[Beth Kon]

Dor Laor wrote:
> On 01/06/2010 12:09 PM, Gleb Natapov wrote:
>> On Wed, Jan 06, 2010 at 05:48:52PM +0800, Sheng Yang wrote:
>>> Hi Beth
>>>
>>> I still found the emulated HPET would result in some boot failure. For
>>> example, on my 2.6.30, with HPET enabled, the kernel would fail 
>>> check_timer(),
>>> especially in timer_irq_works().
>>>
>>> The testing of timer_irq_works() is let 10 ticks pass(using 
>>> mdelay()), and
>>> want to confirm the clock source with at least 5 ticks advanced in 
>>> jiffies.
>>> I've checked that, on my machine, it would mostly get only 4 ticks 
>>> when HPET
>>> enabled, then fail the test. On the other hand, if I using PIT, it 
>>> would get
>>> more than 10 ticks(maybe understandable if some complementary ticks 
>>> there). Of
>>> course, extend the ticks count/mdelay() time can work.
>>>
>>> I think it's a major issue of HPET. And it maybe just due to a too long
>>> userspace path for interrupt injection... If it's true, I think it's 
>>> not easy
>>> to deal with it.
>>>
>> PIT tick are reinjected automatically, HPET should probably do the same
>> although it may just create another set of problems.
>
> Older Linux do automatic adjustment for lost ticks so automatic 
> reinjection causes time to run too fast. This is why we added the 
> -no-kvm-pit-reinject flag...
>
> It took lots of time to pit/rtc to stabilize, in order of seriously 
> consider the hpet emulation, lots of testing should be done.
I will try to look into this. Since HPET is edge-triggered, looks like 
this problem is of a different nature than PIT.  Is this a solid failure 
or intermittent?

>
>>
>> -- 
>>             Gleb.
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Regards,

Beth Kon

Re: The HPET issue on Linux

[Beth Kon]

Beth Kon wrote:
> Dor Laor wrote:
>> On 01/06/2010 12:09 PM, Gleb Natapov wrote:
>>> On Wed, Jan 06, 2010 at 05:48:52PM +0800, Sheng Yang wrote:
>>>> Hi Beth
>>>>
>>>> I still found the emulated HPET would result in some boot failure. For
>>>> example, on my 2.6.30, with HPET enabled, the kernel would fail 
>>>> check_timer(),
>>>> especially in timer_irq_works().
>>>>
>>>> The testing of timer_irq_works() is let 10 ticks pass(using 
>>>> mdelay()), and
>>>> want to confirm the clock source with at least 5 ticks advanced in 
>>>> jiffies.
>>>> I've checked that, on my machine, it would mostly get only 4 ticks 
>>>> when HPET
>>>> enabled, then fail the test. On the other hand, if I using PIT, it 
>>>> would get
>>>> more than 10 ticks(maybe understandable if some complementary ticks 
>>>> there). Of
>>>> course, extend the ticks count/mdelay() time can work.
>>>>
>>>> I think it's a major issue of HPET. And it maybe just due to a too 
>>>> long
>>>> userspace path for interrupt injection... If it's true, I think 
>>>> it's not easy
>>>> to deal with it.
>>>>
>>> PIT tick are reinjected automatically, HPET should probably do the same
>>> although it may just create another set of problems.
>>
>> Older Linux do automatic adjustment for lost ticks so automatic 
>> reinjection causes time to run too fast. This is why we added the 
>> -no-kvm-pit-reinject flag...
>>
>> It took lots of time to pit/rtc to stabilize, in order of seriously 
>> consider the hpet emulation, lots of testing should be done.
> I will try to look into this. Since HPET is edge-triggered, looks like 
> this problem is of a different nature than PIT.  Is this a solid 
> failure or intermittent?
Anthony just explained that on x86, even edge-triggered interrupts are 
queued in the apic and an eoi will occur, so this is not different than 
the PIT.
>
>>
>>>
>>> -- 
>>>             Gleb.
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>


-- 
Regards,

Beth Kon

Re: The HPET issue on Linux

[Anthony Liguori]

On 01/06/2010 01:20 PM, Beth Kon wrote:
> Beth Kon wrote:
>> I will try to look into this. Since HPET is edge-triggered, looks 
>> like this problem is of a different nature than PIT.  Is this a solid 
>> failure or intermittent?
> Anthony just explained that on x86, even edge-triggered interrupts are 
> queued in the apic and an eoi will occur, so this is not different 
> than the PIT.

Not quite queued in the sense that multiple events will be delivered in 
order, but I think the point is that you can still detect whether 
delivery succeeded by counting APIC EOIs.

The trouble is that historically we've struggled with doing this in 
userspace.  Maybe it's time to revisit.

Regards,

Anthony Liguori

Re: The HPET issue on Linux

[Gleb Natapov]

On Wed, Jan 06, 2010 at 01:23:28PM -0600, Anthony Liguori wrote:
> On 01/06/2010 01:20 PM, Beth Kon wrote:
> >Beth Kon wrote:
> >>I will try to look into this. Since HPET is edge-triggered,
> >>looks like this problem is of a different nature than PIT.  Is
> >>this a solid failure or intermittent?
> >Anthony just explained that on x86, even edge-triggered interrupts
> >are queued in the apic and an eoi will occur, so this is not
> >different than the PIT.
> 
> Not quite queued in the sense that multiple events will be delivered
> in order, but I think the point is that you can still detect whether
> delivery succeeded by counting APIC EOIs.
> 
> The trouble is that historically we've struggled with doing this in
> userspace.  Maybe it's time to revisit.
> 
We reinject PIT interrupts from kernel and RTC interrupts from
userspace.

--
			Gleb.

Re: The HPET issue on Linux

[Anthony Liguori]

On 01/06/2010 01:44 PM, Gleb Natapov wrote:
> On Wed, Jan 06, 2010 at 01:23:28PM -0600, Anthony Liguori wrote:
>    
>> On 01/06/2010 01:20 PM, Beth Kon wrote:
>>      
>>> Beth Kon wrote:
>>>        
>>>> I will try to look into this. Since HPET is edge-triggered,
>>>> looks like this problem is of a different nature than PIT.  Is
>>>> this a solid failure or intermittent?
>>>>          
>>> Anthony just explained that on x86, even edge-triggered interrupts
>>> are queued in the apic and an eoi will occur, so this is not
>>> different than the PIT.
>>>        
>> Not quite queued in the sense that multiple events will be delivered
>> in order, but I think the point is that you can still detect whether
>> delivery succeeded by counting APIC EOIs.
>>
>> The trouble is that historically we've struggled with doing this in
>> userspace.  Maybe it's time to revisit.
>>
>>      
> We reinject PIT interrupts from kernel and RTC interrupts from
> userspace.
>    

Because we can determine that we've missed an RTC interrupt in 
userspace.  We cannot determine this with the PIT without adding a hook 
into the userspace apic that lets us know whether an injection failed or 
not.

Regards,

Anthony Liguori

Re: The HPET issue on Linux

[Gleb Natapov]

On Wed, Jan 06, 2010 at 01:51:54PM -0600, Anthony Liguori wrote:
> On 01/06/2010 01:44 PM, Gleb Natapov wrote:
> >On Wed, Jan 06, 2010 at 01:23:28PM -0600, Anthony Liguori wrote:
> >>On 01/06/2010 01:20 PM, Beth Kon wrote:
> >>>Beth Kon wrote:
> >>>>I will try to look into this. Since HPET is edge-triggered,
> >>>>looks like this problem is of a different nature than PIT.  Is
> >>>>this a solid failure or intermittent?
> >>>Anthony just explained that on x86, even edge-triggered interrupts
> >>>are queued in the apic and an eoi will occur, so this is not
> >>>different than the PIT.
> >>Not quite queued in the sense that multiple events will be delivered
> >>in order, but I think the point is that you can still detect whether
> >>delivery succeeded by counting APIC EOIs.
> >>
> >>The trouble is that historically we've struggled with doing this in
> >>userspace.  Maybe it's time to revisit.
> >>
> >We reinject PIT interrupts from kernel and RTC interrupts from
> >userspace.
> 
> Because we can determine that we've missed an RTC interrupt in
> userspace.  We cannot determine this with the PIT without adding a
> hook into the userspace apic that lets us know whether an injection
> failed or not.
> 
We have exactly that hook in apic already and that's how RTC determines
that interrupt was coalesced.

--
			Gleb.

Re: The HPET issue on Linux

[Anthony Liguori]

On 01/06/2010 02:37 PM, Gleb Natapov wrote:
> We have exactly that hook in apic already and that's how RTC determines
> that interrupt was coalesced.
>    

AFAICT, apic_irq_delivered is only reset explicitly by the RTC when the 
line is lowered.  It's not currently lowered based on EOI.

How can this mechanism be used with the HPET when operating in edge 
triggered mode?

Regards,

Anthony Liguori

> --
> 			Gleb.
>    

Re: The HPET issue on Linux

[Gleb Natapov]

On Wed, Jan 06, 2010 at 04:42:30PM -0600, Anthony Liguori wrote:
> On 01/06/2010 02:37 PM, Gleb Natapov wrote:
> >We have exactly that hook in apic already and that's how RTC determines
> >that interrupt was coalesced.
> 
> AFAICT, apic_irq_delivered is only reset explicitly by the RTC when
> the line is lowered.  It's not currently lowered based on EOI.
> 
Correct. We can expose ACK notifiers to userspace (and if we want to
move assigned devices into userspace we have to), but I'd rather avoid
it.

> How can this mechanism be used with the HPET when operating in edge
> triggered mode?
>
If interrupt is coalesced increment counter and double HPET timer
frequency. When counter is zeroed return HPET timer to normal frequency.

--
			Gleb.

Re: The HPET issue on Linux

[Sheng Yang]

On Thursday 07 January 2010 02:36:26 Beth Kon wrote:
> Dor Laor wrote:
> > On 01/06/2010 12:09 PM, Gleb Natapov wrote:
> >> On Wed, Jan 06, 2010 at 05:48:52PM +0800, Sheng Yang wrote:
> >>> Hi Beth
> >>>
> >>> I still found the emulated HPET would result in some boot failure. For
> >>> example, on my 2.6.30, with HPET enabled, the kernel would fail
> >>> check_timer(),
> >>> especially in timer_irq_works().
> >>>
> >>> The testing of timer_irq_works() is let 10 ticks pass(using
> >>> mdelay()), and
> >>> want to confirm the clock source with at least 5 ticks advanced in
> >>> jiffies.
> >>> I've checked that, on my machine, it would mostly get only 4 ticks
> >>> when HPET
> >>> enabled, then fail the test. On the other hand, if I using PIT, it
> >>> would get
> >>> more than 10 ticks(maybe understandable if some complementary ticks
> >>> there). Of
> >>> course, extend the ticks count/mdelay() time can work.
> >>>
> >>> I think it's a major issue of HPET. And it maybe just due to a too long
> >>> userspace path for interrupt injection... If it's true, I think it's
> >>> not easy
> >>> to deal with it.
> >>
> >> PIT tick are reinjected automatically, HPET should probably do the same
> >> although it may just create another set of problems.
> >
> > Older Linux do automatic adjustment for lost ticks so automatic
> > reinjection causes time to run too fast. This is why we added the
> > -no-kvm-pit-reinject flag...
> >
> > It took lots of time to pit/rtc to stabilize, in order of seriously
> > consider the hpet emulation, lots of testing should be done.
> 
> I will try to look into this. Since HPET is edge-triggered, looks like
> this problem is of a different nature than PIT.  Is this a solid failure
> or intermittent?
> 
At least for v2.6.30 in my box, it always fails... Of course, I believe the 
chance of successful injecting enough interrupt depends on the many factors. 
So I think out target can be: not far behind what PIT can do...

-- 
regards
Yang, Sheng

No longer working on HPET

[Beth Kon]

I have decided to take a job outside of IBM and so will not be involved 
with HPET any longer. Working on KVM has been great fun... top-notch 
people and a top-notch technology. Wishing KVM and you all the best!

-- 
Regards,

Beth Kon