Make QEmu HPET disabled by default for KVM?

Make QEmu HPET disabled by default for KVM?

[Sheng Yang]

I think we have already suffered enough timer issues due to this(e.g. I can't 
boot up well on 2.6.18 kernel)... I have kept --no-hpet in my setup for 
months...

-- 
regards
Yang, Sheng

Re: Make QEmu HPET disabled by default for KVM?

[Avi Kivity]

On 03/11/2010 09:52 AM, Sheng Yang wrote:
> I think we have already suffered enough timer issues due to this(e.g. I can't
> boot up well on 2.6.18 kernel)...

2.6.18 as guest or as host?

> I have kept --no-hpet in my setup for
> months...
>    

Any details about the problems?  HPET is important to some guests.

-- 
error compiling committee.c: too many arguments to function

Re: Make QEmu HPET disabled by default for KVM?

[Sheng Yang]

On Thursday 11 March 2010 15:58:12 Avi Kivity wrote:
> On 03/11/2010 09:52 AM, Sheng Yang wrote:
> > I think we have already suffered enough timer issues due to this(e.g. I
> > can't boot up well on 2.6.18 kernel)...
> 
> 2.6.18 as guest or as host?

Guest
 
> > I have kept --no-hpet in my setup for
> > months...
> 
> Any details about the problems?  HPET is important to some guests.
> 

Seems like HPET reaction is too slow to satisfy some guests(for it would 
replace PIT).

Here is the thread last time.

http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899

-- 
regards
Yang, Sheng

Re: Make QEmu HPET disabled by default for KVM?

[Avi Kivity]

On 03/11/2010 10:23 AM, Sheng Yang wrote:
>>> I have kept --no-hpet in my setup for
>>> months...
>>>        
>> Any details about the problems?  HPET is important to some guests.
>>
>>      
> Seems like HPET reaction is too slow to satisfy some guests(for it would
> replace PIT).
>
> Here is the thread last time.
>
> http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899
>
>    

Thanks.  We can address this in three ways: first, adjust the guest not 
to do timing related tests when virtualized (since no matter what we do, 
the tests may fail).  Second, I think we should implement userspace ack 
notifiers (similar to tpr access notifiers already present).  Third, we 
can implement a kernel hpet, which, after we solve the zillion bug it 
introduces, will also give a nice performance improvement for hpet 
intensive workloads.

-- 
error compiling committee.c: too many arguments to function

Re: Make QEmu HPET disabled by default for KVM?

[Gleb Natapov]

On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote:
> On 03/11/2010 10:23 AM, Sheng Yang wrote:
> >>>I have kept --no-hpet in my setup for
> >>>months...
> >>Any details about the problems?  HPET is important to some guests.
> >>
> >Seems like HPET reaction is too slow to satisfy some guests(for it would
> >replace PIT).
> >
> >Here is the thread last time.
> >
> >http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899
> >
> 
> Thanks.  We can address this in three ways: first, adjust the guest
> not to do timing related tests when virtualized (since no matter
> what we do, the tests may fail).  Second, I think we should
> implement userspace ack notifiers (similar to tpr access notifiers
> already present).  Third, we can implement a kernel hpet, which,
> after we solve the zillion bug it introduces, will also give a nice
> performance improvement for hpet intensive workloads.
> 
Second will not solve the problem. Presence of ack notifiers will not
make HPET interrupt arrive faster.

--
			Gleb.

Re: Make QEmu HPET disabled by default for KVM?

[Avi Kivity]

On 03/11/2010 10:31 AM, Gleb Natapov wrote:
> On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote:
>    
>> On 03/11/2010 10:23 AM, Sheng Yang wrote:
>>      
>>>>> I have kept --no-hpet in my setup for
>>>>> months...
>>>>>            
>>>> Any details about the problems?  HPET is important to some guests.
>>>>
>>>>          
>>> Seems like HPET reaction is too slow to satisfy some guests(for it would
>>> replace PIT).
>>>
>>> Here is the thread last time.
>>>
>>> http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899
>>>
>>>        
>> Thanks.  We can address this in three ways: first, adjust the guest
>> not to do timing related tests when virtualized (since no matter
>> what we do, the tests may fail).  Second, I think we should
>> implement userspace ack notifiers (similar to tpr access notifiers
>> already present).  Third, we can implement a kernel hpet, which,
>> after we solve the zillion bug it introduces, will also give a nice
>> performance improvement for hpet intensive workloads.
>>
>>      
> Second will not solve the problem. Presence of ack notifiers will not
> make HPET interrupt arrive faster.
>    

But it will allow us to compensate for interrupts being coalesced, which 
may be the root of the problem.

-- 
error compiling committee.c: too many arguments to function

Re: Make QEmu HPET disabled by default for KVM?

[Sheng Yang]

On Thursday 11 March 2010 16:31:57 Gleb Natapov wrote:
> On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote:
> > On 03/11/2010 10:23 AM, Sheng Yang wrote:
> > >>>I have kept --no-hpet in my setup for
> > >>>months...
> > >>
> > >>Any details about the problems?  HPET is important to some guests.
> > >
> > >Seems like HPET reaction is too slow to satisfy some guests(for it would
> > >replace PIT).
> > >
> > >Here is the thread last time.
> > >
> > >http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899
> >
> > Thanks.  We can address this in three ways: first, adjust the guest
> > not to do timing related tests when virtualized (since no matter
> > what we do, the tests may fail).  Second, I think we should
> > implement userspace ack notifiers (similar to tpr access notifiers
> > already present).  Third, we can implement a kernel hpet, which,
> > after we solve the zillion bug it introduces, will also give a nice
> > performance improvement for hpet intensive workloads.
> 
> Second will not solve the problem. Presence of ack notifiers will not
> make HPET interrupt arrive faster.

The slow may also due to lost tick. And with the lost tick, hpet is still 
unusable...

-- 
regards
Yang, Sheng

Re: Make QEmu HPET disabled by default for KVM?

[Gleb Natapov]

On Thu, Mar 11, 2010 at 04:38:48PM +0800, Sheng Yang wrote:
> On Thursday 11 March 2010 16:31:57 Gleb Natapov wrote:
> > On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote:
> > > On 03/11/2010 10:23 AM, Sheng Yang wrote:
> > > >>>I have kept --no-hpet in my setup for
> > > >>>months...
> > > >>
> > > >>Any details about the problems?  HPET is important to some guests.
> > > >
> > > >Seems like HPET reaction is too slow to satisfy some guests(for it would
> > > >replace PIT).
> > > >
> > > >Here is the thread last time.
> > > >
> > > >http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899
> > >
> > > Thanks.  We can address this in three ways: first, adjust the guest
> > > not to do timing related tests when virtualized (since no matter
> > > what we do, the tests may fail).  Second, I think we should
> > > implement userspace ack notifiers (similar to tpr access notifiers
> > > already present).  Third, we can implement a kernel hpet, which,
> > > after we solve the zillion bug it introduces, will also give a nice
> > > performance improvement for hpet intensive workloads.
> > 
> > Second will not solve the problem. Presence of ack notifiers will not
> > make HPET interrupt arrive faster.
> 
> The slow may also due to lost tick. And with the lost tick, hpet is still 
> unusable...
> 
If the problem it due to lost ticks reinjection may solve it, but only partially.
What if IO thread haven't run even once during the time vcpu did clock
source check? IIRC sometimes we trigger this even with in kernel PIT.

--
			Gleb.

Re: Make QEmu HPET disabled by default for KVM?

[Avi Kivity]

On 03/11/2010 10:42 AM, Gleb Natapov wrote:
> On Thu, Mar 11, 2010 at 04:38:48PM +0800, Sheng Yang wrote:
>    
>> On Thursday 11 March 2010 16:31:57 Gleb Natapov wrote:
>>      
>>> On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote:
>>>        
>>>> On 03/11/2010 10:23 AM, Sheng Yang wrote:
>>>>          
>>>>>>> I have kept --no-hpet in my setup for
>>>>>>> months...
>>>>>>>                
>>>>>> Any details about the problems?  HPET is important to some guests.
>>>>>>              
>>>>> Seems like HPET reaction is too slow to satisfy some guests(for it would
>>>>> replace PIT).
>>>>>
>>>>> Here is the thread last time.
>>>>>
>>>>> http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899
>>>>>            
>>>> Thanks.  We can address this in three ways: first, adjust the guest
>>>> not to do timing related tests when virtualized (since no matter
>>>> what we do, the tests may fail).  Second, I think we should
>>>> implement userspace ack notifiers (similar to tpr access notifiers
>>>> already present).  Third, we can implement a kernel hpet, which,
>>>> after we solve the zillion bug it introduces, will also give a nice
>>>> performance improvement for hpet intensive workloads.
>>>>          
>>> Second will not solve the problem. Presence of ack notifiers will not
>>> make HPET interrupt arrive faster.
>>>        
>> The slow may also due to lost tick. And with the lost tick, hpet is still
>> unusable...
>>
>>      
> If the problem it due to lost ticks reinjection may solve it, but only partially.
> What if IO thread haven't run even once during the time vcpu did clock
> source check? IIRC sometimes we trigger this even with in kernel PIT.
>    

That is true.  Reinjection can correct problems in the long term, but 
may fail in the short term.  10 ticks is easily short term in a heavily 
loaded system.

How does it happen with kernel PIT?  I could understand it if we had a 
work item doing the injection, but everything happens either from 
hrtimer context or vcpu context.

-- 
error compiling committee.c: too many arguments to function

Re: Make QEmu HPET disabled by default for KVM?

[Gleb Natapov]

On Thu, Mar 11, 2010 at 10:46:06AM +0200, Avi Kivity wrote:
> On 03/11/2010 10:42 AM, Gleb Natapov wrote:
> >On Thu, Mar 11, 2010 at 04:38:48PM +0800, Sheng Yang wrote:
> >>On Thursday 11 March 2010 16:31:57 Gleb Natapov wrote:
> >>>On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote:
> >>>>On 03/11/2010 10:23 AM, Sheng Yang wrote:
> >>>>>>>I have kept --no-hpet in my setup for
> >>>>>>>months...
> >>>>>>Any details about the problems?  HPET is important to some guests.
> >>>>>Seems like HPET reaction is too slow to satisfy some guests(for it would
> >>>>>replace PIT).
> >>>>>
> >>>>>Here is the thread last time.
> >>>>>
> >>>>>http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899
> >>>>Thanks.  We can address this in three ways: first, adjust the guest
> >>>>not to do timing related tests when virtualized (since no matter
> >>>>what we do, the tests may fail).  Second, I think we should
> >>>>implement userspace ack notifiers (similar to tpr access notifiers
> >>>>already present).  Third, we can implement a kernel hpet, which,
> >>>>after we solve the zillion bug it introduces, will also give a nice
> >>>>performance improvement for hpet intensive workloads.
> >>>Second will not solve the problem. Presence of ack notifiers will not
> >>>make HPET interrupt arrive faster.
> >>The slow may also due to lost tick. And with the lost tick, hpet is still
> >>unusable...
> >>
> >If the problem it due to lost ticks reinjection may solve it, but only partially.
> >What if IO thread haven't run even once during the time vcpu did clock
> >source check? IIRC sometimes we trigger this even with in kernel PIT.
> 
> That is true.  Reinjection can correct problems in the long term,
> but may fail in the short term.  10 ticks is easily short term in a
> heavily loaded system.
> 
> How does it happen with kernel PIT?  I could understand it if we had
> a work item doing the injection, but everything happens either from
> hrtimer context or vcpu context.
> 
Do we kick vcpu out of guest mode when hrtimer triggers? I don't see us doing it in
__kvm_timer_fn().

--
			Gleb.

Re: Make QEmu HPET disabled by default for KVM?

[Avi Kivity]

On 03/11/2010 12:23 PM, Gleb Natapov wrote:
>>>
>>> If the problem it due to lost ticks reinjection may solve it, but only partially.
>>> What if IO thread haven't run even once during the time vcpu did clock
>>> source check? IIRC sometimes we trigger this even with in kernel PIT.
>>>        
>> That is true.  Reinjection can correct problems in the long term,
>> but may fail in the short term.  10 ticks is easily short term in a
>> heavily loaded system.
>>
>> How does it happen with kernel PIT?  I could understand it if we had
>> a work item doing the injection, but everything happens either from
>> hrtimer context or vcpu context.
>>
>>      
> Do we kick vcpu out of guest mode when hrtimer triggers? I don't see us doing it in
> __kvm_timer_fn().
>
>    

We're always running on the same cpu as vcpu 0, so no need.

-- 
error compiling committee.c: too many arguments to function

Re: Make QEmu HPET disabled by default for KVM?

[Avi Kivity]

On 03/11/2010 01:56 PM, Avi Kivity wrote:
> On 03/11/2010 12:23 PM, Gleb Natapov wrote:
>>>>
>>>> If the problem it due to lost ticks reinjection may solve it, but 
>>>> only partially.
>>>> What if IO thread haven't run even once during the time vcpu did clock
>>>> source check? IIRC sometimes we trigger this even with in kernel PIT.
>>> That is true.  Reinjection can correct problems in the long term,
>>> but may fail in the short term.  10 ticks is easily short term in a
>>> heavily loaded system.
>>>
>>> How does it happen with kernel PIT?  I could understand it if we had
>>> a work item doing the injection, but everything happens either from
>>> hrtimer context or vcpu context.
>>>
>> Do we kick vcpu out of guest mode when hrtimer triggers? I don't see 
>> us doing it in
>> __kvm_timer_fn().
>>
>
> We're always running on the same cpu as vcpu 0, so no need.
>

Would be better to do it, though, in case we have migration races.

-- 
error compiling committee.c: too many arguments to function

Re: Make QEmu HPET disabled by default for KVM?

[Marcelo Tosatti]

On Thu, Mar 11, 2010 at 09:58:12AM +0200, Avi Kivity wrote:
> On 03/11/2010 09:52 AM, Sheng Yang wrote:
> >I think we have already suffered enough timer issues due to this(e.g. I can't
> >boot up well on 2.6.18 kernel)...
> 
> 2.6.18 as guest or as host?
> 
> >I have kept --no-hpet in my setup for
> >months...
> 
> Any details about the problems?  HPET is important to some guests.

As Gleb mentioned in the other thread, reinjection will introduce
another set of problems.

Ideally all this timer related problems should be fixed by correlating
timer interrupts and time source reads.

Since one already has to use special timer parameters (-rtc-td-hack,
-no-kvm-pit-reinjection), using -no-hpet for problematic Linux 
guests seems fine?

Re: Make QEmu HPET disabled by default for KVM?

[Avi Kivity]

On 03/11/2010 09:08 PM, Marcelo Tosatti wrote:
>
>>
>>> I have kept --no-hpet in my setup for
>>> months...
>>>        
>> Any details about the problems?  HPET is important to some guests.
>>      
> As Gleb mentioned in the other thread, reinjection will introduce
> another set of problems.
>
> Ideally all this timer related problems should be fixed by correlating
> timer interrupts and time source reads.
>    

This still needs reinjection (or slewing of the timer frequency).  
Correlation doesn't fix drift.

> Since one already has to use special timer parameters (-rtc-td-hack,
> -no-kvm-pit-reinjection), using -no-hpet for problematic Linux
> guests seems fine?
>    

Depends on how common the problematic ones are.  If they're common, 
better to have a generic fix.

-- 
error compiling committee.c: too many arguments to function

Re: Make QEmu HPET disabled by default for KVM?

[Gleb Natapov]

On Sun, Mar 14, 2010 at 09:05:50AM +0200, Avi Kivity wrote:
> On 03/11/2010 09:08 PM, Marcelo Tosatti wrote:
> >
> >>
> >>>I have kept --no-hpet in my setup for
> >>>months...
> >>Any details about the problems?  HPET is important to some guests.
> >As Gleb mentioned in the other thread, reinjection will introduce
> >another set of problems.
> >
> >Ideally all this timer related problems should be fixed by correlating
> >timer interrupts and time source reads.
> 
> This still needs reinjection (or slewing of the timer frequency).
> Correlation doesn't fix drift.
> 
But only when all time sources are synchronised and correlated with
interrupts we can slew time frequency without guest noticing (and only
if guest disables NTP)

> >Since one already has to use special timer parameters (-rtc-td-hack,
> >-no-kvm-pit-reinjection), using -no-hpet for problematic Linux
> >guests seems fine?
> 
> Depends on how common the problematic ones are.  If they're common,
> better to have a generic fix.
> 
> -- 
> error compiling committee.c: too many arguments to function

--
			Gleb.

Re: Make QEmu HPET disabled by default for KVM?

[Dor Laor]

On 03/14/2010 09:10 AM, Gleb Natapov wrote:
> On Sun, Mar 14, 2010 at 09:05:50AM +0200, Avi Kivity wrote:
>> On 03/11/2010 09:08 PM, Marcelo Tosatti wrote:
>>>
>>>>
>>>>> I have kept --no-hpet in my setup for
>>>>> months...
>>>> Any details about the problems?  HPET is important to some guests.
>>> As Gleb mentioned in the other thread, reinjection will introduce
>>> another set of problems.
>>>
>>> Ideally all this timer related problems should be fixed by correlating
>>> timer interrupts and time source reads.
>>
>> This still needs reinjection (or slewing of the timer frequency).
>> Correlation doesn't fix drift.
>>
> But only when all time sources are synchronised and correlated with
> interrupts we can slew time frequency without guest noticing (and only
> if guest disables NTP)

In the mean time we should definitely disable hpet by default.
Besides this we need to fully virtualize the tsc, fix win7 64bit rtc 
time drift and some pvclock potential issues. Before we add new timer, 
better fix existing ones.

What about creating a pv time keeping device that will be aware of lost 
ticks and host wall clock time? It's similar to hyper-v enlightenment 
virt timers.

>
>>> Since one already has to use special timer parameters (-rtc-td-hack,
>>> -no-kvm-pit-reinjection), using -no-hpet for problematic Linux
>>> guests seems fine?
>>
>> Depends on how common the problematic ones are.  If they're common,
>> better to have a generic fix.
>>
>> --
>> error compiling committee.c: too many arguments to function
>
> --
> 			Gleb.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Make QEmu HPET disabled by default for KVM?

[Avi Kivity]

On 03/14/2010 12:23 PM, Dor Laor wrote:
> On 03/14/2010 09:10 AM, Gleb Natapov wrote:
>> On Sun, Mar 14, 2010 at 09:05:50AM +0200, Avi Kivity wrote:
>>> On 03/11/2010 09:08 PM, Marcelo Tosatti wrote:
>>>>
>>>>>
>>>>>> I have kept --no-hpet in my setup for
>>>>>> months...
>>>>> Any details about the problems?  HPET is important to some guests.
>>>> As Gleb mentioned in the other thread, reinjection will introduce
>>>> another set of problems.
>>>>
>>>> Ideally all this timer related problems should be fixed by correlating
>>>> timer interrupts and time source reads.
>>>
>>> This still needs reinjection (or slewing of the timer frequency).
>>> Correlation doesn't fix drift.
>>>
>> But only when all time sources are synchronised and correlated with
>> interrupts we can slew time frequency without guest noticing (and only
>> if guest disables NTP)
>
> In the mean time we should definitely disable hpet by default.

Definitely not.  Windows needs it.  Some pre-kvmclock Linux may also 
work with it.

Without hpet, there is no fast high resolution timer in the system.

> Besides this we need to fully virtualize the tsc, fix win7 64bit rtc 
> time drift and some pvclock potential issues. Before we add new timer, 
> better fix existing ones.
>
> What about creating a pv time keeping device that will be aware of 
> lost ticks and host wall clock time? It's similar to hyper-v 
> enlightenment virt timers.

That's kvmclock.

-- 
error compiling committee.c: too many arguments to function

Re: Make QEmu HPET disabled by default for KVM?

[Dor Laor]

On 03/14/2010 12:27 PM, Avi Kivity wrote:
> On 03/14/2010 12:23 PM, Dor Laor wrote:
>> On 03/14/2010 09:10 AM, Gleb Natapov wrote:
>>> On Sun, Mar 14, 2010 at 09:05:50AM +0200, Avi Kivity wrote:
>>>> On 03/11/2010 09:08 PM, Marcelo Tosatti wrote:
>>>>>
>>>>>>
>>>>>>> I have kept --no-hpet in my setup for
>>>>>>> months...
>>>>>> Any details about the problems? HPET is important to some guests.
>>>>> As Gleb mentioned in the other thread, reinjection will introduce
>>>>> another set of problems.
>>>>>
>>>>> Ideally all this timer related problems should be fixed by correlating
>>>>> timer interrupts and time source reads.
>>>>
>>>> This still needs reinjection (or slewing of the timer frequency).
>>>> Correlation doesn't fix drift.
>>>>
>>> But only when all time sources are synchronised and correlated with
>>> interrupts we can slew time frequency without guest noticing (and only
>>> if guest disables NTP)
>>
>> In the mean time we should definitely disable hpet by default.
>
> Definitely not. Windows needs it. Some pre-kvmclock Linux may also work
> with it.
>
> Without hpet, there is no fast high resolution timer in the system.

It's all depends on how hard would it be to re-inject to windows guest.
We still need to fix the win2k3 64 bit and win2k8 64 bit (and not win7 
as I told initially) since the irq is broadcasted to all the vcpus and 
we do not track who acknowledged the irq.

>
>> Besides this we need to fully virtualize the tsc, fix win7 64bit rtc
>> time drift and some pvclock potential issues. Before we add new timer,
>> better fix existing ones.
>>
>> What about creating a pv time keeping device that will be aware of
>> lost ticks and host wall clock time? It's similar to hyper-v
>> enlightenment virt timers.
>
> That's kvmclock.
>

I meant a device that can be used to generate timeouts. We do use today 
pit/rtc along with kvmclock time source but it's not perfect and 
probably the same for hpet. This is why I tough that a pv device will be 
beneficial.