[PATCH] BUG in pv_clock when overflow condition is detected

[PATCH] BUG in pv_clock when overflow condition is detected

[Igor Mammedov]

BUG when overflow occurs at pvclock.c:pvclock_get_nsec_offset

    u64 delta = native_read_tsc() - shadow->tsc_timestamp;

this might happen at an attempt to read an uninitialized yet clock.
It won't prevent stalls and hangs but at least it won't do it silently.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 arch/x86/kernel/pvclock.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/pvclock.c b/arch/x86/kernel/pvclock.c
index 42eb330..35a6190 100644
--- a/arch/x86/kernel/pvclock.c
+++ b/arch/x86/kernel/pvclock.c
@@ -43,7 +43,10 @@ void pvclock_set_flags(u8 flags)
 
 static u64 pvclock_get_nsec_offset(struct pvclock_shadow_time *shadow)
 {
-	u64 delta = native_read_tsc() - shadow->tsc_timestamp;
+	u64 delta;
+	u64 tsc = native_read_tsc();
+	BUG_ON(tsc < shadow->tsc_timestamp);
+	delta = tsc - shadow->tsc_timestamp;
 	return pvclock_scale_delta(delta, shadow->tsc_to_nsec_mul,
 				   shadow->tsc_shift);
 }
-- 
1.7.7.6

Re: [PATCH] BUG in pv_clock when overflow condition is detected

[Avi Kivity]

On 02/13/2012 08:20 PM, Igor Mammedov wrote:
> BUG when overflow occurs at pvclock.c:pvclock_get_nsec_offset
>
>     u64 delta = native_read_tsc() - shadow->tsc_timestamp;
>
> this might happen at an attempt to read an uninitialized yet clock.
> It won't prevent stalls and hangs but at least it won't do it silently.
>
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  arch/x86/kernel/pvclock.c |    5 ++++-
>  1 files changed, 4 insertions(+), 1 deletions(-)
>
> diff --git a/arch/x86/kernel/pvclock.c b/arch/x86/kernel/pvclock.c
> index 42eb330..35a6190 100644
> --- a/arch/x86/kernel/pvclock.c
> +++ b/arch/x86/kernel/pvclock.c
> @@ -43,7 +43,10 @@ void pvclock_set_flags(u8 flags)
>  
>  static u64 pvclock_get_nsec_offset(struct pvclock_shadow_time *shadow)
>  {
> -	u64 delta = native_read_tsc() - shadow->tsc_timestamp;
> +	u64 delta;
> +	u64 tsc = native_read_tsc();
> +	BUG_ON(tsc < shadow->tsc_timestamp);
> +	delta = tsc - shadow->tsc_timestamp;
>  	return pvclock_scale_delta(delta, shadow->tsc_to_nsec_mul,
>  				   shadow->tsc_shift);

Maybe a WARN_ON_ONCE()?  Otherwise a relatively minor hypervisor bug can
kill the guest.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH] BUG in pv_clock when overflow condition is detected

[Igor Mammedov]

On 02/15/2012 11:49 AM, Avi Kivity wrote:
> On 02/13/2012 08:20 PM, Igor Mammedov wrote:
>> BUG when overflow occurs at pvclock.c:pvclock_get_nsec_offset
>>
>>      u64 delta = native_read_tsc() - shadow->tsc_timestamp;
>>
>> this might happen at an attempt to read an uninitialized yet clock.
>> It won't prevent stalls and hangs but at least it won't do it silently.
>>
>> Signed-off-by: Igor Mammedov<imammedo@redhat.com>
>> ---
>>   arch/x86/kernel/pvclock.c |    5 ++++-
>>   1 files changed, 4 insertions(+), 1 deletions(-)
>>
>> diff --git a/arch/x86/kernel/pvclock.c b/arch/x86/kernel/pvclock.c
>> index 42eb330..35a6190 100644
>> --- a/arch/x86/kernel/pvclock.c
>> +++ b/arch/x86/kernel/pvclock.c
>> @@ -43,7 +43,10 @@ void pvclock_set_flags(u8 flags)
>>
>>   static u64 pvclock_get_nsec_offset(struct pvclock_shadow_time *shadow)
>>   {
>> -	u64 delta = native_read_tsc() - shadow->tsc_timestamp;
>> +	u64 delta;
>> +	u64 tsc = native_read_tsc();
>> +	BUG_ON(tsc<  shadow->tsc_timestamp);
>> +	delta = tsc - shadow->tsc_timestamp;
>>   	return pvclock_scale_delta(delta, shadow->tsc_to_nsec_mul,
>>   				   shadow->tsc_shift);
>
> Maybe a WARN_ON_ONCE()?  Otherwise a relatively minor hypervisor bug can
> kill the guest.

An attempt to print from this place is not perfect since it often leads
to recursive calling to this very function and it hang there anyway.
But if you insist I'll re-post it with WARN_ON_ONCE,
It won't make much difference because guest will hang/stall due overflow
anyway.

If there is an intention to keep guest functional after the event then
maybe this patch is a way to go
   http://www.spinics.net/lists/kvm/msg68463.html
this way clock will be re-silent to this kind of errors, like bare-metal
one is.

-- 
Thanks,
  Igor

Re: [PATCH] BUG in pv_clock when overflow condition is detected

[Avi Kivity]

On 02/15/2012 01:23 PM, Igor Mammedov wrote:
>>>   static u64 pvclock_get_nsec_offset(struct pvclock_shadow_time
>>> *shadow)
>>>   {
>>> -    u64 delta = native_read_tsc() - shadow->tsc_timestamp;
>>> +    u64 delta;
>>> +    u64 tsc = native_read_tsc();
>>> +    BUG_ON(tsc<  shadow->tsc_timestamp);
>>> +    delta = tsc - shadow->tsc_timestamp;
>>>       return pvclock_scale_delta(delta, shadow->tsc_to_nsec_mul,
>>>                      shadow->tsc_shift);
>>
>> Maybe a WARN_ON_ONCE()?  Otherwise a relatively minor hypervisor bug can
>> kill the guest.
>
>
> An attempt to print from this place is not perfect since it often leads
> to recursive calling to this very function and it hang there anyway.
> But if you insist I'll re-post it with WARN_ON_ONCE,
> It won't make much difference because guest will hang/stall due overflow
> anyway.

Won't a BUG_ON() also result in a printk?

>
> If there is an intention to keep guest functional after the event then
> maybe this patch is a way to go
>   http://www.spinics.net/lists/kvm/msg68463.html
> this way clock will be re-silent to this kind of errors, like bare-metal
> one is.

It's the same patch... do you mean something that detects the overflow
and uses the last value?

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH] BUG in pv_clock when overflow condition is detected

[Igor Mammedov]

----- Original Message -----
> From: "Avi Kivity" <avi@redhat.com>
> To: "Igor Mammedov" <imammedo@redhat.com>
> Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, tglx@linutronix.de, mingo@redhat.com, hpa@zytor.com,
> riel@redhat.com, "amit shah" <amit.shah@redhat.com>, mtosatti@redhat.com
> Sent: Wednesday, February 15, 2012 3:02:04 PM
> Subject: Re: [PATCH] BUG in pv_clock when overflow condition is detected
> 
> On 02/15/2012 01:23 PM, Igor Mammedov wrote:
> >>>   static u64 pvclock_get_nsec_offset(struct pvclock_shadow_time
> >>> *shadow)
> >>>   {
> >>> -    u64 delta = native_read_tsc() - shadow->tsc_timestamp;
> >>> +    u64 delta;
> >>> +    u64 tsc = native_read_tsc();
> >>> +    BUG_ON(tsc<  shadow->tsc_timestamp);
> >>> +    delta = tsc - shadow->tsc_timestamp;
> >>>       return pvclock_scale_delta(delta, shadow->tsc_to_nsec_mul,
> >>>                      shadow->tsc_shift);
> >>
> >> Maybe a WARN_ON_ONCE()?  Otherwise a relatively minor hypervisor
> >> bug can
> >> kill the guest.
> >
> >
> > An attempt to print from this place is not perfect since it often
> > leads
> > to recursive calling to this very function and it hang there
> > anyway.
> > But if you insist I'll re-post it with WARN_ON_ONCE,
> > It won't make much difference because guest will hang/stall due
> > overflow
> > anyway.
> 
> Won't a BUG_ON() also result in a printk?
Yes, it will. But stack will still keep failure point and poking
with crash/gdb at core will always show where it's BUGged.

In case it manages to print dump somehow (saw it couple times from ~
30 test cycles), logs from console or from kernel message buffer
(again poking with gdb) will show where it was called from.

If WARN* is used, it will still totaly screwup clock and 
"last value" and system will become unusable, requiring looking with
gdb/crash at the core any way.

So I've just used more stable failure point that will leave trace
everywhere it manages (maybe in console log, but for sure in stack)
in case of WARN it might leave trace on console or not and probably
won't reflect failure point in stack either leaving only kernel
message buffer for clue.

> 
> >
> > If there is an intention to keep guest functional after the event
> > then
> > maybe this patch is a way to go
> >   http://www.spinics.net/lists/kvm/msg68463.html
> > this way clock will be re-silent to this kind of errors, like
> > bare-metal
> > one is.
> 
> It's the same patch... do you mean something that detects the
> overflow
> and uses the last value?
I'm sorry, pasted wrong link
here it goes: 
  "pvclock: Make pv_clock more robust and fixup it if overflow happens"
     http://www.spinics.net/lists/kvm/msg68440.html

> 
> --
> error compiling committee.c: too many arguments to function
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] BUG in pv_clock when overflow condition is detected

[Avi Kivity]

On 02/15/2012 07:18 PM, Igor Mammedov wrote:
> > On 02/15/2012 01:23 PM, Igor Mammedov wrote:
> > >>>   static u64 pvclock_get_nsec_offset(struct pvclock_shadow_time
> > >>> *shadow)
> > >>>   {
> > >>> -    u64 delta = native_read_tsc() - shadow->tsc_timestamp;
> > >>> +    u64 delta;
> > >>> +    u64 tsc = native_read_tsc();
> > >>> +    BUG_ON(tsc<  shadow->tsc_timestamp);
> > >>> +    delta = tsc - shadow->tsc_timestamp;
> > >>>       return pvclock_scale_delta(delta, shadow->tsc_to_nsec_mul,
> > >>>                      shadow->tsc_shift);
> > >>
> > >> Maybe a WARN_ON_ONCE()?  Otherwise a relatively minor hypervisor
> > >> bug can
> > >> kill the guest.
> > >
> > >
> > > An attempt to print from this place is not perfect since it often
> > > leads
> > > to recursive calling to this very function and it hang there
> > > anyway.
> > > But if you insist I'll re-post it with WARN_ON_ONCE,
> > > It won't make much difference because guest will hang/stall due
> > > overflow
> > > anyway.
> > 
> > Won't a BUG_ON() also result in a printk?
> Yes, it will. But stack will still keep failure point and poking
> with crash/gdb at core will always show where it's BUGged.
>
> In case it manages to print dump somehow (saw it couple times from ~
> 30 test cycles), logs from console or from kernel message buffer
> (again poking with gdb) will show where it was called from.
>
> If WARN* is used, it will still totaly screwup clock and 
> "last value" and system will become unusable, requiring looking with
> gdb/crash at the core any way.
>
> So I've just used more stable failure point that will leave trace
> everywhere it manages (maybe in console log, but for sure in stack)
> in case of WARN it might leave trace on console or not and probably
> won't reflect failure point in stack either leaving only kernel
> message buffer for clue.
>

Makes sense.  But do get an ack from the Xen people to ensure this
doesn't break for them.


-- 
error compiling committee.c: too many arguments to function

Re: [PATCH] BUG in pv_clock when overflow condition is detected

[Igor Mammedov]

On 02/16/2012 03:03 PM, Avi Kivity wrote:
> On 02/15/2012 07:18 PM, Igor Mammedov wrote:
>>> On 02/15/2012 01:23 PM, Igor Mammedov wrote:
>>>>>>    static u64 pvclock_get_nsec_offset(struct pvclock_shadow_time
>>>>>> *shadow)
>>>>>>    {
>>>>>> -    u64 delta = native_read_tsc() - shadow->tsc_timestamp;
>>>>>> +    u64 delta;
>>>>>> +    u64 tsc = native_read_tsc();
>>>>>> +    BUG_ON(tsc<   shadow->tsc_timestamp);
>>>>>> +    delta = tsc - shadow->tsc_timestamp;
>>>>>>        return pvclock_scale_delta(delta, shadow->tsc_to_nsec_mul,
>>>>>>                       shadow->tsc_shift);
>>>>>
>>>>> Maybe a WARN_ON_ONCE()?  Otherwise a relatively minor hypervisor
>>>>> bug can
>>>>> kill the guest.
>>>>
>>>>
>>>> An attempt to print from this place is not perfect since it often
>>>> leads
>>>> to recursive calling to this very function and it hang there
>>>> anyway.
>>>> But if you insist I'll re-post it with WARN_ON_ONCE,
>>>> It won't make much difference because guest will hang/stall due
>>>> overflow
>>>> anyway.
>>>
>>> Won't a BUG_ON() also result in a printk?
>> Yes, it will. But stack will still keep failure point and poking
>> with crash/gdb at core will always show where it's BUGged.
>>
>> In case it manages to print dump somehow (saw it couple times from ~
>> 30 test cycles), logs from console or from kernel message buffer
>> (again poking with gdb) will show where it was called from.
>>
>> If WARN* is used, it will still totaly screwup clock and
>> "last value" and system will become unusable, requiring looking with
>> gdb/crash at the core any way.
>>
>> So I've just used more stable failure point that will leave trace
>> everywhere it manages (maybe in console log, but for sure in stack)
>> in case of WARN it might leave trace on console or not and probably
>> won't reflect failure point in stack either leaving only kernel
>> message buffer for clue.
>>
>
> Makes sense.  But do get an ack from the Xen people to ensure this
> doesn't break for them.
>
Konrad, Ian

Could you please review patch form point of view of xen?
Whole thread could be found here https://lkml.org/lkml/2012/2/13/286

-- 
Thanks,
  Igor

Re: [PATCH] BUG in pv_clock when overflow condition is detected

[Konrad Rzeszutek Wilk]

On Fri, Feb 17, 2012 at 04:25:04PM +0100, Igor Mammedov wrote:
> On 02/16/2012 03:03 PM, Avi Kivity wrote:
> >On 02/15/2012 07:18 PM, Igor Mammedov wrote:
> >>>On 02/15/2012 01:23 PM, Igor Mammedov wrote:
> >>>>>>   static u64 pvclock_get_nsec_offset(struct pvclock_shadow_time
> >>>>>>*shadow)
> >>>>>>   {
> >>>>>>-    u64 delta = native_read_tsc() - shadow->tsc_timestamp;
> >>>>>>+    u64 delta;
> >>>>>>+    u64 tsc = native_read_tsc();
> >>>>>>+    BUG_ON(tsc<   shadow->tsc_timestamp);
> >>>>>>+    delta = tsc - shadow->tsc_timestamp;
> >>>>>>       return pvclock_scale_delta(delta, shadow->tsc_to_nsec_mul,
> >>>>>>                      shadow->tsc_shift);
> >>>>>
> >>>>>Maybe a WARN_ON_ONCE()?  Otherwise a relatively minor hypervisor
> >>>>>bug can
> >>>>>kill the guest.
> >>>>
> >>>>
> >>>>An attempt to print from this place is not perfect since it often
> >>>>leads
> >>>>to recursive calling to this very function and it hang there
> >>>>anyway.
> >>>>But if you insist I'll re-post it with WARN_ON_ONCE,
> >>>>It won't make much difference because guest will hang/stall due
> >>>>overflow
> >>>>anyway.
> >>>
> >>>Won't a BUG_ON() also result in a printk?
> >>Yes, it will. But stack will still keep failure point and poking
> >>with crash/gdb at core will always show where it's BUGged.
> >>
> >>In case it manages to print dump somehow (saw it couple times from ~
> >>30 test cycles), logs from console or from kernel message buffer
> >>(again poking with gdb) will show where it was called from.
> >>
> >>If WARN* is used, it will still totaly screwup clock and
> >>"last value" and system will become unusable, requiring looking with
> >>gdb/crash at the core any way.
> >>
> >>So I've just used more stable failure point that will leave trace
> >>everywhere it manages (maybe in console log, but for sure in stack)
> >>in case of WARN it might leave trace on console or not and probably
> >>won't reflect failure point in stack either leaving only kernel
> >>message buffer for clue.
> >>
> >
> >Makes sense.  But do get an ack from the Xen people to ensure this
> >doesn't break for them.
> >
> Konrad, Ian
> 
> Could you please review patch form point of view of xen?
> Whole thread could be found here https://lkml.org/lkml/2012/2/13/286

What are the conditions under which this happens? You should probably
include that in the git description as well? Is this something that happens
often? If there is an overflow can you synthesize a value instead of
crashing the guest?

Hm, so are you asking for review for this patch or for
http://www.spinics.net/lists/kvm/msg68440.html ?

(which would also entail a early_percpu_clock_init implementation
in the Xen code naturally).

Re: [PATCH] BUG in pv_clock when overflow condition is detected

[Igor Mammedov]

On 02/20/2012 04:28 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Feb 17, 2012 at 04:25:04PM +0100, Igor Mammedov wrote:
>> On 02/16/2012 03:03 PM, Avi Kivity wrote:
>>> On 02/15/2012 07:18 PM, Igor Mammedov wrote:
>>>>> On 02/15/2012 01:23 PM, Igor Mammedov wrote:
>>>>>>>>    static u64 pvclock_get_nsec_offset(struct pvclock_shadow_time
>>>>>>>> *shadow)
>>>>>>>>    {
>>>>>>>> -    u64 delta = native_read_tsc() - shadow->tsc_timestamp;
>>>>>>>> +    u64 delta;
>>>>>>>> +    u64 tsc = native_read_tsc();
>>>>>>>> +    BUG_ON(tsc<    shadow->tsc_timestamp);
>>>>>>>> +    delta = tsc - shadow->tsc_timestamp;
>>>>>>>>        return pvclock_scale_delta(delta, shadow->tsc_to_nsec_mul,
>>>>>>>>                       shadow->tsc_shift);
>>>>>>>
>>>>>>> Maybe a WARN_ON_ONCE()?  Otherwise a relatively minor hypervisor
>>>>>>> bug can
>>>>>>> kill the guest.
>>>>>>
>>>>>>
>>>>>> An attempt to print from this place is not perfect since it often
>>>>>> leads
>>>>>> to recursive calling to this very function and it hang there
>>>>>> anyway.
>>>>>> But if you insist I'll re-post it with WARN_ON_ONCE,
>>>>>> It won't make much difference because guest will hang/stall due
>>>>>> overflow
>>>>>> anyway.
>>>>>
>>>>> Won't a BUG_ON() also result in a printk?
>>>> Yes, it will. But stack will still keep failure point and poking
>>>> with crash/gdb at core will always show where it's BUGged.
>>>>
>>>> In case it manages to print dump somehow (saw it couple times from ~
>>>> 30 test cycles), logs from console or from kernel message buffer
>>>> (again poking with gdb) will show where it was called from.
>>>>
>>>> If WARN* is used, it will still totaly screwup clock and
>>>> "last value" and system will become unusable, requiring looking with
>>>> gdb/crash at the core any way.
>>>>
>>>> So I've just used more stable failure point that will leave trace
>>>> everywhere it manages (maybe in console log, but for sure in stack)
>>>> in case of WARN it might leave trace on console or not and probably
>>>> won't reflect failure point in stack either leaving only kernel
>>>> message buffer for clue.
>>>>
>>>
>>> Makes sense.  But do get an ack from the Xen people to ensure this
>>> doesn't break for them.
>>>
>> Konrad, Ian
>>
>> Could you please review patch form point of view of xen?
>> Whole thread could be found here https://lkml.org/lkml/2012/2/13/286
>
> What are the conditions under which this happens?
> You should probably include that in the git description as well?
This happens on cpu hot-plug in kvm guest:
https://lkml.org/lkml/2012/2/7/222

It probably doesn't affect xen pv guest but issue might affect hvm one.
I'm certainly not xen expert to say it for sure after a cursory look
at the code. If you can confirm that it affects xen hvm I will write
early_percpu_clock_init patch for it as well.

> Is this something that happens often?
Very seldom and unlikely.

> Hm, so are you asking for review for this patch
I was asking for review of subj patch
   "BUG in pv_clock when overflow condition is detected"
I'll update patch description and re-spin it.

>  If there is an overflow can you synthesize a value instead of
> crashing the guest?
> or for http://www.spinics.net/lists/kvm/msg68440.html ?
Probably could, but there was argument that it is fixing the symptoms
and not the root cause. It seems that you've already found patch that
proposes this "pvclock: Make pv_clock more robust and fixup it if overflow happens"

>
> (which would also entail a early_percpu_clock_init implementation
> in the Xen code naturally).
>

-- 
Thanks,
  Igor