soft lockup after stop/cont

soft lockup after stop/cont

[Jim Paris]

If I stop KVM in the monitor with "stop", wait a minute, and do
"cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0".
Is that expected behavior?  What isn't virtualized that allows it to
detect that?  The host is a core 2 duo.

I have bigger problems in the guest after migrating to a file and
restoring from that, but this seems like a good place to start..

-jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Avi Kivity]

Jim Paris wrote:
> If I stop KVM in the monitor with "stop", wait a minute, and do
> "cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0".
> Is that expected behavior?  

No.

> What isn't virtualized that allows it to
> detect that?  The host is a core 2 duo.
>
>   

It may be that the timer correction code detects that zillions of timer
interrupts have not been serviced by the guest so it floods the guest
with these interrupts.  Does -no-kvm-irqchip help?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Dong, Eddie]

kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
> 
> It may be that the timer correction code detects that zillions of
> timer interrupts have not been serviced by the guest so it floods the
> guest with these interrupts.  Does -no-kvm-irqchip help?
> 
In Xen, we decide to froze the guest time after save/restore. In this
way the guest see contiguous timer and thus avoid this issue.

Probably we should do similar here, comments?
thx,eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Avi Kivity]

Dong, Eddie wrote:
> kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
>   
>> It may be that the timer correction code detects that zillions of
>> timer interrupts have not been serviced by the guest so it floods the
>> guest with these interrupts.  Does -no-kvm-irqchip help?
>>
>>     
> In Xen, we decide to froze the guest time after save/restore. In this
> way the guest see contiguous timer and thus avoid this issue.
>
> Probably we should do similar here, comments?
>   

This is pause/resume, not save/restore.  I think save/restore now has
contiguous lapic timer and discontiguous real time.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Dong, Eddie]

Avi Kivity wrote:
> Dong, Eddie wrote:
>> kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
>> 
>>> It may be that the timer correction code detects that zillions of
>>> timer interrupts have not been serviced by the guest so it floods
>>> the guest with these interrupts.  Does -no-kvm-irqchip help?
>>> 
>>> 
>> In Xen, we decide to froze the guest time after save/restore. In this
>> way the guest see contiguous timer and thus avoid this issue.
>> 
>> Probably we should do similar here, comments?
>> 
> 
> This is pause/resume, not save/restore.  I think save/restore now has
> contiguous lapic timer and discontiguous real time.
> 

If save/restore means live migration here, yes you are right.
save to file and later resume from file will have this issue.
thx,eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Avi Kivity]

Dong, Eddie wrote:
> Avi Kivity wrote:
>   
>> Dong, Eddie wrote:
>>     
>>> kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
>>>
>>>       
>>>> It may be that the timer correction code detects that zillions of
>>>> timer interrupts have not been serviced by the guest so it floods
>>>> the guest with these interrupts.  Does -no-kvm-irqchip help?
>>>>
>>>>
>>>>         
>>> In Xen, we decide to froze the guest time after save/restore. In this
>>> way the guest see contiguous timer and thus avoid this issue.
>>>
>>> Probably we should do similar here, comments?
>>>
>>>       
>> This is pause/resume, not save/restore.  I think save/restore now has
>> contiguous lapic timer and discontiguous real time.
>>
>>     
>
> If save/restore means live migration here, yes you are right.
>   

No, I didn't mean that.

> save to file and later resume from file will have this issue.
>   

Why? after saving to a file, the local apic in the kernel is destroyed
when qemu quits.

Or is the timer saved in absolute time?  if so you are right, and yes,
your solution is needed.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Dong, Eddie]

>Or is the timer saved in absolute time?  if so you are right, and yes,
>your solution is needed.
>
Looks like current live migration, also save/restore, doesn't migrate 
guest time. (do I miss something?) So the new guest will 
see a totally different TSC and OS feel stranger or 
many lost ticks etc.

Should we add this one?
thx,eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Avi Kivity]

Dong, Eddie wrote:
>> Or is the timer saved in absolute time?  if so you are right, and yes,
>> your solution is needed.
>>
>>     
> Looks like current live migration, also save/restore, doesn't migrate 
> guest time. (do I miss something?) So the new guest will 
> see a totally different TSC and OS feel stranger or 
> many lost ticks etc.
>
> Should we add this one?
> thx,eddie
>   

cpu_save (qemu/hw/pc.c) has this:

#ifdef USE_KVM
    if (kvm_allowed) {
        for (i = 0; i < NR_IRQ_WORDS ; i++) {
            qemu_put_be32s(f, &env->kvm_interrupt_bitmap[i]);
        }
        qemu_put_be64s(f, &env->tsc);
    }
#endif

And I think the apic code saves the timer?

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Dong, Eddie]

Avi Kivity wrote:
> cpu_save (qemu/hw/pc.c) has this:
> 
> #ifdef USE_KVM
>    if (kvm_allowed) {
>        for (i = 0; i < NR_IRQ_WORDS ; i++) {
>            qemu_put_be32s(f, &env->kvm_interrupt_bitmap[i]);        }
>        qemu_put_be64s(f, &env->tsc);
>    }
> #endif

Mmm, so this is not the root cause.

> 
> And I think the apic code saves the timer?
> 

APIC timer is restored at
kvm_vcpu_ioctl_set_lapic/kvm_apic_post_state_restore/start_apic_timer.

Thx, Eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Jim Paris]

Avi Kivity wrote:
> Jim Paris wrote:
> > If I stop KVM in the monitor with "stop", wait a minute, and do
> > "cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0".
> > Is that expected behavior?  
> 
> No.
> 
> > What isn't virtualized that allows it to
> > detect that?  The host is a core 2 duo.
> 
> It may be that the timer correction code detects that zillions of timer
> interrupts have not been serviced by the guest so it floods the guest
> with these interrupts.  Does -no-kvm-irqchip help?

I can't seem to reproduce the "soft lockup" at the moment, but there's
a "clocksource tsc unstable" that happens pretty reliably when I pause
the VM using stop/cont for about 15 seconds:

$ kvm -M pc -m 256 -smp 1 -boot c -hda test -net none

  stop/cont, guest reports:
    Clocksource tsc unstable (delta = 15877929568 ns)
    Time: acpi_pm clocksource has been installed.

$ kvm -no-kvm-irqchip -M pc -m 256 -smp 1 -boot c -hda test -net none

  stop/cont, guest seems not to notice.

(Host is 2.6.20.4, kvm-45 modules & userspace, guest 2.6.21)

-jim

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Carsten Otte]

Jim Paris wrote:
> If I stop KVM in the monitor with "stop", wait a minute, and do
> "cont", a Linux guest gives me a "BUG: soft lockup detected on CPU#0".
> Is that expected behavior?  
We have the same behavior on s390 when running in a virtual 
environment. The issue is, that the guest physical cpu may be 
scheduled away while a timer runs down. I don't think there's anything 
kvm could do better regerding the soft lockup code. Soft lockup is 
just broken for virtualized environments and needs to be fixed. For 
the time being, just disable it in kconfig.




-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Dong, Eddie]

kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
> Jim Paris wrote:
>> If I stop KVM in the monitor with "stop", wait a minute, and do
>> "cont", a Linux guest gives me a "BUG: soft lockup detected on
>> CPU#0". Is that expected behavior?
> We have the same behavior on s390 when running in a virtual
> environment. The issue is, that the guest physical cpu may be
> scheduled away while a timer runs down. I don't think there's anything
> kvm could do better regerding the soft lockup code. Soft lockup is
> just broken for virtualized environments and needs to be fixed. For
> the time being, just disable it in kconfig.
> 
This is a well known issue, but we can fix by syncing TSC with PIT etc.
But doing this we need to froze guest TSC when a VP is descheduled.

We enable this in Xen but feel it is too complicated so never wanna
to do same thing in KVM per previous discussion. There are some
side effect of course due to this.

In Xen some time you will see PV domain soft lockup, but never in 
hardware HVM :-)  Hopefully Linux will have pv timer soon.

thx, eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Carsten Otte]

Dong, Eddie wrote:
> kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
>> Jim Paris wrote:
>>> If I stop KVM in the monitor with "stop", wait a minute, and do
>>> "cont", a Linux guest gives me a "BUG: soft lockup detected on
>>> CPU#0". Is that expected behavior?
>> We have the same behavior on s390 when running in a virtual
>> environment. The issue is, that the guest physical cpu may be
>> scheduled away while a timer runs down. I don't think there's anything
>> kvm could do better regerding the soft lockup code. Soft lockup is
>> just broken for virtualized environments and needs to be fixed. For
>> the time being, just disable it in kconfig.
>>
> This is a well known issue, but we can fix by syncing TSC with PIT etc.
> But doing this we need to froze guest TSC when a VP is descheduled.
> 
> We enable this in Xen but feel it is too complicated so never wanna
> to do same thing in KVM per previous discussion. There are some
> side effect of course due to this.
> 
> In Xen some time you will see PV domain soft lockup, but never in 
> hardware HVM :-)  Hopefully Linux will have pv timer soon.
Yes, virtual timers will be the right fix for it. I believe Jan has a 
patch for that. Am I mistaken Jan? [adding Jan to CC:]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: soft lockup after stop/cont

[Jan Glauber1]

Carsten Otte/Germany/IBM@IBMDE wrote on 10/11/2007 12:57:38 PM:

> Dong, Eddie wrote:
> > kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
> >> Jim Paris wrote:
> >>> If I stop KVM in the monitor with "stop", wait a minute, and do
> >>> "cont", a Linux guest gives me a "BUG: soft lockup detected on
> >>> CPU#0". Is that expected behavior?
> >> We have the same behavior on s390 when running in a virtual
> >> environment. The issue is, that the guest physical cpu may be
> >> scheduled away while a timer runs down. I don't think there's anything
> >> kvm could do better regerding the soft lockup code. Soft lockup is
> >> just broken for virtualized environments and needs to be fixed. For
> >> the time being, just disable it in kconfig.
> >>
> > This is a well known issue, but we can fix by syncing TSC with PIT etc.
> > But doing this we need to froze guest TSC when a VP is descheduled.
> >
> > We enable this in Xen but feel it is too complicated so never wanna
> > to do same thing in KVM per previous discussion. There are some
> > side effect of course due to this.
> >
> > In Xen some time you will see PV domain soft lockup, but never in
> > hardware HVM :-)  Hopefully Linux will have pv timer soon.
> Yes, virtual timers will be the right fix for it. I believe Jan has a
> patch for that. Am I mistaken Jan? [adding Jan to CC:]

Well, we have an instruction for that ;- (but that's not the whole story).
The idea is simply to make sched_clock using a virtual timer, so softlockup
does only account time the guest was scheduled. Currently we have
softlockup disabled
on s390 too, since there are some open issues with CFS and the virtual
sched_clock.

-- Jan


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/