No kernel interface to reset a VCPU

No kernel interface to reset a VCPU

[Jan Kiszka]

Hi all,

looks to me like there is no way to properly reset the boot processor.
The current pattern used by qemu[-kvm] is to reload all registers with
their reset values. But that does not affect the internal VCPU states
the KVM keeps in the kernel. They are only reset during VCPU setup or
after receiving a SIPI (and the latter only helps with secondary CPUs).

So the only way around it with the current kernel interface is to
destroy/recreate the BSP on reset, right? /me is looking into such an
approach now.


We stumbled over inconsistent VCPU states with our internal qemu-kvm
tree. We have a legacy watchdog emulation here that triggered but failed
to bring up the system again. I wasn't able to create a similar case
with a standard environment yet, but I think it is not unrealistic for
qemu-kvm as well. Hacking kvm_arch_vcpu_reset() into KVM that triggers
on the right register values "solved" the issue here.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: No kernel interface to reset a VCPU

[Gleb Natapov]

On Fri, Sep 25, 2009 at 04:52:05PM +0200, Jan Kiszka wrote:
> Hi all,
> 
> looks to me like there is no way to properly reset the boot processor.
> The current pattern used by qemu[-kvm] is to reload all registers with
> their reset values. But that does not affect the internal VCPU states
> the KVM keeps in the kernel. They are only reset during VCPU setup or
> after receiving a SIPI (and the latter only helps with secondary CPUs).
> 
Userspace should have access to internal VCPU states too, otherwise
migration will not work.

> So the only way around it with the current kernel interface is to
> destroy/recreate the BSP on reset, right? /me is looking into such an
> approach now.
I don't think destroying/recreating vcpu will work. I don't remember 
exact details though.

> 
> We stumbled over inconsistent VCPU states with our internal qemu-kvm
> tree. We have a legacy watchdog emulation here that triggered but failed
> to bring up the system again. I wasn't able to create a similar case
> with a standard environment yet, but I think it is not unrealistic for
> qemu-kvm as well. Hacking kvm_arch_vcpu_reset() into KVM that triggers
> on the right register values "solved" the issue here.
>
Can you find the root cause of the problem? As I said above qemu should
have full access to vcpu state for proper migration support. Not that
kvm_vcpu_reset()/kvm_apic_reset()/kvm_ioapic_reset()/kvm_pit_reset() is
bad idea. Actually I want to add them one day.

--
			Gleb.

Re: No kernel interface to reset a VCPU

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2147 bytes --]

Gleb Natapov wrote:
> On Fri, Sep 25, 2009 at 04:52:05PM +0200, Jan Kiszka wrote:
>> Hi all,
>>
>> looks to me like there is no way to properly reset the boot processor.
>> The current pattern used by qemu[-kvm] is to reload all registers with
>> their reset values. But that does not affect the internal VCPU states
>> the KVM keeps in the kernel. They are only reset during VCPU setup or
>> after receiving a SIPI (and the latter only helps with secondary CPUs).
>>
> Userspace should have access to internal VCPU states too, otherwise
> migration will not work.

Good point.

> 
>> So the only way around it with the current kernel interface is to
>> destroy/recreate the BSP on reset, right? /me is looking into such an
>> approach now.
> I don't think destroying/recreating vcpu will work. I don't remember 
> exact details though.
> 
>> We stumbled over inconsistent VCPU states with our internal qemu-kvm
>> tree. We have a legacy watchdog emulation here that triggered but failed
>> to bring up the system again. I wasn't able to create a similar case
>> with a standard environment yet, but I think it is not unrealistic for
>> qemu-kvm as well. Hacking kvm_arch_vcpu_reset() into KVM that triggers
>> on the right register values "solved" the issue here.
>>
> Can you find the root cause of the problem? As I said above qemu should
> have full access to vcpu state for proper migration support. Not that

I just had a closer look again and found our problem: arch.nmi_pending.

I think the risk to be bitten by this on standard OSes is rather low.
The reset issue we see is widely related to the special NMI-based
watchdog here. The probability to see the pattern NMI-> guest handler ->
NMI -> system-reset on ordinary systems is fairly low. Besides this
hidden state may cause lost NMI events during migration or save/restore.
Again a corner case.

But it should be fixed. Will check where we could add this bit for
userland read-out.

> kvm_vcpu_reset()/kvm_apic_reset()/kvm_ioapic_reset()/kvm_pit_reset() is
> bad idea. Actually I want to add them one day.
> 
> --
> 			Gleb.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]