Save/Restore not working

Save/Restore not working

[Frank Berreth]

Hello,

I am working on save/restore code (independent of Qemu or KVM tools)
and am having trouble doing 'the right thing' with KVM it seems. I do
have trouble finding good documentation on specifically save or
restore (besides the description of IOCTLs) and there is a high chance
I am doing something goofy here.

The way I am save/restoring is simply saving the state I get through
the different KVM IOCTLs and dumping it back on restore. The exception
is restoring of the MSRs, where I first get the list, then get the
MSRS and save/restore only those that were filled in by KVM.

The restore order is the following:

first restore memory, then
KVM_SET_REGS
KVM_SET_FPU
KVM_SET_SREGS
KVM_SET_MSRS
KVM_SET_MPSTATE
KVM_SET_LAPIC
some user mode devices and then
KVM_SET_PIT
KVM_SET_IRQCHIP (3 times, for MASTER, SLAVE and IOAPIC)

I am running different tests to test the save/restore code. One test
for example will compile the linux kernel within the guest while the
VM is saved, destroyed, re-create and finally restored & resumed
during compilation (every 10 seconds) on the same physical machine.

The behavior I get is different depending on KVM version and CPU.

Running with kvm-84 gives only sporadic failures, maybe every thousand
restores a double fault in the guest.

Running with a 2.6.34 kernel gives an 'immediate' double fault in the
guest. The guest does resume work as megabytes of guest data get
modified before the double fault but eventually the guest crashes -
and always it seems in the page fault handler. This is pretty much
100% reproduceable even with a 'workload' that simply does a "sleep
1d" in a shell script.

On Intel CPUs though I get a VMX_INVALID_GUEST_STATE error. The guest
does not run - which tells me I am doing something wrong here.

Assuming that the KVM versions work fine - what state that comes from
KVM through one of the KVM_GET_* IOCTLs has to be modified/sanitized
on restore?

Thanks,
Frank.

Re: Save/Restore not working

[Marcelo Tosatti]

On Mon, Jul 18, 2011 at 03:46:56PM -0700, Frank Berreth wrote:
> Hello,
> 
> I am working on save/restore code (independent of Qemu or KVM tools)
> and am having trouble doing 'the right thing' with KVM it seems. I do
> have trouble finding good documentation on specifically save or
> restore (besides the description of IOCTLs) and there is a high chance
> I am doing something goofy here.
> 
> The way I am save/restoring is simply saving the state I get through
> the different KVM IOCTLs and dumping it back on restore. The exception
> is restoring of the MSRs, where I first get the list, then get the
> MSRS and save/restore only those that were filled in by KVM.
> 
> The restore order is the following:
> 
> first restore memory, then
> KVM_SET_REGS
> KVM_SET_FPU
> KVM_SET_SREGS
> KVM_SET_MSRS
> KVM_SET_MPSTATE
> KVM_SET_LAPIC
> some user mode devices and then
> KVM_SET_PIT
> KVM_SET_IRQCHIP (3 times, for MASTER, SLAVE and IOAPIC)

KVM_SET_VCPU_EVENTS is missing from this list.

Also note this comment from api.txt:

NOTE: For KVM_EXIT_IO, KVM_EXIT_MMIO and KVM_EXIT_OSI, the corresponding
operations are complete (and guest state is consistent) only after
userspace
has re-entered the kernel with KVM_RUN.  The kernel side will first
finish
incomplete operations and then check for pending signals.  Userspace
can re-enter the guest with an unmasked signal pending to complete
pending operations.


> I am running different tests to test the save/restore code. One test
> for example will compile the linux kernel within the guest while the
> VM is saved, destroyed, re-create and finally restored & resumed
> during compilation (every 10 seconds) on the same physical machine.
> 
> The behavior I get is different depending on KVM version and CPU.
> 
> Running with kvm-84 gives only sporadic failures, maybe every thousand
> restores a double fault in the guest.
> 
> Running with a 2.6.34 kernel gives an 'immediate' double fault in the
> guest. The guest does resume work as megabytes of guest data get
> modified before the double fault but eventually the guest crashes -
> and always it seems in the page fault handler. This is pretty much
> 100% reproduceable even with a 'workload' that simply does a "sleep
> 1d" in a shell script.

$ echo kvm_page_fault > /sys/kernel/debug/tracing/set_event

> On Intel CPUs though I get a VMX_INVALID_GUEST_STATE error. The guest
> does not run - which tells me I am doing something wrong here.

You can load kvm-intel.ko with emulate_invalid_guest_state=1, and
inspect guest_state_valid to pinpoint which state is invalid.

> Assuming that the KVM versions work fine - what state that comes from
> KVM through one of the KVM_GET_* IOCTLs has to be modified/sanitized
> on restore?

None. Only ordering must be maintained, follow QEMU there.

Re: Save/Restore not working

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1309 bytes --]

On 2011-07-19 19:20, Marcelo Tosatti wrote:
> On Mon, Jul 18, 2011 at 03:46:56PM -0700, Frank Berreth wrote:
>> Hello,
>>
>> I am working on save/restore code (independent of Qemu or KVM tools)
>> and am having trouble doing 'the right thing' with KVM it seems. I do
>> have trouble finding good documentation on specifically save or
>> restore (besides the description of IOCTLs) and there is a high chance
>> I am doing something goofy here.
>>
>> The way I am save/restoring is simply saving the state I get through
>> the different KVM IOCTLs and dumping it back on restore. The exception
>> is restoring of the MSRs, where I first get the list, then get the
>> MSRS and save/restore only those that were filled in by KVM.
>>
>> The restore order is the following:
>>
>> first restore memory, then
>> KVM_SET_REGS
>> KVM_SET_FPU
>> KVM_SET_SREGS
>> KVM_SET_MSRS
>> KVM_SET_MPSTATE
>> KVM_SET_LAPIC
>> some user mode devices and then
>> KVM_SET_PIT
>> KVM_SET_IRQCHIP (3 times, for MASTER, SLAVE and IOAPIC)
> 
> KVM_SET_VCPU_EVENTS is missing from this list.

...and some more. As Marcelo said, follow QEMU closely. It's the
reference implementation and known to work pretty well.

Just curious: What is the reason for another probably painful rewrite of
QEMU?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: Save/Restore not working

[Frank Berreth]

Marcelo,

thanks for your help.
You were right that we did not complete pending instructions before
saving. However, adding that code did not make much of a difference.
I'll continue investigating with the tracing options you mentioned.

Thanks,
Frank.

On Tue, Jul 19, 2011 at 10:20 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Mon, Jul 18, 2011 at 03:46:56PM -0700, Frank Berreth wrote:
>> Hello,
>>
>> I am working on save/restore code (independent of Qemu or KVM tools)
>> and am having trouble doing 'the right thing' with KVM it seems. I do
>> have trouble finding good documentation on specifically save or
>> restore (besides the description of IOCTLs) and there is a high chance
>> I am doing something goofy here.
>>
>> The way I am save/restoring is simply saving the state I get through
>> the different KVM IOCTLs and dumping it back on restore. The exception
>> is restoring of the MSRs, where I first get the list, then get the
>> MSRS and save/restore only those that were filled in by KVM.
>>
>> The restore order is the following:
>>
>> first restore memory, then
>> KVM_SET_REGS
>> KVM_SET_FPU
>> KVM_SET_SREGS
>> KVM_SET_MSRS
>> KVM_SET_MPSTATE
>> KVM_SET_LAPIC
>> some user mode devices and then
>> KVM_SET_PIT
>> KVM_SET_IRQCHIP (3 times, for MASTER, SLAVE and IOAPIC)
>
> KVM_SET_VCPU_EVENTS is missing from this list.
>
> Also note this comment from api.txt:
>
> NOTE: For KVM_EXIT_IO, KVM_EXIT_MMIO and KVM_EXIT_OSI, the corresponding
> operations are complete (and guest state is consistent) only after
> userspace
> has re-entered the kernel with KVM_RUN.  The kernel side will first
> finish
> incomplete operations and then check for pending signals.  Userspace
> can re-enter the guest with an unmasked signal pending to complete
> pending operations.
>
>
>> I am running different tests to test the save/restore code. One test
>> for example will compile the linux kernel within the guest while the
>> VM is saved, destroyed, re-create and finally restored & resumed
>> during compilation (every 10 seconds) on the same physical machine.
>>
>> The behavior I get is different depending on KVM version and CPU.
>>
>> Running with kvm-84 gives only sporadic failures, maybe every thousand
>> restores a double fault in the guest.
>>
>> Running with a 2.6.34 kernel gives an 'immediate' double fault in the
>> guest. The guest does resume work as megabytes of guest data get
>> modified before the double fault but eventually the guest crashes -
>> and always it seems in the page fault handler. This is pretty much
>> 100% reproduceable even with a 'workload' that simply does a "sleep
>> 1d" in a shell script.
>
> $ echo kvm_page_fault > /sys/kernel/debug/tracing/set_event
>
>> On Intel CPUs though I get a VMX_INVALID_GUEST_STATE error. The guest
>> does not run - which tells me I am doing something wrong here.
>
> You can load kvm-intel.ko with emulate_invalid_guest_state=1, and
> inspect guest_state_valid to pinpoint which state is invalid.
>
>> Assuming that the KVM versions work fine - what state that comes from
>> KVM through one of the KVM_GET_* IOCTLs has to be modified/sanitized
>> on restore?
>
> None. Only ordering must be maintained, follow QEMU there.
>
>