[Qemu-devel] Disabling KVM "on the fly"

[Qemu-devel] Disabling KVM "on the fly"

[Clemens Kolbitsch]

Guys,

I know this is question might seem a bit odd, but I'm curious:

Has anyone ever tried to write code to disable KVM on the fly / is it
at all possible? I have a situation where I need to use TCG for
certain parts of the code, but would love to have acceleration for
everything else. My idea was to pause the VM, then use the
snapshotting mechanism to dump the state, and then to resume the
snapshot, but writing the KVM state into the non-KVM structures.

I know nothing of that sort is implemented at this point. Leaving
aside the issue of certain CPUID instructions suddenly returning
different results, are there obvious problems even trying to go down
that road?

Would love to hear some thoughts on this - don't hesitate to tell me
"that's stupid and impossible because XYZ" (assuming you also fill in
the blanks ;) )

Thanks!
Clemens

-- 
Clemens Kolbitsch
Security Researcher
kolbitsch@lastline.com
+1 (206) 356-7745

Lastline, Inc.
6950 Hollister Avenue, Suite 101
Goleta, CA 93117

www.lastline.com

Re: [Qemu-devel] Disabling KVM "on the fly"

[Paolo Bonzini]

Il 17/10/2012 18:37, Clemens Kolbitsch ha scritto:
> Guys,
> 
> I know this is question might seem a bit odd, but I'm curious:
> 
> Has anyone ever tried to write code to disable KVM on the fly / is it
> at all possible? I have a situation where I need to use TCG for
> certain parts of the code, but would love to have acceleration for
> everything else. My idea was to pause the VM, then use the
> snapshotting mechanism to dump the state, and then to resume the
> snapshot, but writing the KVM state into the non-KVM structures.

As a start, you can try using "migrate exec:cat>foo.save" with a KVM
machine and "-incoming 'exec:cat foo.save'" with a TCG machine.  The
main problem should be that TCG doesn't implement kvmclock.

If you disable the KVM interrupt controller and timer (which is just an
implementation detail, not a hardware difference), the differences
between KVM and TCG are just that KVM doesn't initialize some TCG-only
data structure, and that KVM uses many CPU threads; TCG uses one that
goes through CPUs round-robin.  The CPU threads of course execute
different code.

So no, in theory there is nothing that prevents this from working in
principle, except for kvmclock.

Paolo

> I know nothing of that sort is implemented at this point. Leaving
> aside the issue of certain CPUID instructions suddenly returning
> different results, are there obvious problems even trying to go down
> that road?
> 
> Would love to hear some thoughts on this - don't hesitate to tell me
> "that's stupid and impossible because XYZ" (assuming you also fill in
> the blanks ;) )
> 
> Thanks!
> Clemens
> 

Re: [Qemu-devel] Disabling KVM "on the fly"

[Jan Kiszka]

On 2012-10-17 18:44, Paolo Bonzini wrote:
> Il 17/10/2012 18:37, Clemens Kolbitsch ha scritto:
>> Guys,
>>
>> I know this is question might seem a bit odd, but I'm curious:
>>
>> Has anyone ever tried to write code to disable KVM on the fly / is it
>> at all possible? I have a situation where I need to use TCG for
>> certain parts of the code, but would love to have acceleration for
>> everything else. My idea was to pause the VM, then use the
>> snapshotting mechanism to dump the state, and then to resume the
>> snapshot, but writing the KVM state into the non-KVM structures.
> 
> As a start, you can try using "migrate exec:cat>foo.save" with a KVM
> machine and "-incoming 'exec:cat foo.save'" with a TCG machine.  The
> main problem should be that TCG doesn't implement kvmclock.
> 
> If you disable the KVM interrupt controller and timer (which is just an
> implementation detail, not a hardware difference),

Unnecessary. Both models (KVM in-kernel and QEMU userspace) are
compatible - in the absence of bugs.

> the differences
> between KVM and TCG are just that KVM doesn't initialize some TCG-only
> data structure, and that KVM uses many CPU threads; TCG uses one that
> goes through CPUs round-robin.  The CPU threads of course execute
> different code.
> 
> So no, in theory there is nothing that prevents this from working in
> principle, except for kvmclock.

-cpu qemu64,-kvmclock should solve that.

You also need -global pc-sysfw.rom_only=1 as KVM does not support write
protected memory areas and creates an "old-style" BIOS region.

But loading a KVM image into TCG lets non-trival guests lock up. Likely
due to differences in the CPU virtualization/emulation (MSRs...). Also,
certain KVM specific CPU states cannot be easily translated into TCG
(and are definitely just ignored in TCG so far).

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Disabling KVM "on the fly"

[Paolo Bonzini]

Il 17/10/2012 20:37, Jan Kiszka ha scritto:
> On 2012-10-17 18:44, Paolo Bonzini wrote:
>> Il 17/10/2012 18:37, Clemens Kolbitsch ha scritto:
>>> Guys,
>>>
>>> I know this is question might seem a bit odd, but I'm curious:
>>>
>>> Has anyone ever tried to write code to disable KVM on the fly / is it
>>> at all possible? I have a situation where I need to use TCG for
>>> certain parts of the code, but would love to have acceleration for
>>> everything else. My idea was to pause the VM, then use the
>>> snapshotting mechanism to dump the state, and then to resume the
>>> snapshot, but writing the KVM state into the non-KVM structures.
>>
>> As a start, you can try using "migrate exec:cat>foo.save" with a KVM
>> machine and "-incoming 'exec:cat foo.save'" with a TCG machine.  The
>> main problem should be that TCG doesn't implement kvmclock.
>>
>> If you disable the KVM interrupt controller and timer (which is just an
>> implementation detail, not a hardware difference),
> 
> Unnecessary. Both models (KVM in-kernel and QEMU userspace) are
> compatible - in the absence of bugs.

He wants to really switch it on the fly---not just migrate out and
in---and for that you need to disable the KVM-specific devices.

> But loading a KVM image into TCG lets non-trival guests lock up. Likely
> due to differences in the CPU virtualization/emulation (MSRs...).

Perhaps that can be mitigated by using an older machine model.  Start
with something simple like a pentium2 and work up from there...

Paolo

Re: [Qemu-devel] Disabling KVM "on the fly"

[Jan Kiszka]

On 2012-10-18 08:29, Paolo Bonzini wrote:
> Il 17/10/2012 20:37, Jan Kiszka ha scritto:
>> On 2012-10-17 18:44, Paolo Bonzini wrote:
>>> Il 17/10/2012 18:37, Clemens Kolbitsch ha scritto:
>>>> Guys,
>>>>
>>>> I know this is question might seem a bit odd, but I'm curious:
>>>>
>>>> Has anyone ever tried to write code to disable KVM on the fly / is it
>>>> at all possible? I have a situation where I need to use TCG for
>>>> certain parts of the code, but would love to have acceleration for
>>>> everything else. My idea was to pause the VM, then use the
>>>> snapshotting mechanism to dump the state, and then to resume the
>>>> snapshot, but writing the KVM state into the non-KVM structures.
>>>
>>> As a start, you can try using "migrate exec:cat>foo.save" with a KVM
>>> machine and "-incoming 'exec:cat foo.save'" with a TCG machine.  The
>>> main problem should be that TCG doesn't implement kvmclock.
>>>
>>> If you disable the KVM interrupt controller and timer (which is just an
>>> implementation detail, not a hardware difference),
>>
>> Unnecessary. Both models (KVM in-kernel and QEMU userspace) are
>> compatible - in the absence of bugs.
> 
> He wants to really switch it on the fly---not just migrate out and
> in---and for that you need to disable the KVM-specific devices.

Well, that's even more unrealistic than via migration.

> 
>> But loading a KVM image into TCG lets non-trival guests lock up. Likely
>> due to differences in the CPU virtualization/emulation (MSRs...).
> 
> Perhaps that can be mitigated by using an older machine model.  Start
> with something simple like a pentium2 and work up from there...

Even if, there are still too many untranslated, maybe even
untranslatable states of the KVM CPU model, at least.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Disabling KVM "on the fly"

[Clemens Kolbitsch]

On Thu, Oct 18, 2012 at 10:24 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 2012-10-18 08:29, Paolo Bonzini wrote:
>> Il 17/10/2012 20:37, Jan Kiszka ha scritto:
>>> On 2012-10-17 18:44, Paolo Bonzini wrote:
>>>> Il 17/10/2012 18:37, Clemens Kolbitsch ha scritto:
>>>>> Guys,
>>>>>
>>>>> I know this is question might seem a bit odd, but I'm curious:
>>>>>
>>>>> Has anyone ever tried to write code to disable KVM on the fly / is it
>>>>> at all possible? I have a situation where I need to use TCG for
>>>>> certain parts of the code, but would love to have acceleration for
>>>>> everything else. My idea was to pause the VM, then use the
>>>>> snapshotting mechanism to dump the state, and then to resume the
>>>>> snapshot, but writing the KVM state into the non-KVM structures.
>>>>
>>>> As a start, you can try using "migrate exec:cat>foo.save" with a KVM
>>>> machine and "-incoming 'exec:cat foo.save'" with a TCG machine.  The
>>>> main problem should be that TCG doesn't implement kvmclock.
>>>>
>>>> If you disable the KVM interrupt controller and timer (which is just an
>>>> implementation detail, not a hardware difference),
>>>
>>> Unnecessary. Both models (KVM in-kernel and QEMU userspace) are
>>> compatible - in the absence of bugs.
>>
>> He wants to really switch it on the fly---not just migrate out and
>> in---and for that you need to disable the KVM-specific devices.
>
> Well, that's even more unrealistic than via migration.

I am just looking for a way to do it, regardless whether it causes a
big performance penalty due to migration, etc. Even if this means
preparing a special environment (emulating specific hardware, etc),
this is fine with me.

Thanks a lot for all the feedback, now I know what I will have to look into!

-Clemens