Paravirtualization of the "HLT" instruction (for example) on x386

Paravirtualization of the "HLT" instruction (for example) on x386

[Ian Brown]

Hello,

I am trying to understand a little more the paravirtualization
implementation in Xen.
As I understand, when using paravirtualization, we want to
change the behavior of certain instructions.
So for example, we don't want to enable guest domains to
perform "HLT" instruction (which will stop the machine).
And there are other instructions which their
default behavior in ring 1 should  be modified.

So I tried to find where in the Xen-3.0 code it is done.
I saw in vmx.c the vmx_vmexit_do_hlt() method ,which is called
when "HLT" is performed,  but this is relevant when running a
platform with VT-x.
I am looking for tracing where this handling or modifying of
the HLT instruction is done in a usual x386 (non-VTX) processor.

Can anybody please point where in the code of Xen 3.0 this is done
(for x86 processors)?
IB

Re: Paravirtualization of the "HLT" instruction (for example) on x386

[Keir Fraser]

On 12 Jan 2006, at 09:27, Ian Brown wrote:

> So I tried to find where in the Xen-3.0 code it is done.
> I saw in vmx.c the vmx_vmexit_do_hlt() method ,which is called
> when "HLT" is performed,  but this is relevant when running a
> platform with VT-x.
> I am looking for tracing where this handling or modifying of
> the HLT instruction is done in a usual x386 (non-VTX) processor.
>
> Can anybody please point where in the code of Xen 3.0 this is done
> (for x86 processors)?

The point of paravirtualizetion is that such instructions are replaced 
with explicit traps to the hypervisor in the OS. Emulation of the raw 
instruction is not required.

In fact we do emulate some privileged instructions, just not HLT.

  -- Keir

Re: Paravirtualization of the "HLT" instruction (for example) on x386

[Ian Brown]

Hello,
Thnks, Keir.

>The point of paravirtualizetion is that such instructions are replaced
>with explicit traps to the hypervisor in the OS.

1) Can you please give a pointer to where in code (xen 3.0) such privileged
instruction is  replaced
with a trap to the hypervisor ? (I have some assumption where it is done
but I am really not sure at all)

2) Regarding HLT - what happens if a guest OS issue "HLT" ?
I assume that without intervention in Xen code, issuing HLT from a
guest domain can stop not only the guest domain, but also the
hypervisor (and in fact the machine).
is there somewhere in code special treatment to "HLT"?
IB


On 1/12/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>
> On 12 Jan 2006, at 09:27, Ian Brown wrote:
>
> > So I tried to find where in the Xen-3.0 code it is done.
> > I saw in vmx.c the vmx_vmexit_do_hlt() method ,which is called
> > when "HLT" is performed,  but this is relevant when running a
> > platform with VT-x.
> > I am looking for tracing where this handling or modifying of
> > the HLT instruction is done in a usual x386 (non-VTX) processor.
> >
> > Can anybody please point where in the code of Xen 3.0 this is done
> > (for x86 processors)?
>
> The point of paravirtualizetion is that such instructions are replaced
> with explicit traps to the hypervisor in the OS. Emulation of the raw
> instruction is not required.
>
> In fact we do emulate some privileged instructions, just not HLT.
>
>   -- Keir
>
>

Re: Paravirtualization of the "HLT" instruction (for example) on x386

[Keir Fraser]

On 12 Jan 2006, at 10:39, Ian Brown wrote:

> 1) Can you please give a pointer to where in code (xen 3.0) such 
> privileged
> instruction is  replaced
> with a trap to the hypervisor ? (I have some assumption where it is 
> done
> but I am really not sure at all)

xen_idle() in arch/xen/i386/process.c. Uses the SCHEDOP_block 
hypercall, which is a fairly direct replacement for HLT.

> 2) Regarding HLT - what happens if a guest OS issue "HLT" ?
> I assume that without intervention in Xen code, issuing HLT from a
> guest domain can stop not only the guest domain, but also the
> hypervisor (and in fact the machine).
> is there somewhere in code special treatment to "HLT"?

HLT is only executable in ring 0. If the guest tries to execute it then 
it will receive a general protection fault.

  -- Keir

Re: Paravirtualization of the "HLT" instruction (for example) on x386

[Ian Brown]

Hello,

First, thanks Keir for your answer.

>The point of paravirtualization is that such instructions are replaced
>with explicit traps to the hypervisor in the OS.
>Emulation of the raw instruction is not required.

I had thought again about your answer and after probing the code
and IA32 manuals I am not sure I understood you fully and correctly.

I tried to look for such privileged instructions in the guest OS code
(for IA32)
which are replaced with explicit traps to the hypervisor and didn't find.

I know that CLTS and WBINVD instructions, for example , should cause
#GP(0) if run from CPL which is not 0; but grepping for an asm instruction
which calls CLTS or WBINVD under the sparse tree gives no results.

Can you give one example for such an instruction which cause a trap
to the hypervisor when run in a guest OS and where in the code it causes
such a trap ?

(As far as I understand,if we try to issue a privilege instruction from
CPL1 we should get a #GP(0) and reach the general protection
handler in sparse/arch/xen/i386/kernel/traps.c , do_general_protection().

But I had looked at do_general_protection() in
sparse/arch/xen/i386/kernel/traps.c
and could not find there a mechanism which will trap to the hypervisor;maybe
I had totally missed the point?)


Regards,

IB

On 1/12/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>
> On 12 Jan 2006, at 09:27, Ian Brown wrote:
>
> > So I tried to find where in the Xen-3.0 code it is done.
> > I saw in vmx.c the vmx_vmexit_do_hlt() method ,which is called
> > when "HLT" is performed,  but this is relevant when running a
> > platform with VT-x.
> > I am looking for tracing where this handling or modifying of
> > the HLT instruction is done in a usual x386 (non-VTX) processor.
> >
> > Can anybody please point where in the code of Xen 3.0 this is done
> > (for x86 processors)?
>
> The point of paravirtualizetion is that such instructions are replaced
> with explicit traps to the hypervisor in the OS. Emulation of the raw
> instruction is not required.
>
> In fact we do emulate some privileged instructions, just not HLT.
>
>   -- Keir
>
>

Re: Paravirtualization of the "HLT" instruction (for example) on x386

[Keir Fraser]

On 24 Jan 2006, at 11:27, Ian Brown wrote:

> I know that CLTS and WBINVD instructions, for example , should cause
> #GP(0) if run from CPL which is not 0; but grepping for an asm 
> instruction
> which calls CLTS or WBINVD under the sparse tree gives no results.
>
> Can you give one example for such an instruction which cause a trap
> to the hypervisor when run in a guest OS and where in the code it 
> causes
> such a trap ?
>
> (As far as I understand,if we try to issue a privilege instruction from
> CPL1 we should get a #GP(0) and reach the general protection
> handler in sparse/arch/xen/i386/kernel/traps.c , 
> do_general_protection().
>
> But I had looked at do_general_protection() in
> sparse/arch/xen/i386/kernel/traps.c
> and could not find there a mechanism which will trap to the 
> hypervisor;maybe
> I had totally missed the point?)

The main entry point for GPFs is in the hypervisor at 
do_general_protection() in xen/arch/x86/traps.c. For certain privileged 
instructions we perform emulation (see emulate_privileged_op() in the 
same Xen source file). We emulate both CLTS and WBINVD for example. 
GPFs that are not handled by Xen are indeed then passed to the guest 
and will end up in the function you mentioned in your email.

However, we also have paravirtualised versions of both those 
instructions (for example, CLTS is equivalent to the hypercall 
fpu_taskswitch(0)).

Furthermore, some instructions *have* to be paravirtualised because 
they do not trap (for example, POPF where the restored EFLAGS has a 
different Interrupt-Enable flag value from current EFLAGS).

  -- Keir

Re: Paravirtualization of the "HLT" instruction (for example) on x386

[Ian Brown]

Hello,

Thanks for your answer in such a short time !

I am aware of emulate_privileged_op() in traps.c and also of the
emulation of both CLTS and WBINVD in this method.

you said :
>GPFs that are not handled by Xen are indeed then passed to the guest
>and will end up in the function you mentioned in your email.

I am not sure about something regarding "are indeed then passed to the guest":
suppose a guest OS, running in ring 1, issues a privileged instruction (namely,
an instruction which causes #GP(0) since it was issued in CPL1 ).
I don't know if it is possible at all since as I understand such
instructions were replaced in the guest OS code. But let's say it's
possible, the
"passed to the guest" is the point I am trying to get at.

In such a case, what happens ? there is a #GP(0) of course, but who
handles it in the first place ? is it the OS in ring 0 (with it's
do_general_protection() method in this case ? )
? or is it the OS in ring 1, which also
have do_general_protection() method ?

and by
>GPFs that are not handled by Xen are indeed then passed to the guest
>and will end up in the function you mentioned in your email.

you mean that GPFs that occurred in ring 1 will be handled at the first
place by the guest ? (or ,what seems to me more unlikely, first by ring0
and then somehow "passed" to the guest)

Regards,
IB




On 1/24/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>
> On 24 Jan 2006, at 11:27, Ian Brown wrote:
>
> > I know that CLTS and WBINVD instructions, for example , should cause
> > #GP(0) if run from CPL which is not 0; but grepping for an asm
> > instruction
> > which calls CLTS or WBINVD under the sparse tree gives no results.
> >
> > Can you give one example for such an instruction which cause a trap
> > to the hypervisor when run in a guest OS and where in the code it
> > causes
> > such a trap ?
> >
> > (As far as I understand,if we try to issue a privilege instruction from
> > CPL1 we should get a #GP(0) and reach the general protection
> > handler in sparse/arch/xen/i386/kernel/traps.c ,
> > do_general_protection().
> >
> > But I had looked at do_general_protection() in
> > sparse/arch/xen/i386/kernel/traps.c
> > and could not find there a mechanism which will trap to the
> > hypervisor;maybe
> > I had totally missed the point?)
>
> The main entry point for GPFs is in the hypervisor at
> do_general_protection() in xen/arch/x86/traps.c. For certain privileged
> instructions we perform emulation (see emulate_privileged_op() in the
> same Xen source file). We emulate both CLTS and WBINVD for example.
> GPFs that are not handled by Xen are indeed then passed to the guest
> and will end up in the function you mentioned in your email.
>
> However, we also have paravirtualised versions of both those
> instructions (for example, CLTS is equivalent to the hypercall
> fpu_taskswitch(0)).
>
> Furthermore, some instructions *have* to be paravirtualised because
> they do not trap (for example, POPF where the restored EFLAGS has a
> different Interrupt-Enable flag value from current EFLAGS).
>
>   -- Keir
>
>

Re: Paravirtualization of the "HLT" instruction (for example) on x386

[Keir Fraser]

On 24 Jan 2006, at 12:24, Ian Brown wrote:

> you mean that GPFs that occurred in ring 1 will be handled at the first
> place by the guest ? (or ,what seems to me more unlikely, first by 
> ring0
> and then somehow "passed" to the guest)

That is what happens. The processor does not discriminate between GPFs 
-- they must all be delivered to the same initial handler. That handler 
is the one in Xen in ring 0. Xen then virtualises the GPF to the OS in 
ring 1 if Xen does not want to handle it itself. It does this by 
creating a suitable exception frame on the guest stack and jumping at 
the guest's registered GPF handler.

  -- Keir

Re: Paravirtualization of the "HLT" instruction (for example) on x386

[Mark Ryden]

>Furthermore, some instructions *have* to be paravirtualised because
>they do not trap

So all the instances of such instructions which don't trap and are
problemtaic in some sense , under the linux-xen0 and linux-xenU, are 
replaced ?  Did I get it right ?

Mark



On 1/24/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>
> On 24 Jan 2006, at 11:27, Ian Brown wrote:
>
> > I know that CLTS and WBINVD instructions, for example , should cause
> > #GP(0) if run from CPL which is not 0; but grepping for an asm
> > instruction
> > which calls CLTS or WBINVD under the sparse tree gives no results.
> >
> > Can you give one example for such an instruction which cause a trap
> > to the hypervisor when run in a guest OS and where in the code it
> > causes
> > such a trap ?
> >
> > (As far as I understand,if we try to issue a privilege instruction from
> > CPL1 we should get a #GP(0) and reach the general protection
> > handler in sparse/arch/xen/i386/kernel/traps.c ,
> > do_general_protection().
> >
> > But I had looked at do_general_protection() in
> > sparse/arch/xen/i386/kernel/traps.c
> > and could not find there a mechanism which will trap to the
> > hypervisor;maybe
> > I had totally missed the point?)
>
> The main entry point for GPFs is in the hypervisor at
> do_general_protection() in xen/arch/x86/traps.c. For certain privileged
> instructions we perform emulation (see emulate_privileged_op() in the
> same Xen source file). We emulate both CLTS and WBINVD for example.
> GPFs that are not handled by Xen are indeed then passed to the guest
> and will end up in the function you mentioned in your email.
>
> However, we also have paravirtualised versions of both those
> instructions (for example, CLTS is equivalent to the hypercall
> fpu_taskswitch(0)).
>
> Furthermore, some instructions *have* to be paravirtualised because
> they do not trap (for example, POPF where the restored EFLAGS has a
> different Interrupt-Enable flag value from current EFLAGS).
>
>   -- Keir
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Paravirtualization of the "HLT" instruction (for example) on x386

[Anthony Liguori]

Mark Ryden wrote:

>>Furthermore, some instructions *have* to be paravirtualised because
>>they do not trap
>>    
>>
>
>So all the instances of such instructions which don't trap and are
>problemtaic in some sense , under the linux-xen0 and linux-xenU, are 
>replaced ?  Did I get it right ?
>  
>
Yes, a very thorough explaination of this all is available here:

http://www.cs.nps.navy.mil/people/faculty/irvine/publications/2000/VMM-usenix00-0611.pdf

Regards,

Anthony Liguori

>Mark
>
>
>
>On 1/24/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>  
>
>>On 24 Jan 2006, at 11:27, Ian Brown wrote:
>>
>>    
>>
>>>I know that CLTS and WBINVD instructions, for example , should cause
>>>#GP(0) if run from CPL which is not 0; but grepping for an asm
>>>instruction
>>>which calls CLTS or WBINVD under the sparse tree gives no results.
>>>
>>>Can you give one example for such an instruction which cause a trap
>>>to the hypervisor when run in a guest OS and where in the code it
>>>causes
>>>such a trap ?
>>>
>>>(As far as I understand,if we try to issue a privilege instruction from
>>>CPL1 we should get a #GP(0) and reach the general protection
>>>handler in sparse/arch/xen/i386/kernel/traps.c ,
>>>do_general_protection().
>>>
>>>But I had looked at do_general_protection() in
>>>sparse/arch/xen/i386/kernel/traps.c
>>>and could not find there a mechanism which will trap to the
>>>hypervisor;maybe
>>>I had totally missed the point?)
>>>      
>>>
>>The main entry point for GPFs is in the hypervisor at
>>do_general_protection() in xen/arch/x86/traps.c. For certain privileged
>>instructions we perform emulation (see emulate_privileged_op() in the
>>same Xen source file). We emulate both CLTS and WBINVD for example.
>>GPFs that are not handled by Xen are indeed then passed to the guest
>>and will end up in the function you mentioned in your email.
>>
>>However, we also have paravirtualised versions of both those
>>instructions (for example, CLTS is equivalent to the hypercall
>>fpu_taskswitch(0)).
>>
>>Furthermore, some instructions *have* to be paravirtualised because
>>they do not trap (for example, POPF where the restored EFLAGS has a
>>different Interrupt-Enable flag value from current EFLAGS).
>>
>>  -- Keir
>>
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>
>>    
>>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>

RE: Paravirtualization of the "HLT" instruction ( for example) on x386

[Petersson, Mats]

 

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ian Brown
> Sent: 24 January 2006 12:24
> To: Keir Fraser
> Cc: Xen Mailing List
> Subject: Re: [Xen-devel] Paravirtualization of the "HLT" 
> instruction ( for example) on x386
> 
> Hello,
> 
> Thanks for your answer in such a short time !
> 
> I am aware of emulate_privileged_op() in traps.c and also of 
> the emulation of both CLTS and WBINVD in this method.
> 
> you said :
> >GPFs that are not handled by Xen are indeed then passed to the guest 
> >and will end up in the function you mentioned in your email.
> 
> I am not sure about something regarding "are indeed then 
> passed to the guest":
> suppose a guest OS, running in ring 1, issues a privileged 
> instruction (namely, an instruction which causes #GP(0) since 
> it was issued in CPL1 ).
> I don't know if it is possible at all since as I understand 
> such instructions were replaced in the guest OS code. But 
> let's say it's possible, the "passed to the guest" is the 
> point I am trying to get at.
> 
> In such a case, what happens ? there is a #GP(0) of course, 
> but who handles it in the first place ? is it the OS in ring 
> 0 (with it's
> do_general_protection() method in this case ? ) ? or is it 
> the OS in ring 1, which also have do_general_protection() method ?
> 
> and by
> >GPFs that are not handled by Xen are indeed then passed to the guest 
> >and will end up in the function you mentioned in your email.
> 
> you mean that GPFs that occurred in ring 1 will be handled at 
> the first place by the guest ? (or ,what seems to me more 
> unlikely, first by ring0 and then somehow "passed" to the guest)
> 
> Regards,
> IB
> 
> 
The way it works is that the GP fault is taken by Xen, and if Xen
decides that "I don't know what to do with this", for example because
it's an unexpected scenario, it will PASS it to the Guest. I haven't
looked at how the passing is done in Xen's hypervisor - there are
several ways you could do this, for example parsing the IDT of the guest
and making a fake stack-frame for the guest. 

In a hardware virtualized environment, the GP fault would be passed in
the same way as a virtual interrupt is passed to the guest. 

I just noticed that Keir just answered this one too. 

[snip]

--
Mats