why xen use x86_emulation() in page_fault

why xen use x86_emulation() in page_fault

[cc Luit]

[-- Attachment #1.1: Type: text/plain, Size: 394 bytes --]

Hi, everyone, I have a question,
in the shadow_page_fault or ept mechanism, xen will use the x86_emulation
for some instructions, I'm wondering why it must use it, if after we fix the
SPT or EPT table, just VMEntry to HVM to re-excute this instruction  but not
emulate in xen, is there some problems?

can sb. tell me why?
thanks:)

-- 
- Luit @ Parallel Processing Institute, Fudan University

[-- Attachment #1.2: Type: text/html, Size: 484 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: why xen use x86_emulation() in page_fault

[Tim Deegan]

At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote:
> Hi, everyone, I have a question,
> in the shadow_page_fault or ept mechanism, xen will use the x86_emulation
> for some instructions, I'm wondering why it must use it, if after we fix the
> SPT or EPT table, just VMEntry to HVM to re-excute this instruction  but not
> emulate in xen, is there some problems?

In the shadow pagetable code, we keep the shadows up-to-date by:
 1 - making all shadowed pagetables read-only;
 2 - intercepting the page faults when the guest writes to them; and
 3 - updating the guest pagetable and the shadow at the same time, 
     with whatever change the guest was making. 

For step 3 we need to emulate the instruction that caused the pagefault
so that we can tell what was being written. 

There are other reasons for the emulator to be called (emulating MMIO
instructions, emulating real-mode &c) but that's why the shadow
pagetable code uses it. 

Cheers,

Tim.

Re: why xen use x86_emulation() in page_fault

[cc Luit]

[-- Attachment #1.1: Type: text/plain, Size: 1649 bytes --]

On Tue, Oct 11, 2011 at 4:18 PM, Tim Deegan <tim@xen.org> wrote:

> At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote:
> > Hi, everyone, I have a question,
> > in the shadow_page_fault or ept mechanism, xen will use the x86_emulation
> > for some instructions, I'm wondering why it must use it, if after we fix
> the
> > SPT or EPT table, just VMEntry to HVM to re-excute this instruction  but
> not
> > emulate in xen, is there some problems?
>
> > In the shadow pagetable code, we keep the shadows up-to-date by:
> > 1 - making all shadowed pagetables read-only;
> > 2 - intercepting the page faults when the guest writes to them; and
> > 3 - updating the guest pagetable and the shadow at the same time,
> >   with whatever change the guest was making.
>
> > For step 3 we need to emulate the instruction that caused the pagefault
> > so that we can tell what was being written.
>
> > There are other reasons for the emulator to be called (emulating MMIO
> > instructions, emulating real-mode &c) but that's why the shadow
> > pagetable code uses it.
>
> Thanks first of all, I know now it is the Eager mode that SPT is sync up
with GPT when guest want to change the page table using the instructions
emulator,  but if for some reason I don't want xen to emulate such an
instruction, but just VMEntry to HVM to retry, is there any feasibility
problems? in other words, in the old time, the shadow page is the Lazy mode,
that xen will not emulate, and the GPT and SPT is out of sync for some time,
besides the lose in performance, is there other problems?

> > Cheers,
>
> Tim.
>



-- 
- Luit @ Parallel Processing Institute, Fudan University

[-- Attachment #1.2: Type: text/html, Size: 2202 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: why xen use x86_emulation() in page_fault

[Tim Deegan]

At 18:14 +0800 on 11 Oct (1318356859), cc Luit wrote:
> On Tue, Oct 11, 2011 at 4:18 PM, Tim Deegan <tim@xen.org> wrote:
> 
> > At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote:
> > > Hi, everyone, I have a question,
> > > in the shadow_page_fault or ept mechanism, xen will use the x86_emulation
> > > for some instructions, I'm wondering why it must use it, if after we fix
> > the
> > > SPT or EPT table, just VMEntry to HVM to re-excute this instruction  but
> > not
> > > emulate in xen, is there some problems?
> >
> > > In the shadow pagetable code, we keep the shadows up-to-date by:
> > > 1 - making all shadowed pagetables read-only;
> > > 2 - intercepting the page faults when the guest writes to them; and
> > > 3 - updating the guest pagetable and the shadow at the same time,
> > >   with whatever change the guest was making.
> >
> > > For step 3 we need to emulate the instruction that caused the pagefault
> > > so that we can tell what was being written.
> >
> > > There are other reasons for the emulator to be called (emulating MMIO
> > > instructions, emulating real-mode &c) but that's why the shadow
> > > pagetable code uses it.
> >
> > Thanks first of all, I know now it is the Eager mode that SPT is sync up
> with GPT when guest want to change the page table using the instructions
> emulator,  but if for some reason I don't want xen to emulate such an
> instruction, but just VMEntry to HVM to retry, is there any feasibility
> problems?

The emulation can be avoided - in fact the current shadow pagetable
sometimes lets guests write to shadowed pages and fixes up the shadows
afterwards (this is called out-of-sync or oos in the code).

But if you just return to the guest and retry, the guest will take the
same fault again unless you have done something to change that.  If you
_have_ done something to make it OK, then just returning
EXCRET_fault_fixed from sh_page_fault will return to the guest and
retry the instruction.

Why do you want to avoid calling the emulator? What is your overall goal?
It might be that tinkering in the shadow pagetables isn't the best way
to acheive it.

> in other words, in the old time, the shadow page is the Lazy mode,
> that xen will not emulate, and the GPT and SPT is out of sync for some time,
> besides the lose in performance, is there other problems?

No, it was really about the performance cost of syncing up all the
shadows on a TLB flush.  In retrospect, having fixed some nasty bugs in
the OOS code, I suspect the old shadow code was also incorrect in some
ways but that was an implementation detail, noit architectural.

Tim.

Re: why xen use x86_emulation() in page_fault

[cc Luit]

[-- Attachment #1.1: Type: text/plain, Size: 3633 bytes --]

On Tue, Oct 11, 2011 at 7:29 PM, Tim Deegan <tim@xen.org> wrote:

>  At 18:14 +0800 on 11 Oct (1318356859), cc Luit wrote:
> > On Tue, Oct 11, 2011 at 4:18 PM, Tim Deegan <tim@xen.org> wrote:
> >
> > > At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote:
> > > > Hi, everyone, I have a question,
> > > > in the shadow_page_fault or ept mechanism, xen will use the
> x86_emulation
> > > > for some instructions, I'm wondering why it must use it, if after we
> fix
> > > the
> > > > SPT or EPT table, just VMEntry to HVM to re-excute this instruction
>  but
> > > not
> > > > emulate in xen, is there some problems?
> > >
> > > > In the shadow pagetable code, we keep the shadows up-to-date by:
> > > > 1 - making all shadowed pagetables read-only;
> > > > 2 - intercepting the page faults when the guest writes to them; and
> > > > 3 - updating the guest pagetable and the shadow at the same time,
> > > >   with whatever change the guest was making.
> > >
> > > > For step 3 we need to emulate the instruction that caused the
> pagefault
> > > > so that we can tell what was being written.
> > >
> > > > There are other reasons for the emulator to be called (emulating MMIO
> > > > instructions, emulating real-mode &c) but that's why the shadow
> > > > pagetable code uses it.
> > >
> > > Thanks first of all, I know now it is the Eager mode that SPT is sync
> up
> > with GPT when guest want to change the page table using the instructions
> > emulator,  but if for some reason I don't want xen to emulate such an
> > instruction, but just VMEntry to HVM to retry, is there any feasibility
> > problems?
>
> > The emulation can be avoided - in fact the current shadow pagetable
> > sometimes lets guests write to shadowed pages and fixes up the shadows
> > afterwards (this is called out-of-sync or oos in the code).
>
> > But if you just return to the guest and retry, the guest will take the
> > same fault again unless you have done something to change that.  If you
> > _have_ done something to make it OK, then just returning
> > EXCRET_fault_fixed from sh_page_fault will return to the guest and
> > retry the instruction.
>
I've read a slide said that in Lazy mode:
 when guest os modify the GPT, do not emulate (there is no
write-protected PTE, so guest can directly modify it)
 1) when the access right ascension, the guest OS will INVLPG to shootdown
TLB, so hypervisor can catch the INPLPG inst to sync  up the SPT/GPT
   2) when access right down, when guest OS access this page it will trap to
xen, xen will catch #PF to sync up SPT/GPT,

I'm not sure if this can work?


> Why do you want to avoid calling the emulator? What is your overall goal?
> It might be that tinkering in the shadow pagetables isn't the best way
> to acheive it.

because we're doing some research of security aspect about xen, what's our
goal is avoid xen to access the HVM's memory in the page fault situation,
it's hard to say it out in short words, we have thought a lot of ways but
there is no a simpler one than avoiding the emulation in page_fault.

Thanks

>
> > in other words, in the old time, the shadow page is the Lazy mode,
> > that xen will not emulate, and the GPT and SPT is out of sync for some
> time,
> > besides the lose in performance, is there other problems?
>
> No, it was really about the performance cost of syncing up all the
> shadows on a TLB flush.  In retrospect, having fixed some nasty bugs in
> the OOS code, I suspect the old shadow code was also incorrect in some
> ways but that was an implementation detail, noit architectural.
>
> Tim.
>



-- 
- Luit @ Parallel Processing Institute, Fudan University

[-- Attachment #1.2: Type: text/html, Size: 4675 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: why xen use x86_emulation() in page_fault

[Tim Deegan]

At 20:03 +0800 on 11 Oct (1318363429), cc Luit wrote:
> I've read a slide said that in Lazy mode:
>  when guest os modify the GPT, do not emulate (there is no
> write-protected PTE, so guest can directly modify it)
>  1) when the access right ascension, the guest OS will INVLPG to shootdown
> TLB, so hypervisor can catch the INPLPG inst to sync  up the SPT/GPT
>    2) when access right down, when guest OS access this page it will trap to
> xen, xen will catch #PF to sync up SPT/GPT,
> 
> I'm not sure if this can work?

I don't quite understand your English, but yes, you can let the guest
write to page tables and sync up the shadows afterwards.  You need to
sync the shadows whenever real hardware would re-read pagetables.  That
is, at least: 
 - When the guest executes INVLPG
 - When the guest writes to CR3
 - When the shadowing code adds a new l2e, l3e or l4e that points to an
   existing shadow. 
(and possibly some other cases I don;t remember right now).  

> > Why do you want to avoid calling the emulator? What is your overall goal?
> > It might be that tinkering in the shadow pagetables isn't the best way
> > to acheive it.
> 
> because we're doing some research of security aspect about xen, what's our
> goal is avoid xen to access the HVM's memory in the page fault situation,
> it's hard to say it out in short words, we have thought a lot of ways but
> there is no a simpler one than avoiding the emulation in page_fault.

Ah, I see - this is following on from the CloudVisor work, is it? 

Why don't you just use EPT/NPT instead of shadow pagetables?  That code
is a lot simpler and the end result is easier to validate than shadow
pagetables are.  Is it particularly important to support shadow pagetables?

In either case, there are some instructions that Xen will always
emulate - for example: 
 - MMIO accesses to emulated hardware
 - Some real-mode instructions on older Intel hardware

so even if you rewrote the shadow pagetable code not to emulate PT writes
you still have to handle those.  

Cheers,

Tim.

Re: why xen use x86_emulation() in page_fault

[cc Luit]

[-- Attachment #1.1: Type: text/plain, Size: 2757 bytes --]

On Tue, Oct 11, 2011 at 8:21 PM, Tim Deegan <tim@xen.org> wrote:

> At 20:03 +0800 on 11 Oct (1318363429), cc Luit wrote:
> > I've read a slide said that in Lazy mode:
> >  when guest os modify the GPT, do not emulate (there is no
> > write-protected PTE, so guest can directly modify it)
> >  1) when the access right ascension, the guest OS will INVLPG to
> shootdown
> > TLB, so hypervisor can catch the INPLPG inst to sync  up the SPT/GPT
> >    2) when access right down, when guest OS access this page it will trap
> to
> > xen, xen will catch #PF to sync up SPT/GPT,
> >
> > I'm not sure if this can work?
>
> > I don't quite understand your English, but yes, you can let the guest
> > write to page tables and sync up the shadows afterwards.  You need to
> > sync the shadows whenever real hardware would re-read pagetables.  That
> > is, at least:
> >  - When the guest executes INVLPG
> >  - When the guest writes to CR3
> >  - When the shadowing code adds a new l2e, l3e or l4e that points to an
> >   existing shadow.
> > (and possibly some other cases I don;t remember right now).
>
really sorry about my poor english:-) I will consider your suggestions
carefully~ thanks!

>
> > > Why do you want to avoid calling the emulator? What is your overall
> goal?
> > > It might be that tinkering in the shadow pagetables isn't the best way
> > > to acheive it.
> >
> > because we're doing some research of security aspect about xen, what's
> our
> > goal is avoid xen to access the HVM's memory in the page fault situation,
> > it's hard to say it out in short words, we have thought a lot of ways but
> > there is no a simpler one than avoiding the emulation in page_fault.
>
> > Ah, I see - this is following on from the CloudVisor work, is it?
>
yeah~ CloudVisor was just our lab's work, but ours is a little different:-)

>
> > Why don't you just use EPT/NPT instead of shadow pagetables?  That code
> > is a lot simpler and the end result is easier to validate than shadow
> > pagetables are.  Is it particularly important to support shadow
> pagetables?
>
yeah, we have consider that, and of course the EPT is much more convenient,
BTW, do you know is there any simulator that can well support EPT of xen?
the shadow page's problem is a little relevent to the selection of
simulator~

>
> > In either case, there are some instructions that Xen will always
> > emulate - for example:
> >  - MMIO accesses to emulated hardware
> >  - Some real-mode instructions on older Intel hardware
>
yeah, that's true, and we are also considering it:-)

>
> > so even if you rewrote the shadow pagetable code not to emulate PT writes
> > you still have to handle those.
>
> Cheers,
>
> Tim.
>



-- 
- Luit @ Parallel Processing Institute, Fudan University

[-- Attachment #1.2: Type: text/html, Size: 3878 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: why xen use x86_emulation() in page_fault

[Tim Deegan]

At 21:07 +0800 on 11 Oct (1318367233), cc Luit wrote:
> > > Why don't you just use EPT/NPT instead of shadow pagetables?  That code
> > > is a lot simpler and the end result is easier to validate than shadow
> > > pagetables are.  Is it particularly important to support shadow
> > pagetables?
> >
> yeah, we have consider that, and of course the EPT is much more convenient,
> BTW, do you know is there any simulator that can well support EPT of xen?
> the shadow page's problem is a little relevent to the selection of
> simulator~

There's experimental support in Xen itself for running a hypervisor
inside a HVM guest.  On AMD, that includes (and indeed requires) NPT in
the guest hypervisor.  Unfortunately, on Intel it does not include EPT.

There may be some other simulators that have EPT/NPT - I don't know. 

Cheers,

Tim.

Re: why xen use x86_emulation() in page_fault

[Christoph Egger]

On 10/13/11 13:47, Tim Deegan wrote:
> At 21:07 +0800 on 11 Oct (1318367233), cc Luit wrote:
>>>> Why don't you just use EPT/NPT instead of shadow pagetables?  That code
>>>> is a lot simpler and the end result is easier to validate than shadow
>>>> pagetables are.  Is it particularly important to support shadow
>>> pagetables?
>>>
>> yeah, we have consider that, and of course the EPT is much more convenient,
>> BTW, do you know is there any simulator that can well support EPT of xen?
>> the shadow page's problem is a little relevent to the selection of
>> simulator~
>
> There's experimental support in Xen itself for running a hypervisor
> inside a HVM guest.  On AMD, that includes (and indeed requires) NPT in
> the guest hypervisor.

Not quite right: guest hypervisor can use NPT but this is not required.
It may use shadow paging. But an AMD cpu with NPT capability is
required since shadow-on-shadow is not supported.

 >  Unfortunately, on Intel it does not include EPT.
>
> There may be some other simulators that have EPT/NPT - I don't know.
>
> Cheers,
>
> Tim.


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85689 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: why xen use x86_emulation() in page_fault

[Zhengwang Ruan (Royo)]

Hi Tim,

I am a freshman to this member-list and interested in some contents in 
your reply, looking forward to your answer. :)


At 2011-10-11 16:18, Tim Deegan wrote:
> At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote:
>> Hi, everyone, I have a question,
>> in the shadow_page_fault or ept mechanism, xen will use the x86_emulation
>> for some instructions, I'm wondering why it must use it, if after we fix the
>> SPT or EPT table, just VMEntry to HVM to re-excute this instruction  but not
>> emulate in xen, is there some problems?
> In the shadow pagetable code, we keep the shadows up-to-date by:
>   1 - making all shadowed pagetables read-only;
>   2 - intercepting the page faults when the guest writes to them; and
>   3 - updating the guest pagetable and the shadow at the same time,
>       with whatever change the guest was making.
>
> For step 3 we need to emulate the instruction that caused the pagefault
> so that we can tell what was being written.
>
> There are other reasons for the emulator to be called (emulating MMIO
> instructions, emulating real-mode&c) but that's why the shadow
> pagetable code uses it.

1, Does this emulator set up a emulation environment to execute 
instructions for a VM(guest OS)?
2, Real-mode code you talked above is used to execute the first 16-bit 
initial code of a guest OS? Does it have other usages in current Xen 
implementation?

Thanks,
Royo (Zhengwang)

> Cheers,
>
> Tim.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: why xen use x86_emulation() in page_fault

[cc Luit]

[-- Attachment #1.1: Type: text/plain, Size: 2335 bytes --]

On Tue, Oct 11, 2011 at 8:04 PM, Zhengwang Ruan (Royo) <
zhengwang.ruan@gmail.com> wrote:

> Hi Tim,
>
> I am a freshman to this member-list and interested in some contents in your
> reply, looking forward to your answer. :)
>
>
>
> At 2011-10-11 16:18, Tim Deegan wrote:
>
>>  At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote:
>>
>>> Hi, everyone, I have a question,
>>> in the shadow_page_fault or ept mechanism, xen will use the x86_emulation
>>> for some instructions, I'm wondering why it must use it, if after we fix
>>> the
>>> SPT or EPT table, just VMEntry to HVM to re-excute this instruction  but
>>> not
>>> emulate in xen, is there some problems?
>>>
>> In the shadow pagetable code, we keep the shadows up-to-date by:
>>  1 - making all shadowed pagetables read-only;
>>  2 - intercepting the page faults when the guest writes to them; and
>>  3 - updating the guest pagetable and the shadow at the same time,
>>      with whatever change the guest was making.
>>
>> For step 3 we need to emulate the instruction that caused the pagefault
>> so that we can tell what was being written.
>>
>> There are other reasons for the emulator to be called (emulating MMIO
>> instructions, emulating real-mode&c) but that's why the shadow
>> pagetable code uses it.
>>
>
> 1, Does this emulator set up a emulation environment to execute
> instructions for a VM(guest OS)?
>
there is a x86_emulate() in xen/arch/x86/x86_emulate/x86_emulate.c(my
version is xen-4.0.1), and it will emulate some instructions (I think both
guest OS and Domain-0) in some conditions(like what Tim said)


> 2, Real-mode code you talked above is used to execute the first 16-bit
> initial code of a guest OS? Does it have other usages in current Xen
> implementation?
>
I'm also curious about the real-mode, looking forward to the replay~

>
> Thanks,
> Royo (Zhengwang)
>
> Cheers,
>>
>> Tim.
>>
>>
>> ______________________________**_________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/**xen-devel<http://lists.xensource.com/xen-devel>
>>
>
> ______________________________**_________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/**xen-devel<http://lists.xensource.com/xen-devel>
>



-- 
- Luit @ Parallel Processing Institute, Fudan University

[-- Attachment #1.2: Type: text/html, Size: 3691 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: why xen use x86_emulation() in page_fault

[Tim Deegan]

Hi, 

At 21:46 +0800 on 11 Oct (1318369564), cc Luit wrote:
> On Tue, Oct 11, 2011 at 8:04 PM, Zhengwang Ruan (Royo) <zhengwang.ruan@gmail.com> wrote:
> > 2, Real-mode code you talked above is used to execute the first 16-bit
> > initial code of a guest OS? Does it have other usages in current Xen
> > implementation?
> >
> I'm also curious about the real-mode, looking forward to the replay~

It is used whenever the guest OS runs in real mode.  That's usually just
for a short while in the boot loader and the start of the kernel, but
that would depend on what OS you run in the guest.  DOS, for example,
would use it all the time. :) 

Linux (like most modern OSes) avoids real mode once it's booted, but if
it ever needs to reset a CPU (e.g. if you support CPU hotplug) then that
CPU will run in real mode briefly after it's reset.

Tim.

Re: why xen use x86_emulation() in page_fault

[Zhengwang Ruan (Royo)]

Hi Tim,

I am a freshman to this member-list and interested in some contents in
your reply, looking forward to your answer. :)


At 2011-10-11 16:18, Tim Deegan wrote:
> At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote:
>> Hi, everyone, I have a question,
>> in the shadow_page_fault or ept mechanism, xen will use the x86_emulation
>> for some instructions, I'm wondering why it must use it, if after we fix the
>> SPT or EPT table, just VMEntry to HVM to re-excute this instruction  but not
>> emulate in xen, is there some problems?
> In the shadow pagetable code, we keep the shadows up-to-date by:
>   1 - making all shadowed pagetables read-only;
>   2 - intercepting the page faults when the guest writes to them; and
>   3 - updating the guest pagetable and the shadow at the same time,
>       with whatever change the guest was making.
>
> For step 3 we need to emulate the instruction that caused the pagefault
> so that we can tell what was being written.
>
> There are other reasons for the emulator to be called (emulating MMIO
> instructions, emulating real-mode&c) but that's why the shadow
> pagetable code uses it.

1, Does this emulator set up a emulation environment to execute
instructions for a VM(guest OS)?
2, Real-mode code you talked above is used to execute the first 16-bit
initial code of a guest OS? Does it have other usages in current Xen
implementation?

Thanks,
Royo (Zhengwang)


> Cheers,
>
> Tim.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel