How KVM sync guest page table with corresponding shadow page table?

How KVM sync guest page table with corresponding shadow page table?

[陳韋任 (Wei-Ren Chen)]

Hi all,

  If guest modify its guest page table entry, KVM should trap that event,
emulate what guest trying to do, and sync with the corresponding shadow
page table entry. Is my understanding correct? If so, does KVM use
something like mprotect to make guest page table read-only, or there is
another way? Besides, how KVM know what guest try to write into the
guest page table entry?

Regards,
chenwj

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667
Homepage: http://people.cs.nctu.edu.tw/~chenwj

Re: How KVM sync guest page table with corresponding shadow page table?

[Xiao Guangrong]

On 08/14/2012 11:54 AM, 陳韋任 (Wei-Ren Chen) wrote:
> Hi all,
> 
>   If guest modify its guest page table entry, KVM should trap that event,
> emulate what guest trying to do, and sync with the corresponding shadow
> page table entry. Is my understanding correct? If so, does KVM use
> something like mprotect to make guest page table read-only, or there is
> another way? 

It is controlled by shadow page table, guest-page-tables are write-protected
on shadow pages (the W bit on PTE is cleared).

There has a special case, called unsync shadow page, if the page only used
as guest page structure on the lowest level (level = 1), we allow it to be
writable, it will be sync-ed when the guest flush the tlb (e.g: CR3 reload,
invlpg...) because according to x86 TLB rules, it needs to flush tlb to apply
the change.

> Besides, how KVM know what guest try to write into the
> guest page table entry?

In the normal case, guest writes its page table will generate #PF since the
page is write-protected as we mention above.

For the special case (unsync shadow page), we can happily write its page
table without #PF.

Re: How KVM sync guest page table with corresponding shadow page table?

[陳韋任 (Wei-Ren Chen)]

Hi Guangrong,

  I am not familiar with the term used in paging world, so I need to ask
some dumb questions.

> It is controlled by shadow page table, guest-page-tables are write-protected
> on shadow pages (the W bit on PTE is cleared).

  O.K.
 
> There has a special case, called unsync shadow page, if the page only used
> as guest page structure on the lowest level (level = 1), we allow it to be
> writable, it will be sync-ed when the guest flush the tlb (e.g: CR3 reload,
> invlpg...) because according to x86 TLB rules, it needs to flush tlb to apply
> the change.

  What "guest page structure on the lowest level" means? Take a simple two-level
page hierarchy as an example, do you mean the page table (not page directory)?
In short, if the architecture requires when OS modifies Nth-level page table
entry, the OS has to flush tlb, then we can unsync shadow page since the modifying
operations can be trapped by executing tlb flush instruction. Is that right?

> In the normal case, guest writes its page table will generate #PF since the
> page is write-protected as we mention above.

  Let me try to illustrate what happen after HW generating #PF, please correct
me if I am wrong. :)

  #PF causes an VMExit, and the control returned to KVM. KVM will use guest
virtual address (I think this information is accompanied with #PF, stored in cr4
on x86 for example) to walk guest page table, and KVM will find guest allows
this access, so something must be wrong in shadow page table. Then KVM will
go through shadow page table, found the corresponding entry has been set
to write-protected. Now KVM knows it has to sync guest page table and the
shadow page table, but how KVM knows what guest OS want to write into guest
page table entry? 

  Let's assume KVM knows, what's next? KVM will make the guest page table
writable by setting shadow page table entry, contine to execute guest
instruction which triggers #PF, then sync shadow page table at the same
time? It would be great if you can point me out where should I look into
in KVM source code as a starting code, that makes me understand shadow
page table stuff more concrete, and I don't have to guess how things
work. :)

  Thanks!

Regards,
chenwj

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667
Homepage: http://people.cs.nctu.edu.tw/~chenwj

Re: How KVM sync guest page table with corresponding shadow page table?

[Xiao Guangrong]

On 08/24/2012 03:29 PM, 陳韋任 (Wei-Ren Chen) wrote:
> Hi Guangrong,
> 
>   I am not familiar with the term used in paging world, so I need to ask
> some dumb questions.
> 
>> It is controlled by shadow page table, guest-page-tables are write-protected
>> on shadow pages (the W bit on PTE is cleared).
> 
>   O.K.
> 
>> There has a special case, called unsync shadow page, if the page only used
>> as guest page structure on the lowest level (level = 1), we allow it to be
>> writable, it will be sync-ed when the guest flush the tlb (e.g: CR3 reload,
>> invlpg...) because according to x86 TLB rules, it needs to flush tlb to apply
>> the change.
> 
>   What "guest page structure on the lowest level" means? Take a simple two-level
> page hierarchy as an example, do you mean the page table (not page directory)?

Yes.

> In short, if the architecture requires when OS modifies Nth-level page table
> entry, the OS has to flush tlb, then we can unsync shadow page since the modifying
> operations can be trapped by executing tlb flush instruction. Is that right?
> 

Right. Note, as I mentioned above, only Page Table (N0) can be unsync.

>> In the normal case, guest writes its page table will generate #PF since the
>> page is write-protected as we mention above.
> 
>   Let me try to illustrate what happen after HW generating #PF, please correct
> me if I am wrong. :)
> 
>   #PF causes an VMExit, and the control returned to KVM. KVM will use guest
> virtual address (I think this information is accompanied with #PF, stored in cr4
> on x86 for example) to walk guest page table, and KVM will find guest allows

CR2 if no virtualization, Exit qualification if it is running in guest mode


> this access, so something must be wrong in shadow page table. Then KVM will
> go through shadow page table, found the corresponding entry has been set
> to write-protected. Now KVM knows it has to sync guest page table and the
> shadow page table, but how KVM knows what guest OS want to write into guest
> page table entry? 

If shadow page table can mark to writable, we do not need care
"what guest OS want to write", just set the shadow page table to writable
and return to guest, let the guest reexecute the instruction again.

If shadow page table can not mark to writable (such as MMIO or the page is used
as Page Directory), we can emulate the instruction.
> 
>   Let's assume KVM knows, what's next? KVM will make the guest page table
> writable by setting shadow page table entry, contine to execute guest
> instruction which triggers #PF, then sync shadow page table at the same

Yes, guest reexecutes the instruction, no #PF again since the shadow page table
has already been make writable.

> time? It would be great if you can point me out where should I look into
> in KVM source code as a starting code, that makes me understand shadow
> page table stuff more concrete, and I don't have to guess how things
> work. :)


Welcome! :)