Does KVM use one EPT table per Guest CR3?

Does KVM use one EPT table per Guest CR3?

[Lok Kwong Yan]

After some testing and digging around the 2.6.32-26 Kernel, Ubuntu port, , it seems to me that KVM creates a separate EPT table for each separate guest CR3 value. So, if there are 100 guest processes, there are essentially 100 EPT tables. Is this correct? If so, can someone please tell me where these tables are actually being created? Is this design decision a historical artifact from how QEMU/KVM (without EPT/NPT) created multiple shadow page tables so that each guest/virtual CR3 value has a corresponding real CR3 value so that memory based separation for the guest is enforced?

Thanks!

Re: Does KVM use one EPT table per Guest CR3?

[Anthony Liguori]

On 12/07/2010 03:47 PM, Lok Kwong Yan wrote:
> After some testing and digging around the 2.6.32-26 Kernel, Ubuntu port, , it seems to me that KVM creates a separate EPT table for each separate guest CR3 value. So, if there are 100 guest processes, there are essentially 100 EPT tables. Is this correct?

No, it's not correct.

Regards,

Anthony Liguori

>   If so, can someone please tell me where these tables are actually being created? Is this design decision a historical artifact from how QEMU/KVM (without EPT/NPT) created multiple shadow page tables so that each guest/virtual CR3 value has a corresponding real CR3 value so that memory based separation for the guest is enforced?
>
> Thanks!--
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

RE: Does KVM use one EPT table per Guest CR3?

[Lok Kwong Yan]

Thanks for the quick response. It is greatly appreciated. 

Can you please point me in the right direction on finding out why the EPTP (EPT_POINTER & mmu.root_hpa) have so many different values? 

Thanks again.

________________________________________
From: Anthony Liguori [anthony@codemonkey.ws]
Sent: Tuesday, December 07, 2010 4:57 PM
To: Lok Kwong Yan
Cc: kvm@vger.kernel.org
Subject: Re: Does KVM use one EPT table per Guest CR3?

On 12/07/2010 03:47 PM, Lok Kwong Yan wrote:
> After some testing and digging around the 2.6.32-26 Kernel, Ubuntu port, , it seems to me that KVM creates a separate EPT table for each separate guest CR3 value. So, if there are 100 guest processes, there are essentially 100 EPT tables. Is this correct?

No, it's not correct.

Regards,

Anthony Liguori

>   If so, can someone please tell me where these tables are actually being created? Is this design decision a historical artifact from how QEMU/KVM (without EPT/NPT) created multiple shadow page tables so that each guest/virtual CR3 value has a corresponding real CR3 value so that memory based separation for the guest is enforced?
>
> Thanks!--
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Does KVM use one EPT table per Guest CR3?

[Anthony Liguori]

On 12/07/2010 04:00 PM, Lok Kwong Yan wrote:
> Thanks for the quick response. It is greatly appreciated.
>
> Can you please point me in the right direction on finding out why the EPTP (EPT_POINTER&  mmu.root_hpa) have so many different values?
>    

The table is built dynamically as memory is faulted in so it changes 
over time.  But the rate is significantly less than the rate you'd see 
with shadow paging.

Unlike Xen, KVM doesn't just built an EPT table all at once and leave it 
alone.  The has to be updated as the guest is swapped in and out of memory.

Regards,

Anthony Liguori

> Thanks again.
>
> ________________________________________
> From: Anthony Liguori [anthony@codemonkey.ws]
> Sent: Tuesday, December 07, 2010 4:57 PM
> To: Lok Kwong Yan
> Cc: kvm@vger.kernel.org
> Subject: Re: Does KVM use one EPT table per Guest CR3?
>
> On 12/07/2010 03:47 PM, Lok Kwong Yan wrote:
>    
>> After some testing and digging around the 2.6.32-26 Kernel, Ubuntu port, , it seems to me that KVM creates a separate EPT table for each separate guest CR3 value. So, if there are 100 guest processes, there are essentially 100 EPT tables. Is this correct?
>>      
> No, it's not correct.
>
> Regards,
>
> Anthony Liguori
>
>    
>>    If so, can someone please tell me where these tables are actually being created? Is this design decision a historical artifact from how QEMU/KVM (without EPT/NPT) created multiple shadow page tables so that each guest/virtual CR3 value has a corresponding real CR3 value so that memory based separation for the guest is enforced?
>>
>> Thanks!--
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>      
>    

RE: Does KVM use one EPT table per Guest CR3?

[Lok Kwong Yan]

Thanks again. 

I understand that the table can change over time, but it doesn't explain why the top-most table gets zapped and seemingly recreated with a different page frame (and thus the different EPTP values) so frequently. Why isn't the table being updated instead of being destroyed and then recreated? Which piece of the puzzle and I missing?

Thanks.


________________________________________
From: Anthony Liguori [anthony@codemonkey.ws]
Sent: Tuesday, December 07, 2010 5:11 PM
To: Lok Kwong Yan
Cc: kvm@vger.kernel.org
Subject: Re: Does KVM use one EPT table per Guest CR3?

On 12/07/2010 04:00 PM, Lok Kwong Yan wrote:
> Thanks for the quick response. It is greatly appreciated.
>
> Can you please point me in the right direction on finding out why the EPTP (EPT_POINTER&  mmu.root_hpa) have so many different values?
>

The table is built dynamically as memory is faulted in so it changes
over time.  But the rate is significantly less than the rate you'd see
with shadow paging.

Unlike Xen, KVM doesn't just built an EPT table all at once and leave it
alone.  The has to be updated as the guest is swapped in and out of memory.

Regards,

Anthony Liguori

> Thanks again.
>
> ________________________________________
> From: Anthony Liguori [anthony@codemonkey.ws]
> Sent: Tuesday, December 07, 2010 4:57 PM
> To: Lok Kwong Yan
> Cc: kvm@vger.kernel.org
> Subject: Re: Does KVM use one EPT table per Guest CR3?
>
> On 12/07/2010 03:47 PM, Lok Kwong Yan wrote:
>
>> After some testing and digging around the 2.6.32-26 Kernel, Ubuntu port, , it seems to me that KVM creates a separate EPT table for each separate guest CR3 value. So, if there are 100 guest processes, there are essentially 100 EPT tables. Is this correct?
>>
> No, it's not correct.
>
> Regards,
>
> Anthony Liguori
>
>
>>    If so, can someone please tell me where these tables are actually being created? Is this design decision a historical artifact from how QEMU/KVM (without EPT/NPT) created multiple shadow page tables so that each guest/virtual CR3 value has a corresponding real CR3 value so that memory based separation for the guest is enforced?
>>
>> Thanks!--
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>

Re: Does KVM use one EPT table per Guest CR3?

[Avi Kivity]

On 12/10/2010 09:44 AM, Lok Kwong Yan wrote:
> Thanks again.
>
> I understand that the table can change over time, but it doesn't explain why the top-most table gets zapped and seemingly recreated with a different page frame (and thus the different EPTP values) so frequently. Why isn't the table being updated instead of being destroyed and then recreated? Which piece of the puzzle and I missing?

When the physical memory map changes the EPT tables are zapped.  This 
happens when PCI BARs are remapped, or when VGA remaps the framebuffer 
windows at 0xa0000-0xc0000.  Both of these happen during boot.  Do you 
see zaps happening after boot is complete?

-- 
error compiling committee.c: too many arguments to function

RE: Does KVM use one EPT table per Guest CR3?

[Lok Kwong Yan]

Thanks for the reply and it makes a lot of sense.

I am not seeing any EPT tables being zapped after the guest has fully started up although the value of EPTP continuously changes as the guest is running. 

________________________________________
From: Avi Kivity [avi@redhat.com]
Sent: Sunday, December 12, 2010 5:47 AM
To: Lok Kwong Yan
Cc: Anthony Liguori; kvm@vger.kernel.org
Subject: Re: Does KVM use one EPT table per Guest CR3?

On 12/10/2010 09:44 AM, Lok Kwong Yan wrote:
> Thanks again.
>
> I understand that the table can change over time, but it doesn't explain why the top-most table gets zapped and seemingly recreated with a different page frame (and thus the different EPTP values) so frequently. Why isn't the table being updated instead of being destroyed and then recreated? Which piece of the puzzle and I missing?

When the physical memory map changes the EPT tables are zapped.  This
happens when PCI BARs are remapped, or when VGA remaps the framebuffer
windows at 0xa0000-0xc0000.  Both of these happen during boot.  Do you
see zaps happening after boot is complete?

--
error compiling committee.c: too many arguments to function

Re: Does KVM use one EPT table per Guest CR3?

[Avi Kivity]

On 12/17/2010 12:14 AM, Lok Kwong Yan wrote:
> Thanks for the reply and it makes a lot of sense.
>
> I am not seeing any EPT tables being zapped after the guest has fully started up although the value of EPTP continuously changes as the guest is running.

Really strange, this is likely a bug.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Does KVM use one EPT table per Guest CR3?

[Avi Kivity]

On 12/17/2010 05:24 PM, Avi Kivity wrote:
> On 12/17/2010 12:14 AM, Lok Kwong Yan wrote:
>> Thanks for the reply and it makes a lot of sense.
>>
>> I am not seeing any EPT tables being zapped after the guest has fully 
>> started up although the value of EPTP continuously changes as the 
>> guest is running.
>
> Really strange, this is likely a bug.
>

I tried to reproduce, the only times I see eptp changes are when the 
guest reprograms the vga adapter:

  qemu-system-x86-20944 [033]  1327.151819: kvm_pio:              
pio_write at 0x3ce size 2 count 1
  qemu-system-x86-20944 [033]  1327.151819: kvm_userspace_exit:   reason 
KVM_EXIT_IO (2)
  qemu-system-x86-20944 [033]  1327.152405: kvm_mmu_prepare_zap_page: 
[FAILED TO PARSE] gfn=237568 role=122881 root_count=0 unsync=0
...
  qemu-system-x86-20944 [033]  1327.153230: kvm_mmu_prepare_zap_page: 
[FAILED TO PARSE] gfn=0 role=253956 root_count=2 unsync=0
  qemu-system-x86-20944 [033]  1327.153339: kvm_mmu_get_page:     sp gfn 
0 0/4 q0 direct --- !pge !nxe root 0sync
  qemu-system-x86-20944 [033]  1327.153344: print:                
a0265cde vmx_set_cr3: eptp fef14101

Under what scenario do you see eptp changing?

-- 
error compiling committee.c: too many arguments to function

RE: Does KVM use one EPT table per Guest CR3?

[Lok Kwong Yan]

Thanks for the reply. I haven't traced the source of eptp changing and won't have a chance to do that for a couple of weeks. I will do that and respond. Thanks again.

Lok

________________________________________
From: Avi Kivity [avi@redhat.com]
Sent: Sunday, December 19, 2010 9:31 AM
To: Lok Kwong Yan
Cc: Anthony Liguori; kvm@vger.kernel.org
Subject: Re: Does KVM use one EPT table per Guest CR3?

On 12/17/2010 05:24 PM, Avi Kivity wrote:
> On 12/17/2010 12:14 AM, Lok Kwong Yan wrote:
>> Thanks for the reply and it makes a lot of sense.
>>
>> I am not seeing any EPT tables being zapped after the guest has fully
>> started up although the value of EPTP continuously changes as the
>> guest is running.
>
> Really strange, this is likely a bug.
>

I tried to reproduce, the only times I see eptp changes are when the
guest reprograms the vga adapter:

  qemu-system-x86-20944 [033]  1327.151819: kvm_pio:
pio_write at 0x3ce size 2 count 1
  qemu-system-x86-20944 [033]  1327.151819: kvm_userspace_exit:   reason
KVM_EXIT_IO (2)
  qemu-system-x86-20944 [033]  1327.152405: kvm_mmu_prepare_zap_page:
[FAILED TO PARSE] gfn=237568 role=122881 root_count=0 unsync=0
...
  qemu-system-x86-20944 [033]  1327.153230: kvm_mmu_prepare_zap_page:
[FAILED TO PARSE] gfn=0 role=253956 root_count=2 unsync=0
  qemu-system-x86-20944 [033]  1327.153339: kvm_mmu_get_page:     sp gfn
0 0/4 q0 direct --- !pge !nxe root 0sync
  qemu-system-x86-20944 [033]  1327.153344: print:
a0265cde vmx_set_cr3: eptp fef14101

Under what scenario do you see eptp changing?

--
error compiling committee.c: too many arguments to function

RE: Does KVM use one EPT table per Guest CR3?

[Lok Kwong Yan]

Sorry for the late reply.

Seems to me that the EPTP pointer is changing because of kvm_set_cr0.

Here is what I did and please correct me if I am doing the trace incorrectly:

- Added a trace entry in vmx_set_cr3 where a trace message is outputted whenever vmcs_read64(EPT_POINTER) != eptp after construct_eptp(cr3).

I then looked at the trace log and seems to show up with 

kvm_exit: reason cr_access rip 0xc0122003
kvm_cr: cr_write 0 = 0x8005003b

I also noticed that kvm_mmu_reset_context(vcpu) is being called at the end of kvm_set_cr0. 

The CR0 value of 0x8005003b doesn't seem to trigger any of the if cases which would indicate that kvm_mmu_reset_context(vcpu) is being called and could be the reason why eptp is changing.

Thanks for your help again.

Enjoy,

Lok




________________________________________
From: Avi Kivity [avi@redhat.com]
Sent: Sunday, December 19, 2010 9:31 AM
To: Lok Kwong Yan
Cc: Anthony Liguori; kvm@vger.kernel.org
Subject: Re: Does KVM use one EPT table per Guest CR3?

On 12/17/2010 05:24 PM, Avi Kivity wrote:
> On 12/17/2010 12:14 AM, Lok Kwong Yan wrote:
>> Thanks for the reply and it makes a lot of sense.
>>
>> I am not seeing any EPT tables being zapped after the guest has fully
>> started up although the value of EPTP continuously changes as the
>> guest is running.
>
> Really strange, this is likely a bug.
>

I tried to reproduce, the only times I see eptp changes are when the
guest reprograms the vga adapter:

  qemu-system-x86-20944 [033]  1327.151819: kvm_pio:
pio_write at 0x3ce size 2 count 1
  qemu-system-x86-20944 [033]  1327.151819: kvm_userspace_exit:   reason
KVM_EXIT_IO (2)
  qemu-system-x86-20944 [033]  1327.152405: kvm_mmu_prepare_zap_page:
[FAILED TO PARSE] gfn=237568 role=122881 root_count=0 unsync=0
...
  qemu-system-x86-20944 [033]  1327.153230: kvm_mmu_prepare_zap_page:
[FAILED TO PARSE] gfn=0 role=253956 root_count=2 unsync=0
  qemu-system-x86-20944 [033]  1327.153339: kvm_mmu_get_page:     sp gfn
0 0/4 q0 direct --- !pge !nxe root 0sync
  qemu-system-x86-20944 [033]  1327.153344: print:
a0265cde vmx_set_cr3: eptp fef14101

Under what scenario do you see eptp changing?

--
error compiling committee.c: too many arguments to function