Trying to switch EPTP for execute-protecting guest pages

Trying to switch EPTP for execute-protecting guest pages

[Estrada, Zachary J]

Hi all,

I'm playing around with EPTs and kvm to track execution in the guest.  I've 
created a separate set of EPTs (and copied the last level entries from the real 
tables, minus execute permissions) but I'm not getting exits where I expect. I 
also have code in handle_ept_violation to preserve those permissions for any 
non-execute ept violations.

Here is what I am calling within a VM Exit handler:
---
kvm_mmu_unload(vcpu);
vcpu->arch.mmu.root_hpa = eptp;
kvm_x86_ops->set_tdp_cr3(vcpu, eptp);
kvm_mmu_load(vcpu);
kvm_flush_remote_tlbs(vcpu->kvm);
---

I think some of this is overkill, but am I missing something? I think I may need 
to flush the rmaps too, but I'm not exactly sure how.

I am using Haswell if that matters.

Thank you very much for your help!
--Zak

Re: Trying to switch EPTP for execute-protecting guest pages

[Paolo Bonzini]

On 23/11/2015 18:11, Estrada, Zachary J wrote:
> I'm playing around with EPTs and kvm to track execution in the guest. 
> I've created a separate set of EPTs (and copied the last level entries
> from the real tables, minus execute permissions) but I'm not getting
> exits where I expect. I also have code in handle_ept_violation to
> preserve those permissions for any non-execute ept violations.
> 
> Here is what I am calling within a VM Exit handler:
> ---
> kvm_mmu_unload(vcpu);
> vcpu->arch.mmu.root_hpa = eptp;
> kvm_x86_ops->set_tdp_cr3(vcpu, eptp);
> kvm_mmu_load(vcpu);
> kvm_flush_remote_tlbs(vcpu->kvm);
> ---
> 
> I think some of this is overkill, but am I missing something? I think I
> may need to flush the rmaps too, but I'm not exactly sure how.

My suggestion is:

1) use tracing and check that kvm_mmu_get_page is being called correctly.

2) there is already code for write protection.  Try copying that code
instead of doing a complete reimplementation.

Paolo

Re: Trying to switch EPTP for execute-protecting guest pages

[Estrada, Zachary J]

On 11/24/2015 05:44 AM, Paolo Bonzini wrote:
>
>
> On 23/11/2015 18:11, Estrada, Zachary J wrote:
>> I'm playing around with EPTs and kvm to track execution in the guest.
>> I've created a separate set of EPTs (and copied the last level entries
>> from the real tables, minus execute permissions) but I'm not getting
>> exits where I expect. I also have code in handle_ept_violation to
>> preserve those permissions for any non-execute ept violations.
>>
>> Here is what I am calling within a VM Exit handler:
>> ---
>> kvm_mmu_unload(vcpu);
>> vcpu->arch.mmu.root_hpa = eptp;
>> kvm_x86_ops->set_tdp_cr3(vcpu, eptp);
>> kvm_mmu_load(vcpu);
>> kvm_flush_remote_tlbs(vcpu->kvm);
>> ---
>>
>> I think some of this is overkill, but am I missing something? I think I
>> may need to flush the rmaps too, but I'm not exactly sure how.
>
> My suggestion is:
>
> 1) use tracing and check that kvm_mmu_get_page is being called correctly.
>
> 2) there is already code for write protection.  Try copying that code
> instead of doing a complete reimplementation.
>
> Paolo
>

1) Will do, thanks!

2) Got it. Let's say I want to work with a copy of the extended page tables 
instead of the original, what would be the best way to do so? Right now I'm 
traversing the full tables using root_hpa, but if there's a better way using the 
spte interface, I would prefer that.

Thanks so much!
--Zak

Re: Trying to switch EPTP for execute-protecting guest pages

[Paolo Bonzini]

On 24/11/2015 15:51, Estrada, Zachary J wrote:
> 2) Got it. Let's say I want to work with a copy of the extended page
> tables instead of the original, what would be the best way to do so?

Why would you want that?  It's difficult to give an answer without
understanding what you're doing.  Notice that KVM pretty much always
leaves the X bit set (__direct_map uses ACC_ALL for the pte_access
parameter) so it's easy to go from your copy of the extended page tables
to the original.

I'm not sure if this is your problem, but perhaps you want to record in
the role whether the page comes from your version or the original?  The
role is like the hash key, if the role is the same you get the same PTE.

Paolo

> Right now I'm traversing the full tables using root_hpa, but if there's
> a better way using the spte interface, I would prefer that.

Re: Trying to switch EPTP for execute-protecting guest pages

[Estrada, Zachary J]

On 11/24/2015 09:13 AM, Paolo Bonzini wrote:
>
>
> On 24/11/2015 15:51, Estrada, Zachary J wrote:
>> 2) Got it. Let's say I want to work with a copy of the extended page
>> tables instead of the original, what would be the best way to do so?
>
> Why would you want that?  It's difficult to give an answer without
> understanding what you're doing.  Notice that KVM pretty much always
> leaves the X bit set (__direct_map uses ACC_ALL for the pte_access
> parameter) so it's easy to go from your copy of the extended page tables
> to the original.
>
Reply sent offlist.

> I'm not sure if this is your problem, but perhaps you want to record in
> the role whether the page comes from your version or the original?  The
> role is like the hash key, if the role is the same you get the same PTE.
>
This is extremely helpful, I had not noticed this. I'm using my new root_hpa as 
the base_role.word - does that make sense? I just tried it and I seem to get 
EPT_VIOLATIONS that I was expecting, but missing.

Thanks a ton, it appears that the role was exactly the thing I was looking for!
--Zak

Re: Trying to switch EPTP for execute-protecting guest pages

[Paolo Bonzini]

On 24/11/2015 16:52, Estrada, Zachary J wrote:
>> I'm not sure if this is your problem, but perhaps you want to record in
>> the role whether the page comes from your version or the original?  The
>> role is like the hash key, if the role is the same you get the same PTE.
>
> This is extremely helpful, I had not noticed this. I'm using my new
> root_hpa as the base_role.word - does that make sense? I just tried it
> and I seem to get EPT_VIOLATIONS that I was expecting, but missing.

I think you should add a new bit to the role meaning "should I clear
some X bits?" :) that is computed based on the VCPU state.  For an
example see commit 699023e2 ("KVM: x86: add SMM to the MMU role, support
SMRAM address space"), which does

+       context->base_role.smm = is_smm(vcpu);

in init_kvm_tdp_mmu.  BTW, based on what you told me offlist, what you
are doing should also just work with shadow page tables.

Paolo

> Thanks a ton, it appears that the role was exactly the thing I was
> looking for!