Patching guest kernel code for better performance from HOST

Patching guest kernel code for better performance from HOST

[Dushyant Bansal]

[-- Attachment #1: Type: text/plain, Size: 2185 bytes --]

Hi all,

     I understand that in order to improve performance by reducing vm 
exits, kvm uses one shared page between kvm and guest. Now, guest tells 
the host to map the magic page to '-4096'.

How does kvm make sure that guest will not use this address ( -4096) for 
other purpose?

Then, guest itself patches its kernel.
I am trying to patch guest from host. So far, I have tried to patch 
MFMSR instruction. Patching is done when kvm tries to emulate MFMSR. 
Mechanism used to generate new instruction is same as used by guest in 
"arch/powerpc/kernel/kvm.c"

--- a/arch/powerpc/kvm/book3s_emulate.c
+++ b/arch/powerpc/kvm/book3s_emulate.c
@@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, 
struct kvm_vcpu *vcpu,
                 case OP_31_XOP_MFMSR:
                         kvmppc_set_gpr(vcpu, get_rt(inst),
                                        vcpu->arch.shared->msr);
+
+                       pc = kvmppc_get_pc(vcpu);
+
+                       //if pte.may_write==false then, kvmppc_st will 
not work
+                       kvmppc_xlate(vcpu,pc,false, &pte);
+                       pte.may_write=true;
+
+                       //Generate new instruction
+                       addr= (-4096L) + offsetof(struct 
kvm_vcpu_arch_shared, msr);
+                       u32 rt1 = inst & 0x03e00000;
+                       u32 new_inst=0x80000000 | rt1 | ((addr + 4) & 
0x0000fffc);
+
+                       //patch only if magic page is mapped; try 
patching only for 0x7c0000a6 inst
+                       if (vcpu->arch.magic_page_pa==0xfffff000 && 
inst==0x7c0000a6) {
+                                       
kvmppc_st(vcpu,&pc,sizeof(u32),&new_inst,true);
+                                       }
+
                         break;
                 case OP_31_XOP_MTMSRD:
                 {


But, the problem is guest hangs during boot. I have attached the 
screenshot of the guest. I am not able to figure out the problem. Is 
there something wrong with the approach?

I have checked that new_inst generated corresponding to 0x7c0000a6 
instruction matches with the instruction that is generated when guest 
patches itself.


Thanks,
Dushyant

[-- Attachment #2: Screenshot.png --]
[-- Type: image/png, Size: 25341 bytes --]

Re: Patching guest kernel code for better performance from HOST

[Alexander Graf]

On 05.04.2011, at 10:59, Dushyant Bansal wrote:

> Hi all,
> 
>    I understand that in order to improve performance by reducing vm exits, kvm uses one shared page between kvm and guest. Now, guest tells the host to map the magic page to '-4096'.
> 
> How does kvm make sure that guest will not use this address ( -4096) for other purpose?
> 
> Then, guest itself patches its kernel.
> I am trying to patch guest from host. So far, I have tried to patch MFMSR instruction. Patching is done when kvm tries to emulate MFMSR. Mechanism used to generate new instruction is same as used by guest in "arch/powerpc/kernel/kvm.c"
> 
> --- a/arch/powerpc/kvm/book3s_emulate.c
> +++ b/arch/powerpc/kvm/book3s_emulate.c
> @@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, struct kvm_vcpu *vcpu,
>                case OP_31_XOP_MFMSR:
>                        kvmppc_set_gpr(vcpu, get_rt(inst),
>                                       vcpu->arch.shared->msr);
> +
> +                       pc = kvmppc_get_pc(vcpu);
> +
> +                       //if pte.may_write=false then, kvmppc_st will not work
> +                       kvmppc_xlate(vcpu,pc,false, &pte);
> +                       pte.may_write=true;

The pte struct here is write-only from xlate's point of view. Any modification to it doesn't get reflected to the real translation layer. But your guess is good, I'd also assume that you're hitting a read-only page.


Alex

Re: Patching guest kernel code for better performance from HOST

[Dushyant Bansal]

On Tuesday 05 April 2011 02:39 PM, Alexander Graf wrote:
> On 05.04.2011, at 10:59, Dushyant Bansal wrote:
>
>    
>> Hi all,
>>
>>     I understand that in order to improve performance by reducing vm exits, kvm uses one shared page between kvm and guest. Now, guest tells the host to map the magic page to '-4096'.
>>
>> How does kvm make sure that guest will not use this address ( -4096) for other purpose?
>>
>> Then, guest itself patches its kernel.
>> I am trying to patch guest from host. So far, I have tried to patch MFMSR instruction. Patching is done when kvm tries to emulate MFMSR. Mechanism used to generate new instruction is same as used by guest in "arch/powerpc/kernel/kvm.c"
>>
>> --- a/arch/powerpc/kvm/book3s_emulate.c
>> +++ b/arch/powerpc/kvm/book3s_emulate.c
>> @@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, struct kvm_vcpu *vcpu,
>>                 case OP_31_XOP_MFMSR:
>>                         kvmppc_set_gpr(vcpu, get_rt(inst),
>>                                        vcpu->arch.shared->msr);
>> +
>> +                       pc = kvmppc_get_pc(vcpu);
>> +
>> +                       //if pte.may_write=false then, kvmppc_st will not work
>> +                       kvmppc_xlate(vcpu,pc,false,&pte);
>> +                       pte.may_write=true;
>>      
> The pte struct here is write-only from xlate's point of view. Any modification to it doesn't get reflected to the real translation layer. But your guess is good, I'd also assume that you're hitting a read-only page.
>    
Yes, you are right. Thanks for catching this.
So, if it is a read-only page, 
'kvmppc_st(vcpu,&pc,sizeof(u32),&new_inst,true)' will not write 
anything. Then, there is no reason for guest to hang. Right?

--
Dushyant

Re: Patching guest kernel code for better performance from HOST

[Alexander Graf]

On 04/05/2011 03:30 PM, Dushyant Bansal wrote:
> On Tuesday 05 April 2011 02:39 PM, Alexander Graf wrote:
>> On 05.04.2011, at 10:59, Dushyant Bansal wrote:
>>
>>> Hi all,
>>>
>>>     I understand that in order to improve performance by reducing vm 
>>> exits, kvm uses one shared page between kvm and guest. Now, guest 
>>> tells the host to map the magic page to '-4096'.
>>>
>>> How does kvm make sure that guest will not use this address ( -4096) 
>>> for other purpose?
>>>
>>> Then, guest itself patches its kernel.
>>> I am trying to patch guest from host. So far, I have tried to patch 
>>> MFMSR instruction. Patching is done when kvm tries to emulate MFMSR. 
>>> Mechanism used to generate new instruction is same as used by guest 
>>> in "arch/powerpc/kernel/kvm.c"
>>>
>>> --- a/arch/powerpc/kvm/book3s_emulate.c
>>> +++ b/arch/powerpc/kvm/book3s_emulate.c
>>> @@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, 
>>> struct kvm_vcpu *vcpu,
>>>                 case OP_31_XOP_MFMSR:
>>>                         kvmppc_set_gpr(vcpu, get_rt(inst),
>>>                                        vcpu->arch.shared->msr);
>>> +
>>> +                       pc = kvmppc_get_pc(vcpu);
>>> +
>>> +                       //if pte.may_write=false then, kvmppc_st 
>>> will not work
>>> +                       kvmppc_xlate(vcpu,pc,false,&pte);
>>> +                       pte.may_write=true;
>> The pte struct here is write-only from xlate's point of view. Any 
>> modification to it doesn't get reflected to the real translation 
>> layer. But your guess is good, I'd also assume that you're hitting a 
>> read-only page.
> Yes, you are right. Thanks for catching this.
> So, if it is a read-only page, 
> 'kvmppc_st(vcpu,&pc,sizeof(u32),&new_inst,true)' will not write 
> anything. Then, there is no reason for guest to hang. Right?

Oh? It should inject a page fault. Maybe it doesn't, but then it behaves 
badly :).

Either way - just enable the debug tracepoints and check the exits you 
get right after you hit an mfmsr emulation. That should give you hints 
on what's going wrong.


Alex

Re: Patching guest kernel code for better performance from HOST

[Dushyant Bansal]

>>>> --- a/arch/powerpc/kvm/book3s_emulate.c
>>>> +++ b/arch/powerpc/kvm/book3s_emulate.c
>>>> @@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, 
>>>> struct kvm_vcpu *vcpu,
>>>>                 case OP_31_XOP_MFMSR:
>>>>                         kvmppc_set_gpr(vcpu, get_rt(inst),
>>>>                                        vcpu->arch.shared->msr);
>>>> +
>>>> +                       pc = kvmppc_get_pc(vcpu);
>>>> +
>>>> +                       //if pte.may_write=false then, kvmppc_st 
>>>> will not work
>>>> +                       kvmppc_xlate(vcpu,pc,false,&pte);
>>>> +                       pte.may_write=true;
>>> The pte struct here is write-only from xlate's point of view. Any 
>>> modification to it doesn't get reflected to the real translation 
>>> layer. But your guess is good, I'd also assume that you're hitting a 
>>> read-only page.
>> Yes, you are right. Thanks for catching this.
>> So, if it is a read-only page, 
>> 'kvmppc_st(vcpu,&pc,sizeof(u32),&new_inst,true)' will not write 
>> anything. Then, there is no reason for guest to hang. Right?
>
> Oh? It should inject a page fault. Maybe it doesn't, but then it 
> behaves badly :).
>
> Either way - just enable the debug tracepoints and check the exits you 
> get right after you hit an mfmsr emulation. That should give you hints 
> on what's going wrong.
I tried a couple of things. But I am still having trouble getting this 
to work.

It seems to me that problem lies in my method of shared page mapping.

Before patching any instruction, shared page needs to be mapped to guest 
effective address. I think, this is all that we need for mapping.

+        struct kvmppc_pte pte;
+        vcpu->arch.mmu.xlate(vcpu,vcpu->arch.magic_page_ea,&pte,true);
+        kvmppc_mmu_map_page(vcpu,&pte);

And, then we can start with patching.

--- a/arch/powerpc/kvm/book3s_emulate.c
+++ b/arch/powerpc/kvm/book3s_emulate.c
@@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, 
struct kvm_vcpu *vcpu,
                 case OP_31_XOP_MFMSR:
                         kvmppc_set_gpr(vcpu, get_rt(inst),
                                        vcpu->arch.shared->msr);
+
+                       pc = kvmppc_get_pc(vcpu);
+                       //Generate new instruction
+                       addr= (-4096L) + offsetof(struct 
kvm_vcpu_arch_shared, msr);
+                       u32 rt1 = inst & 0x03e00000;
+                       u32 new_inst=0x80000000 | rt1 | ((addr + 4) & 
0x0000fffc);
+
+                       //patch only if magic page is mapped; try 
patching only for 0x7c0000a6 inst
+                       if (vcpu->arch.magic_page_pa=0xfffff000 && 
inst=0x7c0000a6) {
+                                       
kvmppc_st(vcpu,&pc,sizeof(u32),&new_inst,true);
+                                       }
+


Thanks,
Dushyant

Re: Patching guest kernel code for better performance from HOST

[Dushyant Bansal]

Hi,

On patching 'mfmsr' instruction with 'lwz', guest exits when it tries to 
execute that 'lwz' instruction. I am looking for possible causes for 
this exit.

Here are the details:
Initially,
pc: 0xc0019420, instruction: 0x7c0000a6 [mfmsr r0]
As this is a privileged instruction, this causes an exit.

qemu-system-ppc-4443  [000] 19733.740013: kvm_book3s_exit: exit=0x700 | 
pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | dar=0xe1736a00 | 
srr1=0x100000000004d032
  qemu-system-ppc-4443  [000] 19733.740029: kvm_book3s_patch: return=0 | 
pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | new_inst=0x8000f05c
  qemu-system-ppc-4443  [000] 19733.740030: kvm_ppc_instr: inst 
2080374950 pc 0xc0019420 emulate 0
  qemu-system-ppc-4443  [000] 19733.740037: kvm_book3s_reenter: reentry 
r=1 | pc=0xc0019420

I patched this instruction with:
0x8000f05c:     lwz    r0, -4096(offset of msr)
This instruction reads the 'msr' field of the magic page into register r0.

Then, I do not increment the pc value, so the guest starts at the same 
pc which now points to the new patched instruction.

This 'lwz' instruction is causing a exit due to 
'BOOK3S_INTERRUPT_PROGRAM' (exit_nr: 0x700).
What could be the reason for this exit? As, 'lwz' is not a privileged 
instruction, I am unable to think of any reason.

  qemu-system-ppc-4443  [000] 19733.740040: kvm_book3s_exit: exit=0x700 
| pc=0xc0019420 | inst=0x8000f05c | msr=0x1032 | dar=0xf00000000105d720 
| srr1=0x100000000004d032
  qemu-system-ppc-4443  [000] 19733.740042: kvm_ppc_instr: inst 
2147545180 pc 0xc0019420 emulate 1
  qemu-system-ppc-4443  [000] 19733.740043: kvm_book3s_reenter: reentry 
r=3 | pc=0xc0019424
  qemu-system-ppc-4443  [000] 19733.740046: kvm_userspace_exit: reason 
KVM_EXIT_MMIO (6)


Any help/clue would be greatly appreciated.

Thanks,
Dushyant

Re: Patching guest kernel code for better performance from HOST

[Alexander Graf]

On 07.05.2011, at 22:32, Dushyant Bansal wrote:

> Hi,
> 
> On patching 'mfmsr' instruction with 'lwz', guest exits when it tries to execute that 'lwz' instruction. I am looking for possible causes for this exit.
> 
> Here are the details:
> Initially,
> pc: 0xc0019420, instruction: 0x7c0000a6 [mfmsr r0]
> As this is a privileged instruction, this causes an exit.
> 
> qemu-system-ppc-4443  [000] 19733.740013: kvm_book3s_exit: exit=0x700 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | dar=0xe1736a00 | srr1=0x100000000004d032
> qemu-system-ppc-4443  [000] 19733.740029: kvm_book3s_patch: return=0 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | new_inst=0x8000f05c
> qemu-system-ppc-4443  [000] 19733.740030: kvm_ppc_instr: inst 2080374950 pc 0xc0019420 emulate 0
> qemu-system-ppc-4443  [000] 19733.740037: kvm_book3s_reenter: reentry r=1 | pc=0xc0019420
> 
> I patched this instruction with:
> 0x8000f05c:     lwz    r0, -4096(offset of msr)
> This instruction reads the 'msr' field of the magic page into register r0.
> 
> Then, I do not increment the pc value, so the guest starts at the same pc which now points to the new patched instruction.
> 
> This 'lwz' instruction is causing a exit due to 'BOOK3S_INTERRUPT_PROGRAM' (exit_nr: 0x700).
> What could be the reason for this exit? As, 'lwz' is not a privileged instruction, I am unable to think of any reason.

Did you flush the icache after you patched the instruction? See the function flush_icache_range. Without, your CPU still has the old instruction in its cache, making it trap again :).


Alex

Re: Patching guest kernel code for better performance from HOST

[Alexander Graf]

On 09.05.2011, at 12:34, Dushyant Bansal wrote:

> On Sunday 08 May 2011 02:22 AM, Alexander Graf wrote:
>> On 07.05.2011, at 22:32, Dushyant Bansal wrote:
>> 
>>> Hi,
>>> 
>>> On patching 'mfmsr' instruction with 'lwz', guest exits when it tries to execute that 'lwz' instruction. I am looking for possible causes for this exit.
>>> 
>>> Here are the details:
>>> Initially,
>>> pc: 0xc0019420, instruction: 0x7c0000a6 [mfmsr r0]
>>> As this is a privileged instruction, this causes an exit.
>>> 
>>> qemu-system-ppc-4443  [000] 19733.740013: kvm_book3s_exit: exit=0x700 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | dar=0xe1736a00 | srr1=0x100000000004d032
>>> qemu-system-ppc-4443  [000] 19733.740029: kvm_book3s_patch: return=0 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | new_inst=0x8000f05c
>>> qemu-system-ppc-4443  [000] 19733.740030: kvm_ppc_instr: inst 2080374950 pc 0xc0019420 emulate 0
>>> qemu-system-ppc-4443  [000] 19733.740037: kvm_book3s_reenter: reentry r=1 | pc=0xc0019420
>>> 
>>> I patched this instruction with:
>>> 0x8000f05c:     lwz    r0, -4096(offset of msr)
>>> This instruction reads the 'msr' field of the magic page into register r0.
>>> 
>>> Then, I do not increment the pc value, so the guest starts at the same pc which now points to the new patched instruction.
>>> 
>>> This 'lwz' instruction is causing a exit due to 'BOOK3S_INTERRUPT_PROGRAM' (exit_nr: 0x700).
>>> What could be the reason for this exit? As, 'lwz' is not a privileged instruction, I am unable to think of any reason.
>> Did you flush the icache after you patched the instruction? See the function flush_icache_range. Without, your CPU still has the old instruction in its cache, making it trap again :).
> Thanks.
> 
> I tried     flush_icache_range((ulong)pc, (ulong)pc + 4);
> The system becomes unresponsive and I have to use force shut down.
> 
> Here, pc will have the address of guest instruction and flush_icache_range is called from host. Maybe, I am not using flush_icache_range in the correct way.
> Also, my host os is ppc64 and the guest is ppc32.
> 
> I also tried:  flush_cache_all()
> But the instruction is still present in the instruction cache.

Just patch the _st function to flush the icache on the host virtual address every time it gets invoked :).


Alex

Re: Patching guest kernel code for better performance from HOST

[Dushyant Bansal]

On Sunday 08 May 2011 02:22 AM, Alexander Graf wrote:
> On 07.05.2011, at 22:32, Dushyant Bansal wrote:
>
>> Hi,
>>
>> On patching 'mfmsr' instruction with 'lwz', guest exits when it tries to execute that 'lwz' instruction. I am looking for possible causes for this exit.
>>
>> Here are the details:
>> Initially,
>> pc: 0xc0019420, instruction: 0x7c0000a6 [mfmsr r0]
>> As this is a privileged instruction, this causes an exit.
>>
>> qemu-system-ppc-4443  [000] 19733.740013: kvm_book3s_exit: exit=0x700 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | dar=0xe1736a00 | srr1=0x100000000004d032
>> qemu-system-ppc-4443  [000] 19733.740029: kvm_book3s_patch: return=0 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | new_inst=0x8000f05c
>> qemu-system-ppc-4443  [000] 19733.740030: kvm_ppc_instr: inst 2080374950 pc 0xc0019420 emulate 0
>> qemu-system-ppc-4443  [000] 19733.740037: kvm_book3s_reenter: reentry r=1 | pc=0xc0019420
>>
>> I patched this instruction with:
>> 0x8000f05c:     lwz    r0, -4096(offset of msr)
>> This instruction reads the 'msr' field of the magic page into register r0.
>>
>> Then, I do not increment the pc value, so the guest starts at the same pc which now points to the new patched instruction.
>>
>> This 'lwz' instruction is causing a exit due to 'BOOK3S_INTERRUPT_PROGRAM' (exit_nr: 0x700).
>> What could be the reason for this exit? As, 'lwz' is not a privileged instruction, I am unable to think of any reason.
> Did you flush the icache after you patched the instruction? See the function flush_icache_range. Without, your CPU still has the old instruction in its cache, making it trap again :).
Thanks.

I tried     flush_icache_range((ulong)pc, (ulong)pc + 4);
The system becomes unresponsive and I have to use force shut down.

Here, pc will have the address of guest instruction and 
flush_icache_range is called from host. Maybe, I am not using 
flush_icache_range in the correct way.
Also, my host os is ppc64 and the guest is ppc32.

I also tried:  flush_cache_all()
But the instruction is still present in the instruction cache.


Dushyant

Re: Patching guest kernel code for better performance from HOST

[Dushyant Bansal]

On Monday 09 May 2011 04:08 PM, Alexander Graf wrote:
> On 09.05.2011, at 12:34, Dushyant Bansal wrote:
>
>> On Sunday 08 May 2011 02:22 AM, Alexander Graf wrote:
>>> On 07.05.2011, at 22:32, Dushyant Bansal wrote:
>>>
>>>> Hi,
>>>>
>>>> On patching 'mfmsr' instruction with 'lwz', guest exits when it tries to execute that 'lwz' instruction. I am looking for possible causes for this exit.
>>>>
>>>> Here are the details:
>>>> Initially,
>>>> pc: 0xc0019420, instruction: 0x7c0000a6 [mfmsr r0]
>>>> As this is a privileged instruction, this causes an exit.
>>>>
>>>> qemu-system-ppc-4443  [000] 19733.740013: kvm_book3s_exit: exit=0x700 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | dar=0xe1736a00 | srr1=0x100000000004d032
>>>> qemu-system-ppc-4443  [000] 19733.740029: kvm_book3s_patch: return=0 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | new_inst=0x8000f05c
>>>> qemu-system-ppc-4443  [000] 19733.740030: kvm_ppc_instr: inst 2080374950 pc 0xc0019420 emulate 0
>>>> qemu-system-ppc-4443  [000] 19733.740037: kvm_book3s_reenter: reentry r=1 | pc=0xc0019420
>>>>
>>>> I patched this instruction with:
>>>> 0x8000f05c:     lwz    r0, -4096(offset of msr)
>>>> This instruction reads the 'msr' field of the magic page into register r0.
>>>>
>>>> Then, I do not increment the pc value, so the guest starts at the same pc which now points to the new patched instruction.
>>>>
>>>> This 'lwz' instruction is causing a exit due to 'BOOK3S_INTERRUPT_PROGRAM' (exit_nr: 0x700).
>>>> What could be the reason for this exit? As, 'lwz' is not a privileged instruction, I am unable to think of any reason.
>>> Did you flush the icache after you patched the instruction? See the function flush_icache_range. Without, your CPU still has the old instruction in its cache, making it trap again :).
>> Thanks.
>>
>> I tried     flush_icache_range((ulong)pc, (ulong)pc + 4);
>> The system becomes unresponsive and I have to use force shut down.
>>
>> Here, pc will have the address of guest instruction and flush_icache_range is called from host. Maybe, I am not using flush_icache_range in the correct way.
>> Also, my host os is ppc64 and the guest is ppc32.
>>
>> I also tried:  flush_cache_all()
>> But the instruction is still present in the instruction cache.
> Just patch the _st function to flush the icache on the host virtual address every time it gets invoked :).
Sorry for replying so late.
It worked. Thanks a lot :)


Dushyant