[PATCH] clflush instruction emulation

[PATCH] clflush instruction emulation

[Wei Huang]

[-- Attachment #1: Type: text/plain, Size: 874 bytes --]

We recently found that FreeBSD 8.0 guest failed to install and boot on 
Xen. The reason was that FreeBSD detected clflush feature and invoked 
this instruction to flush MMIO space. This caused a page fault; but 
x86_emulate.c failed to emulate this instruction (not supported). As a 
result, a page fault was detected inside FreeBSD. A similar issue was 
reported earlier.

http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00362.html

I created a patch which enables clflush emulation. I have verified that 
this patch solves FreeBSD issue. This patch returns immediately without 
doing anything. Note that we don't have to intercept clflush in SVM. So 
the only time we have page_fault for clflush instruction is when guest 
VM flushes MMIO space. In this case, do we need to send this command 
over to QEMU? I didn't do it in this patch anyway.

Best,
-Wei



[-- Attachment #2: xen_clflush_emul.txt --]
[-- Type: text/plain, Size: 1023 bytes --]

diff -r 12a610b600b0 xen/arch/x86/x86_emulate/x86_emulate.c
--- a/xen/arch/x86/x86_emulate/x86_emulate.c	Wed Apr 14 11:50:00 2010 -0500
+++ b/xen/arch/x86/x86_emulate/x86_emulate.c	Wed Apr 14 16:11:36 2010 -0500
@@ -227,7 +227,8 @@
     DstMem|SrcReg|ModRM, DstMem|SrcReg|ModRM, 0, 0,
     /* 0xA8 - 0xAF */
     ImplicitOps, ImplicitOps, 0, DstBitBase|SrcReg|ModRM,
-    DstMem|SrcReg|ModRM, DstMem|SrcReg|ModRM, 0, DstReg|SrcMem|ModRM,
+    DstMem|SrcReg|ModRM, DstMem|SrcReg|ModRM, DstMem|SrcReg|ModRM, 
+    DstReg|SrcMem|ModRM,
     /* 0xB0 - 0xB7 */
     ByteOp|DstMem|SrcReg|ModRM, DstMem|SrcReg|ModRM,
     DstReg|SrcMem|ModRM|Mov, DstBitBase|SrcReg|ModRM,
@@ -3948,6 +3949,11 @@
         src.val = x86_seg_gs;
         goto pop_seg;
 
+    case 0xae: /* clflush mem8 */
+        /* we don't need to do anything here */
+        rc = X86EMUL_OKAY;
+        break;
+
     case 0xb0 ... 0xb1: /* cmpxchg */
         /* Save real source value, then compare EAX against destination. */
         src.orig_val = src.val;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] clflush instruction emulation

[Keir Fraser]

On 15/04/2010 17:38, "Wei Huang" <wei.huang2@amd.com> wrote:

> I created a patch which enables clflush emulation. I have verified that
> this patch solves FreeBSD issue. This patch returns immediately without
> doing anything. Note that we don't have to intercept clflush in SVM. So
> the only time we have page_fault for clflush instruction is when guest
> VM flushes MMIO space. In this case, do we need to send this command
> over to QEMU? I didn't do it in this patch anyway.

I don't think we need to anything since all guest reads/writes to the mmio
space will get intercepted to qemu, and hence qemu's view of the space is
coherent with the guest with no need for special handling of CLFLUSH.

Still the emulator should call out and let the caller decide what to do. We
could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook).
Do you know if these CLFLUSH emulations happen often or only very rarely?
That might help us decide how smart we need to be about emulating CLFLUSH
(i.e., add a hook specifically for clflush, separately from wbinvd).

 -- Keir

Re: [PATCH] clflush instruction emulation

[Wei Huang]

WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of 
CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170 
is not huge though.

-Wei

Keir Fraser wrote:
> On 15/04/2010 17:38, "Wei Huang" <wei.huang2@amd.com> wrote:
>
>   
>> I created a patch which enables clflush emulation. I have verified that
>> this patch solves FreeBSD issue. This patch returns immediately without
>> doing anything. Note that we don't have to intercept clflush in SVM. So
>> the only time we have page_fault for clflush instruction is when guest
>> VM flushes MMIO space. In this case, do we need to send this command
>> over to QEMU? I didn't do it in this patch anyway.
>>     
>
> I don't think we need to anything since all guest reads/writes to the mmio
> space will get intercepted to qemu, and hence qemu's view of the space is
> coherent with the guest with no need for special handling of CLFLUSH.
>
> Still the emulator should call out and let the caller decide what to do. We
> could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook).
> Do you know if these CLFLUSH emulations happen often or only very rarely?
> That might help us decide how smart we need to be about emulating CLFLUSH
> (i.e., add a hook specifically for clflush, separately from wbinvd).
>
>  -- Keir
>
>
>
>   

Re: [PATCH] clflush instruction emulation

[Keir Fraser]

Our WBINVD emulation doesn't do anything unless the guest has some devices
passed through to it. I think that will do for now then.

 Thanks,
 Keir

On 15/04/2010 18:25, "Wei Huang" <wei.huang2@amd.com> wrote:

> WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of
> CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170
> is not huge though.
> 
> -Wei
> 
> Keir Fraser wrote:
>> On 15/04/2010 17:38, "Wei Huang" <wei.huang2@amd.com> wrote:
>> 
>>   
>>> I created a patch which enables clflush emulation. I have verified that
>>> this patch solves FreeBSD issue. This patch returns immediately without
>>> doing anything. Note that we don't have to intercept clflush in SVM. So
>>> the only time we have page_fault for clflush instruction is when guest
>>> VM flushes MMIO space. In this case, do we need to send this command
>>> over to QEMU? I didn't do it in this patch anyway.
>>>     
>> 
>> I don't think we need to anything since all guest reads/writes to the mmio
>> space will get intercepted to qemu, and hence qemu's view of the space is
>> coherent with the guest with no need for special handling of CLFLUSH.
>> 
>> Still the emulator should call out and let the caller decide what to do. We
>> could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook).
>> Do you know if these CLFLUSH emulations happen often or only very rarely?
>> That might help us decide how smart we need to be about emulating CLFLUSH
>> (i.e., add a hook specifically for clflush, separately from wbinvd).
>> 
>>  -- Keir
>> 
>> 
>> 
>>   
> 
> 

Re: [PATCH] clflush instruction emulation

[Keir Fraser]

See what you think of xen-unstable:21189. It is also a bit more precise in
its decoding of CLFLUSH, as a member of Grp15.

 -- Keir

On 15/04/2010 18:32, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

> Our WBINVD emulation doesn't do anything unless the guest has some devices
> passed through to it. I think that will do for now then.
> 
>  Thanks,
>  Keir
> 
> On 15/04/2010 18:25, "Wei Huang" <wei.huang2@amd.com> wrote:
> 
>> WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of
>> CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170
>> is not huge though.
>> 
>> -Wei
>> 
>> Keir Fraser wrote:
>>> On 15/04/2010 17:38, "Wei Huang" <wei.huang2@amd.com> wrote:
>>> 
>>>   
>>>> I created a patch which enables clflush emulation. I have verified that
>>>> this patch solves FreeBSD issue. This patch returns immediately without
>>>> doing anything. Note that we don't have to intercept clflush in SVM. So
>>>> the only time we have page_fault for clflush instruction is when guest
>>>> VM flushes MMIO space. In this case, do we need to send this command
>>>> over to QEMU? I didn't do it in this patch anyway.
>>>>     
>>> 
>>> I don't think we need to anything since all guest reads/writes to the mmio
>>> space will get intercepted to qemu, and hence qemu's view of the space is
>>> coherent with the guest with no need for special handling of CLFLUSH.
>>> 
>>> Still the emulator should call out and let the caller decide what to do. We
>>> could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook).
>>> Do you know if these CLFLUSH emulations happen often or only very rarely?
>>> That might help us decide how smart we need to be about emulating CLFLUSH
>>> (i.e., add a hook specifically for clflush, separately from wbinvd).
>>> 
>>>  -- Keir
>>> 
>>> 
>>> 
>>>   
>> 
>> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: [PATCH] clflush instruction emulation

[Huang2, Wei]

I can confirm that 21189 fixed FreeBSD issue.

Thanks,
-Wei

-----Original Message-----
From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
Sent: Thursday, April 15, 2010 12:50 PM
To: Huang2, Wei
Cc: 'xen-devel@lists.xensource.com'
Subject: Re: [Xen-devel] [PATCH] clflush instruction emulation

See what you think of xen-unstable:21189. It is also a bit more precise in
its decoding of CLFLUSH, as a member of Grp15.

 -- Keir

On 15/04/2010 18:32, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

> Our WBINVD emulation doesn't do anything unless the guest has some devices
> passed through to it. I think that will do for now then.
> 
>  Thanks,
>  Keir
> 
> On 15/04/2010 18:25, "Wei Huang" <wei.huang2@amd.com> wrote:
> 
>> WBINVD is a bit heavy since it flushes all caches. I saw 1170 times of
>> CLFLUSH during FreeBSD booting. After guest boots, 0 was observed. 1170
>> is not huge though.
>> 
>> -Wei
>> 
>> Keir Fraser wrote:
>>> On 15/04/2010 17:38, "Wei Huang" <wei.huang2@amd.com> wrote:
>>> 
>>>   
>>>> I created a patch which enables clflush emulation. I have verified that
>>>> this patch solves FreeBSD issue. This patch returns immediately without
>>>> doing anything. Note that we don't have to intercept clflush in SVM. So
>>>> the only time we have page_fault for clflush instruction is when guest
>>>> VM flushes MMIO space. In this case, do we need to send this command
>>>> over to QEMU? I didn't do it in this patch anyway.
>>>>     
>>> 
>>> I don't think we need to anything since all guest reads/writes to the mmio
>>> space will get intercepted to qemu, and hence qemu's view of the space is
>>> coherent with the guest with no need for special handling of CLFLUSH.
>>> 
>>> Still the emulator should call out and let the caller decide what to do. We
>>> could safely turn CLFLUSH into WBINVD (i.e., call the existing wbinvd hook).
>>> Do you know if these CLFLUSH emulations happen often or only very rarely?
>>> That might help us decide how smart we need to be about emulating CLFLUSH
>>> (i.e., add a hook specifically for clflush, separately from wbinvd).
>>> 
>>>  -- Keir
>>> 
>>> 
>>> 
>>>   
>> 
>> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel