[PATCH] Allow removing writable mappings from splintered page tables.

[PATCH] Allow removing writable mappings from splintered page tables.

[Gianluca Guida]

[-- Attachment #1: Type: text/plain, Size: 632 bytes --]

Hello,

The moving of the pagetable mapping in the linux kernel exposed the fact 
that under the linux kernel sh_rm_write_access_from_sl1p was always failing.

Linux seems to use big pages to access page tables, so we should 
instruct the shadow code to be able to remove writable mappings from 
splintered pagetables as well, avoiding using OS heuristic (which were 
failing in 2.6.27 before George patch, leading to brute-force search at 
each resync).

Todd: Can you try try the linux-2.6.27 kernbench with the following patch?
Also, are you using ioemu stub-domains? That should also give a 
performance gain.

Thanks,
Gianluca

[-- Attachment #2: fix_linux_hvm.patch --]
[-- Type: text/x-patch, Size: 1719 bytes --]

diff -r dbac9ee4d761 xen/arch/x86/mm/shadow/common.c
--- a/xen/arch/x86/mm/shadow/common.c	Mon Sep 08 16:02:13 2008 +0100
+++ b/xen/arch/x86/mm/shadow/common.c	Fri Sep 12 17:40:31 2008 +0100
@@ -2462,22 +2462,25 @@ int sh_remove_write_access_from_sl1p(str
     ASSERT(mfn_valid(smfn));
     ASSERT(mfn_valid(gmfn));
     
-    if ( sp->type == SH_type_l1_32_shadow )
+    if ( sp->type == SH_type_l1_32_shadow
+         || sp->type == SH_type_fl1_32_shadow )
     {
         return SHADOW_INTERNAL_NAME(sh_rm_write_access_from_sl1p,2)
             (v, gmfn, smfn, off);
     }
 #if CONFIG_PAGING_LEVELS >= 3
-    else if ( sp->type == SH_type_l1_pae_shadow )
+    else if ( sp->type == SH_type_l1_pae_shadow
+              || sp->type == SH_type_fl1_pae_shadow )
         return SHADOW_INTERNAL_NAME(sh_rm_write_access_from_sl1p,3)
             (v, gmfn, smfn, off);
 #if CONFIG_PAGING_LEVELS >= 4
-    else if ( sp->type == SH_type_l1_64_shadow )
+    else if ( sp->type == SH_type_l1_64_shadow
+              || sp->type == SH_type_fl1_64_shadow )
         return SHADOW_INTERNAL_NAME(sh_rm_write_access_from_sl1p,4)
             (v, gmfn, smfn, off);
 #endif
 #endif
-    
+
     return 0;
 }
 #endif 
diff -r dbac9ee4d761 xen/arch/x86/mm/shadow/multi.c
--- a/xen/arch/x86/mm/shadow/multi.c	Mon Sep 08 16:02:13 2008 +0100
+++ b/xen/arch/x86/mm/shadow/multi.c	Fri Sep 12 17:40:31 2008 +0100
@@ -4539,7 +4539,8 @@ int sh_rm_write_access_from_sl1p(struct 
     sp = mfn_to_shadow_page(smfn);
 
     if ( sp->mbz != 0
-         || (sp->type != SH_type_l1_shadow) )
+         || (sp->type != SH_type_l1_shadow
+             && sp->type != SH_type_fl1_shadow) )
         goto fail;
 
     sl1p = sh_map_domain_page(smfn);

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Allow removing writable mappings from splintered page tables.

[Todd Deshane]

On Fri, Sep 12, 2008 at 12:51 PM, Gianluca Guida
<gianluca.guida@eu.citrix.com> wrote:
> Hello,
>
> The moving of the pagetable mapping in the linux kernel exposed the fact
> that under the linux kernel sh_rm_write_access_from_sl1p was always failing.
>
> Linux seems to use big pages to access page tables, so we should instruct
> the shadow code to be able to remove writable mappings from splintered
> pagetables as well, avoiding using OS heuristic (which were failing in
> 2.6.27 before George patch, leading to brute-force search at each resync).
>
> Todd: Can you try try the linux-2.6.27 kernbench with the following patch?
> Also, are you using ioemu stub-domains? That should also give a performance
> gain.
>

Sure, I can give it a try. Should I also apply George's patch right?

I am not using ioemu stub-domains yet, would you recommend them in a general
sense or are they more of a special case?

Cheers,
Todd

Re: [PATCH] Allow removing writable mappings from splintered page tables.

[Gianluca Guida]

Todd Deshane wrote:
>> Todd: Can you try try the linux-2.6.27 kernbench with the following patch?
>> Also, are you using ioemu stub-domains? That should also give a performance
>> gain.
>>
> 
> Sure, I can give it a try. Should I also apply George's patch right?

Yes.

> I am not using ioemu stub-domains yet, would you recommend them in a general
> sense or are they more of a special case?

I think they're in general recommendable. Both Samuel and Stefano will 
be able to give more information about it.

Thanks,
Gianluca

Re: Re: [PATCH] Allow removing writable mappings from splintered page tables.

[Samuel Thibault]

Gianluca Guida, le Fri 12 Sep 2008 18:14:17 +0100, a écrit :
> >I am not using ioemu stub-domains yet, would you recommend them in a 
> >general
> >sense or are they more of a special case?
> 
> I think they're in general recommendable. Both Samuel and Stefano will 
> be able to give more information about it.

That brings more security and performance, so yes it's recommended :)

Samuel

Re: [PATCH] Allow removing writable mappings from splintered page tables.

[Todd Deshane]

On Fri, Sep 12, 2008 at 1:14 PM, Gianluca Guida
<gianluca.guida@eu.citrix.com> wrote:
>
>
> Todd Deshane wrote:
>>>
>>> Todd: Can you try try the linux-2.6.27 kernbench with the following
>>> patch?
>>> Also, are you using ioemu stub-domains? That should also give a
>>> performance
>>> gain.
>>>
>>
>> Sure, I can give it a try. Should I also apply George's patch right?
>

Much faster, I'll also test on debian 4.0 for a regression test.

Average Optimal load -j 4 Run (std deviation):
Elapsed Time 519.056 (0.098387)
User Time 426.226 (0.815555)
System Time 92.716 (0.858563)
Percent CPU 99 (0)
Context Switches 99125.2 (71.3667)
Sleeps 31199.2 (41.3062)

I also want to look into the ioemu stub domains after reading more about them.

I'll report the debian (older kernel results) when they come out.

Todd

Re: [PATCH] Allow removing writable mappings from splintered page tables.

[Todd Deshane]

> I'll report the debian (older kernel results) when they come out.

Roughly the same as before, so the patches look good.

HVM guest (Debian 4.0 vanilla 2.6.18.6-amd64)

Average Optimal load -j 4 Run (std deviation):
Elapsed Time 492.826 (1.89446)
User Time 390.316 (0.571341)
System Time 101.396 (2.09772)
Percent CPU 99 (0)
Context Switches 29514.4 (53.8266)
Sleeps 37991.4 (572.437)




-- 
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com

Re: Re: [PATCH] Allow removing writable mappings from splintered page tables.

[George Dunlap]

And your original numbers showed elapsed time to be 527s for KVM, so
now Xen is 8 seconds in the lead for HVM Linux. :-)  Thanks for the
help tracking this down!

If you have time, could you take another 30-second trace with the new
changes in, just for fun?  I'll take a quick look and see if there's
any other low-hanging fruit to grab.

 -George

On Fri, Sep 12, 2008 at 8:52 PM, Todd Deshane <deshantm@gmail.com> wrote:
> On Fri, Sep 12, 2008 at 1:14 PM, Gianluca Guida
> <gianluca.guida@eu.citrix.com> wrote:
>>
>>
>> Todd Deshane wrote:
>>>>
>>>> Todd: Can you try try the linux-2.6.27 kernbench with the following
>>>> patch?
>>>> Also, are you using ioemu stub-domains? That should also give a
>>>> performance
>>>> gain.
>>>>
>>>
>>> Sure, I can give it a try. Should I also apply George's patch right?
>>
>
> Much faster, I'll also test on debian 4.0 for a regression test.
>
> Average Optimal load -j 4 Run (std deviation):
> Elapsed Time 519.056 (0.098387)
> User Time 426.226 (0.815555)
> System Time 92.716 (0.858563)
> Percent CPU 99 (0)
> Context Switches 99125.2 (71.3667)
> Sleeps 31199.2 (41.3062)
>
> I also want to look into the ioemu stub domains after reading more about them.
>
> I'll report the debian (older kernel results) when they come out.
>
> Todd
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Re: [PATCH] Allow removing writable mappings from splintered page tables.

[Todd Deshane]

On Mon, Sep 15, 2008 at 6:38 AM, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
> And your original numbers showed elapsed time to be 527s for KVM, so
> now Xen is 8 seconds in the lead for HVM Linux. :-)  Thanks for the
> help tracking this down!
>

KVM is also working on improved page table algorithms
http://www.mail-archive.com/kvm@vger.kernel.org/msg03562.html

I think the competition is a good thing.

> If you have time, could you take another 30-second trace with the new
> changes in, just for fun?  I'll take a quick look and see if there's
> any other low-hanging fruit to grab.
>

Sent the trace to you with another service called sendspace, since, for
some reason, the trace file was much bigger.

Todd

-- 
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com

Re: Re: [PATCH] Allow removing writable mappings from splintered page tables.

[George Dunlap]

Heh... the blatant copying is flattering and annoying at the same
time. :-)  Ah, the beauty of open-source...

I've got your trace, and I'll take a look at it tomorrow. Thanks!

 -George

On Mon, Sep 15, 2008 at 5:30 PM, Todd Deshane <deshantm@gmail.com> wrote:
> On Mon, Sep 15, 2008 at 6:38 AM, George Dunlap
> <George.Dunlap@eu.citrix.com> wrote:
>> And your original numbers showed elapsed time to be 527s for KVM, so
>> now Xen is 8 seconds in the lead for HVM Linux. :-)  Thanks for the
>> help tracking this down!
>>
>
> KVM is also working on improved page table algorithms
> http://www.mail-archive.com/kvm@vger.kernel.org/msg03562.html
>
> I think the competition is a good thing.
>
>> If you have time, could you take another 30-second trace with the new
>> changes in, just for fun?  I'll take a quick look and see if there's
>> any other low-hanging fruit to grab.
>>
>
> Sent the trace to you with another service called sendspace, since, for
> some reason, the trace file was much bigger.
>
> Todd
>
> --
> Todd Deshane
> http://todddeshane.net
> check out our book: http://runningxen.com
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Re: [PATCH] Allow removing writable mappings from splintered page tables.

[George Dunlap]

Hmm, no really obvious low-hanging fruit.  The Xen-HVM was about 9%
slower than your reported numbers for Xen-PV, and the trace shows that
the guest spent about that much inside the hypervisor.  The breakdown:
* 3.6% propagating page faults to guest
* 3.0% pulling through entries from out-of-sync guest pt's to shadow pagetables
* 1.4% Marking pages out of sync (of which 75% was in unsyncs that had
to re-sync another page)
* 0.9% cr3 switches
* 0.9% handling I/O

(Rounding may cause the numbers not to add up exactly.)

So one of the biggest things, really, is that Linux seems to insist on
mapping pages one-at-a-time as they're demand-faulted, rather than
doing a batch of them.  Unfortunately, having pages out-of-sync means
that we must use the slow propagate path rather than the
fast-propagate path, which is at least 25% slower.

The only avenues for optimization I can see are:
* See if there's a way to reduce the number of unsyncs that cause
resyncs.  Allowing more pages to go out-of-sync *might* do this; or it
might just shift the same overhead into cr3 switch.
* Reduce the time of "hot paths" through the hypervisor by profiling, &c.

 -George

On Mon, Sep 15, 2008 at 6:03 PM, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
> Heh... the blatant copying is flattering and annoying at the same
> time. :-)  Ah, the beauty of open-source...
>
> I've got your trace, and I'll take a look at it tomorrow. Thanks!
>
>  -George
>
> On Mon, Sep 15, 2008 at 5:30 PM, Todd Deshane <deshantm@gmail.com> wrote:
>> On Mon, Sep 15, 2008 at 6:38 AM, George Dunlap
>> <George.Dunlap@eu.citrix.com> wrote:
>>> And your original numbers showed elapsed time to be 527s for KVM, so
>>> now Xen is 8 seconds in the lead for HVM Linux. :-)  Thanks for the
>>> help tracking this down!
>>>
>>
>> KVM is also working on improved page table algorithms
>> http://www.mail-archive.com/kvm@vger.kernel.org/msg03562.html
>>
>> I think the competition is a good thing.
>>
>>> If you have time, could you take another 30-second trace with the new
>>> changes in, just for fun?  I'll take a quick look and see if there's
>>> any other low-hanging fruit to grab.
>>>
>>
>> Sent the trace to you with another service called sendspace, since, for
>> some reason, the trace file was much bigger.
>>
>> Todd
>>
>> --
>> Todd Deshane
>> http://todddeshane.net
>> check out our book: http://runningxen.com
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>