Even faster page copy for Xen?

Even faster page copy for Xen?

[Dan Magenheimer]

Hi Jan, Keir --

My x86 assembly skills are much too poor to carefully evaluate
and, if of value, implement this in Xen but given your previous
interest, such as:

http://xenbits.xensource.com/xen-unstable.hg?rev/8de4b4e9a435 

the following might be worth looking at.

Intel has just posted memcpy improvements for glibc for recent
popular Intel processor families here:

http://article.gmane.org/gmane.comp.lib.glibc.alpha/15278 

The preface to the above patch looks very enticing...

Semi-related, I wonder if you know, if there were a
"copy_page_from_other_node()" to be used if the
caller is fairly sure that the page is being copied
between nodes, could this be made significantly faster
than a normal copy_page()?

Thanks,
Dan

Re: Even faster page copy for Xen?

[Keir Fraser]

It has to be said, possibly tmem excepted, there is very little page copying
in Xen.

 -- Keir

On 15/07/2010 19:15, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> Hi Jan, Keir --
> 
> My x86 assembly skills are much too poor to carefully evaluate
> and, if of value, implement this in Xen but given your previous
> interest, such as:
> 
> http://xenbits.xensource.com/xen-unstable.hg?rev/8de4b4e9a435
> 
> the following might be worth looking at.
> 
> Intel has just posted memcpy improvements for glibc for recent
> popular Intel processor families here:
> 
> http://article.gmane.org/gmane.comp.lib.glibc.alpha/15278
> 
> The preface to the above patch looks very enticing...
> 
> Semi-related, I wonder if you know, if there were a
> "copy_page_from_other_node()" to be used if the
> caller is fairly sure that the page is being copied
> between nodes, could this be made significantly faster
> than a normal copy_page()?
> 
> Thanks,
> Dan

RE: Even faster page copy for Xen?

[Dan Magenheimer]

I wasn't sure about that... Jan's patch to speed up
copy_page (by 12%) went in before tmem was in-tree,
so I assumed otherwise.  Clearly my interest is for
tmem, especially if 2x-4x improvement is possible,
but if there really is no significant advantage for
non-tmem code, I will put it on my list... for sometime
in the next century when I am a good x86 assembly
programmer :-)

> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Thursday, July 15, 2010 3:35 PM
> To: Dan Magenheimer; Jan Beulich
> Cc: xen-devel@lists.xensource.com
> Subject: Re: Even faster page copy for Xen?
> 
> It has to be said, possibly tmem excepted, there is very little page
> copying
> in Xen.
> 
>  -- Keir
> 
> On 15/07/2010 19:15, "Dan Magenheimer" <dan.magenheimer@oracle.com>
> wrote:
> 
> > Hi Jan, Keir --
> >
> > My x86 assembly skills are much too poor to carefully evaluate
> > and, if of value, implement this in Xen but given your previous
> > interest, such as:
> >
> > http://xenbits.xensource.com/xen-unstable.hg?rev/8de4b4e9a435
> >
> > the following might be worth looking at.
> >
> > Intel has just posted memcpy improvements for glibc for recent
> > popular Intel processor families here:
> >
> > http://article.gmane.org/gmane.comp.lib.glibc.alpha/15278
> >
> > The preface to the above patch looks very enticing...
> >
> > Semi-related, I wonder if you know, if there were a
> > "copy_page_from_other_node()" to be used if the
> > caller is fairly sure that the page is being copied
> > between nodes, could this be made significantly faster
> > than a normal copy_page()?
> >
> > Thanks,
> > Dan
> 
> 

Re: Even faster page copy for Xen?

[Keir Fraser]

Jan also added a copy_page hypercall, so he may have been optimising for
that, since I guess the Novell kernels must use it.

 -- Keir

On 16/07/2010 00:36, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> I wasn't sure about that... Jan's patch to speed up
> copy_page (by 12%) went in before tmem was in-tree,
> so I assumed otherwise.  Clearly my interest is for
> tmem, especially if 2x-4x improvement is possible,
> but if there really is no significant advantage for
> non-tmem code, I will put it on my list... for sometime
> in the next century when I am a good x86 assembly
> programmer :-)
> 
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>> Sent: Thursday, July 15, 2010 3:35 PM
>> To: Dan Magenheimer; Jan Beulich
>> Cc: xen-devel@lists.xensource.com
>> Subject: Re: Even faster page copy for Xen?
>> 
>> It has to be said, possibly tmem excepted, there is very little page
>> copying
>> in Xen.
>> 
>>  -- Keir
>> 
>> On 15/07/2010 19:15, "Dan Magenheimer" <dan.magenheimer@oracle.com>
>> wrote:
>> 
>>> Hi Jan, Keir --
>>> 
>>> My x86 assembly skills are much too poor to carefully evaluate
>>> and, if of value, implement this in Xen but given your previous
>>> interest, such as:
>>> 
>>> http://xenbits.xensource.com/xen-unstable.hg?rev/8de4b4e9a435
>>> 
>>> the following might be worth looking at.
>>> 
>>> Intel has just posted memcpy improvements for glibc for recent
>>> popular Intel processor families here:
>>> 
>>> http://article.gmane.org/gmane.comp.lib.glibc.alpha/15278
>>> 
>>> The preface to the above patch looks very enticing...
>>> 
>>> Semi-related, I wonder if you know, if there were a
>>> "copy_page_from_other_node()" to be used if the
>>> caller is fairly sure that the page is being copied
>>> between nodes, could this be made significantly faster
>>> than a normal copy_page()?
>>> 
>>> Thanks,
>>> Dan
>> 
>> 

Re: Even faster page copy for Xen?

[Jan Beulich]

>>> Keir Fraser  07/16/10 9:57 AM >>>
>Jan also added a copy_page hypercall, so he may have been optimising for
>that, since I guess the Novell kernels must use it.

Yes, that was the very goal of that change.

Jan

Re: Even faster page copy for Xen?

[Jan Beulich]

>>> On 15.07.10 at 20:15, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
> Hi Jan, Keir --
> 
> My x86 assembly skills are much too poor to carefully evaluate
> and, if of value, implement this in Xen but given your previous
> interest, such as:
> 
> http://xenbits.xensource.com/xen-unstable.hg?rev/8de4b4e9a435 
> 
> the following might be worth looking at.
> 
> Intel has just posted memcpy improvements for glibc for recent
> popular Intel processor families here:
> 
> http://article.gmane.org/gmane.comp.lib.glibc.alpha/15278 
> 
> The preface to the above patch looks very enticing...

I'm not sure how much of this applies to the much more specific
case of copying pages... Additionally, I don't think trying to
use XMM registers in Xen would be a good idea.

> Semi-related, I wonder if you know, if there were a
> "copy_page_from_other_node()" to be used if the
> caller is fairly sure that the page is being copied
> between nodes, could this be made significantly faster
> than a normal copy_page()?

I would think that this should mostly be taken care of by
using non-temporal stores (non-temporal loads unfortunately
aren't available without using XMM registers). The only other
meaningful tuning one could do would be to increase the
prefetch distances and grow the distance between loads and
stores. The latter would require the use of more registers
and hence have other drawbacks.

Jan