Re: [RFC PATCH 0/4]: affinity-on-next-touch

[Stefan Lankes <lankes@lfbs.rwth-aachen.de>]

Lee Schermerhorn wrote:
> On Mon, 2009-06-22 at 16:32 +0200, Stefan Lankes wrote:
>> Brice Goglin wrote:
>>> Lee Schermerhorn wrote:
>>>> On Wed, 2009-06-17 at 09:45 +0200, Stefan Lankes wrote:
>>>>   
>>>>
>>>> Today I rebased the migrate on fault patches to 2.6.30-mmotm-090612...
>>>> [along with my shared policy series atop which they sit in my tree].
>>>> Patches reside in:
>>>>
>>>> http://free.linux.hp.com/~lts/Patches/PageMigration/2.6.30-mmotm-090612-1220/
>>>>
>>>>   
>>> I gave this patchset a try and indeed it seems to work fine, thanks a
>>> lot. But the migration performance isn't very good. I am seeing about
>>> 540MB/s when doing mbind+touch_all_pages on large buffers on a
>>> quad-barcelona machines. move_pages gets 640MB/s there. And my own
>>> next-touch implementation were near 800MB/s in the past.
>> I used a modified stream benchmark to evaluate the performance of Lee's 
>> and my version of the next-touch implementation. In this low-level 
>> benchmark is Lee's patch better than my patch. I think that Brice and I 
>> use the same technique to realize affinity-on-next-touch. Do you use 
>> another kernel version to evaluate the performance?
> 
> Hi, Stefan:
> 
> I also used a [modified!] stream benchmark to test my patches.  One of
> the modifications was to dump the time it takes for one pass over the
> data arrays to a specific file description, if that file description was
> open at start time--e.g., via something like "4>stream_times".  Then, I
> increased the number of iterations to something large so that I could
> run other tests during the stream run.  I plotted the "time per
> iteration" vs iteration number and could see that after any transient
> load, the stream benchmark returned to a good [not sure if maximal]
> locality state.  The time per interation was comparable to hand
> affinitized of the threads.  Without automigration and hand
> affinitization, any transient load would scramble the location of the
> threads relative to the data region they were operating on due to load
> balancing.  The more nodes you have, the less likely you'll end up in a
> good state.
> 
> I was using a parallel kernel make [-j <2*nr_cpus>] as the load.  In
> addition to the stream returning to good locality, I noticed that the
> kernel build completed much faster in the presence of the stream load
> with automigration enabled.  I reported these results in a presentation
> at LCA'07.  Slides and video [yuck! :)] are available on line at the
> LCA'07 site.

I think that you use migration-on-fault in the context of automigration. 
Brice and I use affinity-on-next-touch/migration-on-fault in another 
context. If the access pattern of an application changed, we want to 
redistribute the pages in "nearly" ideal matter. Sometimes it is 
difficult to determine the ideal page distribution. In such cases, 
affinity-on-next-touch could be an attractive solution. In our test 
applications, we add at some certain points the system call to use 
affinity-on-next-touch and redistribute the pages. Assumed that the next 
thread use these pages very often, we improve the performance of our 
test applications.

Regards,

Stefan