Re: Variation in measure_migration_cost() with scheduler-cache-hot-autodetect.patch in -mm

[Ingo Molnar <mingo@elte.hu>]

* Chen, Kenneth W <kenneth.w.chen@intel.com> wrote:

> I'm consistently getting a smaller than expected cache migration cost 
> as measured by Ingo's scheduler-cache-hot-autodetect.patch currently 
> in -mm tree.  In this patch, the memory used to calibrate migration 
> cost is obtained by vmalloc call.  Would it make sense to use 
> __get_free_pages() instead?  I did the following experiments on a 
> variety of machines I have access to:
> 
> 				migration cost		migration cost
> 				with vmalloc mem		with __get_free_pages
> 3.0GHz Xeon, 8MB cache		6.23 ms		6.32 ms
> 3.4GHz Xeon, 2MB cache		1.62 ms		2.00 ms
> 1.6GHz Itanium2, 9MB		9.2 ms		10.2 ms
> 1.4GHz Itanium2, 4MB		4.2 ms		 4.4 ms
> 
> Why the discrepancy?  Possible cache coloring issue?

probably coloring effects, yes. Another reason could be that 
touch_cache() touches 6 separate areas of memory, which combined with 
the stack give a minimum of 7 hugepage TLBs. How many are there in these 
Xeons? If there are say 4 of them then we could be trashing these TLB 
entries. There are much more 4K TLBs. To reduce the number of TLBs
utilized, could you change touch_cache() to do something like:

        unsigned long size = __size/sizeof(long), chunk1 = size/2;
        unsigned long *cache = __cache;
        int i;

        for (i = 0; i < size/4; i += 8) {
                switch (i % 4) {
                        case 0: cache[i]++;
                        case 1: cache[size-1-i]++;
                        case 2: cache[chunk1-i]++;
                        case 3: cache[chunk1+i]++;
                }
        }

does this change the migration-cost values? Btw., how did you determine 
the value of the 'ideal' migration cost? Was this based on the database 
benchmark measurements?

There are a couple of reasons vmalloc() is better than gfp(): 1) it has 
no size limit in the measured range, and 2) it more accurately mimics 
migration costs of userspace apps, which typically have most of their 
cache-footprint in paged memory, not in hugepage memory.

	Ingo