Re: [PATCH] sched,numa: document and fix numa_preferred_nid setting

[Srikar Dronamraju <srikar@linux.vnet.ibm.com>]

> 
> OK, so we are looking at two multi-threaded processes
> on a 4 node system, and waiting for them to converge?
> 
> It may make sense to add my patch in with your patch
> 1/4 from last week, as well as the correct part of
> your patch 4/4, and see how they all work together.
> 

Tested specjbb and autonumabenchmark on 4 kernels.

Plain 4.1.0-rc7-tip (i)
tip + only Rik's patch (ii)
tip + Rik's ++ (iii)
tip + Srikar's ++ (iv)

(i) = Plain 4.1.0-rc7-tip = tip = 4.1.0-rc7 (b7ca96b)

(ii) =  tip + only Rik's patch =  (i) + Rik's fix numa_preferred_nid setting

(iii)  =  tip + Rik's ++ (iii) = (ii) + Srikar's numa hotness + correct nid for evaluating task weight

(iv) =  tip + Srikar's ++ (iv) = (i) + Srikar's  numa hotness + correct nid for evaluating task weight +
       numa_has_capacity fix +  always update preferred node


Plain 4.1.0-rc7-tip (i)
		Testcase:         Min         Max         Avg      StdDev
	  elapsed_numa01:      858.85      949.18      915.64       33.06
	  elapsed_numa02:       23.09       29.89       26.43        2.18
		Testcase:         Min         Max         Avg      StdDev
	   system_numa01:     1516.72     1855.08     1686.24      113.95
	   system_numa02:       63.69       79.06       70.35        5.87
		Testcase:         Min         Max         Avg      StdDev
	     user_numa01:    73284.76    80818.21    78060.88     2773.60
	     user_numa02:     1690.18     2071.07     1821.64      140.25
		Testcase:         Min         Max         Avg      StdDev
	    total_numa01:    74801.50    82572.60    79747.12     2875.61
	    total_numa02:     1753.87     2142.77     1891.99      143.59

tip + only Rik's patch (ii)
		Testcase:         Min         Max         Avg      StdDev     %Change
	  elapsed_numa01:      665.26      877.47      776.77       79.23      15.83%
	  elapsed_numa02:       24.59       31.30       28.17        2.48      -5.56%
		Testcase:         Min         Max         Avg      StdDev     %Change
	   system_numa01:      659.57     1220.99      942.36      234.92      60.92%
	   system_numa02:       44.62       86.01       64.64       14.24       6.64%
		Testcase:         Min         Max         Avg      StdDev     %Change
	     user_numa01:    56280.95    75908.81    64993.57     7764.30      17.21%
	     user_numa02:     1790.35     2155.02     1916.12      132.57      -4.38%
		Testcase:         Min         Max         Avg      StdDev     %Change
	    total_numa01:    56940.50    77128.20    65935.92     7993.49      17.91%
	    total_numa02:     1834.97     2227.03     1980.76      136.51      -3.99%

tip + Rik's ++ (iii)
		Testcase:         Min         Max         Avg      StdDev     %Change
	  elapsed_numa01:      630.60      860.06      760.07       74.33      18.09%
	  elapsed_numa02:       21.92       34.42       27.72        4.49      -3.75%
		Testcase:         Min         Max         Avg      StdDev     %Change
	   system_numa01:      474.31     1379.49      870.12      296.35      59.16%
	   system_numa02:       63.74      120.25       86.69       20.69     -13.59%
		Testcase:         Min         Max         Avg      StdDev     %Change
	     user_numa01:    53004.03    68125.84    61697.01     5011.38      24.02%
	     user_numa02:     1650.82     2278.71     1941.26      224.59      -5.25%
		Testcase:         Min         Max         Avg      StdDev     %Change
	    total_numa01:    53478.30    69505.30    62567.12     5288.18      24.72%
	    total_numa02:     1714.56     2398.96     2027.95      238.08      -5.67%


tip + Srikar's ++ (iv)
		Testcase:         Min         Max         Avg      StdDev     %Change
	  elapsed_numa01:      690.74      919.49      782.67       78.51      14.46%
	  elapsed_numa02:       21.78       29.57       26.02        2.65       1.39%
		Testcase:         Min         Max         Avg      StdDev     %Change
	   system_numa01:      659.12     1041.19      870.15      143.13      78.38%
	   system_numa02:       52.20       78.73       64.18       11.28       7.84%
		Testcase:         Min         Max         Avg      StdDev     %Change
	     user_numa01:    56410.39    71492.31    62514.78     5444.90      21.75%
	     user_numa02:     1594.27     1934.40     1754.37      126.41       3.48%
		Testcase:         Min         Max         Avg      StdDev     %Change
	    total_numa01:    57069.50    72509.90    63384.94     5567.71      22.57%
	    total_numa02:     1647.85     2010.87     1818.55      136.88       3.65%


5 interations of Specjbb on 4 node, 24 core powerpc machine.
Ran 1 instance per system.

For specjbb (higher bops per JVM is better)

Plain 4.1.0-rc7-tip (i)
	  Metric:         Min         Max         Avg      StdDev
      bopsperJVM:   265519.00   272466.00   269377.80     2391.04

tip + only Rik's patch (ii)
	  Metric:         Min         Max         Avg      StdDev     %Change
      bopsperJVM:   263393.00   269660.00   266920.20     2792.07      -0.91%

tip + Rik's ++ (iii)
	  Metric:         Min         Max         Avg      StdDev     %Change
      bopsperJVM:   264298.00   271236.00   266818.20     2579.62      -0.94%

tip + Srikar's ++ (iv)
	  Metric:         Min         Max         Avg      StdDev     %Change
      bopsperJVM:   266774.00   272434.00   269839.60 	  2083.19      +0.17%


So fix for numa_has_capacity and always setting preferred node based on
fault stats seems to help autonuma and specjbb.


-- 
Thanks and Regards
Srikar Dronamraju

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/