Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing

[Bharata B Rao <bharata@amd.com>]

On 17-Feb-23 11:33 AM, Huang, Ying wrote:
> Bharata B Rao <bharata@amd.com> writes:
> 
>> On 14-Feb-23 10:25 AM, Bharata B Rao wrote:
>>> On 13-Feb-23 12:00 PM, Huang, Ying wrote:
>>>>> I have a microbenchmark where two sets of threads bound to two 
>>>>> NUMA nodes access the two different halves of memory which is
>>>>> initially allocated on the 1st node.
>>>>>
>>>>> On a two node Zen4 system, with 64 threads in each set accessing
>>>>> 8G of memory each from the initial allocation of 16G, I see that
>>>>> IBS driven NUMA balancing (i,e., this patchset) takes 50% less time
>>>>> to complete a fixed number of memory accesses. This could well
>>>>> be the best case and real workloads/benchmarks may not get this much
>>>>> uplift, but it does show the potential gain to be had.
>>>>
>>>> Can you find a way to show the overhead of the original implementation
>>>> and your method?  Then we can compare between them?  Because you think
>>>> the improvement comes from the reduced overhead.
>>>
>>> Sure, will measure the overhead.
>>
>> I used ftrace function_graph tracer to measure the amount of time (in us)
>> spent in fault handling and task_work handling in both the methods when
>> the above mentioned benchmark was running.
>>
>> 			Default		IBS
>> Fault handling		29879668.71	1226770.84
>> Task work handling	24878.894	10635593.82
>> Sched switch handling			78159.846
>>
>> Total			29904547.6	11940524.51
> 
> Thanks!  You have shown the large overhead difference between the
> original method and your method.  Can you show the number of the pages
> migrated too?  I think the overhead / page can be a good overhead
> indicator too.
> 
> Can it be translated to the performance improvement?  Per my
> understanding, the total overhead is small compared with total run time.

I captured some of the numbers that you wanted for two different runs.
The first case shows the data for a short run (less number of memory access
iterations) and the second one is for a long run (more number of iterations)

Short-run
=========
Time taken or overhead (us) for fault, task_work and sched_switch
handling

			Default		IBS
Fault handling		29017953.99	1196828.67
Task work handling	10354.40	10356778.53
Sched switch handling			56572.21
Total overhead		29028308.39	11610179.41

Benchmark score(us)	194050290	53963650
numa_pages_migrated	2097256		662755
Overhead / page		13.84		17.51
Pages migrated per sec	72248.64	57083.95

Default
-------
			Total		Min	Max		Avg
do_numa_page		29017953.99	0.1	307.63		15.97
task_numa_work		10354.40	2.86	4573.60		175.50
Total			29028308.39

IBS
---
			Total		Min	Max		Avg
ibs_overflow_handler	1196828.67	0.15	100.28		1.26
task_ibs_access_work	10356778.53	0.21	10504.14	28.42
hw_access_sched_in	56572.21	0.15	16.94		1.45
Total			11610179.41


Long-run
========
Time taken or overhead (us) for fault, task_work and sched_switch
handling
			Default		IBS
Fault handling		27437756.73	901406.37
Task work handling	1741.66		4902935.32
Sched switch handling			100590.33
Total overhead		27439498.38	5904932.02

Benchmark score(us)	306786210.0	153422489.0
numa_pages_migrated	2097218		1746099
Overhead / page		13.08		3.38
Pages migrated per sec	6836.08		11380.98

Default
-------
			Total		Min	Max		Avg
do_numa_page		27437756.73	0.08	363.475		15.03
task_numa_work		1741.66		3.294	1200.71		42.48
Total			27439498.38

IBS
---
			Total		Min	Max		Avg
ibs_overflow_handler	901406.37	0.15	95.51		1.06
task_ibs_access_work	4902935.32	0.22	11013.68	9.64
hw_access_sched_in	100590.33	0.14	91.97		1.52
Total			5904932.02

Regards,
Bharata.