Remote Cache Accesses (1 Hop)

Remote Cache Accesses (1 Hop)

[Manuel Selva]

Hi all,

I wrote a benchmarking program, in order to play with perf_event_open 
memory sampling capabilities (as discussed earlier on this list).

My benchmark is allocating a large array of memory (120 megas by 
default), then I am starting sampling with perf_event_open and the 
MEM_INST_RETIRED_LATENCY_ABOVE_THRESHOLD event (with threshold = 3, the 
minimal one according to Intel's doc) along with memory accesses 
counting (uncore event QMC_NORMAL_READ.ANY) and I access all the 
allocated memory either sequentially or randomly. The benchmark is mono 
thread, pinned to a given core, and memory is allocate don the core 
associated with this node using numa_alloc functions. perf_event_open is 
thus called to monitor only this core/numa node. The code is compiled 
without any optimization and I tried as possible to increase the ratio 
of memory accesses codes vs branching, and other stuff code.

I can clearly see with the QMC_NORMAL_READ.ANY events count that my 
random accesses test case generate far more memory accesses than the 
sequential one (I guess the prefetcher and sequential access are 
responsible for that).

Regarding the sampled event, I successfully mmap the result of perf 
event open and I am able to read the samples. My problem is that even 
for the random accesses test case on the 120 megas, I don't have any 
samples served by the RAM (I am using the PERF_SAMPLE_DATA_SRC field) 
(the sampling period is 1000 events and I only get ~700 samples). 
Nevertheless, in the sequential case I have 0,01% of my samples that are 
remote cache accesses (1 Hop) where as this percentage is ~20% in the 
random case.

So my questions are:

- What is exactly remote cache (1 hop) ? A data found in another core's 
private cache (L1 or L2 in my case) on the same processor ?

- How can I interpret my results, I was expecting to have local memory 
accesses samples increasing in the random case instead of remote cache 
(1 hop). How can I have remote cache accesses with malloced data not 
shared, and used by a thread pinned to a given core.

- I didn't yet look at that on details, but 700~ samples for accessing 
120 megas with sampling every 1000 events seems small. I am going to 
check the assembly code generated and also count the total number of 
memory requests (a core event) and compare that to the 700 samples * 
1000 events.

Thanks in advance for any suggestions you may have in order to help 
understand what's happening there.

-- 
Manu

Re: Remote Cache Accesses (1 Hop)

[Manuel Selva]

Hi all,

Today I followed up my investigations on this subject.

Concerning the first question about, remote cache samples, I am still
not able to understand  why i get this kind of events instead of Local
ram accesses. I modified my memory allocation function to use
numa_alloc function to be sure the memory is physically allocated
where I want. When the memory is allocated on the same node as the
thread running my program, I still have remote cache accesses and no
ram accesses. When it's on another node, I have remote ram accesses as
expected.

Does anybody have an explanation of the exact meaning of these remote
cache events that could explain why I get them ?

Regarding the the number of samples, I first checked the number of
memory loads generated by the sampled function (I mean here
statically, looking at the code generated by gcc). I then compared
this number with the core event: MEM_INST_RETIRED.LOADS, they are very
close (measured = 15001804, expected = 15001804). The difference must
come from the loads generated by the start/stop ioctl calls and maybe
other thing happening behind the scene I must miss, anyway this is
coherent. With a sampling period of 1000 events, I only have 455
samples. For period = 5000 I have 91 samples, and for period = 10000
45 samples.

So my question is what is the reason for this low number of samples ?

1- Some memory accesses have a threshold smaller than 3 cycles and are
thus not counted at all ?

2- Is there any relation to the time taken by  the PMI handler being
too large. My benchmarking code is only doing loads and no computation
at all ?

3- any other idea ?

Manu

PS: Because this list is called perf-users, I should maybe ask this
kind of question on another list. If it's the case, please let me know
where ? Thanks.

2014/1/17 Manuel Selva <selva.manuel@gmail.com>:
> Hi all,
>
> I wrote a benchmarking program, in order to play with perf_event_open memory
> sampling capabilities (as discussed earlier on this list).
>
> My benchmark is allocating a large array of memory (120 megas by default),
> then I am starting sampling with perf_event_open and the
> MEM_INST_RETIRED_LATENCY_ABOVE_THRESHOLD event (with threshold = 3, the
> minimal one according to Intel's doc) along with memory accesses counting
> (uncore event QMC_NORMAL_READ.ANY) and I access all the allocated memory
> either sequentially or randomly. The benchmark is mono thread, pinned to a
> given core, and memory is allocate don the core associated with this node
> using numa_alloc functions. perf_event_open is thus called to monitor only
> this core/numa node. The code is compiled without any optimization and I
> tried as possible to increase the ratio of memory accesses codes vs
> branching, and other stuff code.
>
> I can clearly see with the QMC_NORMAL_READ.ANY events count that my random
> accesses test case generate far more memory accesses than the sequential one
> (I guess the prefetcher and sequential access are responsible for that).
>
> Regarding the sampled event, I successfully mmap the result of perf event
> open and I am able to read the samples. My problem is that even for the
> random accesses test case on the 120 megas, I don't have any samples served
> by the RAM (I am using the PERF_SAMPLE_DATA_SRC field) (the sampling period
> is 1000 events and I only get ~700 samples). Nevertheless, in the sequential
> case I have 0,01% of my samples that are remote cache accesses (1 Hop) where
> as this percentage is ~20% in the random case.
>
> So my questions are:
>
> - What is exactly remote cache (1 hop) ? A data found in another core's
> private cache (L1 or L2 in my case) on the same processor ?
>
> - How can I interpret my results, I was expecting to have local memory
> accesses samples increasing in the random case instead of remote cache (1
> hop). How can I have remote cache accesses with malloced data not shared,
> and used by a thread pinned to a given core.
>
> - I didn't yet look at that on details, but 700~ samples for accessing 120
> megas with sampling every 1000 events seems small. I am going to check the
> assembly code generated and also count the total number of memory requests
> (a core event) and compare that to the 700 samples * 1000 events.
>
> Thanks in advance for any suggestions you may have in order to help
> understand what's happening there.
>
> --
> Manu

Re: Remote Cache Accesses (1 Hop)

[Manuel Selva]

Hi all,

Anyone can help on this subject about low number of sample and remote
cache samples meaning ?

Thanks,

Manu

2014-01-23 Manuel Selva <selva.manuel@gmail.com>:
> Hi all,
>
> Today I followed up my investigations on this subject.
>
> Concerning the first question about, remote cache samples, I am still
> not able to understand  why i get this kind of events instead of Local
> ram accesses. I modified my memory allocation function to use
> numa_alloc function to be sure the memory is physically allocated
> where I want. When the memory is allocated on the same node as the
> thread running my program, I still have remote cache accesses and no
> ram accesses. When it's on another node, I have remote ram accesses as
> expected.
>
> Does anybody have an explanation of the exact meaning of these remote
> cache events that could explain why I get them ?
>
> Regarding the the number of samples, I first checked the number of
> memory loads generated by the sampled function (I mean here
> statically, looking at the code generated by gcc). I then compared
> this number with the core event: MEM_INST_RETIRED.LOADS, they are very
> close (measured = 15001804, expected = 15001804). The difference must
> come from the loads generated by the start/stop ioctl calls and maybe
> other thing happening behind the scene I must miss, anyway this is
> coherent. With a sampling period of 1000 events, I only have 455
> samples. For period = 5000 I have 91 samples, and for period = 10000
> 45 samples.
>
> So my question is what is the reason for this low number of samples ?
>
> 1- Some memory accesses have a threshold smaller than 3 cycles and are
> thus not counted at all ?
>
> 2- Is there any relation to the time taken by  the PMI handler being
> too large. My benchmarking code is only doing loads and no computation
> at all ?
>
> 3- any other idea ?
>
> Manu
>
> PS: Because this list is called perf-users, I should maybe ask this
> kind of question on another list. If it's the case, please let me know
> where ? Thanks.
>
> 2014/1/17 Manuel Selva <selva.manuel@gmail.com>:
>> Hi all,
>>
>> I wrote a benchmarking program, in order to play with perf_event_open memory
>> sampling capabilities (as discussed earlier on this list).
>>
>> My benchmark is allocating a large array of memory (120 megas by default),
>> then I am starting sampling with perf_event_open and the
>> MEM_INST_RETIRED_LATENCY_ABOVE_THRESHOLD event (with threshold = 3, the
>> minimal one according to Intel's doc) along with memory accesses counting
>> (uncore event QMC_NORMAL_READ.ANY) and I access all the allocated memory
>> either sequentially or randomly. The benchmark is mono thread, pinned to a
>> given core, and memory is allocate don the core associated with this node
>> using numa_alloc functions. perf_event_open is thus called to monitor only
>> this core/numa node. The code is compiled without any optimization and I
>> tried as possible to increase the ratio of memory accesses codes vs
>> branching, and other stuff code.
>>
>> I can clearly see with the QMC_NORMAL_READ.ANY events count that my random
>> accesses test case generate far more memory accesses than the sequential one
>> (I guess the prefetcher and sequential access are responsible for that).
>>
>> Regarding the sampled event, I successfully mmap the result of perf event
>> open and I am able to read the samples. My problem is that even for the
>> random accesses test case on the 120 megas, I don't have any samples served
>> by the RAM (I am using the PERF_SAMPLE_DATA_SRC field) (the sampling period
>> is 1000 events and I only get ~700 samples). Nevertheless, in the sequential
>> case I have 0,01% of my samples that are remote cache accesses (1 Hop) where
>> as this percentage is ~20% in the random case.
>>
>> So my questions are:
>>
>> - What is exactly remote cache (1 hop) ? A data found in another core's
>> private cache (L1 or L2 in my case) on the same processor ?
>>
>> - How can I interpret my results, I was expecting to have local memory
>> accesses samples increasing in the random case instead of remote cache (1
>> hop). How can I have remote cache accesses with malloced data not shared,
>> and used by a thread pinned to a given core.
>>
>> - I didn't yet look at that on details, but 700~ samples for accessing 120
>> megas with sampling every 1000 events seems small. I am going to check the
>> assembly code generated and also count the total number of memory requests
>> (a core event) and compare that to the 700 samples * 1000 events.
>>
>> Thanks in advance for any suggestions you may have in order to help
>> understand what's happening there.
>>
>> --
>> Manu

Re: Remote Cache Accesses (1 Hop)

[Manuel Selva]

Hi,

I just read information about Intel MESIF cache coherency protocol, and 
I learned that the protocol tries as possible to satisfy memory requests 
from remote caches before using memory for performances purpose.

As a consequence I am wondering if the hardware may be able to identify 
that my second (remote) L3 cache is not used and thus use it for 
performances purpose even if it's on another Node than the memory I am 
using. This could explain why I am getting remote caches samples while 
sampling a mono thread application using only local memory (allocated 
with numa_alloc_on_node()).

Is this hypothesis possible ?

----
Manu

On 01/29/2014 02:47 PM, Manuel Selva wrote:
> Hi all,
>
> Anyone can help on this subject about low number of sample and remote
> cache samples meaning ?
>
> Thanks,
>
> Manu
>
> 2014-01-23 Manuel Selva <selva.manuel@gmail.com>:
>> Hi all,
>>
>> Today I followed up my investigations on this subject.
>>
>> Concerning the first question about, remote cache samples, I am still
>> not able to understand  why i get this kind of events instead of Local
>> ram accesses. I modified my memory allocation function to use
>> numa_alloc function to be sure the memory is physically allocated
>> where I want. When the memory is allocated on the same node as the
>> thread running my program, I still have remote cache accesses and no
>> ram accesses. When it's on another node, I have remote ram accesses as
>> expected.
>>
>> Does anybody have an explanation of the exact meaning of these remote
>> cache events that could explain why I get them ?
>>
>> Regarding the the number of samples, I first checked the number of
>> memory loads generated by the sampled function (I mean here
>> statically, looking at the code generated by gcc). I then compared
>> this number with the core event: MEM_INST_RETIRED.LOADS, they are very
>> close (measured = 15001804, expected = 15001804). The difference must
>> come from the loads generated by the start/stop ioctl calls and maybe
>> other thing happening behind the scene I must miss, anyway this is
>> coherent. With a sampling period of 1000 events, I only have 455
>> samples. For period = 5000 I have 91 samples, and for period = 10000
>> 45 samples.
>>
>> So my question is what is the reason for this low number of samples ?
>>
>> 1- Some memory accesses have a threshold smaller than 3 cycles and are
>> thus not counted at all ?
>>
>> 2- Is there any relation to the time taken by  the PMI handler being
>> too large. My benchmarking code is only doing loads and no computation
>> at all ?
>>
>> 3- any other idea ?
>>
>> Manu
>>
>> PS: Because this list is called perf-users, I should maybe ask this
>> kind of question on another list. If it's the case, please let me know
>> where ? Thanks.
>>
>> 2014/1/17 Manuel Selva <selva.manuel@gmail.com>:
>>> Hi all,
>>>
>>> I wrote a benchmarking program, in order to play with perf_event_open memory
>>> sampling capabilities (as discussed earlier on this list).
>>>
>>> My benchmark is allocating a large array of memory (120 megas by default),
>>> then I am starting sampling with perf_event_open and the
>>> MEM_INST_RETIRED_LATENCY_ABOVE_THRESHOLD event (with threshold = 3, the
>>> minimal one according to Intel's doc) along with memory accesses counting
>>> (uncore event QMC_NORMAL_READ.ANY) and I access all the allocated memory
>>> either sequentially or randomly. The benchmark is mono thread, pinned to a
>>> given core, and memory is allocate don the core associated with this node
>>> using numa_alloc functions. perf_event_open is thus called to monitor only
>>> this core/numa node. The code is compiled without any optimization and I
>>> tried as possible to increase the ratio of memory accesses codes vs
>>> branching, and other stuff code.
>>>
>>> I can clearly see with the QMC_NORMAL_READ.ANY events count that my random
>>> accesses test case generate far more memory accesses than the sequential one
>>> (I guess the prefetcher and sequential access are responsible for that).
>>>
>>> Regarding the sampled event, I successfully mmap the result of perf event
>>> open and I am able to read the samples. My problem is that even for the
>>> random accesses test case on the 120 megas, I don't have any samples served
>>> by the RAM (I am using the PERF_SAMPLE_DATA_SRC field) (the sampling period
>>> is 1000 events and I only get ~700 samples). Nevertheless, in the sequential
>>> case I have 0,01% of my samples that are remote cache accesses (1 Hop) where
>>> as this percentage is ~20% in the random case.
>>>
>>> So my questions are:
>>>
>>> - What is exactly remote cache (1 hop) ? A data found in another core's
>>> private cache (L1 or L2 in my case) on the same processor ?
>>>
>>> - How can I interpret my results, I was expecting to have local memory
>>> accesses samples increasing in the random case instead of remote cache (1
>>> hop). How can I have remote cache accesses with malloced data not shared,
>>> and used by a thread pinned to a given core.
>>>
>>> - I didn't yet look at that on details, but 700~ samples for accessing 120
>>> megas with sampling every 1000 events seems small. I am going to check the
>>> assembly code generated and also count the total number of memory requests
>>> (a core event) and compare that to the 700 samples * 1000 events.
>>>
>>> Thanks in advance for any suggestions you may have in order to help
>>> understand what's happening there.
>>>
>>> --
>>> Manu