Sample memory references with at least a given latency?

Sample memory references with at least a given latency?

[Harald Servat]

Hello list,

   is there any way to configure perf to sample memory references 
through PEBS and enable the Load Latency Performance Monitoring Facility 
(section 18.7.1.2 from the Intel 64 and IA-32 Architectures Software 
Developer's Manual).

Thank you very much in advance.

WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

http://www.bsc.es/disclaimer

Re: Sample memory references with at least a given latency?

[Harald Servat]

Hello Manuel,

    that's a very nice interesting piece of code! Thank you very much!

    Now two different questions arise:

     * still, is this possible through regular perf binary from the 
linux kernel?

     * with respect to your code, I observe that the data region for the 
PEBS collected samples is pointed by metadata_page. However, when 
printing its contents through print_samples the output does not print 
time-stamps. Are the time-stamps collected? If so, how can one access to 
that information?

Thank you very much in advance!!

On 04/02/15 14:06, Manuel Selva wrote:
> Yes you can. I wrote a small "memory not" for how to do this while
> trying to enable memory sampling:
>
> https://manuelselva.wordpress.com/2013/10/31/how-to-use-the-linux-perf_event_open-system-call/
>
> I also wrote some code exampl here:
> https://github.com/ManuelSelva/c4fun/tree/master/pebs_tests
>
> 2015-02-04 13:48 GMT+01:00 Harald Servat <harald.servat@bsc.es
> <mailto:harald.servat@bsc.es>>:
>
>     Hello list,
>
>        is there any way to configure perf to sample memory references
>     through PEBS and enable the Load Latency Performance Monitoring
>     Facility (section 18.7.1.2 from the Intel 64 and IA-32 Architectures
>     Software Developer's Manual).
>
>     Thank you very much in advance.
>
>     WARNING / LEGAL TEXT: This message is intended only for the use of the
>     individual or entity to which it is addressed and may contain
>     information which is privileged, confidential, proprietary, or exempt
>     from disclosure under applicable law. If you are not the intended
>     recipient or the person responsible for delivering the message to the
>     intended recipient, you are strictly prohibited from disclosing,
>     distributing, copying, or in any way using this message. If you have
>     received this communication in error, please notify the sender and
>     destroy and delete any copies you may have received.
>
>     http://www.bsc.es/disclaimer
>     --
>     To unsubscribe from this list: send the line "unsubscribe
>     linux-perf-users" in
>     the body of a message to majordomo@vger.kernel.org
>     <mailto:majordomo@vger.kernel.org>
>     More majordomo info at http://vger.kernel.org/__majordomo-info.html
>     <http://vger.kernel.org/majordomo-info.html>
>
>


WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

http://www.bsc.es/disclaimer

Re: Sample memory references with at least a given latency?

[Harald Servat]

Hello Manuel, et al,

   I've been digging a bit into the example code you mention below. It 
look like lines 246-271 setup the PEBS infrastructure given through perf.

   I notice, and correct if I'm wrong, that you don't setup any 
interrupt service, so you expect that the buffer for the PEBS entries is 
sufficiently large for the application run. Do you know how to program 
an interrupt service if the buffer cannot hold all the PEBS entries? And 
is that possible from user-space?

Best regards and thank you.

On 04/02/15 14:06, Manuel Selva wrote:
> Yes you can. I wrote a small "memory not" for how to do this while
> trying to enable memory sampling:
>
> https://manuelselva.wordpress.com/2013/10/31/how-to-use-the-linux-perf_event_open-system-call/
>
> I also wrote some code exampl here:
> https://github.com/ManuelSelva/c4fun/tree/master/pebs_tests
>
> 2015-02-04 13:48 GMT+01:00 Harald Servat <harald.servat@bsc.es
> <mailto:harald.servat@bsc.es>>:
>
>     Hello list,
>
>        is there any way to configure perf to sample memory references
>     through PEBS and enable the Load Latency Performance Monitoring
>     Facility (section 18.7.1.2 from the Intel 64 and IA-32 Architectures
>     Software Developer's Manual).
>
>     Thank you very much in advance.
>
>     WARNING / LEGAL TEXT: This message is intended only for the use of the
>     individual or entity to which it is addressed and may contain
>     information which is privileged, confidential, proprietary, or exempt
>     from disclosure under applicable law. If you are not the intended
>     recipient or the person responsible for delivering the message to the
>     intended recipient, you are strictly prohibited from disclosing,
>     distributing, copying, or in any way using this message. If you have
>     received this communication in error, please notify the sender and
>     destroy and delete any copies you may have received.
>
>     http://www.bsc.es/disclaimer
>     --
>     To unsubscribe from this list: send the line "unsubscribe
>     linux-perf-users" in
>     the body of a message to majordomo@vger.kernel.org
>     <mailto:majordomo@vger.kernel.org>
>     More majordomo info at http://vger.kernel.org/__majordomo-info.html
>     <http://vger.kernel.org/majordomo-info.html>
>
>


WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

http://www.bsc.es/disclaimer

Re: Sample memory references with at least a given latency?

[Harald Servat]

Hello Manuel,

   I'm running Linux 3.11 but I don't see a way to setup the latency 
threshold there. My option flags include:

        -t, --type=
        -D, --dump-raw-samples=
        -x, --field-separator
        -C, --cpu-list

   so I guess that this option is not implemented in perf (at least on 
3.11). So I need to capture all the samples and then discard those that 
surpass that threshold.

   Going back into your example, is it possible to capture / show the 
timestamp for the captured samples? I've seen the Intel documentation, 
and there aren't timestamps in PEBS entries so that may require 
additional work. IIRC, someone of the list told me that perf allocates a 
1-entry PEBS buffer so every time PEBS fills it, perf somehow gets an 
interrupt indicating that it has to flush it. This way, perf can 
attribute a timestamp to that sample also.

Thank you very much!


On 04/02/15 15:00, Manuel Selva wrote:
> Hi Harald,
>
> Yes you can with the "perf mem" tool. It has been introduced in kernel
> 3.10 (correct me if I am wrong, not very sure about that). It's built on
> top of perf mem record and perf mem report but hide for you the
> complxity of choosing the memory events . See "man perf mem" for th details.
>
> ----------
> Manuel
>
>
>
> 2015-02-04 13:48 GMT+01:00 Harald Servat <harald.servat@bsc.es
> <mailto:harald.servat@bsc.es>>:
>
>     Hello list,
>
>        is there any way to configure perf to sample memory references
>     through PEBS and enable the Load Latency Performance Monitoring
>     Facility (section 18.7.1.2 from the Intel 64 and IA-32 Architectures
>     Software Developer's Manual).
>
>     Thank you very much in advance.
>
>     WARNING / LEGAL TEXT: This message is intended only for the use of the
>     individual or entity to which it is addressed and may contain
>     information which is privileged, confidential, proprietary, or exempt
>     from disclosure under applicable law. If you are not the intended
>     recipient or the person responsible for delivering the message to the
>     intended recipient, you are strictly prohibited from disclosing,
>     distributing, copying, or in any way using this message. If you have
>     received this communication in error, please notify the sender and
>     destroy and delete any copies you may have received.
>
>     http://www.bsc.es/disclaimer
>     --
>     To unsubscribe from this list: send the line "unsubscribe
>     linux-perf-users" in
>     the body of a message to majordomo@vger.kernel.org
>     <mailto:majordomo@vger.kernel.org>
>     More majordomo info at http://vger.kernel.org/__majordomo-info.html
>     <http://vger.kernel.org/majordomo-info.html>
>
>


WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

http://www.bsc.es/disclaimer

Re: Sample memory references with at least a given latency?

[Vince Weaver]

On Wed, 4 Feb 2015, Harald Servat wrote:

>   I'm running Linux 3.11 but I don't see a way to setup the latency threshold
> there. My option flags include:
> 
>        -t, --type=
>        -D, --dump-raw-samples=
>        -x, --field-separator
>        -C, --cpu-list
> 
>   so I guess that this option is not implemented in perf (at least on 3.11).
> So I need to capture all the samples and then discard those that surpass that
> threshold.

No, PEBS support is very lacking with perf.

The soon-to-be-released 3.19 kernel does add support for 
PERF_SAMPLE_REGS_INTR which finally lets you get the PEBS register state 
from the PEBS sample, but I think many of the other advanced PEBS 
features are still missing.

Vince

Re: Sample memory references with at least a given latency?

[Manuel Selva]

Hi Harald,

You are right, the perf mem tool dosn't support stting the latency 
threshold. It should be a very easy first contribution to kernel for 
myself ;-)

Regarding the timestamp, I nver need such information, but as you said 
you can't tll to the hardware (on Intel I mean) to record it along with 
processor state information. I don't know how th kernel part of perf 
configure PEBS, but why would the kernel pay the overhead of an 
interrupt by using 1 slot buffers ? Only for recording timestamp ?

----
Manu

On 02/04/2015 03:20 PM, Harald Servat wrote:
>
> Hello Manuel,
>
>    I'm running Linux 3.11 but I don't see a way to setup the latency
> threshold there. My option flags include:
>
>         -t, --type=
>         -D, --dump-raw-samples=
>         -x, --field-separator
>         -C, --cpu-list
>
>    so I guess that this option is not implemented in perf (at least on
> 3.11). So I need to capture all the samples and then discard those that
> surpass that threshold.
>
>    Going back into your example, is it possible to capture / show the
> timestamp for the captured samples? I've seen the Intel documentation,
> and there aren't timestamps in PEBS entries so that may require
> additional work. IIRC, someone of the list told me that perf allocates a
> 1-entry PEBS buffer so every time PEBS fills it, perf somehow gets an
> interrupt indicating that it has to flush it. This way, perf can
> attribute a timestamp to that sample also.
>
> Thank you very much!
>
>
> On 04/02/15 15:00, Manuel Selva wrote:
>> Hi Harald,
>>
>> Yes you can with the "perf mem" tool. It has been introduced in kernel
>> 3.10 (correct me if I am wrong, not very sure about that). It's built on
>> top of perf mem record and perf mem report but hide for you the
>> complxity of choosing the memory events . See "man perf mem" for th
>> details.
>>
>> ----------
>> Manuel
>>
>>
>>
>> 2015-02-04 13:48 GMT+01:00 Harald Servat <harald.servat@bsc.es
>> <mailto:harald.servat@bsc.es>>:
>>
>>     Hello list,
>>
>>        is there any way to configure perf to sample memory references
>>     through PEBS and enable the Load Latency Performance Monitoring
>>     Facility (section 18.7.1.2 from the Intel 64 and IA-32 Architectures
>>     Software Developer's Manual).
>>
>>     Thank you very much in advance.
>>
>>     WARNING / LEGAL TEXT: This message is intended only for the use of
>> the
>>     individual or entity to which it is addressed and may contain
>>     information which is privileged, confidential, proprietary, or exempt
>>     from disclosure under applicable law. If you are not the intended
>>     recipient or the person responsible for delivering the message to the
>>     intended recipient, you are strictly prohibited from disclosing,
>>     distributing, copying, or in any way using this message. If you have
>>     received this communication in error, please notify the sender and
>>     destroy and delete any copies you may have received.
>>
>>     http://www.bsc.es/disclaimer
>>     --
>>     To unsubscribe from this list: send the line "unsubscribe
>>     linux-perf-users" in
>>     the body of a message to majordomo@vger.kernel.org
>>     <mailto:majordomo@vger.kernel.org>
>>     More majordomo info at http://vger.kernel.org/__majordomo-info.html
>>     <http://vger.kernel.org/majordomo-info.html>
>>
>>
>
>
> WARNING / LEGAL TEXT: This message is intended only for the use of the
> individual or entity to which it is addressed and may contain
> information which is privileged, confidential, proprietary, or exempt
> from disclosure under applicable law. If you are not the intended
> recipient or the person responsible for delivering the message to the
> intended recipient, you are strictly prohibited from disclosing,
> distributing, copying, or in any way using this message. If you have
> received this communication in error, please notify the sender and
> destroy and delete any copies you may have received.
>
> http://www.bsc.es/disclaimer

Re: Sample memory references with at least a given latency?

[Harald Servat]

Hello,

On 05/02/15 12:33, Manuel Selva wrote:
> Hi Harald,
>
> You are right, the perf mem tool dosn't support stting the latency
> threshold. It should be a very easy first contribution to kernel for
> myself ;-)

   that would be great. If I had some spare time & a machine to mess a 
bit, I'd gladly help you. BTW, do you know if there's any alternative 
for developing the kernel?

> Regarding the timestamp, I nver need such information, but as you said
> you can't tll to the hardware (on Intel I mean) to record it along with
> processor state information. I don't know how th kernel part of perf
> configure PEBS, but why would the kernel pay the overhead of an
> interrupt by using 1 slot buffers ? Only for recording timestamp ?

   Personally, I play with time-stamped trace-files and find capturing 
the time-stamp extremely important! Andi Kleen (also from this list) 
confirmed this (1 slot buffer) but he didn't mention the exact reason. 
Still, and let me insist, this is perfect for my purposes so I'm ok with 
it. See previous mails from this ML:

http://article.gmane.org/gmane.linux.kernel.perf.user/1398/match=information+regarding+perf+tool

http://article.gmane.org/gmane.linux.kernel.perf.user/1375/match=information+regarding+perf+tool

Regards.


> ----
> Manu
>
> On 02/04/2015 03:20 PM, Harald Servat wrote:
>>
>> Hello Manuel,
>>
>>    I'm running Linux 3.11 but I don't see a way to setup the latency
>> threshold there. My option flags include:
>>
>>         -t, --type=
>>         -D, --dump-raw-samples=
>>         -x, --field-separator
>>         -C, --cpu-list
>>
>>    so I guess that this option is not implemented in perf (at least on
>> 3.11). So I need to capture all the samples and then discard those that
>> surpass that threshold.
>>
>>    Going back into your example, is it possible to capture / show the
>> timestamp for the captured samples? I've seen the Intel documentation,
>> and there aren't timestamps in PEBS entries so that may require
>> additional work. IIRC, someone of the list told me that perf allocates a
>> 1-entry PEBS buffer so every time PEBS fills it, perf somehow gets an
>> interrupt indicating that it has to flush it. This way, perf can
>> attribute a timestamp to that sample also.
>>
>> Thank you very much!
>>
>>
>> On 04/02/15 15:00, Manuel Selva wrote:
>>> Hi Harald,
>>>
>>> Yes you can with the "perf mem" tool. It has been introduced in kernel
>>> 3.10 (correct me if I am wrong, not very sure about that). It's built on
>>> top of perf mem record and perf mem report but hide for you the
>>> complxity of choosing the memory events . See "man perf mem" for th
>>> details.
>>>
>>> ----------
>>> Manuel
>>>
>>>
>>>
>>> 2015-02-04 13:48 GMT+01:00 Harald Servat <harald.servat@bsc.es
>>> <mailto:harald.servat@bsc.es>>:
>>>
>>>     Hello list,
>>>
>>>        is there any way to configure perf to sample memory references
>>>     through PEBS and enable the Load Latency Performance Monitoring
>>>     Facility (section 18.7.1.2 from the Intel 64 and IA-32 Architectures
>>>     Software Developer's Manual).
>>>
>>>     Thank you very much in advance.
>>>
>>>     WARNING / LEGAL TEXT: This message is intended only for the use of
>>> the
>>>     individual or entity to which it is addressed and may contain
>>>     information which is privileged, confidential, proprietary, or
>>> exempt
>>>     from disclosure under applicable law. If you are not the intended
>>>     recipient or the person responsible for delivering the message to
>>> the
>>>     intended recipient, you are strictly prohibited from disclosing,
>>>     distributing, copying, or in any way using this message. If you have
>>>     received this communication in error, please notify the sender and
>>>     destroy and delete any copies you may have received.
>>>
>>>     http://www.bsc.es/disclaimer
>>>     --
>>>     To unsubscribe from this list: send the line "unsubscribe
>>>     linux-perf-users" in
>>>     the body of a message to majordomo@vger.kernel.org
>>>     <mailto:majordomo@vger.kernel.org>
>>>     More majordomo info at http://vger.kernel.org/__majordomo-info.html
>>>     <http://vger.kernel.org/majordomo-info.html>
>>>
>>>
>>
>>
>> WARNING / LEGAL TEXT: This message is intended only for the use of the
>> individual or entity to which it is addressed and may contain
>> information which is privileged, confidential, proprietary, or exempt
>> from disclosure under applicable law. If you are not the intended
>> recipient or the person responsible for delivering the message to the
>> intended recipient, you are strictly prohibited from disclosing,
>> distributing, copying, or in any way using this message. If you have
>> received this communication in error, please notify the sender and
>> destroy and delete any copies you may have received.
>>
>> http://www.bsc.es/disclaimer
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-perf-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

http://www.bsc.es/disclaimer

Re: Sample memory references with at least a given latency?

[Andi Kleen]

Harald Servat <harald.servat@bsc.es> writes:

>   is there any way to configure perf to sample memory references
> through PEBS and enable the Load Latency Performance Monitoring
> Facility (section 18.7.1.2 from the Intel 64 and IA-32 Architectures
> Software Developer's Manual).

$ ocperf.py record -e mem_trans_retired.load_latency_gt_128 -a sleep 1
perf record -e
cpu/event=0xcd,umask=0x1,ldlat=0x80,name=mem_trans_retired_load_latency_gt_128/
-a sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.438 MB perf.data (~19125 samples) ]

-- 
ak@linux.intel.com -- Speaking for myself only

Re: Sample memory references with at least a given latency?

[Manuel Selva]

What information is present in the samples recorded by perf record (on 
which ocperf relies) ?

Manu

On 02/05/2015 05:58 PM, Andi Kleen wrote:
> Harald Servat <harald.servat@bsc.es> writes:
>
>>    is there any way to configure perf to sample memory references
>> through PEBS and enable the Load Latency Performance Monitoring
>> Facility (section 18.7.1.2 from the Intel 64 and IA-32 Architectures
>> Software Developer's Manual).
>
> $ ocperf.py record -e mem_trans_retired.load_latency_gt_128 -a sleep 1
> perf record -e
> cpu/event=0xcd,umask=0x1,ldlat=0x80,name=mem_trans_retired_load_latency_gt_128/
> -a sleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.438 MB perf.data (~19125 samples) ]
>

Re: Sample memory references with at least a given latency?

[Andi Kleen]

On Thu, Feb 05, 2015 at 07:54:13PM +0100, Manuel Selva wrote:
> What information is present in the samples recorded by perf record
> (on which ocperf relies) ?

man perf_event_open
man perf-record

It depends on what options you pass to record. As of the latest kernel
practically everything in PEBS is supported. Before that the GPRs were missing.
You can get for example memory source, address or TSX information.

-Andi