How to trace threads created by a process after perf has started recording, using the -p PID flag?

How to trace threads created by a process after perf has started recording, using the -p PID flag?

[Mikolaj Konarski]

Hi,

Could someone clarify the behaviour of the -p PID flag of perf-record?
In our experiments (3.2.0-27 #43-Ubuntu SMP x86_64),
it ignores events on threads spawned after perf-record is started.
Is this the intended behaviour, and is there any work-around?

Generally, what is the best (vs CPU/IO, distortion of profiling results)
way to record only events of a given OS process, with all its threads?

Thanks a lot,
Mikolaj

Re: How to trace threads created by a process after perf has started recording, using the -p PID flag?

[chenggang qin]

HI:
    I submitted a patch to fix this problem. and the patch is waitting for
reviewing.

                        chenggang qin

On Sat, Aug 25, 2012 at 8:52 PM, Mikolaj Konarski <
mikolaj.konarski@gmail.com> wrote:

> Hi,
>
> Could someone clarify the behaviour of the -p PID flag of perf-record?
> In our experiments (3.2.0-27 #43-Ubuntu SMP x86_64),
> it ignores events on threads spawned after perf-record is started.
> Is this the intended behaviour, and is there any work-around?
>
> Generally, what is the best (vs CPU/IO, distortion of profiling results)
> way to record only events of a given OS process, with all its threads?
>
> Thanks a lot,
> Mikolaj
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-perf-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: How to trace threads created by a process after perf has started recording, using the -p PID flag?

[Mikolaj Konarski]

>> Could someone clarify the behaviour of the -p PID flag of perf-record?
>> In our experiments (3.2.0-27 #43-Ubuntu SMP x86_64),
>> it ignores events on threads spawned after perf-record is started.
>> Is this the intended behaviour, and is there any work-around?

Anybody? We'd love to use a proven, standard workaround
for this issue, preferably without forcing our users to upgrade
the kernel or wait for any future release. Do we even diagnose
the problem correctly? The best workaround we can come up with
right now is to use the "--filter 'pid==NNNN'" option, but it has
a couple of downsides. Any help would be sincerely appreciated.

> Can you try perf from Linus' current tree? perf should be tracking threads
> created after the record starts.

Thanks for the tip, David. But are you sure there's
anything new and relevant in the Linus' current tree?
Could you perhaps point me to the commit that contains
the relevant changes?

The version of perf-record that we use already contains the separate
-p PID and -t TID options, so the initial PID patch is already in.
It's just that the -p PID option ignores all threads spawned
after the process is started, AFAICT.

OTOTH, chenggang qin, you say

> I submitted a patch to fix this problem. and the patch is waitting for
> reviewing.

Could you give me a link to this patch? Does it improve
the -p PID option or replace it or solve our problem
in some other way? Thank you for your kind interest.

Cheers,
Mikolaj

Re: How to trace threads created by a process after perf has started recording, using the -p PID flag?

[Namhyung Kim]

Hi, Mikolaj

On Tue, 4 Sep 2012 00:17:50 +0200, Mikolaj Konarski wrote:
>>> Could someone clarify the behaviour of the -p PID flag of perf-record?
>>> In our experiments (3.2.0-27 #43-Ubuntu SMP x86_64),
>>> it ignores events on threads spawned after perf-record is started.
>>> Is this the intended behaviour, and is there any work-around?
>
> Anybody? We'd love to use a proven, standard workaround
> for this issue, preferably without forcing our users to upgrade
> the kernel or wait for any future release. Do we even diagnose
> the problem correctly? The best workaround we can come up with
> right now is to use the "--filter 'pid==NNNN'" option, but it has
> a couple of downsides. Any help would be sincerely appreciated.
>

AFAIK perf events aren't inherited when -p, -t or -u option is used due
to a performance reason.  But if you start the process on the command
line with perf record, it'll be inherited and counted properly.

I guess your problem might be solved by using a cgroup:

  # mkdir ${cgroup_root}/mygroup
  # echo NNNN > ${cgroup_root}/mygroup/tasks
  # perf record -a -e cycles -G mygroup


>> Can you try perf from Linus' current tree? perf should be tracking threads
>> created after the record starts.
>
> Thanks for the tip, David. But are you sure there's
> anything new and relevant in the Linus' current tree?
> Could you perhaps point me to the commit that contains
> the relevant changes?
>
> The version of perf-record that we use already contains the separate
> -p PID and -t TID options, so the initial PID patch is already in.
> It's just that the -p PID option ignores all threads spawned
> after the process is started, AFAICT.
>
> OTOTH, chenggang qin, you say
>
>> I submitted a patch to fix this problem. and the patch is waitting for
>> reviewing.
>
> Could you give me a link to this patch? Does it improve
> the -p PID option or replace it or solve our problem
> in some other way? Thank you for your kind interest.
>

https://lkml.org/lkml/2012/8/22/340

But it only addresses a case of perf top.

Thanks,
Namhyung

Re: How to trace threads created by a process after perf has started recording, using the -p PID flag?

[David Ahern]

On 9/3/12 4:17 PM, Mikolaj Konarski wrote:
>>> Could someone clarify the behaviour of the -p PID flag of perf-record?
>>> In our experiments (3.2.0-27 #43-Ubuntu SMP x86_64),
>>> it ignores events on threads spawned after perf-record is started.
>>> Is this the intended behaviour, and is there any work-around?
>
> Anybody? We'd love to use a proven, standard workaround
> for this issue, preferably without forcing our users to upgrade
> the kernel or wait for any future release. Do we even diagnose
> the problem correctly? The best workaround we can come up with
> right now is to use the "--filter 'pid==NNNN'" option, but it has
> a couple of downsides. Any help would be sincerely appreciated.

I see the problem now. The perf tool gets FORK events as threads are 
created, but it does not respond to them and create new counters. Some 
refactoring is needed to make that happen.

David

Re: How to trace threads created by a process after perf has started recording, using the -p PID flag?

[Mikolaj Konarski]

>> Anybody? We'd love to use a proven, standard workaround
>> for this issue, preferably without forcing our users to upgrade
>> the kernel or wait for any future release. Do we even diagnose
>> the problem correctly?
[snip lots of useful responses]

Thank you very much for your tips, they helped a lot.

Now our main problem is relating the perf timestamps
to a public clock API (like posix_gettime, etc.) in order
to chronologically merge the perf log with our external event logs.
We've so far determined experimentally that even the
clock_gettime(CLOCK_MONOTONIC, ...) values are not identical
to the perf timestamps at the same places, though they are quite close.
Are we missing anything? Is the "--timestamp" option of perf-record
related in any way? Any other tips?

We're working around right now by syncing with a timestamp
of a known event, but that leaves us vulnerable to a time drift.
If there's no way currently to generate externally meaningful
timestamps, could we file a feature request for the sake of the future?

Thanks a lot,
Mikolaj

Re: How to trace threads created by a process after perf has started recording, using the -p PID flag?

[David Ahern]

On 9/22/12 1:58 AM, Mikolaj Konarski wrote:
> Are we missing anything? Is the "--timestamp" option of perf-record
> related in any way? Any other tips?

That option adds the timestamps to the samples if it were not otherwise 
added.

>
> We're working around right now by syncing with a timestamp
> of a known event, but that leaves us vulnerable to a time drift.
> If there's no way currently to generate externally meaningful
> timestamps, could we file a feature request for the sake of the future?

You mean like this: https://lkml.org/lkml/2011/6/7/638

David

Re: How to trace threads created by a process after perf has started recording, using the -p PID flag?

[Mikolaj Konarski]

On Sat, Sep 22, 2012 at 8:39 PM, David Ahern <dsahern@gmail.com> wrote:
> On 9/22/12 1:58 AM, Mikolaj Konarski wrote:
>> the "--timestamp" option of perf-record
>
> That option adds the timestamps to the samples if it were not otherwise
> added.

Thanks. Worth adding to our calls, then, even if
the timestamp is not that trustworthy right now.

>> If there's no way currently to generate externally meaningful
>> timestamps, could we file a feature request for the sake of the future?
>
> You mean like this: https://lkml.org/lkml/2011/6/7/638

Precisely. How did you know? What happened to those
patches? Is there a place I can +1 and *Like* them publicly? :)

Thanks again,
Mikolaj

Re: How to trace threads created by a process after perf has started recording, using the -p PID flag?

[David Ahern]

On 9/22/12 1:57 PM, Mikolaj Konarski wrote:
>>> If there's no way currently to generate externally meaningful
>>> timestamps, could we file a feature request for the sake of the future?
>>
>> You mean like this: https://lkml.org/lkml/2011/6/7/638
>
> Precisely. How did you know? What happened to those
> patches? Is there a place I can +1 and *Like* them publicly? :)

I'm still waiting for certain changes to core perf before I try again to 
get it committed. I haven't forgotten; it's just not ready yet.

And using the time-of-day patch against a newer kernel (tried 3.2, 3.4 
and 3.6) I do see the offset you mentioned. I'll need to look into it 
what to do. I'm traveling this week, so next week is the earliest.

David