Using a new perf tool against an older kernel

Using a new perf tool against an older kernel

[Arun Sharma]

I ran into a few strange issues today using a new perf tool (3.0-rcX) 
against a 2.6.38 based kernel. Basic things seem to work:

# perf record -aR -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.443 MB perf.data (~19334 samples) ]

But:

# perf record -agR -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.558 MB perf.data (~68077 samples) ]
Can't find id 9's machine
Found 1 unknown events!

Is this an older tool processing a perf.data file generated by a more 
recent tool?

If that is not the case, consider reporting to linux-kernel@vger.kernel.org.

I couldn't tell if this is kernel/user space incompatibility or tighter 
error checking in the new perf tool. Before I start digging further, 
thought I'd ask if this combination is supported. The error message 
above doesn't ask me:

Are you using an old kernel?

  -Arun

Re: Using a new perf tool against an older kernel

[David Ahern]

On 06/22/2011 03:39 PM, Arun Sharma wrote:
> 
> I ran into a few strange issues today using a new perf tool (3.0-rcX)
> against a 2.6.38 based kernel. Basic things seem to work:
> 
> # perf record -aR -- sleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.443 MB perf.data (~19334 samples) ]
> 
> But:
> 
> # perf record -agR -- sleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 1.558 MB perf.data (~68077 samples) ]
> Can't find id 9's machine
> Found 1 unknown events!
> 
> Is this an older tool processing a perf.data file generated by a more
> recent tool?
> 
> If that is not the case, consider reporting to
> linux-kernel@vger.kernel.org.
> 
> I couldn't tell if this is kernel/user space incompatibility or tighter
> error checking in the new perf tool. Before I start digging further,
> thought I'd ask if this combination is supported. The error message
> above doesn't ask me:
> 
> Are you using an old kernel?

I have not seen issues like this using newer perf userspace against
older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
testing patches).

David

> 
>  -Arun
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe
> linux-perf-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Using a new perf tool against an older kernel

[Arun Sharma]

On 6/23/11 7:22 AM, David Ahern wrote:

> I have not seen issues like this using newer perf userspace against
> older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
> now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
> testing patches).

I narrowed it down to PERF_SAMPLE_RAW:

perf record -ag -- sleep 1

is fine, but:

perf record -agR -- sleep 1

fails for me most of the time. The reason I needed to use the -R in the 
first place is that "perf script" fails on older kernels with:

Samples do not contain timestamps.

With the newer perf, I don't get errors, but the timestamp field is 
invalid. So I need to use the -R flag to get valid timestamps + 
stacktraces out of "perf script".

  -Arun

Re: Using a new perf tool against an older kernel

[David Ahern]

On 06/23/2011 01:39 PM, Arun Sharma wrote:
> On 6/23/11 7:22 AM, David Ahern wrote:
> 
>> I have not seen issues like this using newer perf userspace against
>> older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
>> now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
>> testing patches).
> 
> I narrowed it down to PERF_SAMPLE_RAW:
> 
> perf record -ag -- sleep 1
> 
> is fine, but:
> 
> perf record -agR -- sleep 1
> 
> fails for me most of the time. The reason I needed to use the -R in the
> first place is that "perf script" fails on older kernels with:
> 
> Samples do not contain timestamps.
> 
> With the newer perf, I don't get errors, but the timestamp field is
> invalid. So I need to use the -R flag to get valid timestamps +
> stacktraces out of "perf script".

That should have been fixed.

And -T on record gets the timestamps.

David

> 
>  -Arun

Re: Using a new perf tool against an older kernel

[Frederic Weisbecker]

On Thu, Jun 23, 2011 at 12:39:54PM -0700, Arun Sharma wrote:
> On 6/23/11 7:22 AM, David Ahern wrote:
> 
> >I have not seen issues like this using newer perf userspace against
> >older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
> >now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
> >testing patches).
> 
> I narrowed it down to PERF_SAMPLE_RAW:
> 
> perf record -ag -- sleep 1
> 
> is fine, but:
> 
> perf record -agR -- sleep 1
> 
> fails for me most of the time. The reason I needed to use the -R in
> the first place is that "perf script" fails on older kernels with:
> 
> Samples do not contain timestamps.
> 
> With the newer perf, I don't get errors, but the timestamp field is
> invalid. So I need to use the -R flag to get valid timestamps +
> stacktraces out of "perf script".

I'm confused, you first said it happens with new tools on older kernel.

Can you tell us which combination of kernel/user raises the error? and
which error.

Thanks!

Re: Using a new perf tool against an older kernel

[Frederic Weisbecker]

On Thu, Jun 23, 2011 at 02:02:15PM -0600, David Ahern wrote:
> 
> 
> On 06/23/2011 01:39 PM, Arun Sharma wrote:
> > On 6/23/11 7:22 AM, David Ahern wrote:
> > 
> >> I have not seen issues like this using newer perf userspace against
> >> older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
> >> now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
> >> testing patches).
> > 
> > I narrowed it down to PERF_SAMPLE_RAW:
> > 
> > perf record -ag -- sleep 1
> > 
> > is fine, but:
> > 
> > perf record -agR -- sleep 1
> > 
> > fails for me most of the time. The reason I needed to use the -R in the
> > first place is that "perf script" fails on older kernels with:
> > 
> > Samples do not contain timestamps.
> > 
> > With the newer perf, I don't get errors, but the timestamp field is
> > invalid. So I need to use the -R flag to get valid timestamps +
> > stacktraces out of "perf script".
> 
> That should have been fixed.
> 
> And -T on record gets the timestamps.
> 
> David

Right, it would be nice to suggest that from perf script when timestamps
are not recorded.

Re: Using a new perf tool against an older kernel

[Arun Sharma]

On 6/23/11 5:11 PM, Frederic Weisbecker wrote:

>
> I'm confused, you first said it happens with new tools on older kernel.
>
> Can you tell us which combination of kernel/user raises the error? and
> which error.
>

perf-3.0 + 2.6.38

perf record -agR gives:

Can't find id 9's machine
Found 1 unknown events!

perf-3.0 + 2.6.32

perf record -ag
perf script

samples have bogus timestamps

perf-3.0 + 2.6.32

perf record -agR
perf script

error: Samples do not contain timestamps

perf-3.0 + any kernel

perf record -agT
perf script

is happy everywhere. Thanks!

  -Arun

Re: Using a new perf tool against an older kernel

[David Ahern]

On 06/23/2011 06:43 PM, Arun Sharma wrote:
> On 6/23/11 5:11 PM, Frederic Weisbecker wrote:
> 
>>
>> I'm confused, you first said it happens with new tools on older kernel.
>>
>> Can you tell us which combination of kernel/user raises the error? and
>> which error.
>>

I tried to repeat your examples below using perf-core tip (3.0-rc3,
af07ce3e77).

> 
> perf-3.0 + 2.6.38
> 
> perf record -agR gives:
> 
> Can't find id 9's machine
> Found 1 unknown events!

This does not reproduce on 2.6.38.8-32.fc15.x86_64 running bare metal or
2.6.38 in a VM. If you are running bare metal try -e cpu-clock and see
if it is cycles related. I doubt it, but that should be the only
difference in the tests.

That said, I did see that message while working on the gettimeofday
patches a few weeks ago. I believe the root cause was the early mixture
of cycles and tracepoints -- something I fixed before sending out the
patches (sample_type hack as well as how the time-of-day trace events
were added to the event list).

> 
> perf-3.0 + 2.6.32
> 
> perf record -ag
> perf script
> 
> samples have bogus timestamps

I'm surprised by this one. I tried with an older Fedora 12 VM (2.6.32.21
kernel). I don't get timestamps with perf-script and perf-report -D
shows -1 which is what I expect given that SAMPLE_TIME attribute is not
set by default. One difference here is that VM's default to cpu-clock
for the event.

> 
> perf-3.0 + 2.6.32
> 
> perf record -agR
> perf script
> 
> error: Samples do not contain timestamps

And this one did not work out so well with the F-12 VM. It caused a
kernel panic - top line on console is perf_swevent_hrtimer (lines
scrolled off the top).

> 
> perf-3.0 + any kernel
> 
> perf record -agT
> perf script
> 
> is happy everywhere. Thanks!

Which is what I would expect. Glad to see my version of reality apply
elsewhere. ;-)

David

> 
>  -Arun
> 

Re: Using a new perf tool against an older kernel

[David Ahern]

On 06/23/2011 06:11 PM, Frederic Weisbecker wrote:
> On Thu, Jun 23, 2011 at 02:02:15PM -0600, David Ahern wrote:
>>
>>
>> On 06/23/2011 01:39 PM, Arun Sharma wrote:
>>> On 6/23/11 7:22 AM, David Ahern wrote:
>>>
>>>> I have not seen issues like this using newer perf userspace against
>>>> older kernels. For example, my laptop was running Fedora 14 (2.6.35) and
>>>> now Fedora 15 (2.6.38.8) and I typically use latest perf builds (e.g.,
>>>> testing patches).
>>>
>>> I narrowed it down to PERF_SAMPLE_RAW:
>>>
>>> perf record -ag -- sleep 1
>>>
>>> is fine, but:
>>>
>>> perf record -agR -- sleep 1
>>>
>>> fails for me most of the time. The reason I needed to use the -R in the
>>> first place is that "perf script" fails on older kernels with:
>>>
>>> Samples do not contain timestamps.
>>>
>>> With the newer perf, I don't get errors, but the timestamp field is
>>> invalid. So I need to use the -R flag to get valid timestamps +
>>> stacktraces out of "perf script".
>>
>> That should have been fixed.
>>
>> And -T on record gets the timestamps.
>>
>> David
> 
> Right, it would be nice to suggest that from perf script when timestamps
> are not recorded.

Timestamps are enabled by default, but that output option is removed if
the samples do not have timestamps. That message is generated if the
user requests timestamps (-f time) in the perf-script output and the
samples do not have timestamps, but Arun did not request that. Arun's
mileage with perf-3.0 definitely varies from what I've seen.

David