[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Adrien Vergé]

Usage of ETM tracing facility is currently very limited: user can
only start/stop tracing. This set of patches enables management of
address combinations and PIDs that trigger tracing.

ETM management was done via sysfs entries (trace_info,
trace_running...), this code adds trace_addrrange and trace_pid to
let the user read/write custom values.

This series of patches apply to v3.13-rc2.

Signed-off-by: Adrien Verg? <adrienverge@gmail.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Ben Dooks <ben.dooks@codethink.co.uk>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "zhangwei(Jovi)" <jovi.zhangwei@huawei.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
---
Adrien Verg? (3):
  ARM Coresight: Rename 'comparator' to 'address comparator' in ETM
  ARM Coresight: Add address control support for ETM tracing
  ARM Coresight: Add PID control support for ETM tracing

 arch/arm/Kconfig.debug                    |   1 +
 arch/arm/include/asm/hardware/coresight.h |   7 +-
 arch/arm/kernel/etm.c                     | 139 ++++++++++++++++++++++++++----
 3 files changed, 129 insertions(+), 18 deletions(-)

-- 
1.8.3.1

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Greg Kroah-Hartman]

On Tue, Dec 03, 2013 at 11:39:21PM -0500, Adrien Verg? wrote:
> Usage of ETM tracing facility is currently very limited: user can
> only start/stop tracing. This set of patches enables management of
> address combinations and PIDs that trigger tracing.
> 
> ETM management was done via sysfs entries (trace_info,
> trace_running...), this code adds trace_addrrange and trace_pid to
> let the user read/write custom values.

I have lots to say about this from a sysfs point of view, but first, why
is it in sysfs at all?  Shouldn't all of this be in debugfs?

greg k-h

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Adrien Vergé]

Hi all,

First, please excuse me for these misformatted patches that my email
client messed up. Next time I will use git send-email. The 3 patches
can be found on
http://git.dorsal.polymtl.ca/~averge?p=linux.git;a=shortlog;h=refs/heads/etm_with_addr_and_pid

Greg: you're right, sysfs is a questionable place for tracing/debug
code. Debugfs or /dev entries would be the right place.

Though, the only support for ETM in Linux is there, and currently very
limited because tracing options such as address and pid are not
configurable. The code I propose would make it usable, until someone
with time (and more experience than me ;)) moves it out from sysfs.

Thanks,

Adrien

2013/12/4 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
> On Tue, Dec 03, 2013 at 11:39:21PM -0500, Adrien Verg? wrote:
>> Usage of ETM tracing facility is currently very limited: user can
>> only start/stop tracing. This set of patches enables management of
>> address combinations and PIDs that trigger tracing.
>>
>> ETM management was done via sysfs entries (trace_info,
>> trace_running...), this code adds trace_addrrange and trace_pid to
>> let the user read/write custom values.
>
> I have lots to say about this from a sysfs point of view, but first, why
> is it in sysfs at all?  Shouldn't all of this be in debugfs?
>
> greg k-h

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Greg Kroah-Hartman]

On Wed, Dec 04, 2013 at 11:52:34AM -0500, Adrien Verg? wrote:
> Hi all,
> 
> First, please excuse me for these misformatted patches that my email
> client messed up. Next time I will use git send-email. The 3 patches
> can be found on
> http://git.dorsal.polymtl.ca/~averge?p=linux.git;a=shortlog;h=refs/heads/etm_with_addr_and_pid
> 
> Greg: you're right, sysfs is a questionable place for tracing/debug
> code. Debugfs or /dev entries would be the right place.

Why would /dev be correct?

What's wrong with the existing tracing infrastructure that you need to
create a new one?

> Though, the only support for ETM in Linux is there, and currently very
> limited because tracing options such as address and pid are not
> configurable. The code I propose would make it usable, until someone
> with time (and more experience than me ;)) moves it out from sysfs.

Your pid implementation is broken, see my other email about that :(

And again, what's wrong with the existing tracing functionalty that is
processor agnostic?  Why can't we just delete this driver today and use
the existing trace code?

thanks,

greg k-h

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Adrien Vergé]

2013/12/4 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
> Your pid implementation is broken, see my other email about that :(

Thank you for your remarks on pid. I'll try to correct that.

> And again, what's wrong with the existing tracing functionalty that is
> processor agnostic?  Why can't we just delete this driver today and use
> the existing trace code?

As far as I know, the tracing functionality in
/sys/kernel/debug/tracing does not take advantage of ETM. ETM is a
dedicated hardware that greatly reduces tracing overhead. It only
exists on ARM platforms.

I understand using sysfs here is not the cleanest way. I patched my
kernel to meet my needs (trace a specific process or address range),
and I thought these small modifications could be useful -- until
bigger work is done to remove ETM control from sysfs.

Do you think it's worth correcting my patch (for pid namespaces) and
re-submitting it? Or should we wait for someone to port ETM tracing to
debugfs?

Thanks,

Adrien

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Greg Kroah-Hartman]

On Wed, Dec 04, 2013 at 04:12:56PM -0500, Adrien Verg? wrote:
> 2013/12/4 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
> > Your pid implementation is broken, see my other email about that :(
> 
> Thank you for your remarks on pid. I'll try to correct that.
> 
> > And again, what's wrong with the existing tracing functionalty that is
> > processor agnostic?  Why can't we just delete this driver today and use
> > the existing trace code?
> 
> As far as I know, the tracing functionality in
> /sys/kernel/debug/tracing does not take advantage of ETM. ETM is a
> dedicated hardware that greatly reduces tracing overhead. It only
> exists on ARM platforms.

How much overhead does the existing tracing code have on ARM?  Is ETM
still even needed?  Why not just use ETM for the core tracing code
instead?

> I understand using sysfs here is not the cleanest way. I patched my
> kernel to meet my needs (trace a specific process or address range),
> and I thought these small modifications could be useful -- until
> bigger work is done to remove ETM control from sysfs.
> 
> Do you think it's worth correcting my patch (for pid namespaces) and
> re-submitting it? Or should we wait for someone to port ETM tracing to
> debugfs?

What's wrong with the in-kernel tracing logic that you can't use that
instead of the ETM stuff?

And no, I don't think adding more functionality to this will be good to
do, ideally it could just be dropped...

thanks,

greg k-h

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Adrien Vergé]

2013/12/4 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
> How much overhead does the existing tracing code have on ARM?  Is ETM
> still even needed?  Why not just use ETM for the core tracing code
> instead?

Coresight ETM is not just faster than /sys/kernel/debug/tracing, it
provides more detailed and customisable info. For instance, you can
trace every load, store, instruction fetch, along with the number of
cycles taken, with almost zero-overhead.

> What's wrong with the in-kernel tracing logic that you can't use that
> instead of the ETM stuff?

ETM has a different purpose. Integrating it in
/sys/kernel/debug/tracing would not take advantage of all its
features.

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Greg Kroah-Hartman]

On Wed, Dec 04, 2013 at 10:49:25PM -0500, Adrien Verg? wrote:
> 2013/12/4 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
> > How much overhead does the existing tracing code have on ARM?  Is ETM
> > still even needed?  Why not just use ETM for the core tracing code
> > instead?
> 
> Coresight ETM is not just faster than /sys/kernel/debug/tracing, it
> provides more detailed and customisable info. For instance, you can
> trace every load, store, instruction fetch, along with the number of
> cycles taken, with almost zero-overhead.

Can't you already do that with the 'perf' tool the kernel provides
without the ETM driver?

> > What's wrong with the in-kernel tracing logic that you can't use that
> > instead of the ETM stuff?
> 
> ETM has a different purpose. Integrating it in
> /sys/kernel/debug/tracing would not take advantage of all its
> features.

What is it's purpose then?  At first glance, this seems to be exactly
what 'perf' provides already.  Doesn't perf work on ARM today?

thanks,

greg k-h

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Christopher Covington]

Hi Greg,

On 12/04/2013 11:01 PM, Greg Kroah-Hartman wrote:
> On Wed, Dec 04, 2013 at 10:49:25PM -0500, Adrien Verg? wrote:
>> 2013/12/4 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
>>> How much overhead does the existing tracing code have on ARM?  Is ETM
>>> still even needed?  Why not just use ETM for the core tracing code
>>> instead?

I think support for the Embedded Trace Macrocell is desirable. (Maybe it's not
necesarily *needed*, but in the same way that graphics and audio aren't
necessarily needed when using a desktop machine.) Plugging the ETM into the
core tracing code or maybe into the perf events framework would be
interesting, but do these patches make that work any more difficult?

>> Coresight ETM is not just faster than /sys/kernel/debug/tracing, it
>> provides more detailed and customisable info. For instance, you can
>> trace every load, store, instruction fetch, along with the number of
>> cycles taken, with almost zero-overhead.
> 
> Can't you already do that with the 'perf' tool the kernel provides
> without the ETM driver?

With perf one can get a count of how many instructions have been executed,
with little overhead, but not the full list of opcodes and addresses. (One can
also sample the Program Counter intermittently, which might suffice for
performance analysis, but probably doesn't for most debugging use cases.) I
think with perf one can have a handful of watchpoints looking at a very few
loads and stores, with large overhead. As I understand it, ETM can handle
arbitrarily large regions, with little overhead.

Regards,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Greg Kroah-Hartman]

On Thu, Dec 05, 2013 at 03:12:50PM -0500, Christopher Covington wrote:
> Hi Greg,
> 
> On 12/04/2013 11:01 PM, Greg Kroah-Hartman wrote:
> > On Wed, Dec 04, 2013 at 10:49:25PM -0500, Adrien Verg? wrote:
> >> 2013/12/4 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
> >>> How much overhead does the existing tracing code have on ARM?  Is ETM
> >>> still even needed?  Why not just use ETM for the core tracing code
> >>> instead?
> 
> I think support for the Embedded Trace Macrocell is desirable. (Maybe it's not
> necesarily *needed*, but in the same way that graphics and audio aren't
> necessarily needed when using a desktop machine.) Plugging the ETM into the
> core tracing code or maybe into the perf events framework would be
> interesting, but do these patches make that work any more difficult?

Well, these patches were incorrect, so that's not really a valid
question :)

And adding new features to code that is "dead" and should probably be
removed isn't a good idea, as I'm sure you can understand.

> >> Coresight ETM is not just faster than /sys/kernel/debug/tracing, it
> >> provides more detailed and customisable info. For instance, you can
> >> trace every load, store, instruction fetch, along with the number of
> >> cycles taken, with almost zero-overhead.
> > 
> > Can't you already do that with the 'perf' tool the kernel provides
> > without the ETM driver?
> 
> With perf one can get a count of how many instructions have been executed,
> with little overhead, but not the full list of opcodes and addresses.

Is that a limitation of perf on ARM or perf in general?  For some reason
I thought I had seen this using perf on x86, but it's been a while since
I last used it.

> (One can also sample the Program Counter intermittently, which might
> suffice for performance analysis, but probably doesn't for most
> debugging use cases.) I think with perf one can have a handful of
> watchpoints looking at a very few loads and stores, with large
> overhead. As I understand it, ETM can handle arbitrarily large
> regions, with little overhead.

How much work is it to incorportate ETM into the perf framework?  Don't
you think that this is a better thing to do overall, instead of having
duplicating interfaces for the same thing?

thanks,

greg k-h

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Christopher Covington]

On 12/05/2013 03:16 PM, Greg Kroah-Hartman wrote:
> On Thu, Dec 05, 2013 at 03:12:50PM -0500, Christopher Covington wrote:
[...]
> And adding new features to code that is "dead" and should probably be
> removed isn't a good idea, as I'm sure you can understand.

I would consider feature additions to be a sign of life. Maybe the
architecture or user interface isn't ideal, but would you suggest just as
quickly for media codec or cryptography hardware support be removed? Those
operations can also be performed purely in software, and of course the pure
software implementation is easier to integrate cleanly across multiple
architectures, but as with tracing hardware, software implementations don't
perform at the rate and scale that some use cases require.

>>>> Coresight ETM is not just faster than /sys/kernel/debug/tracing, it
>>>> provides more detailed and customisable info. For instance, you can
>>>> trace every load, store, instruction fetch, along with the number of
>>>> cycles taken, with almost zero-overhead.
>>>
>>> Can't you already do that with the 'perf' tool the kernel provides
>>> without the ETM driver?
>>
>> With perf one can get a count of how many instructions have been executed,
>> with little overhead, but not the full list of opcodes and addresses.
> 
> Is that a limitation of perf on ARM or perf in general?  For some reason
> I thought I had seen this using perf on x86, but it's been a while since
> I last used it.

Try this:

perf record -e instructions:u -c 1 echo hello && perf script

At least on my machines this clearly does not produce an instruction trace.
What you probably are familiar with is the periodic sampling I mentioned below.

>> (One can also sample the Program Counter intermittently, which might
>> suffice for performance analysis, but probably doesn't for most
>> debugging use cases.) I think with perf one can have a handful of
>> watchpoints looking at a very few loads and stores, with large
>> overhead. As I understand it, ETM can handle arbitrarily large
>> regions, with little overhead.
> 
> How much work is it to incorportate ETM into the perf framework?  Don't
> you think that this is a better thing to do overall, instead of having
> duplicating interfaces for the same thing?

I'm not familiar enough with the ETM hardware (and the ETB, the buffer where
the data is stored) and driver to say. One factor may be whether the perf
events framework would need to be extended for complete functionality or could
be used as-is.

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Greg Kroah-Hartman]

On Thu, Dec 05, 2013 at 05:45:12PM -0500, Christopher Covington wrote:
> On 12/05/2013 03:16 PM, Greg Kroah-Hartman wrote:
> > On Thu, Dec 05, 2013 at 03:12:50PM -0500, Christopher Covington wrote:
> [...]
> > And adding new features to code that is "dead" and should probably be
> > removed isn't a good idea, as I'm sure you can understand.
> 
> I would consider feature additions to be a sign of life. Maybe the
> architecture or user interface isn't ideal, but would you suggest just as
> quickly for media codec or cryptography hardware support be removed?

No, but I would ask that they be moved to use the same userspace api
for the same functionality, and not create custom ones just because
"they can".

> > How much work is it to incorportate ETM into the perf framework?  Don't
> > you think that this is a better thing to do overall, instead of having
> > duplicating interfaces for the same thing?
> 
> I'm not familiar enough with the ETM hardware (and the ETB, the buffer where
> the data is stored) and driver to say. One factor may be whether the perf
> events framework would need to be extended for complete functionality or could
> be used as-is.

How about moving this conversation to the proper mailing list for this
type of thing then?

thanks,

greg k-h

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Adrien Vergé]

Hi Greg and Christopher,

Thank you for your feedback.

2013/12/4 Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> What is it's purpose then?  At first glance, this seems to be exactly
> what 'perf' provides already.  Doesn't perf work on ARM today?

On ARM, perf is unable to trace every instruction, and certainly not
recording the number of cycles taken by every of them. Also, ETM is
not just tracing the execution flow. It can trace data, trigger
tracing on events such as address matching, data matching, context ID
change, monitor CPRT... It would be sad not to use these capabilities.

If we want perf to use ETM's full functionality, its framework clearly
needs to be extended. Future Intel processors will embed Processor
Trace [1], which seems to have similar capabilities as ETM on ARM. At
that point, it will be easier to decide what should be in the kernel
or in perf.

I the meantime, while ETM is not supported elsewhere than in sysfs, it
should at least be configurable. The current implementation is almost
not usable.

> Well, these patches were incorrect, so that's not really a valid
> question :)

Should I correct the pid size and send them again?

Thanks,

Adrien

[1]: http://software.intel.com/en-us/blogs/2013/09/18/processor-tracing

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Greg Kroah-Hartman]

On Thu, Dec 05, 2013 at 06:54:49PM -0500, Adrien Verg? wrote:
> > Well, these patches were incorrect, so that's not really a valid
> > question :)
> 
> Should I correct the pid size and send them again?

I'm not the ARM maintainer, so I can't say if I would reject them or
not.

And I don't know if you can fix up the pid issue due to namespaces, how
are you going to specify the namespace of the pid you are interested in?

Oh, and someone should fix up the sysfs files no matter what, they
shouldn't be using "raw" kobjects, but that's not your issue.  I can
send a patch to do that after your additions go in if needed.

thanks,

greg k-h

[PATCH 0/3] ARM Coresight: Enhance ETM tracing control

[Russell King - ARM Linux]

On Thu, Dec 05, 2013 at 04:06:10PM -0800, Greg Kroah-Hartman wrote:
> On Thu, Dec 05, 2013 at 06:54:49PM -0500, Adrien Verg? wrote:
> > > Well, these patches were incorrect, so that's not really a valid
> > > question :)
> > 
> > Should I correct the pid size and send them again?
> 
> I'm not the ARM maintainer, so I can't say if I would reject them or
> not.

Neither am I someone who knows this code; it was contributed by TI, and
I assume TI had I use for it.  I've no idea what that was or how it's
even used - it's not something I particularly care about.

I'd much rather someone else who has a use for ETM (and thus knows what
it should/can do) look after it.