Perf counters on SiFive FU540-C000

Perf counters on SiFive FU540-C000

[Wladimir J. van der Laan]

Hello,

I'm planning on doing some optimization work for RISC-V, so I'd like to add
perf support for the performance counters on the FU540-C000—as described in
Section 4.10 of the "SiFive FU540-C000 Manual".

These are very basic: CSRs mhpmevent3 and mhpmevent4 can be used to choose a category
(Instruction Commit Events, Microarchitectural Events, Memory System Events), then
from these categories a bitmask of events can be set that will increase the associated
counter when they happen. I don't understand the rationale for counting multiple
kinds of events in one register, so from what I understand, two counters can be
supported at once in the perf interface.

I found some documentation about platform specfic counters in Documentation/riscv/pmu.txt,
hopefully this is enough to get me started.
(If someone started work on this, please let me know)

Kind regards,
Wladimir van der Laan

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Perf counters on SiFive FU540-C000

[Palmer Dabbelt]

On Sun, 31 Mar 2019 07:37:32 PDT (-0700), laanwj@gmail.com wrote:
> Hello,
>
> I'm planning on doing some optimization work for RISC-V, so I'd like to add
> perf support for the performance counters on the FU540-C000—as described in
> Section 4.10 of the "SiFive FU540-C000 Manual".

That would be great, thanks!

> These are very basic: CSRs mhpmevent3 and mhpmevent4 can be used to choose a category
> (Instruction Commit Events, Microarchitectural Events, Memory System Events), then
> from these categories a bitmask of events can be set that will increase the associated
> counter when they happen. I don't understand the rationale for counting multiple
> kinds of events in one register, so from what I understand, two counters can be
> supported at once in the perf interface.

The idea here is to allow users to have different performance counter 
granularities.  For example, users might want to count all pipeline flushes 
(maybe their framework defines that as a possible counter type) or may want to 
specifically indicate which sort of pipeline flush should be counted.

> I found some documentation about platform specfic counters in Documentation/riscv/pmu.txt,
> hopefully this is enough to get me started.
> (If someone started work on this, please let me know)

Someone from Andes had patches out to add support for their performance 
counters a while ago.  The issue there was that the performance counter 
implementation was chosen at compile time, but it should really be detected at 
run time via some platform-specific mechanism (ie, device tree).

Aside from that I think the only issue was how to expose these to userspace in 
a sane fashion, which is something I thought would be quite tricky.  IIRC 
someone managed to convince me that there was a really simple scheme for doing 
this, but I don't remember what it was.

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Perf counters on SiFive FU540-C000

[Alan Kao]

Hi,

On Tue, Apr 02, 2019 at 10:57:14AM -0700, Palmer Dabbelt wrote:
> On Sun, 31 Mar 2019 07:37:32 PDT (-0700), laanwj@gmail.com wrote:
> >Hello,
> >
> >I'm planning on doing some optimization work for RISC-V, so I'd like to add
> >perf support for the performance counters on the FU540-C000—as described in
> >Section 4.10 of the "SiFive FU540-C000 Manual".
> 
> That would be great, thanks!
> 
> >These are very basic: CSRs mhpmevent3 and mhpmevent4 can be used to choose a category
> >(Instruction Commit Events, Microarchitectural Events, Memory System Events), then
> >from these categories a bitmask of events can be set that will increase the associated
> >counter when they happen. I don't understand the rationale for counting multiple
> >kinds of events in one register, so from what I understand, two counters can be
> >supported at once in the perf interface.
> 
> The idea here is to allow users to have different performance counter
> granularities.  For example, users might want to count all pipeline flushes
> (maybe their framework defines that as a possible counter type) or may want
> to specifically indicate which sort of pipeline flush should be counted.
> 
> >I found some documentation about platform specfic counters in Documentation/riscv/pmu.txt,
> >hopefully this is enough to get me started.
> >(If someone started work on this, please let me know)
> 
> Someone from Andes had patches out to add support for their performance
> counters a while ago.  The issue there was that the performance counter
> implementation was chosen at compile time, but it should really be detected
> at run time via some platform-specific mechanism (ie, device tree).

PMU detection was one of the problems, and should have been solved in previous
patches.  For the rest of the problems, Documentation/riscv/pmu.txt should
serve as a good guide.

> Aside from that I think the only issue was how to expose these to userspace
> in a sane fashion, which is something I thought would be quite tricky.  IIRC
> someone managed to convince me that there was a really simple scheme for
> doing this, but I don't remember what it was.

The corresponding userspace tool is in tools/perf.  Once one finishes the kernel
part, the work left for userspace perf will be the symbolization of PMU raw
events.  The preliminary patch supports common counters only, so I believe this
will be the first platform-specific PMU in RISC-V.

Alan

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Perf counters on SiFive FU540-C000

[Wladimir J. van der Laan]

Hello,

Thanks Alan and Palmer for the input!

On Wed, Apr 03, 2019 at 09:04:26AM +0800, Alan Kao wrote:

> > >These are very basic: CSRs mhpmevent3 and mhpmevent4 can be used to choose a category
> > >(Instruction Commit Events, Microarchitectural Events, Memory System Events), then
> > >from these categories a bitmask of events can be set that will increase the associated
> > >counter when they happen. I don't understand the rationale for counting multiple
> > >kinds of events in one register, so from what I understand, two counters can be
> > >supported at once in the perf interface.
> > 
> > The idea here is to allow users to have different performance counter
> > granularities.  For example, users might want to count all pipeline flushes
> > (maybe their framework defines that as a possible counter type) or may want
> > to specifically indicate which sort of pipeline flush should be counted.

I didn't realize some of them were conceptually hierarchical, seen in that light
it makes a lot of sense and it's a good way of being able to count at various levels.

> > >I found some documentation about platform specfic counters in Documentation/riscv/pmu.txt,
> > >hopefully this is enough to get me started.
> > >(If someone started work on this, please let me know)
> > 
> > Someone from Andes had patches out to add support for their performance
> > counters a while ago.  The issue there was that the performance counter
> > implementation was chosen at compile time, but it should really be detected
> > at run time via some platform-specific mechanism (ie, device tree).
> 
> PMU detection was one of the problems, and should have been solved in previous
> patches.  For the rest of the problems, Documentation/riscv/pmu.txt should
> serve as a good guide.

Looking at the device tree (for 'sifive,fu540g' compatibility?) seems, to me,
to be the right answer here.

It'd be possible to really-detect by configuring a specific event type then
running that instruction and looking at the counters, but that seems
unnecessary.

> > Aside from that I think the only issue was how to expose these to userspace
> > in a sane fashion, which is something I thought would be quite tricky.  IIRC
> > someone managed to convince me that there was a really simple scheme for
> > doing this, but I don't remember what it was.
> 
> The corresponding userspace tool is in tools/perf.  Once one finishes the kernel
> part, the work left for userspace perf will be the symbolization of PMU raw
> events.  The preliminary patch supports common counters only, so I believe this
> will be the first platform-specific PMU in RISC-V.

Right-the user-space interface would be 'perf', the idea would be to make that
work for all the counters so that 'perf top' and profiling work for them.

So it seems to me that on the kernel side, apart from the setup and allocation
logic, that these platform-specific counters can be handled in the same way as
`mcycle` and `minstret` are, they're also simple counters without overrun
notification logic or other fancy features.

Also with the same limitation that they cannot be written to from the kernel
in S-mode, so need to keep track of deltas.

Speaking of which, can the kernel write to the selectors? Oh crap I guess not.

Wladimir

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Perf counters on SiFive FU540-C000

[Alan Kao]

Hi Wladimir,

On Wed, Apr 03, 2019 at 10:42:37AM +0200, Wladimir J. van der Laan wrote:
> Hello,
> 
> Thanks Alan and Palmer for the input!
> 
> On Wed, Apr 03, 2019 at 09:04:26AM +0800, Alan Kao wrote:
> 
> > > >These are very basic: CSRs mhpmevent3 and mhpmevent4 can be used to choose a category
> > > >(Instruction Commit Events, Microarchitectural Events, Memory System Events), then
> > > >from these categories a bitmask of events can be set that will increase the associated
> > > >counter when they happen. I don't understand the rationale for counting multiple
> > > >kinds of events in one register, so from what I understand, two counters can be
> > > >supported at once in the perf interface.
> > > 
> > > The idea here is to allow users to have different performance counter
> > > granularities.  For example, users might want to count all pipeline flushes
> > > (maybe their framework defines that as a possible counter type) or may want
> > > to specifically indicate which sort of pipeline flush should be counted.
> 
> I didn't realize some of them were conceptually hierarchical, seen in that light
> it makes a lot of sense and it's a good way of being able to count at various levels.
> 
> > > >I found some documentation about platform specfic counters in Documentation/riscv/pmu.txt,
> > > >hopefully this is enough to get me started.
> > > >(If someone started work on this, please let me know)
> > > 
> > > Someone from Andes had patches out to add support for their performance
> > > counters a while ago.  The issue there was that the performance counter
> > > implementation was chosen at compile time, but it should really be detected
> > > at run time via some platform-specific mechanism (ie, device tree).
> > 
> > PMU detection was one of the problems, and should have been solved in previous
> > patches.  For the rest of the problems, Documentation/riscv/pmu.txt should
> > serve as a good guide.
> 
> Looking at the device tree (for 'sifive,fu540g' compatibility?) seems, to me,
> to be the right answer here.

Ideally there should be a PMU node in the DTS.  I don't have the DTS for FU540
so you have to check if something alike exists.

> 
> It'd be possible to really-detect by configuring a specific event type then
> running that instruction and looking at the counters, but that seems
> unnecessary.
> 

Sure.  beleiving-reading-parsing the DTS you got is the norm.

> > > Aside from that I think the only issue was how to expose these to userspace
> > > in a sane fashion, which is something I thought would be quite tricky.  IIRC
> > > someone managed to convince me that there was a really simple scheme for
> > > doing this, but I don't remember what it was.
> > 
> > The corresponding userspace tool is in tools/perf.  Once one finishes the kernel
> > part, the work left for userspace perf will be the symbolization of PMU raw
> > events.  The preliminary patch supports common counters only, so I believe this
> > will be the first platform-specific PMU in RISC-V.
> 
> Right-the user-space interface would be 'perf', the idea would be to make that
> work for all the counters so that 'perf top' and profiling work for them.
> 

Oops ... current spec gives you no interrupt when a counter overflows, so
basically you cannot do the profiling by perf even if you finish the event
mapping.  The best you can get is "perf stat" to work.

> So it seems to me that on the kernel side, apart from the setup and allocation
> logic, that these platform-specific counters can be handled in the same way as
> `mcycle` and `minstret` are, they're also simple counters without overrun
> notification logic or other fancy features.
> 
> Also with the same limitation that they cannot be written to from the kernel
> in S-mode, so need to keep track of deltas.
> 
> Speaking of which, can the kernel write to the selectors? Oh crap I guess not.

Unfortunately you are right.  You have to manually hack some SBIs for writing
event masks.

> 
> Wladimir

Alan

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Perf counters on SiFive FU540-C000

[Wladimir J. van der Laan]

Hello Alan,

On Wed, Apr 03, 2019 at 05:27:58PM +0800, Alan Kao wrote:

> > > > >I found some documentation about platform specfic counters in Documentation/riscv/pmu.txt,
> > > > >hopefully this is enough to get me started.
> > > > >(If someone started work on this, please let me know)
> > > > 
> > > > Someone from Andes had patches out to add support for their performance
> > > > counters a while ago.  The issue there was that the performance counter
> > > > implementation was chosen at compile time, but it should really be detected
> > > > at run time via some platform-specific mechanism (ie, device tree).
> > > 
> > > PMU detection was one of the problems, and should have been solved in previous
> > > patches.  For the rest of the problems, Documentation/riscv/pmu.txt should
> > > serve as a good guide.
> > 
> > Looking at the device tree (for 'sifive,fu540g' compatibility?) seems, to me,
> > to be the right answer here.
> 
> Ideally there should be a PMU node in the DTS.  I don't have the DTS for FU540
> so you have to check if something alike exists.

I think this is the DTS embedded into FSBL of the SiFive Unleashed board
(haven't cross-checked with mine):
https://github.com/riscv/riscv-device-tree-doc/blob/master/examples/sifive-hifive_unleashed-microsemi.dts

Cannot find anything about a PMU, but also don't have a grasp of how something
like this (part of CPU registers, but really part of the platform) would even
be described.

> > It'd be possible to really-detect by configuring a specific event type then
> > running that instruction and looking at the counters, but that seems
> > unnecessary.
> > 
> 
> Sure.  beleiving-reading-parsing the DTS you got is the norm.

OK, good.

> > > > Aside from that I think the only issue was how to expose these to userspace
> > > > in a sane fashion, which is something I thought would be quite tricky.  IIRC
> > > > someone managed to convince me that there was a really simple scheme for
> > > > doing this, but I don't remember what it was.
> > > 
> > > The corresponding userspace tool is in tools/perf.  Once one finishes the kernel
> > > part, the work left for userspace perf will be the symbolization of PMU raw
> > > events.  The preliminary patch supports common counters only, so I believe this
> > > will be the first platform-specific PMU in RISC-V.
> > 
> > Right-the user-space interface would be 'perf', the idea would be to make that
> > work for all the counters so that 'perf top' and profiling work for them.
> > 
> 
> Oops ... current spec gives you no interrupt when a counter overflows, so
> basically you cannot do the profiling by perf even if you finish the event
> mapping.  The best you can get is "perf stat" to work.

I don't grasp why the interrupts are so important—aren't the 64-bit counters,
even if they trigger every cycle, unlikely to overflow within the entire
lifespan of the hardware? (and that edge case it could even be detected by
having the new value be smaller than the old one)

> > So it seems to me that on the kernel side, apart from the setup and allocation
> > logic, that these platform-specific counters can be handled in the same way as
> > `mcycle` and `minstret` are, they're also simple counters without overrun
> > notification logic or other fancy features.
> > 
> > Also with the same limitation that they cannot be written to from the kernel
> > in S-mode, so need to keep track of deltas.
> > 
> > Speaking of which, can the kernel write to the selectors? Oh crap I guess not.
> 
> Unfortunately you are right.  You have to manually hack some SBIs for writing
> event masks.

Hehe. Here was me thinking this would be fairly easy and straightforward, but
apparently I would need to go all the way up the stack. I'm not even using
OpenSBI yet.

Wladimir

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: Perf counters on SiFive FU540-C000

[Palmer Dabbelt]

On Wed, 03 Apr 2019 01:42:37 PDT (-0700), laanwj@gmail.com wrote:
> Hello,
>
> Thanks Alan and Palmer for the input!
>
> On Wed, Apr 03, 2019 at 09:04:26AM +0800, Alan Kao wrote:
>
>> > >These are very basic: CSRs mhpmevent3 and mhpmevent4 can be used to choose a category
>> > >(Instruction Commit Events, Microarchitectural Events, Memory System Events), then
>> > >from these categories a bitmask of events can be set that will increase the associated
>> > >counter when they happen. I don't understand the rationale for counting multiple
>> > >kinds of events in one register, so from what I understand, two counters can be
>> > >supported at once in the perf interface.
>> >
>> > The idea here is to allow users to have different performance counter
>> > granularities.  For example, users might want to count all pipeline flushes
>> > (maybe their framework defines that as a possible counter type) or may want
>> > to specifically indicate which sort of pipeline flush should be counted.
>
> I didn't realize some of them were conceptually hierarchical, seen in that light
> it makes a lot of sense and it's a good way of being able to count at various levels.
>
>> > >I found some documentation about platform specfic counters in Documentation/riscv/pmu.txt,
>> > >hopefully this is enough to get me started.
>> > >(If someone started work on this, please let me know)
>> >
>> > Someone from Andes had patches out to add support for their performance
>> > counters a while ago.  The issue there was that the performance counter
>> > implementation was chosen at compile time, but it should really be detected
>> > at run time via some platform-specific mechanism (ie, device tree).
>>
>> PMU detection was one of the problems, and should have been solved in previous
>> patches.  For the rest of the problems, Documentation/riscv/pmu.txt should
>> serve as a good guide.
>
> Looking at the device tree (for 'sifive,fu540g' compatibility?) seems, to me,
> to be the right answer here.
>
> It'd be possible to really-detect by configuring a specific event type then
> running that instruction and looking at the counters, but that seems
> unnecessary.

Ya, that seems crazy.  I think sticking something in the device tree is the 
right way to go.  For now I suppose just looking at the compat string on the 
CPU node should be fine, like you mentioned above, but we'll probably end up 
with a lot of those so coming up with a better scheme makes sense to me.

>> > Aside from that I think the only issue was how to expose these to userspace
>> > in a sane fashion, which is something I thought would be quite tricky.  IIRC
>> > someone managed to convince me that there was a really simple scheme for
>> > doing this, but I don't remember what it was.
>>
>> The corresponding userspace tool is in tools/perf.  Once one finishes the kernel
>> part, the work left for userspace perf will be the symbolization of PMU raw
>> events.  The preliminary patch supports common counters only, so I believe this
>> will be the first platform-specific PMU in RISC-V.
>
> Right-the user-space interface would be 'perf', the idea would be to make that
> work for all the counters so that 'perf top' and profiling work for them.
>
> So it seems to me that on the kernel side, apart from the setup and allocation
> logic, that these platform-specific counters can be handled in the same way as
> `mcycle` and `minstret` are, they're also simple counters without overrun
> notification logic or other fancy features.
>
> Also with the same limitation that they cannot be written to from the kernel
> in S-mode, so need to keep track of deltas.
>
> Speaking of which, can the kernel write to the selectors? Oh crap I guess not.

IIRC there's a mechanism for delegating the counters such that they can be 
written from S-mode (or maybe even U-mode?).  It should be in the ISA manual, 
but if you can't find it LMK.

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv