[PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Michael Ellerman]

From: Anshuman Khandual <khandual@linux.vnet.ibm.com>

Right now the config_bhrb() PMU specific call happens after
write_mmcr0(), which actually enables the PMU for event counting and
interrupts. So there is a small window of time where the PMU and BHRB
runs without the required HW branch filter (if any) enabled in BHRB.

This can cause some of the branch samples to be collected through BHRB
without any filter applied and hence affects the correctness of
the results. This patch moves the BHRB config function call before
enabling interrupts.

Here are some data points captured via trace prints which depicts how we
could get PMU interrupts with BHRB filter NOT enabled with a standard
perf record command line (asking for branch record information as well).

    $ perf record -j any_call ls

Before the patch:-

    ls-1962  [003] d...  2065.299590: .perf_event_interrupt: MMCRA: 40000000000
    ls-1962  [003] d...  2065.299603: .perf_event_interrupt: MMCRA: 40000000000
    ...

    All the PMU interrupts before this point did not have the requested
    HW branch filter enabled in the MMCRA.

    ls-1962  [003] d...  2065.299647: .perf_event_interrupt: MMCRA: 40040000000
    ls-1962  [003] d...  2065.299662: .perf_event_interrupt: MMCRA: 40040000000

After the patch:-

    ls-1850  [008] d...   190.311828: .perf_event_interrupt: MMCRA: 40040000000
    ls-1850  [008] d...   190.311848: .perf_event_interrupt: MMCRA: 40040000000

    All the PMU interrupts have the requested HW BHRB branch filter
    enabled in MMCRA.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
[mpe: Fixed up whitespace and cleaned up changelog]
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
---
 arch/powerpc/perf/core-book3s.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Ben this is a bug fix so would be nice for 3.13.

diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c
index 29b89e8..67cf220 100644
--- a/arch/powerpc/perf/core-book3s.c
+++ b/arch/powerpc/perf/core-book3s.c
@@ -1147,6 +1147,9 @@ static void power_pmu_enable(struct pmu *pmu)
 	mmcr0 = ebb_switch_in(ebb, cpuhw->mmcr[0]);
 
 	mb();
+	if (cpuhw->bhrb_users)
+		ppmu->config_bhrb(cpuhw->bhrb_filter);
+
 	write_mmcr0(cpuhw, mmcr0);
 
 	/*
@@ -1158,8 +1161,6 @@ static void power_pmu_enable(struct pmu *pmu)
 	}
 
  out:
-	if (cpuhw->bhrb_users)
-		ppmu->config_bhrb(cpuhw->bhrb_filter);
 
 	local_irq_restore(flags);
 }
-- 
1.8.3.2

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Anshuman Khandual]

On 12/18/2013 07:44 AM, Michael Ellerman wrote:
> From: Anshuman Khandual <khandual@linux.vnet.ibm.com>
> 
> Right now the config_bhrb() PMU specific call happens after
> write_mmcr0(), which actually enables the PMU for event counting and
> interrupts. So there is a small window of time where the PMU and BHRB
> runs without the required HW branch filter (if any) enabled in BHRB.
> 
> This can cause some of the branch samples to be collected through BHRB
> without any filter applied and hence affects the correctness of
> the results. This patch moves the BHRB config function call before
> enabling interrupts.
> 
> Here are some data points captured via trace prints which depicts how we
> could get PMU interrupts with BHRB filter NOT enabled with a standard
> perf record command line (asking for branch record information as well).
> 
>     $ perf record -j any_call ls
> 
> Before the patch:-
> 
>     ls-1962  [003] d...  2065.299590: .perf_event_interrupt: MMCRA: 40000000000
>     ls-1962  [003] d...  2065.299603: .perf_event_interrupt: MMCRA: 40000000000
>     ...
> 
>     All the PMU interrupts before this point did not have the requested
>     HW branch filter enabled in the MMCRA.
> 
>     ls-1962  [003] d...  2065.299647: .perf_event_interrupt: MMCRA: 40040000000
>     ls-1962  [003] d...  2065.299662: .perf_event_interrupt: MMCRA: 40040000000
> 
> After the patch:-
> 
>     ls-1850  [008] d...   190.311828: .perf_event_interrupt: MMCRA: 40040000000
>     ls-1850  [008] d...   190.311848: .perf_event_interrupt: MMCRA: 40040000000
> 
>     All the PMU interrupts have the requested HW BHRB branch filter
>     enabled in MMCRA.
> 
> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
> [mpe: Fixed up whitespace and cleaned up changelog]

Thanks Michael for cleaning and resending this out.

[PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Anshuman Khandual]

Right now the `config_bhrb` PMU specific call happens after write_mmcr0
which actually enables the PMU for event counting and interrupt. So
there is a small window of time where the PMU and BHRB runs without the
required HW branch filter (if any) enabled in BHRB. This can cause some
of the branch samples to be collected through BHRB without any filter
being applied and hence affecting the correctness of the results. This
patch moves the BHRB config function call before enabling the interrupts.

Here are some data points captured via trace prints which depicts how we
could get PMU interrupts with BHRB filter NOT enabled with a standard
perf record command line (asking for branch record information as well).

	perf record -j any_call ls

Before the patch:-

              ls-1962  [003] d...  2065.299590: .perf_event_interrupt: MMCRA: 40000000000
              ls-1962  [003] d...  2065.299603: .perf_event_interrupt: MMCRA: 40000000000
              ls-1962  [003] d...  2065.299611: .perf_event_interrupt: MMCRA: 40000000000
              ls-1962  [003] d...  2065.299618: .perf_event_interrupt: MMCRA: 40000000000
              ls-1962  [003] d...  2065.299625: .perf_event_interrupt: MMCRA: 40000000000
              ls-1962  [003] d...  2065.299632: .perf_event_interrupt: MMCRA: 40000000000
              ls-1962  [003] d...  2065.299639: .perf_event_interrupt: MMCRA: 40000000000

	      --> All the PMU interrupts before this point did not have the
	      --> requested HW branch filter enabled in the MMCRA.

              ls-1962  [003] d...  2065.299647: .perf_event_interrupt: MMCRA: 40040000000
              ls-1962  [003] d...  2065.299662: .perf_event_interrupt: MMCRA: 40040000000
              ls-1962  [003] d...  2065.299700: .perf_event_interrupt: MMCRA: 40040000000
              ls-1962  [003] d...  2065.299798: .perf_event_interrupt: MMCRA: 40040000000
              ls-1962  [003] d...  2065.299956: .perf_event_interrupt: MMCRA: 40040000000
              ls-1962  [003] d...  2065.300145: .perf_event_interrupt: MMCRA: 40040000000
              ls-1962  [003] d...  2065.300347: .perf_event_interrupt: MMCRA: 40040000000
              ls-1962  [003] d...  2065.300556: .perf_event_interrupt: MMCRA: 40040000000
              ls-1962  [003] d...  2065.300771: .perf_event_interrupt: MMCRA: 40040000000

After the patch:-

              ls-1850  [008] d...   190.311828: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.311848: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.311856: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.311863: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.311869: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.311876: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.311884: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.311892: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.311907: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.311945: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.312044: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.312206: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.312397: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d..2   190.312626: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.312814: .perf_event_interrupt: MMCRA: 40040000000
              ls-1850  [008] d...   190.313004: .perf_event_interrupt: MMCRA: 40040000000

	      --> All the PMU interrupts have the requested
	      --> HW BHRB branch filter enabled in MMCRA.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 arch/powerpc/perf/core-book3s.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c
index 29b89e8..7d2f13c 100644
--- a/arch/powerpc/perf/core-book3s.c
+++ b/arch/powerpc/perf/core-book3s.c
@@ -1147,6 +1147,9 @@ static void power_pmu_enable(struct pmu *pmu)
 	mmcr0 = ebb_switch_in(ebb, cpuhw->mmcr[0]);
 
 	mb();
+	if (cpuhw->bhrb_users)
+		ppmu->config_bhrb(cpuhw->bhrb_filter);
+	
 	write_mmcr0(cpuhw, mmcr0);
 
 	/*
@@ -1158,8 +1161,6 @@ static void power_pmu_enable(struct pmu *pmu)
 	}
 
  out:
-	if (cpuhw->bhrb_users)
-		ppmu->config_bhrb(cpuhw->bhrb_filter);
 
 	local_irq_restore(flags);
 }
-- 
1.7.11.7

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Michael Ellerman]

On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote:
> Right now the `config_bhrb` PMU specific call happens after write_mmcr0
> which actually enables the PMU for event counting and interrupt. So
> there is a small window of time where the PMU and BHRB runs without the
> required HW branch filter (if any) enabled in BHRB. This can cause some
> of the branch samples to be collected through BHRB without any filter
> being applied and hence affecting the correctness of the results. This
> patch moves the BHRB config function call before enabling the interrupts.

Patch looks good.

But it reminds me I have an item in my TODO list:
 - "Why can't config_bhrb() be done in compute_mmcr()" ?

cheers

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Anshuman Khandual]

On 10/08/2013 09:51 AM, Michael Ellerman wrote:
> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote:
>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0
>> which actually enables the PMU for event counting and interrupt. So
>> there is a small window of time where the PMU and BHRB runs without the
>> required HW branch filter (if any) enabled in BHRB. This can cause some
>> of the branch samples to be collected through BHRB without any filter
>> being applied and hence affecting the correctness of the results. This
>> patch moves the BHRB config function call before enabling the interrupts.
> 
> Patch looks good.
> 
> But it reminds me I have an item in my TODO list:
>  - "Why can't config_bhrb() be done in compute_mmcr()" ?
> 

compute_mmcr() function deals with generic MMCR* configs for normal PMU
events. Even if BHRB config touches MMCRA register, it's configuration
does not interfere with the PMU config for general events. So its best
to keep them separate. Besides, we can always look at these code consolidation
issues in future. But this patch solves a problem which is happening right now.

Regards
Anshuman

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Michael Ellerman]

On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote:
> On 10/08/2013 09:51 AM, Michael Ellerman wrote:
> > On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote:
> >> Right now the `config_bhrb` PMU specific call happens after write_mmcr0
> >> which actually enables the PMU for event counting and interrupt. So
> >> there is a small window of time where the PMU and BHRB runs without the
> >> required HW branch filter (if any) enabled in BHRB. This can cause some
> >> of the branch samples to be collected through BHRB without any filter
> >> being applied and hence affecting the correctness of the results. This
> >> patch moves the BHRB config function call before enabling the interrupts.
> > 
> > Patch looks good.
> > 
> > But it reminds me I have an item in my TODO list:
> >  - "Why can't config_bhrb() be done in compute_mmcr()" ?
> > 
> 
> compute_mmcr() function deals with generic MMCR* configs for normal PMU
> events. Even if BHRB config touches MMCRA register, it's configuration
> does not interfere with the PMU config for general events. So its best
> to keep them separate. 

I'm unconvinced. If they'd been together to begin with this bug never
would have happened.

And there's the added overhead of another indirect function call.

> Besides, we can always look at these code consolidation
> issues in future. 

The future is now.

> But this patch solves a problem which is happening right now.

Sure, I'm not saying we shouldn't merge it as a fix. But I think we
should do the cleanup to move it into compute_mmcr() for 3.13.

cheers

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Anshuman Khandual]

On 10/09/2013 06:51 AM, Michael Ellerman wrote:
> On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote:
>> On 10/08/2013 09:51 AM, Michael Ellerman wrote:
>>> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote:
>>>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0
>>>> which actually enables the PMU for event counting and interrupt. So
>>>> there is a small window of time where the PMU and BHRB runs without the
>>>> required HW branch filter (if any) enabled in BHRB. This can cause some
>>>> of the branch samples to be collected through BHRB without any filter
>>>> being applied and hence affecting the correctness of the results. This
>>>> patch moves the BHRB config function call before enabling the interrupts.
>>>
>>> Patch looks good.
>>>
>>> But it reminds me I have an item in my TODO list:
>>>  - "Why can't config_bhrb() be done in compute_mmcr()" ?
>>>
>>
>> compute_mmcr() function deals with generic MMCR* configs for normal PMU
>> events. Even if BHRB config touches MMCRA register, it's configuration
>> does not interfere with the PMU config for general events. So its best
>> to keep them separate. 
> 
> I'm unconvinced. If they'd been together to begin with this bug never
> would have happened.

This is an ordering of configuration problem. Putting them together in the
same function does not rule out the chances of this ordering problem. Could
you please kindly explain how this could have been avoided ?

> 
> And there's the added overhead of another indirect function call.
>

This overhead should be minimal given the fact that we already call so
many PMU specific indirect calls. BHRB is a different part of the PMU
hardware, so a separate call for this purpose is not a bad idea. AFAIK,
X86 does that too for LBR. But yes it is debatable.

>> Besides, we can always look at these code consolidation
>> issues in future. 
> 
> The future is now.

What I meant was functional correctness has always more priority than
code consolidation efforts. Yes I will look into this after book3s
software branch filtering code has been merged.

> 
>> But this patch solves a problem which is happening right now.
> 
> Sure, I'm not saying we shouldn't merge it as a fix. But I think we
> should do the cleanup to move it into compute_mmcr() for 3.13.

yeah that sounds reasonable.

Regards
Anshuman

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Michael Ellerman]

On Wed, Oct 09, 2013 at 10:16:32AM +0530, Anshuman Khandual wrote:
> On 10/09/2013 06:51 AM, Michael Ellerman wrote:
> > On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote:
> >> On 10/08/2013 09:51 AM, Michael Ellerman wrote:
> >>> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote:
> >>>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0
> >>>> which actually enables the PMU for event counting and interrupt. So
> >>>> there is a small window of time where the PMU and BHRB runs without the
> >>>> required HW branch filter (if any) enabled in BHRB. This can cause some
> >>>> of the branch samples to be collected through BHRB without any filter
> >>>> being applied and hence affecting the correctness of the results. This
> >>>> patch moves the BHRB config function call before enabling the interrupts.
> >>>
> >>> Patch looks good.
> >>>
> >>> But it reminds me I have an item in my TODO list:
> >>>  - "Why can't config_bhrb() be done in compute_mmcr()" ?
> >>>
> >>
> >> compute_mmcr() function deals with generic MMCR* configs for normal PMU
> >> events. Even if BHRB config touches MMCRA register, it's configuration
> >> does not interfere with the PMU config for general events. So its best
> >> to keep them separate. 
> > 
> > I'm unconvinced. If they'd been together to begin with this bug never
> > would have happened.
> 
> This is an ordering of configuration problem. Putting them together in the
> same function does not rule out the chances of this ordering problem. Could
> you please kindly explain how this could have been avoided ?

The existing code already makes sure to write MMCRA before MMCR0.

cheers

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Anshuman Khandual]

On 10/09/2013 11:33 AM, Michael Ellerman wrote:
> On Wed, Oct 09, 2013 at 10:16:32AM +0530, Anshuman Khandual wrote:
>> On 10/09/2013 06:51 AM, Michael Ellerman wrote:
>>> On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote:
>>>> On 10/08/2013 09:51 AM, Michael Ellerman wrote:
>>>>> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote:
>>>>>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0
>>>>>> which actually enables the PMU for event counting and interrupt. So
>>>>>> there is a small window of time where the PMU and BHRB runs without the
>>>>>> required HW branch filter (if any) enabled in BHRB. This can cause some
>>>>>> of the branch samples to be collected through BHRB without any filter
>>>>>> being applied and hence affecting the correctness of the results. This
>>>>>> patch moves the BHRB config function call before enabling the interrupts.
>>>>>
>>>>> Patch looks good.
>>>>>
>>>>> But it reminds me I have an item in my TODO list:
>>>>>  - "Why can't config_bhrb() be done in compute_mmcr()" ?
>>>>>
>>>>
>>>> compute_mmcr() function deals with generic MMCR* configs for normal PMU
>>>> events. Even if BHRB config touches MMCRA register, it's configuration
>>>> does not interfere with the PMU config for general events. So its best
>>>> to keep them separate. 
>>>
>>> I'm unconvinced. If they'd been together to begin with this bug never
>>> would have happened.
>>
>> This is an ordering of configuration problem. Putting them together in the
>> same function does not rule out the chances of this ordering problem. Could
>> you please kindly explain how this could have been avoided ?
> 
> The existing code already makes sure to write MMCRA before MMCR0.
> 

Thats not true. One example being here at power_pmu_enable function.

        write_mmcr0(cpuhw, mmcr0);

        /*
         * Enable instruction sampling if necessary
         */
        if (cpuhw->mmcr[2] & MMCRA_SAMPLE_ENABLE) {
                mb();
                mtspr(SPRN_MMCRA, cpuhw->mmcr[2]);
        }

Even I think this is not right. Instruction sampling should have been
enabled before we enable PMU interrupts. Else there is a small window
of time where we could have the PMU enabled with events (which requires
sampling) without the sampling itself being enabled in MMCRA.

The only dependency BHRB and generic events have with each other is that
they both are ready for action once the PMU interrupt has been enabled
with MMCR0_PMXE bit.

Regards
Anshuman

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Michael Ellerman]

On Thu, Oct 10, 2013 at 02:20:22PM +0530, Anshuman Khandual wrote:
> On 10/09/2013 11:33 AM, Michael Ellerman wrote:
> > On Wed, Oct 09, 2013 at 10:16:32AM +0530, Anshuman Khandual wrote:
> >> On 10/09/2013 06:51 AM, Michael Ellerman wrote:
> >>> On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote:
> >>>> On 10/08/2013 09:51 AM, Michael Ellerman wrote:
> >>>>> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote:
> >>>>>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0
> >>>>>> which actually enables the PMU for event counting and interrupt. So
> >>>>>> there is a small window of time where the PMU and BHRB runs without the
> >>>>>> required HW branch filter (if any) enabled in BHRB. This can cause some
> >>>>>> of the branch samples to be collected through BHRB without any filter
> >>>>>> being applied and hence affecting the correctness of the results. This
> >>>>>> patch moves the BHRB config function call before enabling the interrupts.
> >>>>>
> >>>>> Patch looks good.
> >>>>>
> >>>>> But it reminds me I have an item in my TODO list:
> >>>>>  - "Why can't config_bhrb() be done in compute_mmcr()" ?
> >>>>>
> >>>>
> >>>> compute_mmcr() function deals with generic MMCR* configs for normal PMU
> >>>> events. Even if BHRB config touches MMCRA register, it's configuration
> >>>> does not interfere with the PMU config for general events. So its best
> >>>> to keep them separate. 
> >>>
> >>> I'm unconvinced. If they'd been together to begin with this bug never
> >>> would have happened.
> >>
> >> This is an ordering of configuration problem. Putting them together in the
> >> same function does not rule out the chances of this ordering problem. Could
> >> you please kindly explain how this could have been avoided ?
> > 
> > The existing code already makes sure to write MMCRA before MMCR0.
> 
> Thats not true. One example being here at power_pmu_enable function.
>
>         write_mmcr0(cpuhw, mmcr0);
> 
>         /*
>          * Enable instruction sampling if necessary
>          */
>         if (cpuhw->mmcr[2] & MMCRA_SAMPLE_ENABLE) {
>                 mb();
>                 mtspr(SPRN_MMCRA, cpuhw->mmcr[2]);
>         }

The only example.

The BHRB config would have been applied prior to that:

	mtspr(SPRN_MMCRA, cpuhw->mmcr[2] & ~MMCRA_SAMPLE_ENABLE);
	mtspr(SPRN_MMCR1, cpuhw->mmcr[1]);
	mtspr(SPRN_MMCR0, (cpuhw->mmcr[0] & ~(MMCR0_PMC1CE | MMCR0_PMCjCE))
				| MMCR0_FC);


So as I said, if the BHRB config was in cpuhw->mmcr[2] then the ordering would
have been correct.

> Even I think this is not right. Instruction sampling should have been
> enabled before we enable PMU interrupts. Else there is a small window
> of time where we could have the PMU enabled with events (which requires
> sampling) without the sampling itself being enabled in MMCRA.

Yes I agree. That's a separate bug, which we'll need to test on all the book3s
platforms we have perf support for.

cheers

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Anshuman Khandual]

On 10/11/2013 07:41 AM, Michael Ellerman wrote:
> On Thu, Oct 10, 2013 at 02:20:22PM +0530, Anshuman Khandual wrote:
>> On 10/09/2013 11:33 AM, Michael Ellerman wrote:
>>> On Wed, Oct 09, 2013 at 10:16:32AM +0530, Anshuman Khandual wrote:
>>>> On 10/09/2013 06:51 AM, Michael Ellerman wrote:
>>>>> On Tue, Oct 08, 2013 at 12:51:18PM +0530, Anshuman Khandual wrote:
>>>>>> On 10/08/2013 09:51 AM, Michael Ellerman wrote:
>>>>>>> On Mon, Oct 07, 2013 at 10:00:26AM +0530, Anshuman Khandual wrote:
>>>>>>>> Right now the `config_bhrb` PMU specific call happens after write_mmcr0
>>>>>>>> which actually enables the PMU for event counting and interrupt. So
>>>>>>>> there is a small window of time where the PMU and BHRB runs without the
>>>>>>>> required HW branch filter (if any) enabled in BHRB. This can cause some
>>>>>>>> of the branch samples to be collected through BHRB without any filter
>>>>>>>> being applied and hence affecting the correctness of the results. This
>>>>>>>> patch moves the BHRB config function call before enabling the interrupts.
>>>>>>>
>>>>>>> Patch looks good.
>>>>>>>
>>>>>>> But it reminds me I have an item in my TODO list:
>>>>>>>  - "Why can't config_bhrb() be done in compute_mmcr()" ?
>>>>>>>
>>>>>>
>>>>>> compute_mmcr() function deals with generic MMCR* configs for normal PMU
>>>>>> events. Even if BHRB config touches MMCRA register, it's configuration
>>>>>> does not interfere with the PMU config for general events. So its best
>>>>>> to keep them separate. 
>>>>>
>>>>> I'm unconvinced. If they'd been together to begin with this bug never
>>>>> would have happened.
>>>>
>>>> This is an ordering of configuration problem. Putting them together in the
>>>> same function does not rule out the chances of this ordering problem. Could
>>>> you please kindly explain how this could have been avoided ?
>>>
>>> The existing code already makes sure to write MMCRA before MMCR0.
>>
>> Thats not true. One example being here at power_pmu_enable function.
>>
>>         write_mmcr0(cpuhw, mmcr0);
>>
>>         /*
>>          * Enable instruction sampling if necessary
>>          */
>>         if (cpuhw->mmcr[2] & MMCRA_SAMPLE_ENABLE) {
>>                 mb();
>>                 mtspr(SPRN_MMCRA, cpuhw->mmcr[2]);
>>         }
> 
> The only example.
> 
> The BHRB config would have been applied prior to that:
> 
> 	mtspr(SPRN_MMCRA, cpuhw->mmcr[2] & ~MMCRA_SAMPLE_ENABLE);
> 	mtspr(SPRN_MMCR1, cpuhw->mmcr[1]);
> 	mtspr(SPRN_MMCR0, (cpuhw->mmcr[0] & ~(MMCR0_PMC1CE | MMCR0_PMCjCE))
> 				| MMCR0_FC);
> 
> 
> So as I said, if the BHRB config was in cpuhw->mmcr[2] then the ordering would
> have been correct.

I got your point. But ...

> 
>> Even I think this is not right. Instruction sampling should have been
>> enabled before we enable PMU interrupts. Else there is a small window
>> of time where we could have the PMU enabled with events (which requires
>> sampling) without the sampling itself being enabled in MMCRA.
> 
> Yes I agree. That's a separate bug, which we'll need to test on all the book3s
> platforms we have perf support for.

Okay, I guess any platform which supports sampling will definitely want to have
it enabled before we can set the events to count on PMU. Can you think of any
problem which can arise if we move it before the enabling the PMU back ? Else
we can fix this easily.

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Michael Ellerman]

On Fri, Oct 11, 2013 at 10:02:28AM +0530, Anshuman Khandual wrote:
> On 10/11/2013 07:41 AM, Michael Ellerman wrote:
> > On Thu, Oct 10, 2013 at 02:20:22PM +0530, Anshuman Khandual wrote:
> > 
> >> Even I think this is not right. Instruction sampling should have been
> >> enabled before we enable PMU interrupts. Else there is a small window
> >> of time where we could have the PMU enabled with events (which requires
> >> sampling) without the sampling itself being enabled in MMCRA.
> > 
> > Yes I agree. That's a separate bug, which we'll need to test on all the book3s
> > platforms we have perf support for.
> 
> Okay, I guess any platform which supports sampling will definitely want to have
> it enabled before we can set the events to count on PMU. Can you think of any
> problem which can arise if we move it before the enabling the PMU back ? Else
> we can fix this easily.

In theory it should be a trivial change. But hardware can behave in
strange ways, it's possible on some old chip we need to do it the
current way for some reason.

So although I don't think it will be a problem, it could be, so we
will need to test it thoroughly.

cheers

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Anshuman Khandual]

On 10/14/2013 11:49 AM, Michael Ellerman wrote:
> On Fri, Oct 11, 2013 at 10:02:28AM +0530, Anshuman Khandual wrote:
>> On 10/11/2013 07:41 AM, Michael Ellerman wrote:
>>> On Thu, Oct 10, 2013 at 02:20:22PM +0530, Anshuman Khandual wrote:
>>>
>>>> Even I think this is not right. Instruction sampling should have been
>>>> enabled before we enable PMU interrupts. Else there is a small window
>>>> of time where we could have the PMU enabled with events (which requires
>>>> sampling) without the sampling itself being enabled in MMCRA.
>>>
>>> Yes I agree. That's a separate bug, which we'll need to test on all the book3s
>>> platforms we have perf support for.
>>
>> Okay, I guess any platform which supports sampling will definitely want to have
>> it enabled before we can set the events to count on PMU. Can you think of any
>> problem which can arise if we move it before the enabling the PMU back ? Else
>> we can fix this easily.
> 
> In theory it should be a trivial change. But hardware can behave in
> strange ways, it's possible on some old chip we need to do it the
> current way for some reason.
> 
> So although I don't think it will be a problem, it could be, so we
> will need to test it thoroughly.

So which are the HW chips, you would like to test this fix for possible problems ? 

Re: [PATCH] powerpc, perf: Configure BHRB filter before enabling PMU interrupts

[Anshuman Khandual]

On 10/16/2013 10:00 AM, Anshuman Khandual wrote:
> On 10/14/2013 11:49 AM, Michael Ellerman wrote:
>> On Fri, Oct 11, 2013 at 10:02:28AM +0530, Anshuman Khandual wrote:
>>> On 10/11/2013 07:41 AM, Michael Ellerman wrote:
>>>> On Thu, Oct 10, 2013 at 02:20:22PM +0530, Anshuman Khandual wrote:
>>>>
>>>>> Even I think this is not right. Instruction sampling should have been
>>>>> enabled before we enable PMU interrupts. Else there is a small window
>>>>> of time where we could have the PMU enabled with events (which requires
>>>>> sampling) without the sampling itself being enabled in MMCRA.
>>>>
>>>> Yes I agree. That's a separate bug, which we'll need to test on all the book3s
>>>> platforms we have perf support for.
>>>
>>> Okay, I guess any platform which supports sampling will definitely want to have
>>> it enabled before we can set the events to count on PMU. Can you think of any
>>> problem which can arise if we move it before the enabling the PMU back ? Else
>>> we can fix this easily.
>>
>> In theory it should be a trivial change. But hardware can behave in
>> strange ways, it's possible on some old chip we need to do it the
>> current way for some reason.
>>
>> So although I don't think it will be a problem, it could be, so we
>> will need to test it thoroughly.
> 
> So which are the HW chips, you would like to test this fix for possible problems ? 
> 

Michael,
	Any updates on this patch ?