PL310 and QoS logic

PL310 and QoS logic

[Armando VISCONTI]

Catalin, All,

I work on the SPEAr1300 SoC, which is a device based on Cortex A9 MP 
(r1p1) and
is using a PL310 L2 cache controller.

We are using the Linux version 2.6.35 and we have also ported some 
patches coming
from you.

Even with this latest version of kernel we are not able to make the L2 
cache working.
It looks like that we are having a problem particularly when using it in 
combination
with MP.

Just to add a very important information, the situation looks to clearly 
stabilize
when  we set the bit9 of PL310 Auxiliary Control Register to '1'. THis 
bit is reserved,
and ARM support told us to avoid setting it.

In fact, this bit when set seems to disable a  QoS logic whose purpose 
is to avoid issues
for which one core is not able to release a semaphore because the other 
core is
continuously trying to take it. So, this QoS logic has been introduce 
inside the
PL310 to set a kind of priority (a QoS).

Strangely enough this issue looks exactly what we are seeing.
It explains all of our observations.

So, it seems that for us writing this bit9 to '1' has the effect of
enabling (and not disabling) the QoS logic.

Have you ever heard an issue like that?

Thx,
Armando

PL310 and QoS logic

[shiraz hashim]

Hello Russell, Catalin,

On Mon, Aug 2, 2010 at 8:14 PM, Armando VISCONTI
<armando.visconti@st.com> wrote:
> Catalin, All,
>
> I work on the SPEAr1300 SoC, which is a device based on Cortex A9 MP (r1p1)
> and
> is using a PL310 L2 cache controller.
>
> We are using the Linux version 2.6.35 and we have also ported some patches
> coming
> from you.
>
> Even with this latest version of kernel we are not able to make the L2 cache
> working.
> It looks like that we are having a problem particularly when using it in
> combination
> with MP.
>
> Just to add a very important information, the situation looks to clearly
> stabilize
> when ?we set the bit9 of PL310 Auxiliary Control Register to '1'. THis bit
> is reserved,
> and ARM support told us to avoid setting it.
>
> In fact, this bit when set seems to disable a ?QoS logic whose purpose is to
> avoid issues
> for which one core is not able to release a semaphore because the other core
> is
> continuously trying to take it. So, this QoS logic has been introduce inside
> the
> PL310 to set a kind of priority (a QoS).
>
> Strangely enough this issue looks exactly what we are seeing.
> It explains all of our observations.
>
> So, it seems that for us writing this bit9 to '1' has the effect of
> enabling (and not disabling) the QoS logic.
>
> Have you ever heard an issue like that?

Any help on this.

regards
Shiraz

PL310 and QoS logic

[shiraz hashim]

Hello Catalin,

On Wed, Oct 6, 2010 at 2:18 PM, shiraz hashim
<shiraz.linux.kernel@gmail.com> wrote:
> Hello Russell, Catalin,
>
> On Mon, Aug 2, 2010 at 8:14 PM, Armando VISCONTI
> <armando.visconti@st.com> wrote:
>> Catalin, All,
>>
>> I work on the SPEAr1300 SoC, which is a device based on Cortex A9 MP (r1p1)
>> and
>> is using a PL310 L2 cache controller.
>>
>> We are using the Linux version 2.6.35 and we have also ported some patches
>> coming
>> from you.
>>
>> Even with this latest version of kernel we are not able to make the L2 cache
>> working.
>> It looks like that we are having a problem particularly when using it in
>> combination
>> with MP.
>>
>> Just to add a very important information, the situation looks to clearly
>> stabilize
>> when ?we set the bit9 of PL310 Auxiliary Control Register to '1'. THis bit
>> is reserved,
>> and ARM support told us to avoid setting it.
>>
>> In fact, this bit when set seems to disable a ?QoS logic whose purpose is to
>> avoid issues
>> for which one core is not able to release a semaphore because the other core
>> is
>> continuously trying to take it. So, this QoS logic has been introduce inside
>> the
>> PL310 to set a kind of priority (a QoS).
>>
>> Strangely enough this issue looks exactly what we are seeing.
>> It explains all of our observations.
>>
>> So, it seems that for us writing this bit9 to '1' has the effect of
>> enabling (and not disabling) the QoS logic.
>>
>> Have you ever heard an issue like that?
>
> Any help on this.

We were going through ARM PL310 errata list and we came across bug
#738415, which says
"738415:   A cacheable read with address bits [20:5] equal to 0x0000
can be stalled by non-cacheable read traffic targeting other address
region 2"

We believe this could be one of the potential problems which we are
facing, and the
errata says the implication could be that there could be a deadlock if
there are dependent
reads.

Now, how do we ensure that this is the root cause. What is the best
way in which I
can prevent kernel from using cache-lines at each 2MB boundary.
Have you come across this bug? Any information would be really helpful.

thanks

regards
Shiraz

PL310 and QoS logic

[Catalin Marinas]

On Wed, 2010-10-27 at 14:09 +0100, shiraz hashim wrote:
> We were going through ARM PL310 errata list and we came across bug
> #738415, which says
> "738415:   A cacheable read with address bits [20:5] equal to 0x0000
> can be stalled by non-cacheable read traffic targeting other address
> region 2"
> 
> We believe this could be one of the potential problems which we are
> facing, and the
> errata says the implication could be that there could be a deadlock if
> there are dependent
> reads.
> 
> Now, how do we ensure that this is the root cause. What is the best
> way in which I
> can prevent kernel from using cache-lines at each 2MB boundary.
> Have you come across this bug? Any information would be really helpful.

I think more information or conditions for this to happen can be given
by support at arm.com. I don't know the hardware details around this.

In general you don't have any cacheable/non-cacheable dependent reads in
Linux to be able to trigger the deadlock. That's a situation where one
CPU is polling some non-cacheable memory location and a different CPU
cannot continue. The only situation I'm aware of is during secondary CPU
booting or if you boot Linux with CPU with a maxcpus less than the total
number of CPUs (so some of them keep polling the pen_release even at
run-time). But that's easily solvable with a WFE in the pen polling loop
(if your platform has such booting protocol).

On the RealView platforms, the secondary CPUs are waiting in a WFI in
the boot monitor code and waken up one by one via an IPI, so they don't
usually wait in the pen polling loop.

If you want to prevent such memory from being used, you can hack the
memory initialisation code in Linux to reserve a 4K physical page for
every 2MB.

Catalin

PL310 and QoS logic

[shiraz hashim]

Hello Catalin,

On Wed, Oct 27, 2010 at 7:03 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Wed, 2010-10-27 at 14:09 +0100, shiraz hashim wrote:
>> We were going through ARM PL310 errata list and we came across bug
>> #738415, which says
>> "738415: ? A cacheable read with address bits [20:5] equal to 0x0000
>> can be stalled by non-cacheable read traffic targeting other address
>> region 2"
>>
>> We believe this could be one of the potential problems which we are
>> facing, and the
>> errata says the implication could be that there could be a deadlock if
>> there are dependent
>> reads.
>>
>> Now, how do we ensure that this is the root cause. What is the best
>> way in which I
>> can prevent kernel from using cache-lines at each 2MB boundary.
>> Have you come across this bug? Any information would be really helpful.
>
> I think more information or conditions for this to happen can be given
> by support at arm.com. I don't know the hardware details around this.

OK.

> In general you don't have any cacheable/non-cacheable dependent reads in
> Linux to be able to trigger the deadlock. That's a situation where one
> CPU is polling some non-cacheable memory location and a different CPU
> cannot continue. The only situation I'm aware of is during secondary CPU
> booting or if you boot Linux with CPU with a maxcpus less than the total
> number of CPUs (so some of them keep polling the pen_release even at
> run-time). But that's easily solvable with a WFE in the pen polling loop
> (if your platform has such booting protocol).
>
> On the RealView platforms, the secondary CPUs are waiting in a WFI in
> the boot monitor code and waken up one by one via an IPI, so they don't
> usually wait in the pen polling loop.
>
> If you want to prevent such memory from being used, you can hack the
> memory initialisation code in Linux to reserve a 4K physical page for
> every 2MB.

I am not very sure how to do it. Can I use reserve_bootmem_node from
bootmem_init.

-- 
regards
Shiraz Hashim