some questions on new dual 1Gig G4's support?

some questions on new dual 1Gig G4's support?

[Kevin B. Hendricks]

Hi,

I have never played around in the cpu specific parts of the kernel (except
for cache flush code) and don't really know my way around (yet!).

I have collected the 7455 info (pdf files including errata) from the
Motorola site and rsynced benh latest 2.4 kernel.

I grepped for 7450 and found references to it in the following files:

in arch/ppc/kernel/
in l2cr.S, sleep.S, cputable.c, and idle.c

Ben Hinted that simply duplicating the cputable.c entry for the 7450 to
make a 7455 might be a good beginning.

So this now leads me to a few questions:

1. are the cpu specific things I might have to play with all in the files
I found that reference 7450 or are there other files I need to play with

2. I read the errata for the 7455 and it was quite short.  Are errata
cumulative  in some way?  Are these just the new errata since the 7450?
(i.e. do I need to grab and worry about the 7450 errata too?).  Or has the
errata listed in the 7450 been dealt with by changes in the 7455 and this
is all I need to know hardware errata-wise.

3. Are there any other references I should be reading that describe
differences in the 7450 and the 7455 that are important for getting a
kernel up on my new machine?

Thanks,

Kevin


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: some questions on new dual 1Gig G4's support?

[Benjamin Herrenschmidt]

>1. are the cpu specific things I might have to play with all in the files
>I found that reference 7450 or are there other files I need to play with
>
>2. I read the errata for the 7455 and it was quite short.  Are errata
>cumulative  in some way?  Are these just the new errata since the 7450?
>(i.e. do I need to grab and worry about the 7450 errata too?).  Or has the
>errata listed in the 7450 been dealt with by changes in the 7455 and this
>is all I need to know hardware errata-wise.
>
>3. Are there any other references I should be reading that describe
>differences in the 7450 and the 7455 that are important for getting a
>kernel up on my new machine?

Out of my mind, there are 3 places you want to look at:

cputable.c : add an entry & declare a specific setup_cpu function
for the 7455

head.S : add the setup_cpu for the 7455, possibly copying the one
of the 7450. Then go figure if the L3 is also incompatible with
NAP mode or not. If it is not, you can strip the code that clears
the "can DOZE" feature. Note that ideally, we should change that
code to be 2 different feature bits, one for can DOZE, and one for
can NAP (as the 7450/55 can only NAP, not DOZE).
You can change that easily, basically define the new bit for NAP
in cputable.h, set it for all CPUs (except 7450 rev 2.0) in
cputable, and remove DOZE capability from 7450/55. Once done,
tweak the code in arch/ppc/kernel/idle.c so that it does power
save if either of these bits is set, but also make sure it does
not try to NAP if powersave_nap isn't set (as some host bridges
can't handle it, it will be set properly by pmac_feature).
If your CPU can't DOZE but can NAP, but powersave_nap is false,
do nothing.

pmac_feature.c : add an entry in the machine table for your machine
based on OF "model" property.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: some questions on new dual 1Gig G4's support?

[Kevin B. Hendricks]

Hi Ben,

Thanks!  I see you have hit the big time!  The /. inerview article was
great!  Now I can say I actually know someone famous!  ;-)

One last question:

The 7455 errata basically says to disable the L2 hardware prefetch engine
(MSSCR0[L2PFE] = 0 when disabling the L1 data cache.

It goes on to say that the L1 data cache may be disabled during cache
flushing on some systems or be disabled upon startup.

Do we disable L1 cache's during cache flushing?  If so were should I be
looking for this code to get the workaround in place?

Thanks,

Kevin

On February 2, 2002 03:11, Benjamin Herrenschmidt wrote:
> >1. are the cpu specific things I might have to play with all in the
> > files I found that reference 7450 or are there other files I need to
> > play with
> >
> >2. I read the errata for the 7455 and it was quite short.  Are errata
> >cumulative  in some way?  Are these just the new errata since the 7450?
> >(i.e. do I need to grab and worry about the 7450 errata too?).  Or has
> > the errata listed in the 7450 been dealt with by changes in the 7455
> > and this is all I need to know hardware errata-wise.
> >
> >3. Are there any other references I should be reading that describe
> >differences in the 7450 and the 7455 that are important for getting a
> >kernel up on my new machine?
>
> Out of my mind, there are 3 places you want to look at:
>
> cputable.c : add an entry & declare a specific setup_cpu function
> for the 7455
>
> head.S : add the setup_cpu for the 7455, possibly copying the one
> of the 7450. Then go figure if the L3 is also incompatible with
> NAP mode or not. If it is not, you can strip the code that clears
> the "can DOZE" feature. Note that ideally, we should change that
> code to be 2 different feature bits, one for can DOZE, and one for
> can NAP (as the 7450/55 can only NAP, not DOZE).
> You can change that easily, basically define the new bit for NAP
> in cputable.h, set it for all CPUs (except 7450 rev 2.0) in
> cputable, and remove DOZE capability from 7450/55. Once done,
> tweak the code in arch/ppc/kernel/idle.c so that it does power
> save if either of these bits is set, but also make sure it does
> not try to NAP if powersave_nap isn't set (as some host bridges
> can't handle it, it will be set properly by pmac_feature).
> If your CPU can't DOZE but can NAP, but powersave_nap is false,
> do nothing.
>
> pmac_feature.c : add an entry in the machine table for your machine
> based on OF "model" property.
>

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: some questions on new dual 1Gig G4's support?

[Benjamin Herrenschmidt]

>Thanks!  I see you have hit the big time!  The /. inerview article was
>great!  Now I can say I actually know someone famous!  ;-)
>
>One last question:
>
>The 7455 errata basically says to disable the L2 hardware prefetch engine
>(MSSCR0[L2PFE] = 0 when disabling the L1 data cache.
>
>It goes on to say that the L1 data cache may be disabled during cache
>flushing on some systems or be disabled upon startup.
>
>Do we disable L1 cache's during cache flushing?  If so were should I be
>looking for this code to get the workaround in place?

I don't think we disable L1 except when changing L2CR value or when
going to sleep.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: some questions on new dual 1Gig G4's support?

[Kevin B. Hendricks]

Hi Ben,

> Out of my mind, there are 3 places you want to look at:
>
> cputable.c : add an entry & declare a specific setup_cpu function
> for the 7455

Done


> head.S : add the setup_cpu for the 7455, possibly copying the one
> of the 7450.

Done, I have simply copied the 7540 one until I get a chance to study the
pdfs to see what else changed in hid0 (there are 2 new bits I think, 1 to
enable extra bats and one other one).

>Then go figure if the L3 is also incompatible with
> NAP mode or not. If it is not, you can strip the code that clears
> the "can DOZE" feature. Note that ideally, we should change that
> code to be 2 different feature bits, one for can DOZE, and one for
> can NAP (as the 7450/55 can only NAP, not DOZE).
> You can change that easily, basically define the new bit for NAP
> in cputable.h, set it for all CPUs (except 7450 rev 2.0) in
> cputable, and remove DOZE capability from 7450/55. Once done,
> tweak the code in arch/ppc/kernel/idle.c so that it does power
> save if either of these bits is set, but also make sure it does
> not try to NAP if powersave_nap isn't set (as some host bridges
> can't handle it, it will be set properly by pmac_feature).
> If your CPU can't DOZE but can NAP, but powersave_nap is false,
> do nothing.

Is this L3 cache problem with DOZE and NAP an errata for 7450?  I looked
in the updates 7450 manual (with 7455 specifics already added to it) and I
did not see any limitations on NAP or DOZE (it is 840 some pages long so
perhaps I am not looking in the right places yet).

> pmac_feature.c : add an entry in the machine table for your machine
> based on OF "model" property.

According to the Apple developers note just released for this machine its
model ID is simply "PowerMac3,5" which is unchanged from previous versions
and already covered in the pmac_feature.c file.

So I will spend my time reading up on NAP and DOZE and trying to see if
and how that has changed since the 7450.

Thanks for all your help!

Kevin


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: some questions on new dual 1Gig G4's support?

[Timothy A. Seufert]

At 2:02 PM -0500 2/3/02, Kevin B. Hendricks wrote:

>According to the Apple developers note just released for this machine its
>model ID is simply "PowerMac3,5" which is unchanged from previous versions
>and already covered in the pmac_feature.c file.

Yep.  It is exactly the same motherboard as the previous set of
machines (733 / dual 800 / 867).  There might be firmware changes to
support the new cpus but the hardware is identical, and I believe
that Apple's intent with the model IDs is to identify motherboard
architectures, not cpus.

--
Tim Seufert

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: some questions on new dual 1Gig G4's support?

[benh]

>
>Is this L3 cache problem with DOZE and NAP an errata for 7450?  I looked
>in the updates 7450 manual (with 7455 specifics already added to it) and I
>did not see any limitations on NAP or DOZE (it is 840 some pages long so
>perhaps I am not lookingin the right places yet).

No. It's 2 things. First the NAP must be disabled depending on the
errata (7450 + L3), but also, neither the 7450 nor the 7455 support
DOZE, they can only NAP or do nothing. Worst, on the 7455, it's bad
to set HID0:DOZE as it actually has a different meaning.

So let's split the can doze & can nap into 2 different features.

>> pmac_feature.c : add an entry in the machine table for your machine
>> based on OF "model" property.
>
>According to the Apple developers note just released for this machine its
>model ID is simply "PowerMac3,5" which is unchanged from previous versions
>and already covered in the pmac_feature.c file.

Ok.

>So I will spend my time reading up on NAP and DOZE and trying to see if
>and how that has changed since the 7450.
>
>Thanks for all your help!


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: some questions on new dual 1Gig G4's support?

[Kevin B. Hendricks]

Hi Ben,

> No. It's 2 things. First the NAP must be disabled depending on the
> errata (7450 + L3), but also, neither the 7450 nor the 7455 support
> DOZE, they can only NAP or do nothing. Worst, on the 7455, it's bad
> to set HID0:DOZE as it actually has a different meaning.

Yes, that is the HIGH_BAT_EN (bit 8) of HID0 which I think we should leave
as zero to keep the 7455 compatible with the 7450.  I know nothing about
BAT registers or how to take advantage of block sizes up to 4 Gig, and I
also do not know if/how we should handle the 4 extra special purpose
registers (supevisor mode only).  Some sample exception handling code uses
these extra SPRG registers to hand;le BAT table exceptions to speed things
up.  Someone who actually knows a whole lot more about the best way to use
BATs will have to take a look at all of these extra pieces.

> So let's split the can doze & can nap into 2 different features.

I will do that in cputable.h as you said..

But what about the SLEEP mode my last e-mail mentioned.  It is a different
mode from NAP (does no bus snooping) but saves more power.

Thanks,

Kevin

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: some questions on new dual 1Gig G4's support?

[benh]

>But what about the SLEEP mode my last e-mail mentioned.  It is a different
>mode from NAP (does no bus snooping) but saves more power.

sleep already existed in previous CPUs, we almost don't use it. Actually,
we do use it on some laptops, but that's nothing you should have to care
about.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: some questions on new dual 1Gig G4's support?

[Kevin B. Hendricks]

Hi Ben,

Okay the following is a bit over my head.  So any helpful hints welcomed
from anyone!

>Then go figure if the L3 is also incompatible with
> NAP mode or not. If it is not, you can strip the code that clears
> the "can DOZE" feature. Note that ideally, we should change that
> code to be 2 different feature bits, one for can DOZE, and one for
> can NAP (as the 7450/55 can only NAP, not DOZE).
> You can change that easily, basically define the new bit for NAP
> in cputable.h, set it for all CPUs (except 7450 rev 2.0) in
> cputable, and remove DOZE capability from 7450/55. Once done,
> tweak the code in arch/ppc/kernel/idle.c so that it does power
> save if either of these bits is set, but also make sure it does
> not try to NAP if powersave_nap isn't set (as some host bridges
> can't handle it, it will be set properly by pmac_feature).
> If your CPU can't DOZE but can NAP, but powersave_nap is false,
> do nothing.

I just finished reading Section 10.2 of the newly revised 7450 family User
Manual (that thing is over 800 pages long).

They clearly say that 7451 and 7455 do not have a settable DOZE state.
Instead the DOZE state is just an intermediate state between the Full
Power Mode state and the NAP state.   They have a nice event transition
driven state diagram to illustrate all of this.   According to the manual
the purpose of this transitory state (DOZE) is to allow bus snooping.
There is also a SLEEP mode that is diferent from NAP (saves more power)
and that does not ever enter the trasient DOZE state on its way back to
the FULL Power State.

The differnces between NAP and SLEEP are as follows (according to the
manual)

---
10.2.2 Nap Mode
...
 Because the processor is in nap mode, the system
causes the processor to enter doze state by negating QACK(bar). The
processor snoops bus operations and performs appropriate memory coherency
operations while in the doze mode. When the system reasserts QACK(bar),
the processor returns to the nap mode after completing any memory
operations.
---

versus

---
10.2.3 Sleep Mode
The sleep mode disables all clocking in the MPC7451. The PLL remains
locked for fast wake response. The JTAG logic remains operational. Note
that the MPC7451 does not enter doze state to snoop if it is in sleep
mode. Caches must be flushed to guarantee cache coherency if other system
components access memory while the MPC7451 is sleeping.
---


It is interesting to note that the recommended algo to enter either sleep
or nap includes the following (again quoting from the revised manual)


---
10.2.4 Power Management Software Considerations
Because the MPC7451 is a three-issue processor with out-of-order execution
capability, care must be taken in how the power management mode is
entered. Furthermore, nap and sleep modes require all outstanding bus
operations to be completed before these modes are entered. Normally,
during system configuration time, one of these power management
modes would be selected by setting the appropriate HID0 bit. Later on, the
mode is invoked by setting MSR[POW]. To ensure a clean transition into and
out of a power-management mode, the mtmsr which sets the POW bit must be
preceded by the proper cache flushes, instruction cache disable followed
by isync, TLB invalidates, and then dssall and sync
instructions. The sleep mode entry sequence is as follows:

mtHIDO (NAP | SLEEP)
...
...
dssall
... cache flushes... (1)
... instruction cache disable ... (2)
isync (3)
... TLB invalidates ... (4)
loop    sync
mtmsr[POW = 1]
isync
b loop


The dssall instruction is needed to kill any outstanding stream touch
instructions not killed by a sync.
----

So this looks pretty detailed and it clearly says to flush all caches
before entering and disable the instruction cache but it does not  show
any differences between NAP and SLEEP.

So what advantage of bus snooping with NAP (that tranistions to DOZE and
back) if you must flush all of the cache's before entering and leaving NAP
or SLEEP?

I don't understand the advantage of NAP unless the QACK(bar) signal to the
processor is enabled for all memory writes by all other processors, DMA
chips, etc.   Is this true for *everything* that accesses L1, L2, L3 cache?

Can anyone explain what I am misunderstanding here?

Thanks,

Kevin


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: some questions on new dual 1Gig G4's support?

[benh]

>They clearly say that 7451 and 7455 do not have a settable DOZE state.
>Instead the DOZE state is just an intermediate state between the Full
>Power Mode state and the NAP state.   They have a nice event transition
>driven state diagram to illustrate all of this.   According to the manual
>the purpose of this transitory state (DOZE) is to allow bus snooping.
>There is also a SLEEP mode that is diferent from NAP (saves more power)
>and that does not ever enter the trasient DOZE state on its way back to
>the FULL Power State.

Yes, they don't have the specific DOZE mode (DOZE bit in HID0), which
is why we should split those 2 features. Some CPUs can DOZE & NAP,
some can NAP only, but some montherboard don't allow NAP because the
host bridge don't properly assert QACK to enforce cache coherency.

So instead of the current cruft, let's properly say which CPU can
do what in cputable.c. Then, in idle.c, the algorithm should be:

 if (powersave_nap && cpu_can_nap)
        nap
 else if (cpu_can_doze)
        doze
 else
        do nothing

(powersave_nap is set by pmac_feature and indicates if the motherboard
supports NAP mode & cache coherency or not).

>The differnces between NAP and SLEEP are as follows (according to the
>manual)

 (...)

Yup, this basically explains that SLEEP mode won't enforce cache coherency.
We currently don't use SLEEP mode (except eventually when putting the laptops
to sleep, which is a different matter).

>It is interesting to note that the recommended algo to enter either sleep
>or nap includes the following (again quoting from the revised manual)
>
>
>---
>10.2.4 Power Management Software Considerations
>Because the MPC7451 is a three-issue processor with out-of-order execution
>capability, care must be taken in how the power management mode is
>entered. Furthermore, nap and sleep modes require all outstanding bus
>operations to be completed before these modes are entered. Normally,
>during system configuration time, one of these power management
>modes would be selected by setting the appropriate HID0 bit. Later on, the
>mode is invoked by setting MSR[POW]. To ensure a clean transition into and
>out of a power-management mode, the mtmsr which sets the POW bit must be
>preceded by the proper cache flushes, instruction cache disable followed
>by isync, TLB invalidates, and then dssall and sync
>instructions. The sleep mode entry sequence is as follows:
>
>mtHIDO (NAP | SLEEP)
>...
>...
>dssall
>... cache flushes... (1)
>... instruction cache disable ... (2)
>isync (3)
>... TLB invalidates ... (4)
>loop    sync
>mtmsr[POW = 1]
>isync
>b loop

That's interesting. My understanding was that NAP didn't require that.
This might explains (or might not) some stability issues some people
are still having with dual 7450's.

>
>The dssall instruction is needed to kill any outstanding stream touch
>instructions not killed by a sync.

Ok, this one has to be added I beleive, probably also to the laptop
sleep code.
>
>So this looks pretty detailed and it clearly says to flush all caches
>before entering and disable the instruction cache but it does not  show
>any differences between NAP and SLEEP.
>
>So what advantage of bus snooping with NAP (that tranistions to DOZE and
>back) if you must flush all of the cache's before entering and leaving NAP
>or SLEEP?

Which makes few sense as the purpose of bus snooping is actually to keep
cache coherency. Can someone from MOTO explain ? Edward ?

>I don't understand the advantage of NAP unless the QACK(bar) signal to the
>processor is enabled for all memory writes by all other processors, DMA
>chips, etc.   Is this true for *everything* that accesses L1, L2, L3 cache?
>
>Can anyone explain what I am misunderstanding here?

Hrm... not me.... All I can say is that I currently do none of the above and
use NAP mode on my 7450 based laptop with no problem so far.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/