PowerMac PMU programming

PowerMac PMU programming

[Justin Hibbits]

Hello,

I'm working on various PMU bits for FreeBSD (I know, not related to
Linux), but running into some problems, which the Linux crowd may be
able to help with, as it's already solved for Linux.  I've managed to
get it to sleep for performing the sleep operation and for the CPU speed
switch on MPC7455-based machines, but it goes catatonic, not waking
up.  I've installed the reset vector pointer at 0x0080, which both OS X
and Linux do, and flush the caches, but no luck.  During development of
the Linux driver was this ever encountered?  Any assistance would be
greatly appreciated, I've been banging my head against this off and on
for about 8 months now.

Thanks for any assistance.  I'm not subscribed to the list, so please
keep my address in replies.

Justin Hibbits

Re: PowerMac PMU programming

[Benjamin Herrenschmidt]

On Sun, 2012-06-10 at 09:29 -0400, Justin Hibbits wrote:
> Hello,
> 
> I'm working on various PMU bits for FreeBSD (I know, not related to
> Linux), but running into some problems, which the Linux crowd may be
> able to help with, as it's already solved for Linux.  I've managed to
> get it to sleep for performing the sleep operation and for the CPU speed
> switch on MPC7455-based machines, but it goes catatonic, not waking
> up.  I've installed the reset vector pointer at 0x0080, which both OS X
> and Linux do, and flush the caches, but no luck.  During development of
> the Linux driver was this ever encountered?  Any assistance would be
> greatly appreciated, I've been banging my head against this off and on
> for about 8 months now.
> 
> Thanks for any assistance.  I'm not subscribed to the list, so please
> keep my address in replies.

Yes it was :-) pretty much any bug even minor during that process will
cause just that :-)

One thing paulus did that helped us a lot initially was write a little
synchronous version of the PMU send command and use it to blink the PMU
LED very early on when resuming from sleep, which allows us to debug a
bit.

It takes quite a bit of work (see the code in radeonfb) to get the
graphics back, so most of the time you come back with nothing on screen.

If your machine is not a laptop, there are other possible
considerations, such as the need to restore the PCI-PCI bridge that
might be above the macio chip very early on, before you try to restore
macio itself (we don't do that in Linux at this stage).

Cheers,
Ben.
 

Re: PowerMac PMU programming

[Justin Hibbits]

On Mon, 11 Jun 2012 09:03:13 +1000
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> On Sun, 2012-06-10 at 09:29 -0400, Justin Hibbits wrote:
> > Hello,
> > 
> > I'm working on various PMU bits for FreeBSD (I know, not related to
> > Linux), but running into some problems, which the Linux crowd may be
> > able to help with, as it's already solved for Linux.  I've managed
> > to get it to sleep for performing the sleep operation and for the
> > CPU speed switch on MPC7455-based machines, but it goes catatonic,
> > not waking up.  I've installed the reset vector pointer at 0x0080,
> > which both OS X and Linux do, and flush the caches, but no luck.
> > During development of the Linux driver was this ever encountered?
> > Any assistance would be greatly appreciated, I've been banging my
> > head against this off and on for about 8 months now.
> > 
> > Thanks for any assistance.  I'm not subscribed to the list, so
> > please keep my address in replies.
> 
> Yes it was :-) pretty much any bug even minor during that process will
> cause just that :-)
> 
> One thing paulus did that helped us a lot initially was write a little
> synchronous version of the PMU send command and use it to blink the
> PMU LED very early on when resuming from sleep, which allows us to
> debug a bit.
> 
> It takes quite a bit of work (see the code in radeonfb) to get the
> graphics back, so most of the time you come back with nothing on
> screen.
> 
> If your machine is not a laptop, there are other possible
> considerations, such as the need to restore the PCI-PCI bridge that
> might be above the macio chip very early on, before you try to restore
> macio itself (we don't do that in Linux at this stage).
> 
> Cheers,
> Ben.
>  
> 

Thanks Ben,

I'll settle on just getting the CPU speed change working, losing 333MHz
on my TiBook is a few too many MHz for the poor thing to lose :)  I do
have a desktop, but that's my development platform, so needs to be
at least somewhat stable, so my TiBook is my guinea pig.

I've googled everything I can think of, but do you know of any
definitive PMU documentation?  Or can you give me any hints on what's
needed just for that simple operation?

I noticed that you wind down the ADB queue before killing the PMU.  Is
this necessary?  I only disable the interrupt by writing 0x10 to the
IER, but our interrupt model for the PMU is different from the Linux
one.  Is it also necessary to disable the interrupt input, even though
the register write should be disabling the PMU's asserting of the
line?  There doesn't seem to be much more different than that, and
setting the openpic priority, Linux sets it to 0x0f but I didn't see
anything in OS X doing that (might that be a problem, too?)

I would love to get my TiBook using the full 1GHz.

Thanks,

Justin

Re: PowerMac PMU programming

[Benjamin Herrenschmidt]

On Sun, 2012-06-10 at 23:31 -0400, Justin Hibbits wrote:

> I'll settle on just getting the CPU speed change working, losing 333MHz
> on my TiBook is a few too many MHz for the poor thing to lose :)  I do
> have a desktop, but that's my development platform, so needs to be
> at least somewhat stable, so my TiBook is my guinea pig.
> 
> I've googled everything I can think of, but do you know of any
> definitive PMU documentation?  Or can you give me any hints on what's
> needed just for that simple operation?

No doco no, Darwin and Linux source code are your best bet.

It works like sleep as in: you send the switch command and the PMU will
reboot the processor with the new frequency whenever the processor
enters NAP mode.

This is tricky because there's a loss of cache coherency vs. ongoing
DMAs, hence tricky bits of code in there to deal with the various
caches.

> I noticed that you wind down the ADB queue before killing the PMU.  Is
> this necessary?  I only disable the interrupt by writing 0x10 to the
> IER, but our interrupt model for the PMU is different from the Linux
> one. 

Well that will not disable CB1 (or the external GPIO one). You need to
make sure no other communication with the PMU takes place once you've
sent the sleep command and until you're back.

>  Is it also necessary to disable the interrupt input, even though
> the register write should be disabling the PMU's asserting of the
> line?  There doesn't seem to be much more different than that, and
> setting the openpic priority, Linux sets it to 0x0f but I didn't see
> anything in OS X doing that (might that be a problem, too?)
> 
> I would love to get my TiBook using the full 1GHz.

I don't remember what openpic does in Darwin, I think interrupts are all
masked there as a side effect of the code in KeyLargo.cpp but I'm not
sure that applies to frequency changes.

Cheers,
Ben.

Re: PowerMac PMU programming

[Justin Hibbits]

On Tue, 12 Jun 2012 07:24:22 +1000
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> On Sun, 2012-06-10 at 23:31 -0400, Justin Hibbits wrote:
> 
> > I'll settle on just getting the CPU speed change working, losing
> > 333MHz on my TiBook is a few too many MHz for the poor thing to
> > lose :)  I do have a desktop, but that's my development platform,
> > so needs to be at least somewhat stable, so my TiBook is my guinea
> > pig.
> > 
> > I've googled everything I can think of, but do you know of any
> > definitive PMU documentation?  Or can you give me any hints on
> > what's needed just for that simple operation?
> 
> No doco no, Darwin and Linux source code are your best bet.
> 
> It works like sleep as in: you send the switch command and the PMU
> will reboot the processor with the new frequency whenever the
> processor enters NAP mode.
> 
> This is tricky because there's a loss of cache coherency vs. ongoing
> DMAs, hence tricky bits of code in there to deal with the various
> caches.
> 
> > I noticed that you wind down the ADB queue before killing the PMU.
> > Is this necessary?  I only disable the interrupt by writing 0x10 to
> > the IER, but our interrupt model for the PMU is different from the
> > Linux one. 
> 
> Well that will not disable CB1 (or the external GPIO one). You need to
> make sure no other communication with the PMU takes place once you've
> sent the sleep command and until you're back.

I disable interrupts on the CPU first thing right now with a kernel spin
lock, and don't restore them until I'm ready to return from the speed
switch, so the only communication that should be taking place is what
happens in my thread -- I may be missing something here during the
reset, so I'll have to go over it with a fine-toothed comb, this may be
the wrong thing for me to do.

> >  Is it also necessary to disable the interrupt input, even though
> > the register write should be disabling the PMU's asserting of the
> > line?  There doesn't seem to be much more different than that, and
> > setting the openpic priority, Linux sets it to 0x0f but I didn't see
> > anything in OS X doing that (might that be a problem, too?)
> > 
> > I would love to get my TiBook using the full 1GHz.
> 
> I don't remember what openpic does in Darwin, I think interrupts are
> all masked there as a side effect of the code in KeyLargo.cpp but I'm
> not sure that applies to frequency changes.
> 
> Cheers,
> Ben.
> 
> 

Thanks for all this, it's really helping me understand what needs to
happen.  When I get some free time I'll take a closer look at what I'm
doing and try to fix it with your comments.

Thanks
- Justin

Re: PowerMac PMU programming

[Justin Hibbits]

Hi Ben,

Looking at this again, so bringing up this thread, although it is 4
months old.

On Tue, 12 Jun 2012 07:24:22 +1000
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> On Sun, 2012-06-10 at 23:31 -0400, Justin Hibbits wrote:
>=20
> > I'll settle on just getting the CPU speed change working, losing
> > 333MHz on my TiBook is a few too many MHz for the poor thing to
> > lose :)  I do have a desktop, but that's my development platform,
> > so needs to be at least somewhat stable, so my TiBook is my guinea
> > pig.
> >=20
> > I've googled everything I can think of, but do you know of any
> > definitive PMU documentation?  Or can you give me any hints on
> > what's needed just for that simple operation?
>=20
> No doco no, Darwin and Linux source code are your best bet.
>=20
> It works like sleep as in: you send the switch command and the PMU
> will reboot the processor with the new frequency whenever the
> processor enters NAP mode.
>=20
> This is tricky because there's a loss of cache coherency vs. ongoing
> DMAs, hence tricky bits of code in there to deal with the various
> caches.
>=20
> > I noticed that you wind down the ADB queue before killing the PMU.
> > Is this necessary?  I only disable the interrupt by writing 0x10 to
> > the IER, but our interrupt model for the PMU is different from the
> > Linux one.=20
>=20
> Well that will not disable CB1 (or the external GPIO one). You need to
> make sure no other communication with the PMU takes place once you've
> sent the sleep command and until you're back.
>=20
> >  Is it also necessary to disable the interrupt input, even though
> > the register write should be disabling the PMU's asserting of the
> > line?  There doesn't seem to be much more different than that, and
> > setting the openpic priority, Linux sets it to 0x0f but I didn't see
> > anything in OS X doing that (might that be a problem, too?)
> >=20
> > I would love to get my TiBook using the full 1GHz.
>=20
> I don't remember what openpic does in Darwin, I think interrupts are
> all masked there as a side effect of the code in KeyLargo.cpp but I'm
> not sure that applies to frequency changes.
>=20
> Cheers,
> Ben.

=46rom looking at the Linux PMU source, and the FreeBSD source, I can see
only cosmetic differences.  However, the biggest thing I see between
the Linux PPC code and the FreeBSD code is that Linux quiesces Open
Firmware right away.  I haven't tried this with FreeBSD yet, but from
your experience with development on this platform, would you think that
not quiescing OFW might cause the catatonic state?

- Justin