IMAP_ADDR on PPC 8xx

IMAP_ADDR on PPC 8xx

[Walter L. Wimer III]

Hi Everyone,

I'm looking for wisdom/understanding about IMAP_ADDR on PPC 8xx
platforms.

In particular, I have an MPC885ADS board running "U-Boot 1.1.3 (Apr 19
2005 - 13:39:58)".  It will boot neither 2.6.15 nor 2.6.16.11.  After
U-Boot decompresses the kernel, I get no kernel output at all; it just
hangs.

After some debugging, I think things go awry when the code starts
dereferencing IMAP_ADDR as a direct pointer.  IMAP_ADDR is defined to be
0xFF000000, but the MPC885ADS documentation says that the internal
memory map is supposed to at 0x22000000.  In addition, when I look at
the bd_t pointer from U-Boot, it's saying that 0x22000000 is the correct
address.

We have a 2.6.11.7 kernel that successfully boots on this board.  The
port was done by another engineer who has moved on to another job.  His
port includes a patch that replaces all references to IMAP_ADDR with a
global pointer that is initialized using io_remap() of the internal map
value from the U-Boot bd_t structure.

Why is this a problem for us and apparently not for anyone else on this
list?  Is no one else using U-Boot?  Or does everyone else's U-Boot use
0xFF000000 instead of 0x22000000?  Or do I have a different problem
entirely?  :-)

While reading through the archives, I see that using IMAP_ADDR the way
it is currently used is somewhat frowned upon anyway.  Is this one of
those things that we (the PPC Linux community) should fix the "right
way" once and for all?  I'm happy to submit a patch once I understand
what the "right way" is...  :-)



Thanks!!!

Walt Wimer
TimeSys Corporation

Re: IMAP_ADDR on PPC 8xx

[Wolfgang Denk]

In message <1147108983.27101.63.camel@excalibur.timesys.com> you wrote:
> 
> In particular, I have an MPC885ADS board running "U-Boot 1.1.3 (Apr 19
> 2005 - 13:39:58)".  It will boot neither 2.6.15 nor 2.6.16.11.  After
> U-Boot decompresses the kernel, I get no kernel output at all; it just
> hangs.

Probably you forgot to specify a correct  console  device  with  your
boot arguments.

> After some debugging, I think things go awry when the code starts
> dereferencing IMAP_ADDR as a direct pointer.  IMAP_ADDR is defined to be
> 0xFF000000, but the MPC885ADS documentation says that the internal

Correct.

> memory map is supposed to at 0x22000000.  In addition, when I look at
> the bd_t pointer from U-Boot, it's saying that 0x22000000 is the correct
> address.

No. U-Boot uses 0xFF000000. At least the official U-Boot source  tree
does. I don't know where you got yours from, or who might have broken
it.

> Why is this a problem for us and apparently not for anyone else on this
> list?  Is no one else using U-Boot?  Or does everyone else's U-Boot use
> 0xFF000000 instead of 0x22000000?  Or do I have a different problem

Most probably everybody else who uses U-Boot uses a good version with
a high mapping.

> While reading through the archives, I see that using IMAP_ADDR the way
> it is currently used is somewhat frowned upon anyway.  Is this one of
> those things that we (the PPC Linux community) should fix the "right
> way" once and for all?  I'm happy to submit a patch once I understand
> what the "right way" is...  :-)

The memory map requirements of PowerPC systems  have  been  explained
many  times  before,  so a little search in the archives, HOWTOs etc.
should point you quickly for good description why a low mapping  like
0x22000000 cannot work.

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Hokey religions and ancient weapons are  no  substitute  for  a  good
blaster at your side.                                      - Han Solo

Re: IMAP_ADDR on PPC 8xx

[Walter L. Wimer III]

Hi All,

Thanks for you response, Wolfgang.


On Tue, 2006-05-09 at 00:46 +0200, Wolfgang Denk wrote:
> In message <1147108983.27101.63.camel@excalibur.timesys.com> you wrote:
> > 
> > In particular, I have an MPC885ADS board running "U-Boot 1.1.3 (Apr 19
> > 2005 - 13:39:58)".  It will boot neither 2.6.15 nor 2.6.16.11.  After
> > U-Boot decompresses the kernel, I get no kernel output at all; it just
> > hangs.
> 
> Probably you forgot to specify a correct  console  device  with  your
> boot arguments.

Excellent point, but unfortunately this (at least by itself) didn't
help.


> > After some debugging, I think things go awry when the code starts
> > dereferencing IMAP_ADDR as a direct pointer.  IMAP_ADDR is defined to be
> > 0xFF000000, but the MPC885ADS documentation says that the internal
> 
> Correct.
> 
> > memory map is supposed to at 0x22000000.  In addition, when I look at
> > the bd_t pointer from U-Boot, it's saying that 0x22000000 is the correct
> > address.
> 
> No. U-Boot uses 0xFF000000. At least the official U-Boot source  tree
> does. I don't know where you got yours from, or who might have broken
> it.
> 
> > Why is this a problem for us and apparently not for anyone else on this
> > list?  Is no one else using U-Boot?  Or does everyone else's U-Boot use
> > 0xFF000000 instead of 0x22000000?  Or do I have a different problem
> 
> Most probably everybody else who uses U-Boot uses a good version with
> a high mapping.

Interesting.  Thanks for the info.  I'm not certain where this U-Boot
came from -- it was already loaded onto the board when the board "landed
in my lap".  I suspect that this U-Boot may have come from Freescale.


> > While reading through the archives, I see that using IMAP_ADDR the way
> > it is currently used is somewhat frowned upon anyway.  Is this one of
> > those things that we (the PPC Linux community) should fix the "right
> > way" once and for all?  I'm happy to submit a patch once I understand
> > what the "right way" is...  :-)
> 
> The memory map requirements of PowerPC systems  have  been  explained
> many  times  before,  so a little search in the archives, HOWTOs etc.
> should point you quickly for good description why a low mapping  like
> 0x22000000 cannot work.

Thanks again for the advice.  Interestingly, I gave the wrong address
above.  It wasn't 0x22000000, it was 0x02200000 (i.e. even lower!).  And
yet with the "io_remap()'ed global variable" patch, 2.6.11.7 does indeed
work on this board with this U-Boot....  Perhaps this works because this
particular board only has 8MiB of RAM....

I'll definitely investigate switching to a U-Boot built from official
sources.

Interestingly, I ran into a similar problem on a completely different
Freescale board about a year ago.  We have an MPC8272ADS board that
definitely has a Freescale copy of U-Boot and it fails to boot vanilla
kernels due to different definitions for the BCSR addresses (which if I
recall correctly are within the internal memory map and thus controlled
by the IMMR register).  At that time, Kumar appealed to the community on
this mailing list to figure out the "right" solution to the discrepancy.
I didn't have time to do the "right" thing, so I merely #ifdef'd out the
offending code in the kernel and the problem magically went away (I
suspect that the Freescale U-Boot had already initialized things
sufficiently that it wasn't strictly _necessary_ for the kernel to
re-initialize the same registers (even if it might have been
_desired_)).

Bottom line: I'm wondering what the Linux PPC community thinks is the
correct long term solution to these discrepancies.  Should we the
community declare "Freescale U-Boots are considered harmful; never use
them; always use the official U-Boot sources" ???

Or should we create a kernel mechanism to automatically adapt to the
different U-Boot flavors?

In a related vein, as I alluded to in my previous email, there has been
previous discussion on this list about the fact that it is frowned upon
for device drivers to directly dereference IMAP_ADDR.  Instead, I've
seen a recommendation that each individual driver perform an io_remap()
operation on IMAP_ADDR and save the resulting kernel virtual address in
a driver-specific data structure.  Is this a universally-accepted
viewpoint?  Is this something that the community agrees "should be
fixed" and we're just waiting for someone (like me) to volunteer to fix
all the drivers?

Or are there arguments in favor of keeping the direct IMAP_ADDR
dereferences?  For example, if each driver performs its own separate
io_remap(), doesn't that have potentially negative consequences on the
VM system for some PPC implementations (e.g. increased contention for
TLB entries)?

Are these issues addressed by or otherwise impacted by other ongoing PPC
Linux work such as the "ppc" + "ppc64" --> "powerpc" effort / "flat
device tree" stuff???



> Best regards,
> 
> Wolfgang Denk



Thanks!!!

Walt Wimer
TimeSys Corporation

Re: IMAP_ADDR on PPC 8xx

[Wolfgang Denk]

In message <1147194879.2200.41.camel@excalibur.timesys.com> you wrote:
> 
> Thanks again for the advice.  Interestingly, I gave the wrong address
> above.  It wasn't 0x22000000, it was 0x02200000 (i.e. even lower!).  And
> yet with the "io_remap()'ed global variable" patch, 2.6.11.7 does indeed
> work on this board with this U-Boot....  Perhaps this works because this
> particular board only has 8MiB of RAM....

It does not work. It will certainly crash as soon as you start a  few
user space applications.

> Bottom line: I'm wondering what the Linux PPC community thinks is the
> correct long term solution to these discrepancies.  Should we the
> community declare "Freescale U-Boots are considered harmful; never use
> them; always use the official U-Boot sources" ???

Indeed it would be nice if Freescale worked more  directly  with  the
community.

> Or should we create a kernel mechanism to automatically adapt to the
> different U-Boot flavors?

No, of course not. U-Boot is just one boot  loader,  there  are  many
others, and the kernel hast to stay independent.

And it is definitely not the kernel's fault if the boot  loader  sets
up a braindamaged memory map.

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
As in certain cults it is possible to kill a process if you know  its
true name.                      -- Ken Thompson and Dennis M. Ritchie

Re: IMAP_ADDR on PPC 8xx

[Walter L. Wimer III]

On Tue, 2006-05-09 at 22:22 +0200, Wolfgang Denk wrote:
> In message <1147194879.2200.41.camel@excalibur.timesys.com> you wrote:
> > 
> > Thanks again for the advice.  Interestingly, I gave the wrong
> > address above.  It wasn't 0x22000000, it was 0x02200000 (i.e.
> > even lower!).  And yet with the "io_remap()'ed global variable"
> > patch, 2.6.11.7 does indeed work on this board with this U-Boot....
> > Perhaps this works because this particular board only has 8MiB of
> > RAM....
> 
> It does not work. It will certainly crash as soon as you start a  few
> user space applications.


Well, something "interesting" is certainly going on because our 2.6.11.7
kernel *does* work and *does not* crash when running user space
applications.  It runs BusyBox quite happily with multiple processes
(e.g. 3 incoming telnet sessions, a console shell, etc.).

I can only conclude that there is something more to our 2.6.11.7-based
patch than I currently understand.



Cheers!

Walt

IMAP_ADDR on PPC 8xx

[Kenneth Poole]

[-- Attachment #1: Type: text/plain, Size: 1749 bytes --]

In our build, (currently based on 2.6.14.3) we define IMAP_ADDR as
follows:

#define	IMAP_ADDR	(((bd_t *)__res)->bi_immr_base)

With very few exceptions, nearly all driver code that dereferences
IMAP_ADDR can be used unchanged and the IMMR value is always the value
passed up from the bootloader. We build one image that runs on multiple
platforms and some platforms place the IMMR address space at different
addresses than others. It's not a constant.

Regardless, I see little reason to ioremap() the IMMR address. The MMU
is set up in such a way that IMMR based locations can be accessed
directly.

Ken Poole, MRV Communications, Inc.

> In a related vein, as I alluded to in my previous email, there has
been
> previous discussion on this list about the fact that it is frowned
upon
> for device drivers to directly dereference IMAP_ADDR.  Instead, I've
> seen a recommendation that each individual driver perform an
io_remap()
> operation on IMAP_ADDR and save the resulting kernel virtual address
in
> a driver-specific data structure.  Is this a universally-accepted
> viewpoint?  Is this something that the community agrees "should be
> fixed" and we're just waiting for someone (like me) to volunteer to
fix
> all the drivers?

> Or are there arguments in favor of keeping the direct IMAP_ADDR
> dereferences?  For example, if each driver performs its own separate
> io_remap(), doesn't that have potentially negative consequences on the
> VM system for some PPC implementations (e.g. increased contention for
> TLB entries)?

> Are these issues addressed by or otherwise impacted by other ongoing
PPC
> Linux work such as the "ppc" + "ppc64" --> "powerpc" effort / "flat
> device tree" stuff???


[-- Attachment #2: Type: text/html, Size: 6931 bytes --]

Re: IMAP_ADDR on PPC 8xx

[Walter L. Wimer III]

On Tue, 2006-05-09 at 14:38 -0400, Kenneth Poole wrote:
> In our build, (currently based on 2.6.14.3) we define IMAP_ADDR as
> follows:
>=20
> #define IMAP_ADDR       (((bd_t *)__res)->bi_immr_base)

Yes, this is (part of) what our 2.6.11.7-based patch does.


> With very few exceptions, nearly all driver code that dereferences
> IMAP_ADDR can be used unchanged and the IMMR value is always the value
> passed up from the bootloader. We build one image that runs on
> multiple platforms and some platforms place the IMMR address space at
> different addresses than others. It=FFs not a constant.

Exactly.  I think this kind of "automatic adaption" to the particular
platform is what should be in the vanilla kernel.


> Regardless, I see little reason to ioremap() the IMMR address.

This was the second major part of our 2.6.11.7-based patch.  It
performed a single ioremap(), stored the result in a global pointer, and
then used that pointer in all the drivers instead of using IMAP_ADDR
directly.  Personally, I don't have a strong opinion yet as to whether
this is desirable or not.


> The MMU is set up in such a way that IMMR based locations can be
> accessed directly.

I'm still rather fuzzy on whether one can count on this always being the
case on all PPC variants.  (????)



> Ken Poole, MRV Communications, Inc.


Thanks!

Walt

Re: IMAP_ADDR on PPC 8xx

[Dan Malek]

On May 9, 2006, at 3:52 PM, Walter L. Wimer III wrote:

> Exactly.  I think this kind of "automatic adaption" to the particular
> platform is what should be in the vanilla kernel.

This does not mean you can choose some arbitrary value.
There is a small range of high memory addresses that will
work successfully for IMMR.  You may not see any problems
right away, but depending upon drivers selected and the
software features used, some problems will crop up.
There are also MMU performance enhancements that may
be used with certain values, and guaranteed kernel crashes
at some point in the future when abused.

With Linux, the IMMR should always have a value of
0xf0000000 or 0xff000000 for best results.

> This was the second major part of our 2.6.11.7-based patch.  It
> performed a single ioremap(), stored the result in a global  
> pointer, and
> then used that pointer in all the drivers instead of using IMAP_ADDR
> directly.

This is not an acceptable practice.  We are removing all
global pointers like this, and any driver that needs access to
some or all of the IMMR space should be individually mapping
those regions it needs.  Under the covers of ioremap() we are
performing various alignment and reuse of address spaces
in order to support things like performance enhancements
and cache coherent regions.

Thanks.

	-- Dan

Re: IMAP_ADDR on PPC 8xx

[Walter L. Wimer III]

Thanks, Dan!  This is exactly the kind of feedback I was seeking.

Based on your comments and Wolfgang's comments, I conclude that:

     1. The U-Boots on my MPC885ADS and MPC8272ADS boards are
        unequivocally broken and should be replaced with ones from the
        official U-Boot source repositories that use IMMR values
        matching the Linux kernel source.
     2. For now, there is no pressing need to fix the existing drivers;
        they may continue dereferencing IMAP_ADDR directly as they
        currently do.
     3. In the future, the drivers will ultimately migrate toward
        individually calling ioremap() and maintaining their own private
        pointers.


Thanks again,

Walt 



On Tue, 2006-05-09 at 17:51 -0400, Dan Malek wrote:
> On May 9, 2006, at 3:52 PM, Walter L. Wimer III wrote:
> 
> > Exactly.  I think this kind of "automatic adaption" to the particular
> > platform is what should be in the vanilla kernel.
> 
> This does not mean you can choose some arbitrary value.
> There is a small range of high memory addresses that will
> work successfully for IMMR.  You may not see any problems
> right away, but depending upon drivers selected and the
> software features used, some problems will crop up.
> There are also MMU performance enhancements that may
> be used with certain values, and guaranteed kernel crashes
> at some point in the future when abused.
> 
> With Linux, the IMMR should always have a value of
> 0xf0000000 or 0xff000000 for best results.
> 
> > This was the second major part of our 2.6.11.7-based patch.  It
> > performed a single ioremap(), stored the result in a global  
> > pointer, and
> > then used that pointer in all the drivers instead of using IMAP_ADDR
> > directly.
> 
> This is not an acceptable practice.  We are removing all
> global pointers like this, and any driver that needs access to
> some or all of the IMMR space should be individually mapping
> those regions it needs.  Under the covers of ioremap() we are
> performing various alignment and reuse of address spaces
> in order to support things like performance enhancements
> and cache coherent regions.
> 
> Thanks.
> 
> 	-- Dan
> 

Re: IMAP_ADDR on PPC 8xx

[Walter L. Wimer III]

On Wed, 2006-05-10 at 11:23 -0400, Walter L. Wimer III wrote:
> Based on your comments and Wolfgang's comments, I conclude that:
> 
>      1. The U-Boots on my MPC885ADS and MPC8272ADS boards are
>         unequivocally broken and should be replaced with ones from the
>         official U-Boot source repositories that use IMMR values
>         matching the Linux kernel source.


FYI, here's a table from the "MPC885ADS PowerQUICC(tm) Application
Development System User's Guide", available on Freescale's website.
This table seems to confirm how they've configured their U-Boot -- the
IMMR is set to 0x02200000...

This may be fine for non-Linux purposes, but it looks like we need to
spread some gospel to Freescale regarding the correct IMMR address for
U-Boot / Linux....

(Or they need to convince folks on this list why their way is
right.... :-)

(In any case I'm convinced.  I'm switching over to the community U-Boot
that uses 0xFF000000.)


FYI-ly,

Walt



               Freescale Semiconductor, Inc.

             Table 3-2. Memory Map in MP885ADS

                                                            Port
ADDESS RANGE                       Memory Type              Size
------------------------    -----------------------------   ----
00000000 - 007FFFFF (8M)    SDRAM -- CS4                     32

02000000 - 020FFFFF (1M)    Communication ports -- CS5
     02000000 - 020000FF                         ATM25
     02000100 - 020001FF                        ATM155
     02000200 - 020002FF                  T1/E1 framer
     02000300 - 020003FF       BCSR5, control register
     02000400 - 020004FF                      Reserved

02100000 - 02107FFF (32K)   BCSR[0-4] (1) -- CS1             32 (2)
     02100000 - 02107FE3                         BCSR0
     02100004 - 02107FE7                         BCSR1
     02100008 - 02107FEB                         BCSR2
     0210000C - 02107FEF                         BCSR3
     02100010 - 02107FF3                         BCSR4

02108000 - 021FFFFF         Empty space

02200000 - 02203FFF         MPC885 internal MAP (3)          32

02204000 - 0221FFFF         Empty space

02220000 - 02227FFF         Security engine (SEC)            64

02228000 - 027FFFFF         Empty space

02800000 - 029FFFFF (2M)                                     32
02A00000 - 02BFFFFF (2M)    Flash SIMM -- CS0
02C00000 - 02FFFFFF (4M)

03000000 - FFFFFFFF         Empty space
-----------------------------------------------------------------
 (1) The device appears repeatedly in multiples of its size.
     That is, BCSR0 appears at memory locations 2100000, 2100020,
     2100040..., while BCSR1 appears at 2100004, 2100024,
     2100044, and so on.
 (2) Only the upper 16 bits (D0\u2013D15) are used.
 (3) Refer to the MPC885 PowerQUICC Family Reference Manual for
     a complete description of the MPC885 internal memory map.

Re: IMAP_ADDR on PPC 8xx

[Dan Malek]

On May 10, 2006, at 12:49 PM, Walter L. Wimer III wrote:

> FYI, here's a table from the "MPC885ADS PowerQUICC(tm) Application
> Development System User's Guide", available on Freescale's website.
> This table seems to confirm how they've configured their U-Boot -- the
> IMMR is set to 0x02200000...

Freescale never ported U-Boot to the MPC855ADS platform, and
I don't believe it came with any software at all.  This IMMR value was
chosen to be backward compatible with some old software tools to
get a compact address space without the need of using the MMU,
long before we had Linux running on the platform.

> This may be fine for non-Linux purposes, but it looks like we need to
> spread some gospel to Freescale regarding the correct IMMR address for
> U-Boot / Linux....

I think you better be pointing fingers at the clueless person that
provided the software you have, as it didn't come from Freescale nor
any of the public U-Boot sources.

Thanks.

	-- Dan

Re: IMAP_ADDR on PPC 8xx

[John Rigby]

Thanks Dan for the extra info.

Just to let everyone know even though this turned out to not
be our problem we here at Freescale are looking at how we
can make sure this does not happen in the future.  We want
any uboot shipped with future Freescale boards to have the
correct behavior.  We also understand the board documentation
needs to be changed to mention the appropriate value for IMMR
for linux.

John
jrigby@freescale.com

On 5/10/06, Dan Malek <dan@embeddedalley.com> wrote:
>
> On May 10, 2006, at 12:49 PM, Walter L. Wimer III wrote:
>
> > FYI, here's a table from the "MPC885ADS PowerQUICC(tm) Application
> > Development System User's Guide", available on Freescale's website.
> > This table seems to confirm how they've configured their U-Boot -- the
> > IMMR is set to 0x02200000...
>
> Freescale never ported U-Boot to the MPC855ADS platform, and
> I don't believe it came with any software at all.  This IMMR value was
> chosen to be backward compatible with some old software tools to
> get a compact address space without the need of using the MMU,
> long before we had Linux running on the platform.
>
> > This may be fine for non-Linux purposes, but it looks like we need to
> > spread some gospel to Freescale regarding the correct IMMR address for
> > U-Boot / Linux....
>
> I think you better be pointing fingers at the clueless person that
> provided the software you have, as it didn't come from Freescale nor
> any of the public U-Boot sources.
>
> Thanks.
>
>         -- Dan
>
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>

Re: IMAP_ADDR on PPC 8xx

[Walter L. Wimer III]

On Wed, 2006-05-10 at 13:45 -0400, Dan Malek wrote:
> > This may be fine for non-Linux purposes, but it looks like we need
> > to spread some gospel to Freescale regarding the correct IMMR
> > address for U-Boot / Linux....
> 
> I think you better be pointing fingers at the clueless person that
> provided the software you have, as it didn't come from Freescale nor
> any of the public U-Boot sources.


My sincerest apologies to Freescale.  Upon further research, it appears
that it was another TimeSys engineer (who has since moved on to another
job) who built and installed U-Boot on our MPC885ADS board.  Strangely,
it appears that he started with the community U-Boot 1.1.3 and then
added a patch to change the IMMR value (plus a few other things).  I'm
not sure why.  He's a very bright guy, so I'm very puzzled....

Anyway, mystery solved.

Again, my sincerest apologies to Freescale.

I built the community U-Boot 1.1.4 and flashed it onto our board.  Linux
2.6.16.11 now boots much farther than before (I'm now getting printk()
output).  The kernel is hitting a new problem now, but I suspect it may
be related to the TLB code that Marcello has discussed recently.  I'm
going to go look at that patch next.  :-)


> Thanks.
> 
>         -- Dan


Thanks again to everyone!!!

Walt Wimer
TimeSys Corporation