[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Aaron Williams]

Hi all,

I have 2010.09 U-Boot working fairly well now on our Octeon 63XX processor and 
have a number of observations about U-Boot.

One of our challenges is the fact that the Octeon is a 64-bit multi-core MIPS 
processor that requires 64-bit memory access. In our case, various I/O 
subsystems are mapped to physical addresses outside of the 4GB address space. 
Due to this, we don't even support a 32-bit Linux kernel but require a 64-bit 
kernel. We also support loading applications linked with our SDK onto various 
cores of our processor, which are basically special ELF images.

Another challenge is the fact that we load U-Boot at the top of physical RAM 
which requires that we use TLB mapping due to the fact that it is quite common 
for us to support 4GB or more DRAM in our embedded environment. We need to do 
this since a number of customers have custom operating systems and 
applications that load in the lower 4GB and need all the space they can get. 
The TLB mapping is not nearly as invasive as one might think and it actually 
simplifies the case where we could be loading the same u-boot image from 
different addresses, such as NOR flash, failsafe, NAND and booting over 
PCIe/X. The only places where this does have an impact is in device drivers 
that perform DMA and/or PCI access since we need to make sure to use wrappers 
for mapping the virtual addresses to physical addresses. In our case, I am 
using bus_to_phys and use the 64-bit result to fill in the appropriate 
descriptors. So far I have made changes to EHCI and the e1000 drivers, but 
will likely be working on others like AHCI for SATA. We also don't know where 
we will load U-Boot into physical memory until run-time, even on a per-board 
basis since many boards support DIMMs rather than memory on the board.

I've run into a number of corner cases and bugs in U-Boot which I have fixed 
and hope to soon release patches which are not specific to Octeon:

1. 4GB and larger NAND flash doesn't work. I have a generic patch for this I 
will try and get out this week. Tested with Micron MT29F32G08CBABA.
2. Support for TI TMP42X temperature monitor
3. PCI breaks if we assign the BAR address space to 0xf8000000-0xffffffff (our 
hardware actually maps this to 0x11b00f0000000 or 0x11c00f800000 depending on 
the PCIE bus number). I have a generic fix for this I also hope to release 
this week.
4. Support for Spansion S29GL064N which reports a manufacturer ID of 0x0000 
and requires AMD fixups.
5. Fix for Micron NAND flash MT29F32G08CBABA which erroneously reports a 16-
bit bus when it has an 8-bit BUS.
6. Ability for software to change the u-boot prompt at run-time/boot time. We 
use this to indicate whether we are booting from RAM, FLASH, or failsafe with 
a single U-Boot binary.

Our 64-bit platform is different enough in that it is almost a completely 
different processor compared to the other MIPS platforms. While we were able 
to make use of and expand mipsregs.h and many of the other include files, we 
had to replace almost all of the C code as well as start.S. Even lib/board.c 
could not be reused, though hopefully I can clean this up so it can be used in 
the future.

Due to the vast differences, I have tried to update the MIPS cpu tree to be 
like ARM and PowerPC such that different CPUs have different directories and a 
common mips32 directory.

Just for the CPU portion we have added around 29Kloc of code, not including 
the 440Kloc of code from our SDK which we link with U-Boot (LGPL). We make use 
of our SDK since it abstracts support for all of our chips and makes porting 
to new chips easier, such as our upcoming 68XX series of 32 core processors.

I have tried to minimize our changes to the common code as much as possible 
and have been largely successul. Where I have made changes, I have separated 
out all changes specific to our hardware using #ifdef CONFIG_OCTEON. Some of 
these changes may be useful for other platforms if other platforms choose to 
add 64-bit support.

Moving forward I hope I can get our changes included into the main U-Boot 
release, but this is going to take a while as I still have a lot of cleanup to 
do as well as add support for more of our processors and boards.

We also have a number of custom commands we have added for our platform. Some 
of these may be able to be made more generic, but we use our own commands for 
things like loading our Linux kernel and simple executive applications since 
they require some datastructures to be configured that are tied to our SDK and 
are needed for backwards compatibility.

Most of the code for our platform is also under mips/cpu/octeon since it is 
common between many different boards. The amount of board specific code is 
actually quite small.

I still have a way to go before I can start submitting patches. I need to get 
our code committed to our main trunk in-house then I'll have to clean up some 
of the formatting and break some of our files (our DDR DRAM initialization 
code is over 300K!).

I don't think I can include our SDK as a series of patches on the mailing list 
since it is about 26MB with some of the hardware generated files being 
hundreds of kilobytes to 12MB for our register database file (which 
fortunately isn't needed by u-boot!) It's available under the LGPL but not 
easily accessible through our open-source web site without registration :(

I have also been trying to keep up with the GIT patches to u-boot and I don't 
think it will be all that difficult to move to the latest version since it 
sounds like most of the changes have affected PowerPC and ARM.

-Aaron Williams

[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Wolfgang Denk]

Dear Aaron Williams,

In message <201102071402.37099.Aaron.Williams@caviumnetworks.com> you wrote:
> 
> One of our challenges is the fact that the Octeon is a 64-bit multi-core MIPS 
> processor that requires 64-bit memory access. In our case, various I/O 
> subsystems are mapped to physical addresses outside of the 4GB address space. 
> Due to this, we don't even support a 32-bit Linux kernel but require a 64-bit 
> kernel. We also support loading applications linked with our SDK onto various 
> cores of our processor, which are basically special ELF images.

There are many systems which have a physical address space that
requires more than 32 bit.  So far, we still run a 32 bit U-Boot on
these.  I understand that you are using a 64 bit port of U-Boot?

> Another challenge is the fact that we load U-Boot at the top of physical RAM 
> which requires that we use TLB mapping due to the fact that it is quite common 
> for us to support 4GB or more DRAM in our embedded environment. We need to do 
> this since a number of customers have custom operating systems and 
> applications that load in the lower 4GB and need all the space they can get. 

Existing U-Boot deals with this by mapping just the lower and the
upper parts of available physical memory. See the CONFIG_VERY_BIG_RAM
config option.

> The TLB mapping is not nearly as invasive as one might think and it actually 
> simplifies the case where we could be loading the same u-boot image from 
> different addresses, such as NOR flash, failsafe, NAND and booting over 
> PCIe/X. The only places where this does have an impact is in device drivers 
> that perform DMA and/or PCI access since we need to make sure to use wrappers 
> for mapping the virtual addresses to physical addresses. In our case, I am 
> using bus_to_phys and use the 64-bit result to fill in the appropriate 
> descriptors. So far I have made changes to EHCI and the e1000 drivers, but 
> will likely be working on others like AHCI for SATA. We also don't know where 
> we will load U-Boot into physical memory until run-time, even on a per-board 
> basis since many boards support DIMMs rather than memory on the board.

This is a pretty standard situation - that is what the original design
did, and what all PowerPC (and now all working ARM) boards are doing,
too.

> I've run into a number of corner cases and bugs in U-Boot which I have fixed 
> and hope to soon release patches which are not specific to Octeon:
> 
> 1. 4GB and larger NAND flash doesn't work. I have a generic patch for this I 
> will try and get out this week. Tested with Micron MT29F32G08CBABA.
> 2. Support for TI TMP42X temperature monitor
> 3. PCI breaks if we assign the BAR address space to 0xf8000000-0xffffffff (our 
> hardware actually maps this to 0x11b00f0000000 or 0x11c00f800000 depending on 
> the PCIE bus number). I have a generic fix for this I also hope to release 
> this week.
> 4. Support for Spansion S29GL064N which reports a manufacturer ID of 0x0000 
> and requires AMD fixups.
> 5. Fix for Micron NAND flash MT29F32G08CBABA which erroneously reports a 16-
> bit bus when it has an 8-bit BUS.
> 6. Ability for software to change the u-boot prompt at run-time/boot time. We 
> use this to indicate whether we are booting from RAM, FLASH, or failsafe with 
> a single U-Boot binary.

All these sound pretty harmless and independent of your Octeon
support.

> Our 64-bit platform is different enough in that it is almost a completely 
> different processor compared to the other MIPS platforms. While we were able 
> to make use of and expand mipsregs.h and many of the other include files, we 
> had to replace almost all of the C code as well as start.S. Even lib/board.c 
> could not be reused, though hopefully I can clean this up so it can be used in 
> the future.

We are in the process of overcoming the long-term repercussions of the
original ARM port which did not follow the original (PPC based)
design.  ARM is now fixed, or in the process of being cleaned up.
Unfortunately MIPS folled the ARM example, and here this work still
needs to be done.  So if you are looking for reference
implementations, you rather want to look at PPC code than ARM or MIPS
code.

One part of the unification attempts is that we are trying to
synchronize lib/board.c across architectures, with the goal to use a
single common version soon.  So again, pleasre use PPC as reference
for changes, or at least try not to introduce any additional, new
incompatibilities.

> Just for the CPU portion we have added around 29Kloc of code, not including 
> the 440Kloc of code from our SDK which we link with U-Boot (LGPL). We make use 
> of our SDK since it abstracts support for all of our chips and makes porting 
> to new chips easier, such as our upcoming 68XX series of 32 core processors.

Do I understand correctly t hat this SDK (whatever that might be) will
then have to be included with the U-Boot distribution?

What would be alternatives to static linking, such as to avoid adding
all this code?

> I have tried to minimize our changes to the common code as much as possible 
> and have been largely successul. Where I have made changes, I have separated 
> out all changes specific to our hardware using #ifdef CONFIG_OCTEON. Some of 
> these changes may be useful for other platforms if other platforms choose to 
> add 64-bit support.

Such general changes should then NOT use CONFIG_OCTEON, but some
generic variable name.

> We also have a number of custom commands we have added for our platform. Some 
> of these may be able to be made more generic, but we use our own commands for 
> things like loading our Linux kernel and simple executive applications since 
> they require some datastructures to be configured that are tied to our SDK and 
> are needed for backwards compatibility.

Many boards use board specific commands; I see no problem with having
SoC specific commands either.

> I don't think I can include our SDK as a series of patches on the mailing list 
> since it is about 26MB with some of the hardware generated files being 
> hundreds of kilobytes to 12MB for our register database file (which 
> fortunately isn't needed by u-boot!) It's available under the LGPL but not 
> easily accessible through our open-source web site without registration :(

U-Boot is supposed to be self-sufficient, i. e. to contain all parts
that are required to build a working U-Boot image.  I see a potential
area of conflicts here.

> I have also been trying to keep up with the GIT patches to u-boot and I don't 
> think it will be all that difficult to move to the latest version since it 
> sounds like most of the changes have affected PowerPC and ARM.

Indeed.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Never put off until tomorrow what you can put off indefinitely.

[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Aaron Williams]

On Monday, February 07, 2011 02:26:15 pm Wolfgang Denk wrote:
> Dear Aaron Williams,
> 
> In message <201102071402.37099.Aaron.Williams@caviumnetworks.com> you wrote:
> > One of our challenges is the fact that the Octeon is a 64-bit multi-core
> > MIPS processor that requires 64-bit memory access. In our case, various
> > I/O subsystems are mapped to physical addresses outside of the 4GB
> > address space. Due to this, we don't even support a 32-bit Linux kernel
> > but require a 64-bit kernel. We also support loading applications linked
> > with our SDK onto various cores of our processor, which are basically
> > special ELF images.
> 
> There are many systems which have a physical address space that
> requires more than 32 bit.  So far, we still run a 32 bit U-Boot on
> these.  I understand that you are using a 64 bit port of U-Boot?

No. We are using a 32-bit port since I think trying to make a 64-bit port of 
U-Boot would be far more involved. We do have support for loading and 
executing 64-bit ELF images, however. We use special memcpy/memset functions 
for 64-bit memory addressing in these cases.

> 
> > Another challenge is the fact that we load U-Boot at the top of physical
> > RAM which requires that we use TLB mapping due to the fact that it is
> > quite common for us to support 4GB or more DRAM in our embedded
> > environment. We need to do this since a number of customers have custom
> > operating systems and applications that load in the lower 4GB and need
> > all the space they can get.
> 
> Existing U-Boot deals with this by mapping just the lower and the
> upper parts of available physical memory. See the CONFIG_VERY_BIG_RAM
> config option.
> 
I just looked at this and this. The only place I see this used is in 
arch/powerpc/lib/board.c and it looks like it just limits the effective memory 
size to CONFIG_MAX_MEM_MAPPED. This won't work for us. As I said, we need to 
move it out of the lower 4GB when there's more memory involved. We also don't 
want there being holes in the middle of the memory if we can help it, nor do 
we want to place u-boot at the lower end of memory. At least on MIPS the TLB 
support is quite simple and is performed very early on by adding a single TLB 
entry. We have customers that need to make use of the entire KUSEG, for 
example. Our SDK only reserves a very small portion of this.

> > The TLB mapping is not nearly as invasive as one might think and it
> > actually simplifies the case where we could be loading the same u-boot
> > image from different addresses, such as NOR flash, failsafe, NAND and
> > booting over PCIe/X. The only places where this does have an impact is
> > in device drivers that perform DMA and/or PCI access since we need to
> > make sure to use wrappers for mapping the virtual addresses to physical
> > addresses. In our case, I am using bus_to_phys and use the 64-bit result
> > to fill in the appropriate descriptors. So far I have made changes to
> > EHCI and the e1000 drivers, but will likely be working on others like
> > AHCI for SATA. We also don't know where we will load U-Boot into
> > physical memory until run-time, even on a per-board basis since many
> > boards support DIMMs rather than memory on the board.
> 
> This is a pretty standard situation - that is what the original design
> did, and what all PowerPC (and now all working ARM) boards are doing,
> too.

In our case we just map u-boot to always be at 0xC0000000 regardless of where 
it is located in physical memory. In my current setup it is located at 
physical address 0x10f800000. So far I've found that the e1000 and EHCI 
drivers assume that virtual addresses are physical addresses. Furthermore, the 
e1000 driver assumes that readl can be used to access the PCI BAR address 
space, which on our platform is not the case since that is located at 
0x11b00f0000000 or 0x11c00f80000000 and needs the appropriate wrapper 
function. Alternatively I could add more TLB entries to map the PCI BAR 
address space to 0xf0000000 and 0xf8000000 respectively or modify readl/writel 
to make the appropriate translations.

Would using virt_to_bus to convert pointers to DMA addresses be appropriate 
instead of the current assumption that pointers can be used as DMA addresses 
directly? This seems like a portable solution since on platforms where the 
pointer and DMA address are the same the macro would just do nothing. Even if 
we didn't use virtual addresses and were using, say, KSEG1 the pointer and 
physical address don't match.

> 
> > I've run into a number of corner cases and bugs in U-Boot which I have
> > fixed and hope to soon release patches which are not specific to Octeon:
> > 
> > 1. 4GB and larger NAND flash doesn't work. I have a generic patch for
> > this I will try and get out this week. Tested with Micron
> > MT29F32G08CBABA. 2. Support for TI TMP42X temperature monitor
> > 3. PCI breaks if we assign the BAR address space to 0xf8000000-0xffffffff
> > (our hardware actually maps this to 0x11b00f0000000 or 0x11c00f800000
> > depending on the PCIE bus number). I have a generic fix for this I also
> > hope to release this week.
> > 4. Support for Spansion S29GL064N which reports a manufacturer ID of
> > 0x0000 and requires AMD fixups.
> > 5. Fix for Micron NAND flash MT29F32G08CBABA which erroneously reports a
> > 16- bit bus when it has an 8-bit BUS.
> > 6. Ability for software to change the u-boot prompt at run-time/boot
> > time. We use this to indicate whether we are booting from RAM, FLASH, or
> > failsafe with a single U-Boot binary.
> 
> All these sound pretty harmless and independent of your Octeon
> support.
> 
> > Our 64-bit platform is different enough in that it is almost a completely
> > different processor compared to the other MIPS platforms. While we were
> > able to make use of and expand mipsregs.h and many of the other include
> > files, we had to replace almost all of the C code as well as start.S.
> > Even lib/board.c could not be reused, though hopefully I can clean this
> > up so it can be used in the future.
> 
> We are in the process of overcoming the long-term repercussions of the
> original ARM port which did not follow the original (PPC based)
> design.  ARM is now fixed, or in the process of being cleaned up.
> Unfortunately MIPS folled the ARM example, and here this work still
> needs to be done.  So if you are looking for reference
> implementations, you rather want to look at PPC code than ARM or MIPS
> code.
> 
> One part of the unification attempts is that we are trying to
> synchronize lib/board.c across architectures, with the goal to use a
> single common version soon.  So again, pleasre use PPC as reference
> for changes, or at least try not to introduce any additional, new
> incompatibilities.

That is what I did, though for board.c I basically took our old implementation 
and cleaned it up for the new u-boot. I need to make another pass and do a 
proper implementation once I get some time.
> 
> > Just for the CPU portion we have added around 29Kloc of code, not
> > including the 440Kloc of code from our SDK which we link with U-Boot
> > (LGPL). We make use of our SDK since it abstracts support for all of our
> > chips and makes porting to new chips easier, such as our upcoming 68XX
> > series of 32 core processors.
> 
> Do I understand correctly t hat this SDK (whatever that might be) will
> then have to be included with the U-Boot distribution?
> 
> What would be alternatives to static linking, such as to avoid adding
> all this code?
> 
It would have to be included only for the Octeon processors. It is statically 
linked and we don't want to get away from this. Also, some of our u-boot files 
are in turn statically linked against some of our utilities, such as our 
utility which loads and boots u-boot over the PCI(e) bus. It contains all of 
the register definitions that are used on our board as well as functionality 
to initialize and work with different I/O blocks. It provides an abstraction 
to make it easy to deal with all of our various chips and boards as well as 
errata. At the moment in our internal tree we just create symlinks to all of 
the files. The header files I'm placing under arch/mips/include/asm/arch-
octeon and the .c files under arch/mips/cpu/mips/octeon/cvmx to separate them.

Also note that u-boot for Octeon can only be compiled with our toolchain, 
since there is some dependency on some of the include files from our GCC 
distribution as well, plus our toolchain distribution includes support for 
some of the extensions we make use of.

> > I have tried to minimize our changes to the common code as much as
> > possible and have been largely successul. Where I have made changes, I
> > have separated out all changes specific to our hardware using #ifdef
> > CONFIG_OCTEON. Some of these changes may be useful for other platforms
> > if other platforms choose to add 64-bit support.
> 
> Such general changes should then NOT use CONFIG_OCTEON, but some
> generic variable name.
> 
I agree, though some many cases they are not general, such as some of the 
support for our compact flash in cmd_ide.c and a few other areas.

> > We also have a number of custom commands we have added for our platform.
> > Some of these may be able to be made more generic, but we use our own
> > commands for things like loading our Linux kernel and simple executive
> > applications since they require some datastructures to be configured
> > that are tied to our SDK and are needed for backwards compatibility.
> 
> Many boards use board specific commands; I see no problem with having
> SoC specific commands either.

I am placing these commands under arch/mips/cpu/octeon/commands rather than 
clutter up the common code, unless you feel it's better to put all the 
commands under the common code.
> 
> > I don't think I can include our SDK as a series of patches on the mailing
> > list since it is about 26MB with some of the hardware generated files
> > being hundreds of kilobytes to 12MB for our register database file
> > (which fortunately isn't needed by u-boot!) It's available under the
> > LGPL but not easily accessible through our open-source web site without
> > registration :(
> 
> U-Boot is supposed to be self-sufficient, i. e. to contain all parts
> that are required to build a working U-Boot image.  I see a potential
> area of conflicts here.

We don't have any problem including our SDK with U-Boot. I can work on trying 
to cut down on the files that are needed. One thing to note is that we make 
patches and changes to our SDK fairly regularly to fix errata and to support 
new chips.

The main parts I am using are the sections dealing with memory, PCI/PCIE, 
model identification, Ethernet, twsi (I2C), UART, TLB (for virt_to_bus) and a 
few other areas. The memory part we use to tell our SDK which blocks of memory 
are in use by u-boot. There are many areas that are not used by u-boot but 
this may grow. For example, in the future I may make use of the usb support 
for our earlier chips that have a proprietary USB interface or our zip support 
for faster unzip, md5, sha1, etc.

The biggest portions of our SDK are generated files for dealing with errors,  
a register database (12MB!) and include files which define access functions to 
all the various registers in all of the I/O units for all of our chips (which 
also may not be needed). I need to play with this and see what files I can 
remove from the SDK.

> 
> > I have also been trying to keep up with the GIT patches to u-boot and I
> > don't think it will be all that difficult to move to the latest version
> > since it sounds like most of the changes have affected PowerPC and ARM.
> 
> Indeed.
> 
> Best regards,
> 
> Wolfgang Denk

[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Wolfgang Denk]

Dear Aaron Williams,

In message <201102071524.17440.Aaron.Williams@caviumnetworks.com> you wrote:
>
> > these.  I understand that you are using a 64 bit port of U-Boot?
> 
> No. We are using a 32-bit port since I think trying to make a 64-bit port of 
> U-Boot would be far more involved. We do have support for loading and 
> executing 64-bit ELF images, however. We use special memcpy/memset functions 
> for 64-bit memory addressing in these cases.

OK, let's discuss this when we see your code.

> > Existing U-Boot deals with this by mapping just the lower and the
> > upper parts of available physical memory. See the CONFIG_VERY_BIG_RAM
> > config option.
> > 
> I just looked at this and this. The only place I see this used is in 
> arch/powerpc/lib/board.c and it looks like it just limits the effective memory 
> size to CONFIG_MAX_MEM_MAPPED. This won't work for us. As I said, we need to 
> move it out of the lower 4GB when there's more memory involved. We also don't 

Why?

> want there being holes in the middle of the memory if we can help it, nor do 
> we want to place u-boot at the lower end of memory. At least on MIPS the TLB 

The question is what is the lesser of two evils: not mapping all of
the available memory and stick with a total of mapped memory that is
addressable in 32 bit adress space, or adding the complexity to deal
with 64 bit address spaces - which will require changes in _lots_ of
places.

But again I suggst to defer the discussion until we see your code.

> Would using virt_to_bus to convert pointers to DMA addresses be appropriate 
> instead of the current assumption that pointers can be used as DMA addresses 
> directly? This seems like a portable solution since on platforms where the 
> pointer and DMA address are the same the macro would just do nothing. Even if 
> we didn't use virtual addresses and were using, say, KSEG1 the pointer and 
> physical address don't match.

Yes, I think this would indeed be a better approach.

> > Do I understand correctly t hat this SDK (whatever that might be) will
> > then have to be included with the U-Boot distribution?
> > 
> > What would be alternatives to static linking, such as to avoid adding
> > all this code?
> > 
> It would have to be included only for the Octeon processors. It is statically 

I don't undertsand what you mean. Either it is part of the U-Boot
source tree, or it is not.  You cannot add it "only for the Octeon
processors" - either it's there, or it ain't.

Given that we are talking about code in the order of 30% of the total
U-Boot code, probably not conforming to U-Boot coding standards, I am
anything but happy about such an addition.  Assume ever SoC vendor
comes up with similar ideas...

> linked and we don't want to get away from this. Also, some of our u-boot files 
> are in turn statically linked against some of our utilities, such as our 

I understand that you do not _want_ to change this.  My question is:
what would you do if you _had_ to change it?

> Also note that u-boot for Octeon can only be compiled with our toolchain, 
> since there is some dependency on some of the include files from our GCC 
> distribution as well, plus our toolchain distribution includes support for 
> some of the extensions we make use of.

Can this be fixed?  I mean, copying some header files should probably
be a solvable problem.  What about of these "extensions" - are they
absolutely needed in U-Boot?  Usually such extensions are either
performance optimizations which are not really needed in U-Boot, or
other well-localized operations that can b ehandlef with small
assembler stubs.


> > Such general changes should then NOT use CONFIG_OCTEON, but some
> > generic variable name.
> > 
> I agree, though some many cases they are not general, such as some of the 
> support for our compact flash in cmd_ide.c and a few other areas.

In general we do not want to see board or SoC specific changes to
common code.

> > Many boards use board specific commands; I see no problem with having
> > SoC specific commands either.
> 
> I am placing these commands under arch/mips/cpu/octeon/commands rather than 
> clutter up the common code, unless you feel it's better to put all the 
> commands under the common code.

This sounds OK with me.

> > > I don't think I can include our SDK as a series of patches on the mailing
> > > list since it is about 26MB with some of the hardware generated files
> > > being hundreds of kilobytes to 12MB for our register database file
> > > (which fortunately isn't needed by u-boot!) It's available under the
> > > LGPL but not easily accessible through our open-source web site without
> > > registration :(
> > 
> > U-Boot is supposed to be self-sufficient, i. e. to contain all parts
> > that are required to build a working U-Boot image.  I see a potential
> > area of conflicts here.
> 
> We don't have any problem including our SDK with U-Boot. I can work on trying 

But  we probably will have problems adding tons of such code which is
of no use to anybody else.

> The biggest portions of our SDK are generated files for dealing with errors,  
> a register database (12MB!) and include files which define access functions to 

I'd expect that only a tiny percentage of this is actually needed /
used in U-Boot.  We should restric it to these actually used parts.


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"We don't have to protect the environment -- the Second Coming is  at
hand."                                                   - James Watt

[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Aaron Williams]

On Tuesday, February 08, 2011 12:53:27 am Wolfgang Denk wrote:
> Dear Aaron Williams,
> 
> In message <201102071524.17440.Aaron.Williams@caviumnetworks.com> you wrote:
> > > these.  I understand that you are using a 64 bit port of U-Boot?
> > 
> > No. We are using a 32-bit port since I think trying to make a 64-bit port
> > of U-Boot would be far more involved. We do have support for loading and
> > executing 64-bit ELF images, however. We use special memcpy/memset
> > functions for 64-bit memory addressing in these cases.
> 
> OK, let's discuss this when we see your code.
> 
> > > Existing U-Boot deals with this by mapping just the lower and the
> > > upper parts of available physical memory. See the CONFIG_VERY_BIG_RAM
> > > config option.
> > 
> > I just looked at this and this. The only place I see this used is in
> > arch/powerpc/lib/board.c and it looks like it just limits the effective
> > memory size to CONFIG_MAX_MEM_MAPPED. This won't work for us. As I said,
> > we need to move it out of the lower 4GB when there's more memory
> > involved. We also don't
> 
> Why?

As I have said, we need all of the lower portion of memory and do not want to 
introduce holes in the memory space for u-boot for loading 64-bit 
applications. Also, some of our customers are loading their own proprietary 
operating systems that need all of the memory. The overhead from our SDK in 
the lower memory region is only a few kilobytes.

One other nice thing about the TLB support is all of the fixups go away.
> 
> > want there being holes in the middle of the memory if we can help it, nor
> > do we want to place u-boot at the lower end of memory. At least on MIPS
> > the TLB
> 
> The question is what is the lesser of two evils: not mapping all of
> the available memory and stick with a total of mapped memory that is
> addressable in 32 bit adress space, or adding the complexity to deal
> with 64 bit address spaces - which will require changes in _lots_ of
> places.
> 
> But again I suggst to defer the discussion until we see your code.
> 
> > Would using virt_to_bus to convert pointers to DMA addresses be
> > appropriate instead of the current assumption that pointers can be used
> > as DMA addresses directly? This seems like a portable solution since on
> > platforms where the pointer and DMA address are the same the macro would
> > just do nothing. Even if we didn't use virtual addresses and were using,
> > say, KSEG1 the pointer and physical address don't match.
> 
> Yes, I think this would indeed be a better approach.

This is what I am doing, though I'm having to make changes on a driver by 
driver basis to add this. I'm using dma_addr_t for the address result of this 
then using the same method used in Linux to fill in both the upper and lower 
32-bit descriptor portions, i.e. foo->lower = dma_addr & 0xffffffff; foo-
>upper = ((dma_addr >> 16) >> 16); This is how it's done in the Linux kernel. 
So far the drivers I've modified this way seem to work fine.
> 
> > > Do I understand correctly t hat this SDK (whatever that might be) will
> > > then have to be included with the U-Boot distribution?
> > > 
> > > What would be alternatives to static linking, such as to avoid adding
> > > all this code?
> > 
> > It would have to be included only for the Octeon processors. It is
> > statically
> 
> I don't undertsand what you mean. Either it is part of the U-Boot
> source tree, or it is not.  You cannot add it "only for the Octeon
> processors" - either it's there, or it ain't.
> 
> Given that we are talking about code in the order of 30% of the total
> U-Boot code, probably not conforming to U-Boot coding standards, I am
> anything but happy about such an addition.  Assume ever SoC vendor
> comes up with similar ideas...
> 
It can be included with u-boot, though I think first I should try and see how 
much I can strip it down. We cannot separate u-boot completely from the SDK, 
though. We use it extensively for things like Ethernet and will also likely 
make use of it for USB when I add the Synopsis USB support used in our older 
chips (which is not EHCI/OHCI compatible).

> > linked and we don't want to get away from this. Also, some of our u-boot
> > files are in turn statically linked against some of our utilities, such
> > as our
> 
> I understand that you do not _want_ to change this.  My question is:
> what would you do if you _had_ to change it?

We might be able to make some changes, but it might be difficult since we have 
a lot of developers working on this around the world. I would prefer to not 
have to maintain two separate code bases. I will need to see how much I can 
strip out.

> 
> > Also note that u-boot for Octeon can only be compiled with our toolchain,
> > since there is some dependency on some of the include files from our GCC
> > distribution as well, plus our toolchain distribution includes support
> > for some of the extensions we make use of.
> 
> Can this be fixed?  I mean, copying some header files should probably
> be a solvable problem.  What about of these "extensions" - are they
> absolutely needed in U-Boot?  Usually such extensions are either
> performance optimizations which are not really needed in U-Boot, or
> other well-localized operations that can b ehandlef with small
> assembler stubs.

Support for our Octeon2 processor is not in the mainline trunk yet. We also 
have some changes needed to work with our SDK and the Linux kernel with our 
newer chips.

> 
> > > Such general changes should then NOT use CONFIG_OCTEON, but some
> > > generic variable name.
> > 
> > I agree, though some many cases they are not general, such as some of the
> > support for our compact flash in cmd_ide.c and a few other areas.
> 
> In general we do not want to see board or SoC specific changes to
> common code.

The common code is already chock full of them from what I've seen in a number 
of areas. The number of changes for our SoC is actually quite small. In some 
cases I've just changed some functions to use __attribute__((__weak__)) so we 
can define the functionality elsewhere. I count a total of 21 ifdefs in the 
common code, most of those are in cmd_ide.c, one in cmd_boot.c, one in 
cmd_elf.c, two in env_flash.c and some in main.c, which can be removed (those 
are for adding the support for dynamically changing the boot prompt) and one 
in miiphyutil.c. Right now cmd_ide.c can be cleaned up a bit. The problem is 
that cmd_ide.c includes the driver functionality in it, requiring us to do 
this.  I can clean this up by making some of the functions weak. Is this 
reasonable?

One other change I had to make was for the command table. I have to force the 
alignment to 8 bytes in the compiler or very bad things happen.
> 
> > > Many boards use board specific commands; I see no problem with having
> > > SoC specific commands either.
> > 
> > I am placing these commands under arch/mips/cpu/octeon/commands rather
> > than clutter up the common code, unless you feel it's better to put all
> > the commands under the common code.
> 
> This sounds OK with me.
> 
> > > > I don't think I can include our SDK as a series of patches on the
> > > > mailing list since it is about 26MB with some of the hardware
> > > > generated files being hundreds of kilobytes to 12MB for our register
> > > > database file (which fortunately isn't needed by u-boot!) It's
> > > > available under the LGPL but not easily accessible through our
> > > > open-source web site without registration :(
> > > 
> > > U-Boot is supposed to be self-sufficient, i. e. to contain all parts
> > > that are required to build a working U-Boot image.  I see a potential
> > > area of conflicts here.
> > 
> > We don't have any problem including our SDK with U-Boot. I can work on
> > trying
> 
> But  we probably will have problems adding tons of such code which is
> of no use to anybody else.
As I have said, I will try and cut down on what is included.
> 
> > The biggest portions of our SDK are generated files for dealing with
> > errors, a register database (12MB!) and include files which define
> > access functions to
> 
> I'd expect that only a tiny percentage of this is actually needed /
> used in U-Boot.  We should restric it to these actually used parts.
> 
Agreed.
> 
> Best regards,
> 
> Wolfgang Denk

[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Wolfgang Denk]

Dear Aaron Williams,

In message <201102081927.36497.Aaron.Williams@caviumnetworks.com> you wrote:
>
...
> > > memory size to CONFIG_MAX_MEM_MAPPED. This won't work for us. As I said,
> > > we need to move it out of the lower 4GB when there's more memory
> > > involved. We also don't
> > 
> > Why?
> 
> As I have said, we need all of the lower portion of memory and do not want to 
> introduce holes in the memory space for u-boot for loading 64-bit 

When I ask "why?" then I do so to understand the reasons fdor your
design decisions (and eventually to be able to recomment alternative
solutions).

In such a situation phrases as "we need" or "we [do not] want" are
completely useless und unhelpful, unless they are accompanied with a
"because ..." part that explains _why_ you [think you] "need" or
"want" these things.

> > I understand that you do not _want_ to change this.  My question is:
> > what would you do if you _had_ to change it?
> 
> We might be able to make some changes, but it might be difficult since we have 
> a lot of developers working on this around the world. I would prefer to not 
> have to maintain two separate code bases. I will need to see how much I can 
> strip out.

Well, frankly, this is your problem, not ours. If Cavium had decided
to discuss the design early we might have come up with a solution
that fits without too much problems. Instead, Cavium went on and kept
their stuff closed for years - now they say that it's difficult to
change things. Please understand that our foxus is on the overall
quality and maintainability of the code, so such arguments do not
carry much weight.

> > > since there is some dependency on some of the include files from our GCC
> > > distribution as well, plus our toolchain distribution includes support
> > > for some of the extensions we make use of.
> > 
> > Can this be fixed?  I mean, copying some header files should probably
> > be a solvable problem.  What about of these "extensions" - are they
> > absolutely needed in U-Boot?  Usually such extensions are either
> > performance optimizations which are not really needed in U-Boot, or
> > other well-localized operations that can b ehandlef with small
> > assembler stubs.
> 
> Support for our Octeon2 processor is not in the mainline trunk yet. We also 
> have some changes needed to work with our SDK and the Linux kernel with our 
> newer chips.

You reply, but you do not actually answer my questions.


> > In general we do not want to see board or SoC specific changes to
> > common code.
> 
> The common code is already chock full of them from what I've seen in a number 

Yes, and we've learned that lesson, and now we strive hard not to add
to the mess any more.

> of areas. The number of changes for our SoC is actually quite small. In some 
> cases I've just changed some functions to use __attribute__((__weak__)) so we 
> can define the functionality elsewhere. I count a total of 21 ifdefs in the 
> common code, most of those are in cmd_ide.c, one in cmd_boot.c, one in 
> cmd_elf.c, two in env_flash.c and some in main.c, which can be removed (those 
> are for adding the support for dynamically changing the boot prompt) and one 
> in miiphyutil.c. Right now cmd_ide.c can be cleaned up a bit. The problem is 
> that cmd_ide.c includes the driver functionality in it, requiring us to do 
> this.  I can clean this up by making some of the functions weak. Is this 
> reasonable?

I don't know. I haven't seen any of the code yet, so I really cannot
tell.

> One other change I had to make was for the command table. I have to force the 
> alignment to 8 bytes in the compiler or very bad things happen.

Why?


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Let the programmers be many and the managers few -- then all will  be
productive.               -- Geoffrey James, "The Tao of Programming"

[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Matthias Weißer]

Hi Aaron

Am 07.02.2011 23:02, schrieb Aaron Williams:
> 5. Fix for Micron NAND flash MT29F32G08CBABA which erroneously reports a 16-
> bit bus when it has an 8-bit BUS.

Can you send that patch separately? I have a iMX25 board here with the 
2GiB version of that chip which also reports a 16 bit bus width. I 
didn't had time to look into this and if you have a fix for this...

Thanks,
Matthias

[U-Boot] Some notes and status on porting U-Boot to the Cavium 64-bit Octeon MIPS Processor

[Aaron Williams]

On Tuesday, February 08, 2011 11:39:58 pm Wolfgang Denk wrote:
> Dear Aaron Williams,
> 
> In message <201102081927.36497.Aaron.Williams@caviumnetworks.com> you
> wrote:
> 
> ...
> 
> > > > memory size to CONFIG_MAX_MEM_MAPPED. This won't work for us. As I
> > > > said, we need to move it out of the lower 4GB when there's more
> > > > memory involved. We also don't
> > > 
> > > Why?
> > 
> > As I have said, we need all of the lower portion of memory and do not
> > want to introduce holes in the memory space for u-boot for loading
> > 64-bit
> 
> When I ask "why?" then I do so to understand the reasons fdor your
> design decisions (and eventually to be able to recomment alternative
> solutions).
> 
> In such a situation phrases as "we need" or "we [do not] want" are
> completely useless und unhelpful, unless they are accompanied with a
> "because ..." part that explains _why_ you [think you] "need" or
> "want" these things.
> 
We cannot have u-boot consuming space in KUSEG or in KSEG0. Placing U-Boot at 
the bottom of memory causes problems for some of our customers which load 
proprietary operating systems that don't handle this well. They also need all 
the memory they can get. By using a single TLB entry we map U-Boot to the top 
of physical memory and map it to KSEG2. This makes the entire KUSEG and KSEG0 
available for customer applications and operating systems. We don't have the 
uncached KSEG1 since the Octeon is a cache coherent design. We also can't load 
U-Boot at the top of KUSEG or KSEG0 because that would create a hole in the 
middle of memory for loading 64-bit applications. 

The other alternative would be to make U-Boot fully 64-bit. That, however,  
would be a far larger challenge since there are assumptions everywhere about 
32-bit pointers.

By utilizing the TLB we map U-Boot to the top of physical memory which leaves 
almost all of the memory available for loading large 64-bit applications and 
operating systems without creating a hole in KSEG0/KUSEG (64-bit) where U-Boot 
was loaded.

As a side benefit, the relocation code is no longer needed with the exception 
of gd and the environment.

The drawback is that drivers cannot assume a pointer is always to a physical 
address, but they shouldn't make that assumption anyway. For example, if u-
boot is running out of KSEG0 then the pointer and physical address will not 
match on our platform.

> > > I understand that you do not _want_ to change this.  My question is:
> > > what would you do if you _had_ to change it?
> > 
> > We might be able to make some changes, but it might be difficult since we
> > have a lot of developers working on this around the world. I would
> > prefer to not have to maintain two separate code bases. I will need to
> > see how much I can strip out.
> 
> Well, frankly, this is your problem, not ours. If Cavium had decided
> to discuss the design early we might have come up with a solution
> that fits without too much problems. Instead, Cavium went on and kept
> their stuff closed for years - now they say that it's difficult to
> change things. Please understand that our foxus is on the overall
> quality and maintainability of the code, so such arguments do not
> carry much weight.
> 
I can't say much about that.  That was before my time here.

> > > > since there is some dependency on some of the include files from our
> > > > GCC distribution as well, plus our toolchain distribution includes
> > > > support for some of the extensions we make use of.
> > > 
> > > Can this be fixed?  I mean, copying some header files should probably
> > > be a solvable problem.  What about of these "extensions" - are they
> > > absolutely needed in U-Boot?  Usually such extensions are either
> > > performance optimizations which are not really needed in U-Boot, or
> > > other well-localized operations that can b ehandlef with small
> > > assembler stubs.
> > 
> > Support for our Octeon2 processor is not in the mainline trunk yet. We
> > also have some changes needed to work with our SDK and the Linux kernel
> > with our newer chips.
> 
> You reply, but you do not actually answer my questions.
> 
The code uses features found in the Octeon II extensions that are not 
available in the binutils trunk yet. I know for a fact that the standard MIPS 
toolchain used to compile other MIPS boards in U-Boot will not work at all for 
the Octeon. It doesn't even support the n32 ABI which is absolutely required.

> > > In general we do not want to see board or SoC specific changes to
> > > common code.
> > 
> > The common code is already chock full of them from what I've seen in a
> > number
> 
> Yes, and we've learned that lesson, and now we strive hard not to add
> to the mess any more.
> 
For the most part I have avoided them, and many that are there can be changed 
to be more general. The IDE code will take more work since it's already a 
mess.

> > of areas. The number of changes for our SoC is actually quite small. In
> > some cases I've just changed some functions to use
> > __attribute__((__weak__)) so we can define the functionality elsewhere.
> > I count a total of 21 ifdefs in the common code, most of those are in
> > cmd_ide.c, one in cmd_boot.c, one in cmd_elf.c, two in env_flash.c and
> > some in main.c, which can be removed (those are for adding the support
> > for dynamically changing the boot prompt) and one in miiphyutil.c. Right
> > now cmd_ide.c can be cleaned up a bit. The problem is that cmd_ide.c
> > includes the driver functionality in it, requiring us to do this.  I can
> > clean this up by making some of the functions weak. Is this reasonable?
> 
> I don't know. I haven't seen any of the code yet, so I really cannot
> tell.
> 
> > One other change I had to make was for the command table. I have to force
> > the alignment to 8 bytes in the compiler or very bad things happen.
> 
> Why?

Because otherwise the code generated by the compiler to enumerate the commands 
when looking for a match will crash because the compiler increments the 
pointer by a value that does not match what the linker does. The linker's 
minimum section alignment is 64-bits for n32, but the compiler generates code 
assuming 32-bit alignment. If the commands were just an array then there would 
be no problem. The other alternative is to add padding to force the data 
structure to be a multiple of 64-bits. I have not found any other workaround.  
I don't think this is an issue for a purely 32-bit ABI, such as o32, but it is 
an issue with n32 which supports 64-bit load/store operations and registers. 
Even telling the linker to force 32-bit alignment does not help, it still 
performs 64-bit alignment when using the n32 ABI. I haven't looked into the 
details for the ABI, but I wouldn't be surprised if the linker is doing the 
right thing and the problem is just that u-boot's assumptions break in this 
case.
> 
> 
> Best regards,
> 
> Wolfgang Denk