Calling the kernel from a mini-bootloader

Calling the kernel from a mini-bootloader

[Guillaume Dargaud]

Hello all,
I'm in the process of writing a mini-bootloaler for a custom board and would 
like some feedback on my boot methodology.

Basically the kernel code (elf file) is copied into memory by a JTAG 
debugger. A custom program (yet to be finished) then copies it onto an 
onboard flash memory using serial commands. This flash is not part of the 
flat memory addressing.

When I want to boot, I have a small bootloader put into an eprom that _is_ 
part of the flat memory addressing. This (finished but not tested yet) 
bootloader basically does two things:
- it copies the content of the first flash into RAM
- it launches it using a custom function call like "0x400000();"

Now, I'd like to know if this is a reasonable approach ? I couldn't think of 
a better way, but maybe there are issues I didn't think off.


One additional question, if the above is valid, is that I would like to pass 
a particular argument to the kernel to set its MAC address. I can read an 
external value (similar to jumpers) from the bootloader via GPIO. Is there a 
way to pass this value to the kernel so that a modified 
arch/ppc/boot/simple/embed_config.c can use it to set the MAC address ?

Maybe the main() of the kernel can receive argv/argc the usual way, and I 
just need to call "0x400000(argc, argv);" but I have no idea if that works. 
Thanks for suggestions.
-- 
Guillaume Dargaud
http://www.gdargaud.net/

Re: Calling the kernel from a mini-bootloader

[Milton Miller]

On Thu Jul 17 at 23:22:28 EST in 2008, Guillaume Dargaud wrote:
> Hello all,
> I'm in the process of writing a mini-bootloaler for a custom board and  
> would
> like some feedback on my boot methodology.
>
> Basically the kernel code (elf file) is copied into memory by a JTAG
> debugger. A custom program (yet to be finished) then copies it onto an
> onboard flash memory using serial commands. This flash is not part of  
> the
> flat memory addressing.
>
> When I want to boot, I have a small bootloader put into an eprom that  
> _is_
> part of the flat memory addressing. This (finished but not tested yet)
> bootloader basically does two things:
> - it copies the content of the first flash into RAM
> - it launches it using a custom function call like "0x400000();"
>
> Now, I'd like to know if this is a reasonable approach ? I couldn't  
> think of
> a better way, but maybe there are issues I didn't think off.

This is a very reasonable approach, and is quite similar to what
I do.

You actually have a few choices.   You can put just the loaded
data, or you can put the elf file and parse the header in your
boot loader.  After seeing the advantages, I would keep the elf
header.   One big advantage of keeping the elf header is you can
see how much data will be zeroed for bss when the kernel starts.
Another choice is rather than loading the kernel, build a zImage
of some kind (see the code in arch/powerpc/boot).  This way the
kernel code and static data is compressed.  I often see a 3-fold
reduction in size.  You can also attach an initrd, should you
decide to use initramfs later.  More o this below.

Both the kernel and zImage will build an elf file with a single
load header.  Both can run regardless of where they are loaded,
so you don't have to worry about parsing the load address, just
calculating the relatve entry point.  (The kernel will move
itself down to address 0 (except book-e parts), and the zImage
will run where it is started.  However, the default zImage will
fail to start the kernel if it isn't loaded above the link size
of the kernel (including bss).

> One additional question, if the above is valid, is that I would like  
> to pass
> a particular argument to the kernel to set its MAC address. I can read  
> an
> external value (similar to jumpers) from the bootloader via GPIO. Is  
> there a
> way to pass this value to the kernel so that a modified
> arch/ppc/boot/simple/embed_config.c can use it to set the MAC address ?
>
> Maybe the main() of the kernel can receive argv/argc the usual way,  
> and I
> just need to call "0x400000(argc, argv);" but I have no idea if that  
> works.

Ahh ... you are still on ppc.   Please note that we just merged
the removal of arch/ppc, everyone needs to use powerpc now.  The
good news is: its easier to state the requirements, and its
easier to share the code in vmlinux.  The bad news is you have to
follow the rules for passing data to the kernel.  Its not hard.
We have defined a data structure that is parsed to become a tree
of data describing the machine, and based the contents on open
firmware.  We call this the flattened device tree.

To pass the MAC address, one uses a property to the device node
which also describes what the hardware is (by name) and the
address space it uses.  Similarly, the kernel finds out how fast
the cpu and timebase are running by reading properties in the cpu
node.

Please read Documentation/powerpc/booting-without-of for more
information.  To summarize, the kernel is started with a pointer
to the device tree structure, and it contains all the information
needed.  All the data the kernel needs will be in one flat linear
chunk, but the kernel will move itself in memory to its linked
location.   The 32 bit kernel does require r1 to be set to the
top of a memory block that can be used as a stack.


There are several options to get the data into the device tree.
If you choose to use the zImage format mentioned above, the
device tree library is compiled in and you would just write a bit
of c code to read your structure and tell the library to put it
in the device tree.  The standard initialization preserves r3-r6
so you get 4 32 bit args (or maybe 5) to pass data or pointers.

Ohterwise, there is a tool called dtc (device tree compiler) that
that creates the binary structure from a source file that is
human readable.  From this, there is a library that will modify
the tree at runtime.

Another approach is to modify the device tree directly from your
first stage.   While you can link the library into your first
stage, you can also just tell dtc to output an assembly file with
symbolic labels on the data you need to modify.  You would then
pass this structure to the zImage wrapper or directly to the
kernel.

To see how simple this approach can be when you only need to
modify a few parameters, I will point out my [qemu-rom v2 patch.
Hmm, thats not it, where is it?  I thought I had posted this to
linuxppc-dev but can't find it.]

I will point out the minimal rom image I did for qemu where the
device tree is linked into the assembly, the kernel elf file is
already loaded, but I had to copy the nvram data serially into
ram and then poke the memory size and initrd location into the
device tree.

http://landley.net/hg/firmware/raw-file/92f89c9c9495/sources/toys/make- 
ppc_rom.tar.bz2

> Thanks for suggestions.

You're welcome.
milton

Re: Calling the kernel from a mini-bootloader

[David Gibson]

On Thu, Jul 17, 2008 at 11:14:11AM -0500, Milton Miller wrote:
> On Thu Jul 17 at 23:22:28 EST in 2008, Guillaume Dargaud wrote:
[snip]
>> Maybe the main() of the kernel can receive argv/argc the usual way,  
>> and I
>> just need to call "0x400000(argc, argv);" but I have no idea if that  
>> works.
>
> Ahh ... you are still on ppc.   Please note that we just merged
> the removal of arch/ppc, everyone needs to use powerpc now.  The
> good news is: its easier to state the requirements, and its
> easier to share the code in vmlinux.  The bad news is you have to
> follow the rules for passing data to the kernel.  Its not hard.
> We have defined a data structure that is parsed to become a tree
> of data describing the machine, and based the contents on open
> firmware.  We call this the flattened device tree.

Your firmware doesn't have to deal with this though, although it can.
It's perfectly acceptable - in fact I'd mildly recommend it over doing
this in the firmware - to instead have the kernel's zImage wrapper
supply the flattened device tree, folding information it needs to get
from the firmware into it.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: Calling the kernel from a mini-bootloader

[Milton Miller]

On Jul 17, 2008, at 10:47 PM, David Gibson wrote:

> On Thu, Jul 17, 2008 at 11:14:11AM -0500, Milton Miller wrote:
>> On Thu Jul 17 at 23:22:28 EST in 2008, Guillaume Dargaud wrote:
> [snip]
[more snip]
>> We call this the flattened device tree.
>
> Your firmware doesn't have to deal with this though, although it can.
> It's perfectly acceptable - in fact I'd mildly recommend it over doing
> this in the firmware - to instead have the kernel's zImage wrapper
> supply the flattened device tree, folding information it needs to get
> from the firmware into it.

I thought about adding a section on which to recommend or talk about 
the tradeoffs, but got hungry and hit send before going to lunch.

Yes, I the zImage choice is definitely the least amount of code to 
write, and is a good tradeoff to get future upgradability.  It may not 
be the absolute fastest split of the work to start the kernel but it is 
reasonably good, especially as it will decompress the kernel to its 
run-time destination.

milton

Re: Calling the kernel from a mini-bootloader

[Guillaume Dargaud]

Hello Milton and David, thanks t=for the answers.

> This is a very reasonable approach, and is quite similar to what
> I do.
Makes me feel better !!!

> You actually have a few choices.   You can put just the loaded
> data, or you can put the elf file and parse the header in your
> boot loader.
I cannot do that for memory constrains. I have 4Mb in the flash for the 
kernel and only a few Kb for the eprom of the bootloader. Also the idea is 
to sometimes update the kernel but basically never the bootloader.

> After seeing the advantages, I would keep the elf
> header.   One big advantage of keeping the elf header is you can
> see how much data will be zeroed for bss when the kernel starts.
> Another choice is rather than loading the kernel, build a zImage
> of some kind (see the code in arch/powerpc/boot).  This way the
> kernel code and static data is compressed.  I often see a 3-fold
> reduction in size.  You can also attach an initrd, should you
> decide to use initramfs later.  More o this below.
I'm not too familiar with the sequence of events that unfolds when the 
kernel start.
The JTAG debugger copies the kernel at a given address. A program dumps a 
copy of that address range in flash. A bootloader does the reverse and 
launches it, so it shouldn't really matter what form the kernel takes, as 
long as the original _did_ work.

> Ahh ... you are still on ppc.   Please note that we just merged
> the removal of arch/ppc, everyone needs to use powerpc now.  The
Yes, my code works on the ML405, now I'm just trying to get it to work on 
our custom board... I haven't been able to follow closely the PPC/PowerPC 
change but it got me worried so I stopped updating at 2.6.15. I'm using the 
Xilinx git tree because I rely on a lot of their drivers, so I don't know if 
the change will be seemless. From the change of xparameters.h to and 
entirely different device description method, I'd say I'm not ready to take 
the jump.

Any Xilinx user care to comment on that ?

And I'm supposed to release the alpha version of our code today !!!

> good news is: its easier to state the requirements, and its
Hmmm...

> easier to share the code in vmlinux.  The bad news is you have to
> follow the rules for passing data to the kernel.  Its not hard.
> We have defined a data structure that is parsed to become a tree
> of data describing the machine, and based the contents on open
> firmware.  We call this the flattened device tree.
Our firmware is custom VHDL code. I'll go ask the 'tronics guy if it fits 
open firmware requirements, but I'd be surprised.

> Please read Documentation/powerpc/booting-without-of for more
Will do. Last update is 2005. Is this still relevant  ?

> I will point out the minimal rom image I did for qemu where the
> device tree is linked into the assembly, the kernel elf file is
> already loaded, but I had to copy the nvram data serially into
> ram and then poke the memory size and initrd location into the
> device tree.
Maybe I can just use an elementary approach like that: have the bootloader 
write a single byte with the jumper positions somewhere in RAM, and then 
have the kernel read it to use for the Mac address. With the flat addressing 
it could work if there aren't built-in securities.

Thanks
-- 
Guillaume Dargaud
http://www.gdargaud.net/

Re: Calling the kernel from a mini-bootloader

[Milton Miller]

On Fri Jul 18 18:43:11 EST 2008, Guillaume Dargaud wrote:
> Hello Milton and David, thanks for the answers.
>> This is a very reasonable approach, and is quite similar to what
>> I do.
> Makes me feel better !!!
>
>> You actually have a few choices.   You can put just the loaded
>> data, or you can put the elf file and parse the header in your
>> boot loader.
> I cannot do that for memory constrains. I have 4Mb in the flash for the
> kernel and only a few Kb for the eprom of the bootloader. Also the 
> idea is
> to sometimes update the kernel but basically never the bootloader.

Then the zImage approach is definitly the way to go.  Actually, dtbImage
now, because you want the device tree to be compiled and linked into it.

(It only took me 33 lines of assembly, and 6 were labels or blank, to
skip over the elf header -- the vmlinux and zImage don't require they be
moved into a specific place - but keeping it just a binary blob is 
fine).

>> After seeing the advantages, I would keep the elf
>> header.   One big advantage of keeping the elf header is you can
>> see how much data will be zeroed for bss when the kernel starts.
>> Another choice is rather than loading the kernel, build a zImage
>> of some kind (see the code in arch/powerpc/boot).  This way the
>> kernel code and static data is compressed.  I often see a 3-fold
>> reduction in size.  You can also attach an initrd, should you
>> decide to use initramfs later.  More o this below.
>
> I'm not too familiar with the sequence of events that unfolds when the
> kernel start.
> The JTAG debugger copies the kernel at a given address. A program 
> dumps a
> copy of that address range in flash. A bootloader does the reverse and
> launches it, so it shouldn't really matter what form the kernel takes, 
> as
> long as the original _did_ work.

Ok, should not be a problem.  You will want the wrapper code to create a
flat binary image, so have your platform set the binary variable like
cuboot.  Whatever you specify for your platform file  will be linked at
the beginning of the image.

When choosing where to laod the zImage into memory, you should consider
that the linked kernel size (including its bss) needs to fit below the
load address, and the extracted device tree and random malloc heap 
normally
reside the wrapper.

>
>> Ahh ... you are still on ppc.   Please note that we just merged
>> the removal of arch/ppc, everyone needs to use powerpc now.  The
> Yes, my code works on the ML405, now I'm just trying to get it to work 
> on
> our custom board... I haven't been able to follow closely the 
> PPC/PowerPC
> change but it got me worried so I stopped updating at 2.6.15. I'm 
> using the
> Xilinx git tree because I rely on a lot of their drivers, so I don't 
> know if
> the change will be seemless. From the change of xparameters.h to and
> entirely different device description method, I'd say I'm not ready to 
> take
> the jump.
>
> Any Xilinx user care to comment on that ?

I'm not a user, but from reading the list, they have integrated creating
the dts file into their build system.

>
> And I'm supposed to release the alpha version of our code today !!!
>
>> good news is: its easier to state the requirements, and its
> Hmmm...
>
>> easier to share the code in vmlinux.  The bad news is you have to
>> follow the rules for passing data to the kernel.  Its not hard.
>> We have defined a data structure that is parsed to become a tree
>> of data describing the machine, and based the contents on open
>> firmware.  We call this the flattened device tree.
> Our firmware is custom VHDL code. I'll go ask the 'tronics guy if it 
> fits
> open firmware requirements, but I'd be surprised.

I'm sure your firmware does not presently create this tree, but
the dtbImage format is designed to be a delivery vehicle for it.

>
>> Please read Documentation/powerpc/booting-without-of for more
> Will do. Last update is 2005. Is this still relevant  ?

That's might be the last time we changed the structure of the tree,
but we have been updating the definition of what goes into the tree
structure pretty much every month since then.   In the 2.6.27 kernel,
I think we started breaking out these descriptions into the
dts-bindings directory tree.

>
>> I will point out the minimal rom image I did for qemu where the
>> device tree is linked into the assembly, the kernel elf file is
>> already loaded, but I had to copy the nvram data serially into
>> ram and then poke the memory size and initrd location into the
>> device tree.
>
> Maybe I can just use an elementary approach like that: have the 
> bootloader
> write a single byte with the jumper positions somewhere in RAM, and 
> then
> have the kernel read it to use for the Mac address. With the flat 
> addressing
> it could work if there aren't built-in securities.

We are happy for you to read such information in the zImage (or 
dtbImage).
In fact we have a hook "fixups" that is the designated place for this to
happen.  If the location to read the jumper is well defined, I would
suggest doing all of it from your fixup function, you don't need to save
the jumper position in ram (less code for your bootloader, and probably
a simpler handoff).  (Needing the mac address does mean you can't use 
the
simpleboot.c).

Hope you find these comments helpful,
milton

Re: Calling the kernel from a mini-bootloader

[Guillaume Dargaud]

Thank you Milton for the detailed explanation. It'll take me quite a while 
to digest it.


As a follow up to my previous messages, I now have a working kernel and a 
working bootloader... but not when they are both together.

Case in point:
- load zImage.elf to DRAM 0x400000 with JTAG debugger. Run it. It runs fine.

- Load Bootloader.elf to BRAM 0xFFFF0000 with JTAG debugger. Run it. It runs 
fine (but it doesn't do much yet).

- Now load both of the previous ones, and have the bootloader perform a jump 
to kernel:
     typedef void tFunc(void);
     ((tFunc *)0x400000)();

The kernel seems to start:

loaded at:     00400000 004EA19C
board data at: 004E8120 004E819C
relocated to:  0040405C 004040D8
zimage at:     00404E48 004E7A7E
avail ram:     004EB000 08000000

Linux/PPC load: console=ttyUL0,115200 rw root=/dev/nfs ip=bootp
Uncompressing Linux...done.
Now booting the kernel

But then it hangs for exactly 3 minutes, nothing on the serial port, before 
the bootloader restarts.
Why should the behavior be different whether it's started from the debugger 
or from my bootloader ?
-- 
Guillaume Dargaud
http://www.gdargaud.net/

Re: Calling the kernel from a mini-bootloader

[Guillaume Dargaud]

Answering to myself to try and provide more background info:

> As a follow up to my previous messages, I now have a working kernel and a 
> working bootloader... but not when they are both together.
>
> Case in point:
> - load zImage.elf to DRAM 0x400000 with JTAG debugger. Run it. It runs 
> fine.
>
> - Load Bootloader.elf to BRAM 0xFFFF0000 with JTAG debugger. Run it. It 
> runs fine (but it doesn't do much yet).
>
> - Now load both of the previous ones, and have the bootloader perform a 
> jump to kernel:
void main() {
     typedef void tFunc(void);
     ((tFunc *)0x400000)();
}

Now if the bootloader _only_ contains the above code, the kernel runs 
perfectly.
Unfortunately I need to do some things in the bootloader: I use 3 GPIOs to 
control a frontal set of LEDs, to get a card number and to load/unload a 
serial flash memory. Those operations work. But here is the strange thing 
that leads me to believe there are interrupt interferences.

If I just read from a GPIO before launching the kernel, it boots fine, with 
a few NFS errors when mounting the rootnfs:
[    4.360060] Freeing unused kernel memory: 76k init
[   11.432269] nfs: server 192.168.1.185 not responding, still trying
[   11.441186] nfs: server 192.168.1.185 OK
init started: BusyBox v1.10.1 (2008-04-24 12:26:38 CEST)
genepy login:

Now if I also write to a 2nd GPIO, I get a lot more NFS errors.

And if I also access the flash through the 3rd GPIO, the kernel hangs:

> loaded at:     00400000 004EA19C
> board data at: 004E8120 004E819C
> relocated to:  0040405C 004040D8
> zimage at:     00404E48 004E7A7E
> avail ram:     004EB000 08000000
>
> Linux/PPC load: console=ttyUL0,115200 rw root=/dev/nfs ip=bootp
> Uncompressing Linux...done.
> Now booting the kernel
>
> But then it hangs for exactly 3 minutes, nothing on the serial port, 
> before the bootloader restarts.
> Why should the behavior be different whether it's started from the 
> debugger or from my bootloader ?

Now I've defined my hardware project using interrupts on the GPIOs, which I 
don't need and frankly they don't make sense for an output GPIO. I would 
like to try to compile a kernel without GPIO interrupts, unfortunately the 
compilation fails:
$ make ARCH=ppc CROSS_COMPILE=powerpc-linux-uclibc-
[...]
  CC      arch/ppc/syslib/virtex_devices.o
arch/ppc/syslib/virtex_devices.c:535: error: 'XPAR_INTC_0_GPIO_0_VEC_ID' 
undeclared here (not in a function)

Question breakdown:
- do I need interrupts for GPIOs ?
- how can I compile the ppc 2.6.25 kernel without them ?
- Why do I have problems running the kernel if the bootloader makes use of 
the GPIOs ?
- does the bootloader need to perform some kind of special cleanup before 
calling the kernel ?

Thanks a lot for shedding a light on those mysteries.
-- 
Guillaume Dargaud
http://www.gdargaud.net/