CELF Project Proposal- Refactoring Qi, lightweight bootloader

CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Matt Hsu]

Summary: Refactoring Qi, lightweight bootloader.

Proposer: Matt Hsu
                                                                                                                                                           

Description:

Qi (named by Alan Cox on Openmoko kernel list) is a minimal bootloader that
"breathes life" into Linux.  Its goal is to stay close to the minimum 
needed                                                                                
 
to "load" and then "boot" Linux -- no boot menus, additional peripheral init
or private states.

Qi currently supports samsung s3c24xx series, s3c6410, TI omap3530. The
more support platforms are planned to add on. The purpose of this project
would be to improve Qi's maintainability, portability. Ideally, this would
make people spend less time on bootloader development but be more focus
on Linux system.

Project objectives:

- Make the hierarchy of source files more sensible and clean

- Generalize components which could be used in common such as I2C drivers.
  Example: platform specific I2C driver -> GPIO bitbang driver.

- Remove duplicated, unused code, header definition. Keep Qi as minimum 
as needed.
 
Related work:

http://wiki.openmoko.org/wiki/Qi

Development branches are hosted here:

http://git.warmcat.com/cgi-bin/cgit/qi/
http://gitorious.org/0xlab-bootloader

Scope:
    Unknown.

Thanks.
Matt

Re: CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/17/09 08:31, Somebody in the thread at some point said:

Hi -

Just read about this a little late :-)

I wrote the bulk of Qi while at Openmoko.

> Qi currently supports samsung s3c24xx series, s3c6410, TI omap3530. The

I also ported it to iMX31.

> - Generalize components which could be used in common such as I2C drivers.
> Example: platform specific I2C driver -> GPIO bitbang driver.

I wrote a generic bitbang I2C "driver" for Qi back in 2007, it's used on 
the GTA02 build IIRC to talk to the PMU:

 
http://git.warmcat.com/cgi-bin/cgit/qi/tree/src/drivers/i2c-bitbang.c?h=txtr

> - Remove duplicated, unused code, header definition. Keep Qi as minimum
> as needed.

What I can suggest would be worthwhile goals are:

  - extending the targeted CPU arches

  - folding the OMAP branch (written by Matt) back into the main tree

  - improving the memory test to force burst mode

On the txtr branch (currently they are my customer and we use Qi on 
their iMX31 e-book reader) there's already various tidying up and 
advances like FAT filesystem, PRNG-based memory test.

  http://git.warmcat.com/cgi-bin/cgit/qi/log/?h=txtr

Here is a bit more about the reasoning for bothering with another 
bootloader and the philosophy behind Qi.

While working at Openmoko maintaining their kernel, it became obvious 
that U-Boot was turning into a mini-me for Linux.  Many cut-down Linux 
drivers were appearing there, there was a shell with an environment 
holding private states like booting action commands.  This led to a 
situation where two identical devices with same patchlevel of everything 
including bootloader may act differently according to what's in that 
hidden environment.  Wanting a "boot menu" meant supporting graphics and 
some of the drivers like for Glamo were complex and always forked from 
their Linux counterparts.

The burden of maintaining forked drivers in two places (the U-Boot ones 
had to be "dumbed down" to the point it can't share sources) and the 
additional complexity of trying to manage power and additional device 
state in two places are all actually completely unnecessary.  All the 
bootloader needs is just enough to get hold of a kernel and boot it. 
That is Qi's philosophy in a nutshell: "load [the kernel], then boot". 
Linux is then the single place for good support of all assets on the device.

It can work from NAND or NOR but actually the existing support is 
designed to work with SD Card boot.  Recent devices like s3c6410 and 
i.MX31 can completely boot from SD Card, including getting their 
bootloader from there.  On i.MX31 using this scheme, we are able to boot 
to a bash prompt in 2.5s from cold.

Another big problem with U-Boot was the inability to update the 
bootloader from the Linux world.  Instead the bootloader was treated 
special and had to be updated over USB with DFU (the exclusivity of it 
enforced by an ECC policy disconnected from Linux, meaning U-Boot had to 
write it in there).  In Qi, the bootloader can be updated by a packaged 
update like anything else in the rootfs.

There are no private environment / states, there are per-board 
heuristics for where to look for a kernel, and filesystem files which 
can append to kernel commandlines if present.  Therefore identical 
devices with the same Qi patchlevel will always act the same.

-Andy

Re: CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Tim Bird]

Matt Hsu wrote:
> Summary: Refactoring Qi, lightweight bootloader.
I've created a page for this at:
http://elinux.org/CELF_Project_Proposal/Refactor_the_Qi_lightweight_bootloader

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Rob Landley]

On Thursday 17 December 2009 02:31:36 Matt Hsu wrote:
> Summary: Refactoring Qi, lightweight bootloader.
>
> Proposer: Matt Hsu
>
>
> Description:
>
> Qi (named by Alan Cox on Openmoko kernel list) is a minimal bootloader that
> "breathes life" into Linux.  Its goal is to stay close to the minimum
> needed

Which bits does it do?

Every piece of software needs something to initialize the DRAM controller.  
After that, they could presumably just jump to a known location in flash that 
the kernel lives at, and the kernel can decompress itself and so on.  Doing 
just that much can probably be done in 4k pretty easily.  (I gloss over the 
kernel command line and root filesystem location, which can be hardwired into 
the image if you really don't care about providing the developer with a UI.)

However, if that's your minimum then you can't use the bootloader to re-flash 
the device, which is kind of handy.  (It gives you an un-bricking fallback 
short of pulling out a jtag.)  But doing that requires things like a network 
driver, TCP/IP stack, tftp implementation, serial driver, command line 
interpreter, and so on.  And of course code to erase and write flash blocks for 
your flash chip du jour, plus knowledge of the flash layout.  (In theory, said 
knowledge comes from parsing a device tree.)

I still live in hope that somebody will split the first part (coreboot) out 
from the second part (grub) in embedded bootloaders.  It's sad that the PC has 
a tradition of orthogonality here but the embedded world treats it as a single 
opaque lump.

> http://wiki.openmoko.org/wiki/Qi

Looking at the screen shot there, you've got code to parse ext2 filesystems.  
What is your definition of "minimal"?

Rationale for not providing a boot menu is you don't want to mess with video 
init.  I don't think I've actually seen an embedded bootloader that messes 
with video, they do serial console instead, and you have a screen shot of 
serial console messages so apparently the serial driver part is there...

Confused,

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Matt Hsu]

Rob Landley wrote:
> However, if that's your minimum then you can't use the bootloader to re-flash 
> the device, which is kind of handy.  (It gives you an un-bricking fallback 
> short of pulling out a jtag.) 
    Hi Rob,

    Well, Boot from SD is your good friend.

    If you look at the platform that Qi which is supported, most of them 
all have this feature.
    If you notice the trend of SoC, booting from peripherals becomes a 
must.

    Once you step into kernel via Qi, kernel provides you everything 
such as mtd utils to re-flash device.
    We don't need to support programming the device in the bootloader 
anymore.
    Don't reinvent the wheel.  
>
> Looking at the screen shot there, you've got code to parse ext2 filesystems.  
> What is your definition of "minimal"?
>   
    Enough to boot into Linux.
> Rationale for not providing a boot menu is you don't want to mess with video 
> init.  
    Nope, the centric idea of Qi, is let kernel deal with everything it 
could handle.
    The video init should be handled by kernel stead of bootloader.

    The following clip demonstrate the advantage of Qi bootloader:

    http://www.youtube.com/watch?v=ol9LWBKXXwQ&feature=related

    - Faster booting time
    - Get rid of flash on display device when stepping into kernel

    Hope these could clear your doubt.

    Cheers,  
    Matt

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Rob Landley]

On Sunday 20 December 2009 23:51:23 Matt Hsu wrote:
> Rob Landley wrote:
> > However, if that's your minimum then you can't use the bootloader to
> > re-flash the device, which is kind of handy.  (It gives you an
> > un-bricking fallback short of pulling out a jtag.)
>
>     Hi Rob,
>
>     Well, Boot from SD is your good friend.

Ok, not aiming to be a generic bootloader then.  You're only trying to support 
hardware that has the flash equivalent of a floppy drive.  Got it.

>     If you look at the platform that Qi which is supported, most of them
> all have this feature.

Because if they didn't you wouldn't support them?  Bit of a selection bias 
there...

Life is a bit easier if you can stick a USB port on your device and boot from 
a USB stick or cdrom.  Most of the SoCs coming down the pipe support USB too, 
but your wiki doesn't seem to consider this an interesting case...

>     If you notice the trend of SoC, booting from peripherals becomes a
> must.

Depends how cheap you want your hardware to be, and whether you can afford 
separate development and deployment boards.

Software development is a bit easier when you can spec adding a extra few 
dollars worth of hardware to your device rather than redoing its software.  I 
tend to deal with people who repurpose existing cheap plastic crap already 
being knocked out in huge quantities somewhere in china.  (Their hardware 
contribution _might_ be a different plastic case for the thing.)  I've also 
dealt with highly integrated little suckers that haven't got _space_ for an sd 
card.  Since I one day dream of following Moore's Law down to "disposable 
computing", I'm reluctant to discard these cases as uninteresting.

>     Once you step into kernel via Qi, kernel provides you everything
> such as mtd utils to re-flash device.
>     We don't need to support programming the device in the bootloader
> anymore.

Depending on the kernel to reflash the device means that if you reflash the 
device with a kernel that doesn't work, what you have is a brick.  There's 
lots and lots of reasons for a kernel not to work, and a 2.6 kernel kernel 
takes up around a megabyte on a device that may only have 2 or 4 megs of ram 
so keeping an old one around at all times isn't feasible.  So without some 
kind of fallback (such as a little ~32k bootloader at the start of flash that 
you never overwrite, in its own little erase block), every time you install a 
new kernel you risk bricking the device.  (If you only care about devices that 
have 2 gigs of flash, life is much easier.)

>     Don't reinvent the wheel.

There are how many existing bootloaders out there already?

> > Looking at the screen shot there, you've got code to parse ext2
> > filesystems. What is your definition of "minimal"?
>
>     Enough to boot into Linux.

You need to parse an ext2 filesystem to boot into linux?  (I'm not saying it's 
a _bad_ thing, I'm just not seeing how it's "minimal".)

> > Rationale for not providing a boot menu is you don't want to mess with
> > video init.
>
>     Nope, the centric idea of Qi, is let kernel deal with everything it
> could handle.

So the fact the kernel can provide a serial console means you shouldn't?

>     The video init should be handled by kernel stead of bootloader.

Oh agreed.  What that has to do with a command line interpreter on the serial 
console was my question.

Personally, i'm used to embedded devices being headless.  (I tend to consider 
the iPhone the next stage in the mainframe->micro->mini evolution and it'll 
soon be "just another PC" target when they get the ergonomics worked out.  
Heck, give those suckers a USB port and you could give 'em a big keyboard and 
display today, and they could even charge themselves through the thing.  Way 
more powerful than my first 386 PC.  Actually about as powerful as the laptop I 
had circa 2002 or 2003.)

>     The following clip demonstrate the advantage of Qi bootloader:
>
>     http://www.youtube.com/watch?v=ol9LWBKXXwQ&feature=related
>
>     - Faster booting time

I.E. you enable the cpu cache during kernel decompression.

I believe the u-boot guy just posted that as a todo item, and that falls on 
the coreboot side of bootloading, which really should be broken out into its 
own little project someday...

The other delay in something like u-boot is it pausing to see if it gets an 
"esc" or similar from the serial console, so it can be interrupted and provide 
a prompt.  That's a configurable delay which can be removed if it really bugs 
you.

>     - Get rid of flash on display device when stepping into kernel
>
>     Hope these could clear your doubt.

Your proposal confused me when it said things like "improve portability" and 
"make people spend less time on bootloader development", so I thought you were 
aiming at something more generic.  But this bootloader is not intended to ever 
apply to something like a hammer board, or any existing linksys-class router, 
most current cell phones, devices like handheld game consoles where the sd 
card stores data but not the operating system (due to the thing still needing 
to work when you swap sd cards)...

My mistake.  You're creating a bootloader specifically tailored to booting from 
sd cards.  Might want to make that clear on the web page.

>     Cheers,
>     Matt

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/21/09 08:00, Somebody in the thread at some point said:

Hi Rob -

>>> However, if that's your minimum then you can't use the bootloader to
>>> re-flash the device, which is kind of handy.  (It gives you an
>>> un-bricking fallback short of pulling out a jtag.)

There is a simple and much more flexible alternative to have a "recovery 
rootfs" on the device that can be selected, by holding down a button or 
somesuch which is understood by Qi.  That gives you ALL the options 
possible in Linux like network connectivity for your recovery without 
doubling the support load in having a "smart" bootloader and its 
different drivers to deal with.

>>      Well, Boot from SD is your good friend.
>
> Ok, not aiming to be a generic bootloader then.  You're only trying to support
> hardware that has the flash equivalent of a floppy drive.  Got it.

No, Qi will boot from onboard NAND or NOR as well as SD.

For other reasons -- eg, unbrickability -- true boot from SD is very 
powerful.  By "true boot from SD" I mean the on-CPU ROM is pulling a 
bootloader from SD so the entire software environment is coming from there.

>>      If you look at the platform that Qi which is supported, most of them
>> all have this feature.
>
> Because if they didn't you wouldn't support them?  Bit of a selection bias
> there...

Naturally enough we only added support for devices we are using in the 
real world :-)

All the devices Qi target have what Samsung called "steppingstone", some 
small SRAM on the die that can hold your core bootloader startup from 
NAND or SD (since you cannot execute direct from NAND).  That's probably 
a better one-liner to describe the scope of it.

> Life is a bit easier if you can stick a USB port on your device and boot from
> a USB stick or cdrom.  Most of the SoCs coming down the pipe support USB too,
> but your wiki doesn't seem to consider this an interesting case...

To be clear Openmoko Wiki isn't "our Wiki".  Qi got started there to 
solve the difficulties created by having basically a second Linux on the 
device in the form of U-Boot.  But the main work on Qi is done now at 
warmcat and Matt's OMAP branch.  Openmoko doesn't seem to be in the 
phone business any more.

> Depending on the kernel to reflash the device means that if you reflash the
> device with a kernel that doesn't work, what you have is a brick.  There's

That does not follow if you have a recovery kernel + initrd / rootfs in 
another partition that can be selected by holding down a key.

> new kernel you risk bricking the device.  (If you only care about devices that
> have 2 gigs of flash, life is much easier.)
>
>>      Don't reinvent the wheel.
>
> There are how many existing bootloaders out there already?

So what?

As described the advantages of Qi's debloated and Linux-centric 
philosophy came from having our noses pushed for months into the world 
of crap coming from supporting multiple drivers and power behaviours 
spread over U-Boot and Linux.

We did not find any existing open ARM bootloader that had a solid 
internal structure and avoided the bloat.

Therefore we made one that proves that it is possible to push most of 
the features of U-Boot and other heavy bootloaders to Linux.

>>> Looking at the screen shot there, you've got code to parse ext2
>>> filesystems. What is your definition of "minimal"?

Yes the main tenet is it has got to "load" and "boot" the kernel.  If 
the kernel is on ext* then that's part of its remit.

>>      Enough to boot into Linux.
>
> You need to parse an ext2 filesystem to boot into linux?  (I'm not saying it's
> a _bad_ thing, I'm just not seeing how it's "minimal".)

The ext* code is tight (taken from U-Boot and cleaned up).

With VFAT, ext*, memory test, all of the features Qi comes to 28KBytes 
typical binary.

>>> Rationale for not providing a boot menu is you don't want to mess with
>>> video init.
>>
>>      Nope, the centric idea of Qi, is let kernel deal with everything it
>> could handle.
>
> So the fact the kernel can provide a serial console means you shouldn't?

Qi provides serial output for diagnosis of the boot action.  But not input.

But the answer to your rhetorical question is "yes".  It costs mindspace 
and support effort to support and work with these complex chunks of 
software, it's real money.

Turn your question around, what is your rationale for duplicating the 
stuff that Linux does great into a bloated bootloader with its own 
shell?  If you look at each of the features in a bootloader like U-Boot 
that is not directly needed for "load" and "boot", they each have a more 
powerful counterpart in Linux already.  You have to support the Linux 
version on your Linux box, just simplify it down to that alone.

>>      The video init should be handled by kernel stead of bootloader.
>
> Oh agreed.  What that has to do with a command line interpreter on the serial
> console was my question.

First maybe it's on you to explain why, other than habit, you think that 
is worth the bloat.

> I.E. you enable the cpu cache during kernel decompression.

You are assuming wrongly that the kernel is compressed.  On iMX31 I use 
Qi with uncompressed 3MByte kernel from SD Card, it's booted to shell 
from cold in 2.5s.  In any event with compressed kernels, we use the 
kernel uncompress, it's not done in Qi.

> aiming at something more generic.  But this bootloader is not intended to ever
> apply to something like a hammer board, or any existing linksys-class router,
> most current cell phones, devices like handheld game consoles where the sd
> card stores data but not the operating system (due to the thing still needing
> to work when you swap sd cards)...
>
> My mistake.  You're creating a bootloader specifically tailored to booting from
> sd cards.  Might want to make that clear on the web page.

Apologies for throwing you back into confusion and doubt by the fact Qi 
boots fine from NAND, NOR and SD.

Actually Qi will work fine on most current ARM-based cellphones and game 
consoles with NAND or NOR.

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Wolfgang Denk]

Dear Andy,

In message <4B29F834.90108@warmcat.com> you wrote:
> 
> Another big problem with U-Boot was the inability to update the 
> bootloader from the Linux world.  Instead the bootloader was treated 
> special and had to be updated over USB with DFU (the exclusivity of it 
> enforced by an ECC policy disconnected from Linux, meaning U-Boot had to 
> write it in there).  In Qi, the bootloader can be updated by a packaged 
> update like anything else in the rootfs.

Can you explain to me why it was not possible to update U-Boot from
Linux? I cannot imagine a reason for such a restriction.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
I have made mistakes, but have never made the mistake of  claiming  I
never made one.                                     - James G. Bennet

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Mike Frysinger]

On Mon, Dec 21, 2009 at 14:30, Wolfgang Denk wrote:
> Andy wrote:
>> Another big problem with U-Boot was the inability to update the
>> bootloader from the Linux world.  Instead the bootloader was treated
>> special and had to be updated over USB with DFU (the exclusivity of it
>> enforced by an ECC policy disconnected from Linux, meaning U-Boot had to
>> write it in there).  In Qi, the bootloader can be updated by a packaged
>> update like anything else in the rootfs.
>
> Can you explain to me why it was not possible to update U-Boot from
> Linux? I cannot imagine a reason for such a restriction.

we do it all the time under the Blackfin port on nor/nand/spi flashes.
-mike

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/21/09 19:30, Somebody in the thread at some point said:

Hi -

Let me add back the context for this quote:

''While working at Openmoko maintaining their kernel, it became obvious 
that U-Boot was turning into a mini-me for Linux.  Many cut-down Linux 
drivers were appearing there, there was a shell with an environment 
holding private states ...''

>> Another big problem with U-Boot was the inability to update the
>> bootloader from the Linux world.  Instead the bootloader was treated
>> special and had to be updated over USB with DFU (the exclusivity of it
>> enforced by an ECC policy disconnected from Linux, meaning U-Boot had to
>> write it in there).  In Qi, the bootloader can be updated by a packaged
>> update like anything else in the rootfs.
>
> Can you explain to me why it was not possible to update U-Boot from
> Linux? I cannot imagine a reason for such a restriction.

I was talking about GTA02 specifically here, it was (and still is AFAIK) 
only updateable by DFU for the bootloader.  We had kernels with soft ECC 
that differed from the ECC / bad block marking generated and used by the 
s3c2442 NAND hardware unit.  If U-Boot wrote it, it could at least read 
it again.  So DFU was / is the only official way to update GTA02 bootloader.

As you say nothing generally stops update of U-Boot from Linux same as 
anything else can be updated from there, if Linux can write those NAND 
sectors with ECC / BBT that the bootloader can understand or ignore, and 
Linux can understand and comply with / update the BBT scheme of U-Boot.

Another benefit of SD Card boot is that it regularizes the bootloader 
physical storage, so these issues don't exist and it can always be 
updated from Linux.

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Wookey]

+++ Rob Landley [2009-12-20 20:45 -0600]:
> On Thursday 17 December 2009 02:31:36 Matt Hsu wrote:
> 
> I still live in hope that somebody will split the first part (coreboot) out 
> from the second part (grub) in embedded bootloaders.  

blob took this approach a long time ago (started 2000-ish) trying very
hard not to do any more than was needed in the normal version, and
having a uploadable extra part for interactive stuff but was only
adopted on a fairly small number of platforms.

Popular bootloaders always grow more bits and more hardware support
until they are no longer small. It's a very difficult trend to resist,
although good architecture helps.

The codebase is still there: http://sourceforge.net/projects/blob/

I always thought it was good code (especially since I've had far too
much to do with bootldr (from familiar) for the last few years (yuk!)).

Wookey
-- 
Principal hats:  iEndian - Balloonboard - Toby Churchill - Emdebian
http://wookware.org/

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Wolfgang Denk]

Dear Andy Green,

In message <4B2FD7DB.10008@warmcat.com> you wrote:
> 
> >> Another big problem with U-Boot was the inability to update the
> >> bootloader from the Linux world.  Instead the bootloader was treated
...
> I was talking about GTA02 specifically here, it was (and still is AFAIK) 
> only updateable by DFU for the bootloader.  We had kernels with soft ECC 
> that differed from the ECC / bad block marking generated and used by the 
> s3c2442 NAND hardware unit.  If U-Boot wrote it, it could at least read 
> it again.  So DFU was / is the only official way to update GTA02 bootloader.

This only means that the ports of U-Boot and Linux to that hardware
were inconsistent, which means that at least one of them can be
considered broken. [And I'm tempted to add that this might eventually
be a consequence of the fact that all this work was done without ever
getting in touch with the U-Boot community. Likewise, no code or
fixes have been contributed back into mainline.]


I think it's unfair to blame U-Boot for a poor on incorrect
implementation of the NAND driver on this specific hardware.


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
It's hard to make a program foolproof because fools are so ingenious.

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/21/09 21:38, Somebody in the thread at some point said:

Hi -

>> I was talking about GTA02 specifically here, it was (and still is AFAIK)
>> only updateable by DFU for the bootloader.  We had kernels with soft ECC
>> that differed from the ECC / bad block marking generated and used by the
>> s3c2442 NAND hardware unit.  If U-Boot wrote it, it could at least read
>> it again.  So DFU was / is the only official way to update GTA02 bootloader.
>
> This only means that the ports of U-Boot and Linux to that hardware
> were inconsistent, which means that at least one of them can be

Actually the problem of synchronizing the ECC and BBT rules between 
bootloader and Linux is a generic one where raw NAND is involved.  Soft 
ECC and Hard ECC on a particular chip may differ and the kernel decides 
what policy it has there.  It's a "raw NAND" issue not a U-Boot one but 
if anyone shipped a kernel one way and wanted to change it, they have to 
take care about matching the bootloader's view of the NAND in great 
detail if they want interoperation between Linux and the bootloader.

> considered broken. [And I'm tempted to add that this might eventually
> be a consequence of the fact that all this work was done without ever
> getting in touch with the U-Boot community. Likewise, no code or
> fixes have been contributed back into mainline.]

Well, this is getting a bit off-topic for considering the merits of Qi.

Openmoko started to use U-Boot on GTA01 long before my time, likewise 
the soft ECC was "like that when I got there".

About tapping into the wisdom of the U-Boot community, most of their 
changes were GTAxx-specific.  For example I don't know any other Linux 
device than GTA02 with a Glamo in it, there is a lot of code I ported 
from Linux for that bloating their tree.  With closed docs, that would 
be completely useless for upstream.

Bearing in mind they could only update by DFU and with GTA01, there was 
no bootloader recovery mechanism if it failed, keeping their bootloader 
tree inhouse was an established tradition by the time I got there.

For all those reasons the best way we found was deprecate their U-Boot 
tree and learn from what we had found there.  (A fair amount of Qi is 
cut out of U-Boot so I see the great work it has done as well as the 
problems: respect for your work.)

> I think it's unfair to blame U-Boot for a poor on incorrect
> implementation of the NAND driver on this specific hardware.

Yeah it would be unfair to solely blame U-Boot for the sins of GTA02. 
But it's fair to blame U-Boot with the sins of U-Boot.

The main lessons I took from that was the dollar and time value of 
removing the "unnecessary features" in U-Boot and especially the 
Openmoko tree of it:

  - video drivers

  - shells

  - environments

  - special update mechanisms

  - raw NAND at all

  - duplicating the OS in there

  - private nonvolatile state

  - PMU management when we are already able to run

  - per board variant bootloader image (ie, GTA02 v3 can only run a 
special GTA02 v3 binary of U-Boot that can't run on anything else; Qi 
has a per CPU binary that supports all variants)

After understanding that all those are needless and actively wasteful in 
developer resource, support, and device time the reasoning led to the 
development of Qi in its current form.

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Wookey]

+++ Andy Green [2009-12-21 22:38 +0000]:
> On 12/21/09 21:38, Somebody in the thread at some point said:
>
> Hi -
>
>>> I was talking about GTA02 specifically here, it was (and still is AFAIK)
>>> only updateable by DFU for the bootloader.  We had kernels with soft ECC
>>> that differed from the ECC / bad block marking generated and used by the
>>> s3c2442 NAND hardware unit.  If U-Boot wrote it, it could at least read
>>> it again.  So DFU was / is the only official way to update GTA02 bootloader.
>>
>> This only means that the ports of U-Boot and Linux to that hardware
>> were inconsistent, which means that at least one of them can be
>
> Actually the problem of synchronizing the ECC and BBT rules between  
> bootloader and Linux is a generic one where raw NAND is involved. 

Balloonboard has the same problem (with bootldr and Linux disagreeing
about exactly how the ECC should be done). The kernel can read
bootldr's version but not write it reliably, so userspace nand kernel
upgrades don't work. It's 'simply' a matter of the hacked MTD and yaffs
versions in bootldr getting out of sync with the ones in the kernel
and 'people' not having time to fix it.

It's not bootldr's fault - it's just one of those things no-one's got
round to fixing because it's never the most urgent, and it's quite
painful to do, and generally illustrates the problems of maintaining
separate bootldr and kernel code, which is generally to be avoided as
much as possible. Of course NAND reading/writing is pretty much the
one thing your bootloader has to be able to do so I don't forsee much
relief here (except one day I hope to be able to switch to a more
mainstream bootloader then it's wouldn't be my problem anymore :-)
That's been on my list of things to do for several years now...

I post merely to illustrate that openmoko is not the only
hardware/community with this problem, not to point fingers at anyone
except myself. I sincerely hope all your bootloaders are much better
than ours.

Wookey
-- 
Principal hats:  iEndian - Balloonboard - Toby Churchill - Emdebian
http://wookware.org/

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Robert Schwebel]

Andy,

On Mon, Dec 21, 2009 at 10:38:31PM +0000, Andy Green wrote:
> About tapping into the wisdom of the U-Boot community, most of their
> changes were GTAxx-specific. For example I don't know any other Linux
> device than GTA02 with a Glamo in it, there is a lot of code I ported
> from Linux for that bloating their tree. With closed docs, that would
> be completely useless for upstream.

Hmm, there have been interesting items in the openmoko trees. For
barebox, we took the DFU support, which was done in a device specific
way, cleaned that up and made a generic command out of it:

dfu /dev/self0(bootloader)sr,/dev/nand0.root.bb(root)

You can specify the slots on the command line, not hardcoded. Whereas we
reworked the interfaces, the core code was pretty interesting. So I
think some items it would have been worth to be pushed into u-boot at
the time it was written.

> Bearing in mind they could only update by DFU and with GTA01, there
> was no bootloader recovery mechanism if it failed,

Our DFU scenario goes like "press a button while booting goes into DFU
mode", so you can re-flash as often as you like. However, our use cases
are probably different than yours (deeply embedded systems, which often
don't even have removable stuff like SD or USB sticks).

In general, I like in-system techniques much better than card juggeling,
because it fits better into automated environments like our RemoteLab,
which does our automatic nightly tests. But that's surely a matter of
the use case you have.

> The main lessons I took from that was the dollar and time value of
> removing the "unnecessary features" in U-Boot and especially the
> Openmoko tree of it:

In barebox, we use Kconfig to configure things away; so removing
unnecessary features is just a matter of 'make menuconfig'.

>   - video drivers

I see video drivers in the bootloader as an optimization topic: If you
can effort to get your splash 3 s after power-on, you should leave video
drivers out of the boot loader and do it all in the kernel.

Our competition in industry projects is often the old 2-lines-alpha
displays, which are "instant on" after you hit the power switch. If this
is required, I don't see a way to achieve that with kernel-only at the
moment.

>   - shells

Especially during development, we often see that the hardware people
really like having a very limited shell with hardware bit banging access
in barebox. In a phase where you port Linux to a device, it gives you
something that works while Linux is not ready yet. And in barebox, you
have full scripting capabilities, so hardware people can even use that
for certain qualification scripts.

>   - environments

That was one of our design goals in barebox as well: get rid of the
scripting in the environment, as it was done in u-boot.

>   - special update mechanisms

What do you mean with "special"?

>   - raw NAND at all
>   - duplicating the OS in there

If you want to boot from NAND-only devices, how would you do that
without NAND drivers?

>   - private nonvolatile state

?

>   - PMU management when we are already able to run

Several CPUs need PMU support early in the boot stage, because they come
up in slow-clock mode. So you either boot slow until the kernel is up
far enough (but then the whole kernel loading is slow), or you need
access to the PMU from the bootloader.

In barebox, our design is that we have frameworks for i2c+spi to access
a PMU, but if you don't need that, you can configure it away. The idea
is that *if* you actually need it, then better have a good design for
it.

>   - per board variant bootloader image (ie, GTA02 v3 can only run a
>     special GTA02 v3 binary of U-Boot that can't run on anything else;
>     Qi has a per CPU binary that supports all variants)

I don't know the GTA02 hardware, but it is often a problem to actually
detect a certain CPU or board variant on runtime. But if that's
possible, I don't see a reason why you can't make a single image.

We had one strange case with an MPC5200 where the bootup bus
configuration was so complicated that it was a major issue and we
decided for different bootloaders, but in general it should be possible.

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/21/09 23:19, Somebody in the thread at some point said:

Hi Robert -

Thanks for your reply.

 > mode", so you can re-flash as often as you like. However, our use cases
 > are probably different than yours (deeply embedded systems, which often
 > don't even have removable stuff like SD or USB sticks).

Right, some of what Qi proposes won't work on all systems, like SD boot 
where there is no SD card.  But the core "just load and boot" heuristic 
should work almost anywhere.

 >>    - special update mechanisms
 >
 > What do you mean with "special"?

> Hmm, there have been interesting items in the openmoko trees. For
> barebox, we took the DFU support, which was done in a device specific
> way, cleaned that up and made a generic command out of it:

DFU is a "special update mechanism" which I believe is a bad idea.

I know a lot of people are still putting out full rootfs images as 
updates, and for some platforms that are too resource-constrained that's 
all people can do.

But for modern devices like ARM11+ and the kind of board they typically 
find themselves on with a network connection, these are fundamentally at 
the level of PC from a few years ago.  Linux PCs then and now use 
packaged update systems to manage the software on the device.  And they 
package both the kernel and the bootloader and track and update it like 
any other package, apply packagesets as transactions, etc.  The correct 
approach I believe is to unify the bootloader (and kernel) update path 
with the rest of the system, all done from Linux alone.

(Personally I used Fedora ARM port and RPM, but any distro and 
packagesystem like Debian workable on ARM would be fine).

> dfu /dev/self0(bootloader)sr,/dev/nand0.root.bb(root)
>
> You can specify the slots on the command line, not hardcoded. Whereas we
> reworked the interfaces, the core code was pretty interesting. So I
> think some items it would have been worth to be pushed into u-boot at
> the time it was written.
>
>> Bearing in mind they could only update by DFU and with GTA01, there
>> was no bootloader recovery mechanism if it failed,
>
> Our DFU scenario goes like "press a button while booting goes into DFU
> mode", so you can re-flash as often as you like. However, our use cases
> are probably different than yours (deeply embedded systems, which often
> don't even have removable stuff like SD or USB sticks).

The issue GTA01 faced was that you are updating the thing the button 
takes you to.  If that goes south you have to bust out JTAG / OpenOCD 
and that is definitely not an end-user tool for a consumer product.

In GTA02 a separate NOR was added to contain the "bootloader behind the 
button" which was not updatable in the field, that then caused trouble 
since the updatable NAND bootloaders moved on but that never did.  It 
also acted as the third pole in the love triangle betweeen NAND U-Boot 
and Linux in the NAND ECC / BBT differences since it could only recover 
the NAND bootloader only with the NOR bootloader's fixed idea of what 
ECC and BBT looked like, no matter what we had done with updates to the 
NAND bootloader in the meanwhile (eg, move from soft to incompatible but 
faster hard ECC in Linux).  So we were actually unable to migrate to 
hard ECC in Linux, which is an insane outcome of a broken system.

In contrast if your chip supports it (iMX31 and s3c6410 do and Qi works 
with those) having your bootloader on some sectors of SD card is 
wonderfully simple and easy to dd in on a postinstall scriptlet of your 
bootloader package.

> In general, I like in-system techniques much better than card juggeling,
> because it fits better into automated environments like our RemoteLab,
> which does our automatic nightly tests. But that's surely a matter of
> the use case you have.

Agreed.

But consider this: if your bootloader is on SD, and your bootloader 
completely rejects to hold private state on the board (other than 
onetime individualization, eg MAC address), something awesome happens 
when you pop your SD card and put it in another board, it comes up like 
the previous board did, no ifs or buts.

You can imagine the effect that has on production / test "virgin" board 
bringup.  When you have seen this, you do not want to return to raw 
onboard NAND.

>> The main lessons I took from that was the dollar and time value of
>> removing the "unnecessary features" in U-Boot and especially the
>> Openmoko tree of it:
>
> In barebox, we use Kconfig to configure things away; so removing
> unnecessary features is just a matter of 'make menuconfig'.

That is good, but what I am suggesting is that

  - these things are definitively unnecessary, ie, they deserve 
permanent deselection

  - the config system leads to bootloader-binary-per-variant Hell

Because Qi burns off all the peripheral support and leaves it to Linux, 
actually building in support for multiple boards and multiple variants 
is pretty lightweight.  The CPU bringup is always the same, SDRAM 
bringup may vary slightly and kernel commandlines and paths, amount and 
maybe placement of memory will change.

Qi uses a per-board callback in an API struct to discover at runtime 
which supported board it's on, and the board can check version bits on 
GPIO typically to discover which variant it is (which is passed on to 
Linux in an ATAG).

>>    - video drivers
>
> I see video drivers in the bootloader as an optimization topic: If you
> can effort to get your splash 3 s after power-on, you should leave video
> drivers out of the boot loader and do it all in the kernel.
>
> Our competition in industry projects is often the old 2-lines-alpha
> displays, which are "instant on" after you hit the power switch. If this
> is required, I don't see a way to achieve that with kernel-only at the
> moment.

Yeah that is true.  You are into a 1.8 - 2 second (on iMX31 SD boot) 
delay from hitting the button to your driver starting up in Linux and 
getting your display up.

Given what you get out of that from a project management POV, I don't 
think 2 seconds for startup feedback is a problem for most systems.  If 
your system has a hardwired power LED, then even more so.

But if you have to have the display lit quicker, Qi has per-board API 
callback that lets the board set itself up how it needs.  You could add 
this there if you have to.

Have a look at

http://git.warmcat.com/cgi-bin/cgit/qi/tree/src/cpu/imx31/txtr-steppingstone.c?h=txtr

scroll down to the bottom to see how the per-board setup works.

>>    - shells
>
> Especially during development, we often see that the hardware people
> really like having a very limited shell with hardware bit banging access
> in barebox. In a phase where you port Linux to a device, it gives you
> something that works while Linux is not ready yet. And in barebox, you
> have full scripting capabilities, so hardware people can even use that
> for certain qualification scripts.

Yeah I agree hardware people like doing that.  Here's how that innocent 
pastime can take you to Hell.

I described on the Openmoko list how even normally good programmers 
become "like a fat girl in Ibiza" when they see how it is in (Openmoko 
tree anyway) U-Boot, any wild thing goes.  (It was quite sad to have to 
chop down some of the drivers that had pretty good code quality from 
Linux to fit the simplified world in U-Boot).  And some people who 
describe themselves as "hardware guys" are not good programmers.

What it led to was private bootloader trees that did not track the main 
one, filled with perverted bit-twiddling code that was not understood by 
anyone except the guy who wrote it, and that guy left a while back as 
did the guy after him.

These trees were not even on the radar of the software guys nor did any 
patches come.  But it is these decayed stump versions of the bootloader 
forked years ago that will become the basis of production test in a huge 
expensive factory "because it has the test code in it".  By now it's 
test code nobody really understands (even if they are told The Secret of 
its existence) and they daren't uplevel their tree (even if they know 
such black magic is possible) because they neither have the forked 
version unchanged any more nor have heard of revision control outside 
the context of homework.

Because it was an unknown secret whispered only to new initiates in the 
Hardware Club, nobody in the software world is trying to keep 
compatibility with this forked bootloader with resulting car-crashes. 
And indeed a fourth pole in the NAND / ECC policy love quadrangle if 
we're still counting.

Same thing happens if you allow the existence of "test kernels" as with 
"test bootloaders".

Ultimately, even if that had all been correctly managed, it is still not 
preferable to have anything but truly core hardware tests in the 
bootloader (ie, testing of assets required to boot Linux that may not 
already be working since we are running the bootloader: just SDRAM test 
normally) compared to having them in Linux, since they can be scripted 
and reported easily from Linux.

Therefore the only test code in Qi is SDRAM test, no special bootloader 
version is needed (or allowed in my case) for verification or test.

If rapid asset verification is needed, it should be done in Linux with 
stub drivers or added to machine init code temporarily, and in revision 
control of someone who will write the real driver.

All other test actions should be integrated into the Linux driver and if 
they need to be triggered, exposed down /sys.

All of that should be present in normal shipping kernels, so what you 
take to the factory is simply current shipping version of bootloader and 
kernel with no custom build of anything.

>>    - environments
>
> That was one of our design goals in barebox as well: get rid of the
> scripting in the environment, as it was done in u-boot.
>

>
>>    - raw NAND at all
>>    - duplicating the OS in there
>
> If you want to boot from NAND-only devices, how would you do that
> without NAND drivers?

If all you have is NAND on your board then nothing can be done.

But if you have NAND and SD, it is possible

>>    - private nonvolatile state
>
> ?

Private nonvolatile state is stuff like the U-Boot environment that 
lives on the board itself and is out of any update management.

This leads to the situation where two boards from the same factory can 
act totally differently depending on what opaque different secrets have 
been hidden away in their private nonvolatile state, even if everything 
updatable in the rootfs is at the same patchlevel and even the 
bootloaders themselves at the same patchlevel.

That is "private nonvolatile state Hell".

>>    - PMU management when we are already able to run
>
> Several CPUs need PMU support early in the boot stage, because they come
> up in slow-clock mode. So you either boot slow until the kernel is up
> far enough (but then the whole kernel loading is slow), or you need
> access to the PMU from the bootloader.

Yeah.  But in the PMUs I have seen, Vcore is not by default at the level 
where it can ONLY run at 32kHz or whatever.  Instead it is at some 
intermediate voltage like 1.2V by default that will allow midrange 
operation.  (On this iMX31 board I currently work on in fact the PMU 
comes up by default on Vcore high enough for 532Mhz directly.)

That enables you to complete the boot at a reasonable speed without 
actually having the requirement to touch the PMU in those cases.

> In barebox, our design is that we have frameworks for i2c+spi to access
> a PMU, but if you don't need that, you can configure it away. The idea
> is that *if* you actually need it, then better have a good design for
> it.

Yeah Qi has generic gpio bitbang i2c implemented already and we can do 
the same for SPI if needed.  But I think you find most PMU have Vcore by 
default at a place you can run at a reasonable speed without touching it.

>>    - per board variant bootloader image (ie, GTA02 v3 can only run a
>>      special GTA02 v3 binary of U-Boot that can't run on anything else;
>>      Qi has a per CPU binary that supports all variants)
>
> I don't know the GTA02 hardware, but it is often a problem to actually
> detect a certain CPU or board variant on runtime. But if that's
> possible, I don't see a reason why you can't make a single image.

Yeah if care wasn't taken to reserve some GPIO for the task, it can be 
nontrivial.  But assets like NOR can be detected with a VID / PID and 
used for this to fingerprint a board.

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Robert Schwebel]

Hi Andy,

[is this the right set of lists to discuss these issues? It's not
directly CELF related, but I don't know a better place for general
project independend bootloader discussions]

On Tue, Dec 22, 2009 at 08:22:27AM +0000, Andy Green wrote:
> DFU is a "special update mechanism" which I believe is a bad idea.

For dedicated embedded system in the non-phone league, with small root
filesystems, it works pretty well.

> I know a lot of people are still putting out full rootfs images as
> updates, and for some platforms that are too resource-constrained
> that's all people can do.

Ah, ok, we don't do that.

> But for modern devices like ARM11+ and the kind of board they
> typically find themselves on with a network connection, these are
> fundamentally at the level of PC from a few years ago. Linux PCs then
> and now use packaged update systems to manage the software on the
> device. And they package both the kernel and the bootloader and track
> and update it like any other package, apply packagesets as
> transactions, etc. The correct approach I believe is to unify the
> bootloader (and kernel) update path with the rest of the system, all
> done from Linux alone.
>
> (Personally I used Fedora ARM port and RPM, but any distro and
> packagesystem like Debian workable on ARM would be fine).

Until now, we are using the "build it yourself" approach with ptxdist,
basically because of these reasons:

- If something goes wrong, we want to be able to fix it, which means
  that we must be able to recompile everything. Having the source is no
  value by itself, if you are not able to build it.

- Root filesystems are small; a complete rootfs for a typical industrial
  application with Linux, glibc, dbus and qt is about 20...30 MiB.

- People don't change software that often, and if they do, it has to be
  made sure that it is absolutely well tested. Nobody wants to reboot
  their deeply embedded machine controller at the other end of the world
  if somehting goes wrong. We usually don't have an administrator who
  can interoperate if something goes wrong.

- Customizability. We recently tried Debian on the Neo, and it is an
  absolute mess. About 2.5 minutes of boot time, a lot of flicker and
  almost no reactivity of the system. So for us, the question remains
  how to customize standard distributions in a reproducable way.

So at least at the moment, I prefer ptxdist over a customized debian.
But in general, I respect the argument why people want to use standard
distributions (I know the pain to fix all the cross compiling issues). I
just don't think that today's distributions are there yet. Most embedded
systems I've seen so far which follow the strip-down-standard-distro
pattern have been unreproducable for anyone but the original developer.

> So we were actually unable to migrate to hard ECC in Linux, which is
> an insane outcome of a broken system.
>
> In contrast if your chip supports it (iMX31 and s3c6410 do and Qi
> works with those) having your bootloader on some sectors of SD card is
> wonderfully simple and easy to dd in on a postinstall scriptlet of
> your bootloader package.

Agreed.

> > In general, I like in-system techniques much better than card
> > juggeling, because it fits better into automated environments like
> > our RemoteLab, which does our automatic nightly tests. But that's
> > surely a matter of the use case you have.
>
> Agreed.
>
> But consider this: if your bootloader is on SD, and your bootloader
> completely rejects to hold private state on the board (other than
> onetime individualization, eg MAC address), something awesome happens
> when you pop your SD card and put it in another board, it comes up
> like the previous board did, no ifs or buts.
>
> You can imagine the effect that has on production / test "virgin"
> board bringup. When you have seen this, you do not want to return to
> raw onboard NAND.

In general, I agree with you here (although I think the MAC address
should be glued to the hardware and not change if you change SD cards ->
people will then copy it and you have the same MAC address twice).

However, I think it's more developer friendly to have that "no changable
state" as a policy than a design decision: during development, we quite
often change for example the kernel command line (adding quiet or debug
switches, boot from net/disk...). For delivery, we just make barebox +
it's scripting environment one image and change that to r/o, if
necessary. So you can get best of both worlds.

> > In barebox, we use Kconfig to configure things away; so removing
> > unnecessary features is just a matter of 'make menuconfig'.
>
> That is good, but what I am suggesting is that
>
> - these things are definitively unnecessary, ie, they deserve
> permanent deselection
>
> - the config system leads to bootloader-binary-per-variant Hell

For us, the bootloader is not only something which is delivered with the
product - that's one use case. But there is also quite a long time where
lots of developers work with the board - and in that use case we'd like
to do things like hacking on registers (without the complexity of
Linux), do TFTP/nfsroot, change kernel command lines etc.

Seeing the production case, I'm all with you.

> Qi uses a per-board callback in an API struct to discover at runtime
> which supported board it's on, and the board can check version bits on
> GPIO typically to discover which variant it is (which is passed on to
> Linux in an ATAG).

Unfortunately, not all hardware vendors make different variants
detectable in software. That's quite often a problem for us. So for the
general case, a compile time selection is necessary. If the hardware
behaves, you are right :-)

> > I see video drivers in the bootloader as an optimization topic: If you
> > can effort to get your splash 3 s after power-on, you should leave video
> > drivers out of the boot loader and do it all in the kernel.
> >
> > Our competition in industry projects is often the old 2-lines-alpha
> > displays, which are "instant on" after you hit the power switch. If this
> > is required, I don't see a way to achieve that with kernel-only at the
> > moment.
>
> Yeah that is true.  You are into a 1.8 - 2 second (on iMX31 SD boot)
> delay from hitting the button to your driver starting up in Linux and
> getting your display up.

... and we still do have a lot of ARM9 systems in the 200...400 MHz
range out there.

> I described on the Openmoko list how even normally good programmers
> become "like a fat girl in Ibiza" when they see how it is in (Openmoko
> tree anyway) U-Boot, any wild thing goes.

That's why we went the device model way in barebox. Having a restricted
environment is no excuse for hackery (and people even assumed that the
same binary size wouldn't be possible in the beginning).

> And some people who describe themselves as "hardware guys" are not
> good programmers.

Very true ;)

> What it led to was private bootloader trees that did not track the
> main one, filled with perverted bit-twiddling code that was not
> understood by anyone except the guy who wrote it, and that guy left a
> while back as did the guy after him.

That's solvable by working on mainline integration. You'll get this
problem with Linux as well, if people are not on a mainline strategy. No
tool can change that.

> All other test actions should be integrated into the Linux driver and
> if they need to be triggered, exposed down /sys.

Ack.

> If all you have is NAND on your board then nothing can be done.
>
> But if you have NAND and SD, it is possible

In barebox, we have bootloader images that run from everywhere. So you
can for example write a little script that detects that you run from SD
or USB stick (taken that we'll have drivers for them) and relocate
to somewhere else (NAND on ARM, or soldered SSD on x86).

> >    - private nonvolatile state
>
> Private nonvolatile state is stuff like the U-Boot environment that
> lives on the board itself and is out of any update management.

On modular systems (like phyCORE, Qseven etc) you have a CPU module, a
baseboard, maybe additional addon boards, and requirements where to
store information like MAC addresses, serial numbers etc. are often very
different.

> This leads to the situation where two boards from the same factory can
> act totally differently depending on what opaque different secrets
> have been hidden away in their private nonvolatile state, even if
> everything updatable in the rootfs is at the same patchlevel and even
> the bootloaders themselves at the same patchlevel.

You can make the private nonvolatile state r/o or w/once.

> That enables you to complete the boot at a reasonable speed without
> actually having the requirement to touch the PMU in those cases.

Unfortunately, often people want to boot as fast as possible, which
requires optimization in that area as well. We recently had a board
which refused to boot without the PMIC having switched on some voltages
which are default-off.

> Yeah Qi has generic gpio bitbang i2c implemented already and we can do
> the same for SPI if needed. But I think you find most PMU have Vcore
> by default at a place you can run at a reasonable speed without
> touching it.

My fear is that quite often one starts with "oh, this problem is simple,
let's design simple". Then things move on and you notice that you need
to work on SPI, I2C, you need ext2, jffs2, ubifs, later maybe btrfs,
then SD support or USB. In the end, the problem turns out to be
complicated.

That's why we more or less stayed with the overall look-and-feel of
u-boot in barebox. We just tried to pull in the best ideas from the
Linux and POSIX universe, like the device model, sane scripting etc.
That way, people at least have something where they can put their hacks
if they really need them, without too much damage for the rest of us :-)

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/22/09 11:12, Somebody in the thread at some point said:

Hi Robert -

>> (Personally I used Fedora ARM port and RPM, but any distro and
>> packagesystem like Debian workable on ARM would be fine).
>
> Until now, we are using the "build it yourself" approach with ptxdist,
> basically because of these reasons:
>
> - If something goes wrong, we want to be able to fix it, which means
>    that we must be able to recompile everything. Having the source is no
>    value by itself, if you are not able to build it.

Fedora provides a whole solution there, with the restriction it's 
designed for native build, not cross.  That puts a limit on what you can 
target with it, which is why "ARM11+".  You can just go here

http://fedoraproject.org/wiki/Architectures/ARM#Getting_Started_with_the_Fedora_ARM_Port

Get a rootfs tarball, bring it up and build packages with the compiler 
and build stuff in there already, yum install other stuff, etc.  Again 
SD rootfs makes that simple to consider and do, you can untar it from 
your PC them replug to your device and run it.

> - Root filesystems are small; a complete rootfs for a typical industrial
>    application with Linux, glibc, dbus and qt is about 20...30 MiB.

True, typical Fedora footprint for X and so on will be 200MB.

However.  The developer cost in using these distro packages is next to 
nothing, both acquiring them and their ongoing uplevelling when fc13 
comes.  Quality of the packages is better than I can make :-) and it's 
pretty much anything there is to want

http://ftp.linux.org.uk/pub/linux/arm/fedora/pub/fedora/linux/releases/12/Everything/arm/os/Packages/

And again, if you have SD Card rootfs basis, smallest one of those now 
you can get in a shop is 512MB anyway.

> - People don't change software that often, and if they do, it has to be
>    made sure that it is absolutely well tested. Nobody wants to reboot
>    their deeply embedded machine controller at the other end of the world
>    if somehting goes wrong. We usually don't have an administrator who
>    can interoperate if something goes wrong.

Packages only help any QA effort.  You don't have to release every 
package build to your stable repo, a staged development / testing / 
stable repo scheme is simple.

> - Customizability. We recently tried Debian on the Neo, and it is an
>    absolute mess. About 2.5 minutes of boot time, a lot of flicker and

:-|  Yeah.  But I admired the effort they put in there.  The CPU is too 
weak for that game though and the Glamo in there is the final insult. 
(Actually there is a good rage comic: "Problem, GTA02?")

>    almost no reactivity of the system. So for us, the question remains
>    how to customize standard distributions in a reproducable way.

Well, I can explain what I did with Fedora, we were up in Bash with 
ethernet usb gadget and sshd ready for action in < 5s from cold on iMX31 
SD Card boot.  That's the stock Fedora bash, sshd, etc from their tarball.

The main action is not to use the stock /sbin/init.  (You can still use 
sysV init / upstart to bring services up and down even if init never got 
started since it's basically all sat there in /etc/init.d.)

> So at least at the moment, I prefer ptxdist over a customized debian.
> But in general, I respect the argument why people want to use standard
> distributions (I know the pain to fix all the cross compiling issues). I
> just don't think that today's distributions are there yet. Most embedded
> systems I've seen so far which follow the strip-down-standard-distro
> pattern have been unreproducable for anyone but the original developer.

Right.  Fedora is different though, there are no cross-built packages 
(although they do provide cross compilers, I use them for kernel and Qi 
builds) and if storage is sufficient, there's no need to strip anything 
out.  Just nobble init.

>> But consider this: if your bootloader is on SD, and your bootloader
>> completely rejects to hold private state on the board (other than
>> onetime individualization, eg MAC address), something awesome happens

> In general, I agree with you here (although I think the MAC address
> should be glued to the hardware and not change if you change SD cards ->
> people will then copy it and you have the same MAC address twice).

(That's what I meant by "other than onetime individualization eg MAC 
address" -- that stuff (only) should live on the board not the card)

> However, I think it's more developer friendly to have that "no changable
> state" as a policy than a design decision: during development, we quite
> often change for example the kernel command line (adding quiet or debug
> switches, boot from net/disk...). For delivery, we just make barebox +
> it's scripting environment one image and change that to r/o, if
> necessary. So you can get best of both worlds.

Right.  Qi supports this by allowing the boot source filesystem that 
holds the kernel to also contain an optional "append" text file.  The 
text in this file, if present, is appended to the generated kernel 
commandline.  Qi auto generates the correct root= (based on where it 
found the first workable kernel) and other board-mandated options 
already, but this external append file lets you do things like force 
debuglevel=8 from the filesystem without doing anything to the 
bootloader itself.

It's a plain text file, public, exposed in a standard filesystem and 
will get copied along with the rootfs.  So it's flexibility without the 
private nonvolatile bootloader storage problems.

> Unfortunately, not all hardware vendors make different variants
> detectable in software. That's quite often a problem for us. So for the
> general case, a compile time selection is necessary. If the hardware
> behaves, you are right :-)

Well for my uses of it I have been able to specify that we should have 
versioning GPIOs on the boards.  It's a good idea anyway.

> ... and we still do have a lot of ARM9 systems in the 200...400 MHz
> range out there.

Same as GTA02 class processor.  Qi loads the kernel there in a few 
seconds and brings up its video in Linux as told.

>> What it led to was private bootloader trees that did not track the
>> main one, filled with perverted bit-twiddling code that was not
>> understood by anyone except the guy who wrote it, and that guy left a
>> while back as did the guy after him.
>
> That's solvable by working on mainline integration. You'll get this
> problem with Linux as well, if people are not on a mainline strategy. No
> tool can change that.

It's nothing to do with mainline, just intra-company communication and 
management failure.

>> But if you have NAND and SD, it is possible
>
> In barebox, we have bootloader images that run from everywhere. So you
> can for example write a little script that detects that you run from SD
> or USB stick (taken that we'll have drivers for them) and relocate
> to somewhere else (NAND on ARM, or soldered SSD on x86).

Qi works the same.  The same image can be placed in NAND, NOR or SD and 
boot any supported device.

>> Private nonvolatile state is stuff like the U-Boot environment that
>> lives on the board itself and is out of any update management.
>
> On modular systems (like phyCORE, Qseven etc) you have a CPU module, a
> baseboard, maybe additional addon boards, and requirements where to
> store information like MAC addresses, serial numbers etc. are often very
> different.

What is the point there?

>> This leads to the situation where two boards from the same factory can
>> act totally differently depending on what opaque different secrets
>> have been hidden away in their private nonvolatile state, even if
>> everything updatable in the rootfs is at the same patchlevel and even
>> the bootloaders themselves at the same patchlevel.
>
> You can make the private nonvolatile state r/o or w/once.

You can do that with individualization data, like MAC Addresses.  But 
that is not what the U-Boot environment is.

>> That enables you to complete the boot at a reasonable speed without
>> actually having the requirement to touch the PMU in those cases.
>
> Unfortunately, often people want to boot as fast as possible, which
> requires optimization in that area as well. We recently had a board
> which refused to boot without the PMIC having switched on some voltages
> which are default-off.

If your device is able to run from USB power, there's the issue that you 
are limited to 100mA before enumeration takes place.  So without a USB 
stack, you have to trade speed for power anyway.

>> Yeah Qi has generic gpio bitbang i2c implemented already and we can do
>> the same for SPI if needed. But I think you find most PMU have Vcore
>> by default at a place you can run at a reasonable speed without
>> touching it.
>
> My fear is that quite often one starts with "oh, this problem is simple,
> let's design simple". Then things move on and you notice that you need
> to work on SPI, I2C, you need ext2, jffs2, ubifs, later maybe btrfs,
> then SD support or USB. In the end, the problem turns out to be
> complicated.

You're right to fear it because you are too willing to re-introduce the 
bloat into your bootloader.  For example you mention earlier that 
"Unfortunately, often people want to boot as fast as possible" and that 
is the rationale for re-introducing PMU management and the serial bus 
driver back into the bootloader.  But actually, normal customers don't 
care about 200ms on boot either way.  They can get the thing to market 
quicker and so cheaper and more reliably without that stuff in the 
bootloader.

> That's why we more or less stayed with the overall look-and-feel of
> u-boot in barebox. We just tried to pull in the best ideas from the
> Linux and POSIX universe, like the device model, sane scripting etc.
> That way, people at least have something where they can put their hacks
> if they really need them, without too much damage for the rest of us :-)

No offence but you are basically describing U-Boot there, mini Linux, 
scripting, hacks.  Surely like U-Boot dreams of growing up into Linux, 
you will find your project dreaming of growing up into U-Boot.

Qi rejects all of those and so is different (and smaller) than your 
project, I am sure there is "room in the market" for all of the 
philosophies including U-Boot.

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Robert Schwebel]

Hi Andi,

On Tue, Dec 22, 2009 at 10:23:37PM +0000, Andy Green wrote:
> Fedora provides a whole solution there, with the restriction it's
> designed for native build, not cross.

That's probably also a matter of taste. I still find it a feature to be
able to cross compile the systems - we can still recompile whole systems
consistently in just a few (ten) minutes, i.e. in order to change a
central switch (like softfloat vs. hardfloat, use another toolchain
etc). Out of interest, how long does it take to recompile Fedora on for
example MX31?

> Packages only help any QA effort. You don't have to release every
> package build to your stable repo, a staged development / testing /
> stable repo scheme is simple.

For let's say a telematics box it's almost impossible to test packet
combinations. All you can do is decide for "update application" and
"update platform". If you got your software updates via SMS, it
shouldn't go wrong because of an untested packet combination :-)

> Right. Fedora is different though, there are no cross-built packages
> (although they do provide cross compilers, I use them for kernel and
> Qi builds) and if storage is sufficient, there's no need to strip
> anything out. Just nobble init.

Hmm, I'm still not convinced. But as I don't have any data, I'll keep
quiet :-)

I think a central question is if you want to optimize things like

* early boot splash
* footprint

> Well for my uses of it I have been able to specify that we should have
> versioning GPIOs on the boards. It's a good idea anyway.

Hardware description is still one of the unsolved problems out there. It
works good if you have just a few variants. We often have for example
5-10 assembly options on a cpu module, plus several on the base board,
all not in-system detectable. We tried several variants to describe the
hardware in some config block, we tried to have oftree sniplets in the
hardware, but none of the options did really work well. In the end, you
need the schematics in machine readable form... When oftree comes into
the ARM world, we need a maximum-style bootloader which can provide
oftrees.

> > > What it led to was private bootloader trees that did not track the
> > > main one, filled with perverted bit-twiddling code that was not
> > > understood by anyone except the guy who wrote it, and that guy
> > > left a while back as did the guy after him.
> >
> > That's solvable by working on mainline integration. You'll get this
> > problem with Linux as well, if people are not on a mainline
> > strategy. No tool can change that.
>
> It's nothing to do with mainline, just intra-company communication and
> management failure.

My experience is that mainline exposure of that perverted bit-twiddling
code and ideas helps finding sane solutions, because someone will
allways ask the right questions :-)

> > Unfortunately, often people want to boot as fast as possible, which
> > requires optimization in that area as well. We recently had a board
> > which refused to boot without the PMIC having switched on some
> > voltages which are default-off.
>
> If your device is able to run from USB power, there's the issue that
> you are limited to 100mA before enumeration takes place. So without a
> USB stack, you have to trade speed for power anyway.

My argument is that there are all these different requirements out
there: your use cases, mine, others. What we want to do with barebox is
turning u-boot into something that can fulfill all of these
requirements, not only parts of it. You don't need a network driver in
your bootloader? Just configure it out and don't look at the code. But
somebody else may care, and he has a sane design to put his driver into.
The maximum paradigm works as good as the minimum does.

> > My fear is that quite often one starts with "oh, this problem is
> > simple, let's design simple". Then things move on and you notice
> > that you need to work on SPI, I2C, you need ext2, jffs2, ubifs,
> > later maybe btrfs, then SD support or USB. In the end, the problem
> > turns out to be complicated.
>
> You're right to fear it because you are too willing to re-introduce
> the bloat into your bootloader. For example you mention earlier that
> "Unfortunately, often people want to boot as fast as possible" and
> that is the rationale for re-introducing PMU management and the serial
> bus driver back into the bootloader. But actually, normal customers
> don't care about 200ms on boot either way. They can get the thing to
> market quicker and so cheaper and more reliably without that stuff in
> the bootloader.

That's a matter of the definition of "normal customers" :-)

> No offence but you are basically describing U-Boot there, mini Linux,
> scripting, hacks. Surely like U-Boot dreams of growing up into Linux,
> you will find your project dreaming of growing up into U-Boot.

Well, we think we have found a good design for reducing the hack factor
significantly.

While I'm still convinced that, for our projects, we have to stay with
barebox in maximum-mode, ptxdist's cross-building of self-made
distributions for now, there is a gradual move towards mainstream
distributions, where people can just 'apt-get install kde'. All our
recent experiments seem to suggest that mainstream distros are not able
to do what we need today, but times may change.

My conclusion is that we are on a good track: ptxdist is a 'do things in
a reproducable way, following a configuration' style system which is not
limited to cross building by design. I can imagine that, once the new
world has arrived, we can also use it for customizing standard
distributions as well. And barebox is flexible enough to serve the
maximum-bootloader pattern today, while gradually migrate towards the
minimum one.

However, I found the discussion pretty interesting! Thanks a lot, it
gave me some interesting insights.

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Jamie Lokier]

Rob Landley wrote:
> However, if that's your minimum then you can't use the bootloader to
> re-flash the device, which is kind of handy.  (It gives you an
> un-bricking fallback short of pulling out a jtag.)  But doing that
> requires things like a network driver, TCP/IP stack, tftp
> implementation, serial driver, command line interpreter, and so on.
> And of course code to erase and write flash blocks for your flash
> chip du jour, plus knowledge of the flash layout.  (In theory, said
> knowledge comes from parsing a device tree.)

What a lot of bloat you added for such a basic requirement. :-)
You don't need all that to unbrick.

It's enough to have a serial/USB/network driver (choose one),
obediently listen for a kernel to be sent for booting from RAM,
and boot it.  In the network case it can be a simple UDP protocol,
or even raw ethernet.

No TCP/IP, no TFTP, not even BOOTP (but it's a nice bonus), no command
line interpreter (just a GPIO on board to boot into "unbrick me" mode
:-), and most strikingly _no_ flash driver for flash chip du jour.

To flash it you send a kernel to boot from RAM which is capable of
flashing it.

> > http://wiki.openmoko.org/wiki/Qi
> 
> Looking at the screen shot there, you've got code to parse ext2 filesystems.  
> What is your definition of "minimal"?

Ew, ext2 doesn't even satisfy powerfail-during-kernel-upgrade safety.

I agree it does beg the question of what is "minimal".

The proposal did explain quite well what Qi aims for: not duplicating
lots of kernel drivers badly.  If it succeeds in the area of flash
writing, network drivers, network protocols and so on it would be no
bad thing.

One area for potential common ground among bootloaders could be to
share the code for parsing filesystems.  It'd be great to see that in
a library shared by GRUB, Qi, U-boot and so on as it's not device
specific at all and not particularly large, but not so trivial that
it's good to have lots of clones.

It's possible to boot without parsing filesystems, but that is one
rather nice feature, and with the right filesystems it can make system
updates powerfail-safe.

> Rationale for not providing a boot menu is you don't want to mess with video 
> init.  I don't think I've actually seen an embedded bootloader that messes 
> with video, they do serial console instead, and you have a screen shot of 
> serial console messages so apparently the serial driver part is there...

In perspective, serial is usually quite simple.  Output only serial is
even simpler, though :-)

-- Jamie

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/22/09 23:28, Somebody in the thread at some point said:

Hi -

>> Fedora provides a whole solution there, with the restriction it's
>> designed for native build, not cross.
>
> That's probably also a matter of taste. I still find it a feature to be

The two ways have different advantages and disadvantages.  Cross gives 
you speed of rebuild, but 99% of packages on an ARM11+ type system you 
actually have no interest in rebuilding since you just use the stock 
versions.  Cross can be extremely painful with projects that are not set 
up for it, eg, perl.  Native build gives you compatibility of build with 
the whole package universe.  The slow build action doesn't hurt so much 
because -->

> able to cross compile the systems - we can still recompile whole systems
> consistently in just a few (ten) minutes, i.e. in order to change a
> central switch (like softfloat vs. hardfloat, use another toolchain
> etc). Out of interest, how long does it take to recompile Fedora on for
> example MX31?

I don't know or care when I get the binary packages from a repo where 
they're already built.  The whole point of a distro solution is someone 
did all the work for you.  You're only thinking about mass rebuild 
yourself because it's the buildroot mindset, that whole task disappears 
with a distro basis.

>> Packages only help any QA effort. You don't have to release every
>> package build to your stable repo, a staged development / testing /
>> stable repo scheme is simple.
>
> For let's say a telematics box it's almost impossible to test packet
> combinations. All you can do is decide for "update application" and
> "update platform". If you got your software updates via SMS, it
> shouldn't go wrong because of an untested packet combination :-)

You missed my point I think.  You can emulate "issue 6 monthly rootfs 
tarball updates" by just updating the stable package repo at long 
intervals with well-tested packagesets.  At the same time you can offer 
other repos with newer features earlier, get changed packages tested 
easier, confirm patchlevel on test systems, etc.

>> Right. Fedora is different though, there are no cross-built packages
>> (although they do provide cross compilers, I use them for kernel and
>> Qi builds) and if storage is sufficient, there's no need to strip
>> anything out. Just nobble init.
>
> Hmm, I'm still not convinced. But as I don't have any data, I'll keep
> quiet :-)

You can get data easily enough, use the Fedora tarball and run with 
init=/bin/bash as a starting point.  Then write a script that does the 
minimal jobs like bringup lo, remount rw, config ethernet and service 
sshd start and time that as init.

The point is you're definitively bypassing the *whole* of Fedora bringup 
that way while still getting the advantages of the Fedora rootfs basis 
and most of sysv service management.

> I think a central question is if you want to optimize things like
>
> * early boot splash
> * footprint

Early boot splash is a kernel matter, it has facilities for it already. 
  For footprint, you can't play this game if you don't have say 200MB 
spare as already told it's the cost of entry.

But if you can play the distro game (combined with leaving the 
bootloader alone to only boot Linux) the software engineering and 
maintainence load for a really capable rootfs dwindles to about the same 
level of job as maintaining a Linux server box, and quality goes up.

>> Well for my uses of it I have been able to specify that we should have
>> versioning GPIOs on the boards. It's a good idea anyway.
>
> Hardware description is still one of the unsolved problems out there. It
...
> need the schematics in machine readable form... When oftree comes into
> the ARM world, we need a maximum-style bootloader which can provide
> oftrees.

I take your point but actually there's no reason the *bootloader* needs 
that when the bootloader is focussed solely on booting Linux.  *Linux* 
might want an equipment list from the board, but then typically you 
would build all the drivers and they can simply probe and fail if its 
not there on the board.

>> It's nothing to do with mainline, just intra-company communication and
>> management failure.
>
> My experience is that mainline exposure of that perverted bit-twiddling
> code and ideas helps finding sane solutions, because someone will
> allways ask the right questions :-)

I'm not sure I managed to give the flavour of a bunch of hardware guys 
half a world away rotating in and out on Military service.  Even patches 
internally aren't happening, Mainline isn't an answer.

>>> Unfortunately, often people want to boot as fast as possible, which
>>> requires optimization in that area as well. We recently had a board
>>> which refused to boot without the PMIC having switched on some
>>> voltages which are default-off.
>>
>> If your device is able to run from USB power, there's the issue that
>> you are limited to 100mA before enumeration takes place. So without a
>> USB stack, you have to trade speed for power anyway.
>
> My argument is that there are all these different requirements out
> there: your use cases, mine, others. What we want to do with barebox is

Sure, Qi targets ARM9+ with steppingstone.  If you have something else 
you will have to use something else or port it.

If it's not understood what the advantages are of Qi's strict rejection 
of its list of evil things, then they will likely go with U-Boot I guess 
since it has the mindshare.

>> bus driver back into the bootloader. But actually, normal customers
>> don't care about 200ms on boot either way. They can get the thing to
>> market quicker and so cheaper and more reliably without that stuff in
>> the bootloader.
>
> That's a matter of the definition of "normal customers" :-)

What I mean by it is for geeks like us, it's interesting to see how fast 
it will go.  The actual customer cannot tell 200ms by eye he will accept 
it if it's not passing his threshold of being "too slow".  But he will 
like getting it shipping earlier because the bootloader is almost 
invisible in dev effort and in management of production.

> distributions, where people can just 'apt-get install kde'. All our
> recent experiments seem to suggest that mainstream distros are not able
> to do what we need today, but times may change.

Well it sounds like you didn't try Fedora which should be high on the 
list.  But if "what you need" doesn't match I guess you won't have a 
good time there no matter how great it is.

> However, I found the discussion pretty interesting! Thanks a lot, it
> gave me some interesting insights.

You're welcome, we even mentioned Qi occasionally :-)

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/23/09 02:28, Somebody in the thread at some point said:

Hi -

> No TCP/IP, no TFTP, not even BOOTP (but it's a nice bonus), no command
> line interpreter (just a GPIO on board to boot into "unbrick me" mode
> :-), and most strikingly _no_ flash driver for flash chip du jour.
>
> To flash it you send a kernel to boot from RAM which is capable of
> flashing it.

Sorry I missed where this kernel appears from and the bootloader that 
spawned it, since both could get trashed.  That is actually a conundrum 
on a lot of systems and some of the solutions (write-once backup 
bootloader) in the long run lead to other issues.

True SD Boot does truly deliver unbrickability if you are willing to 
swap out or reformat the SD card.

>>> http://wiki.openmoko.org/wiki/Qi
>>
>> Looking at the screen shot there, you've got code to parse ext2 filesystems.
>> What is your definition of "minimal"?
>
> Ew, ext2 doesn't even satisfy powerfail-during-kernel-upgrade safety.

It's just misleading (but accurate).  ext2 is the "lowest common 
denominator" read-only parsing that actually supports ext3 and ext4 if 
you are careful about the formatting options.  So the actual filesystem 
is ext3 or ext4 typically (ext3 in GTA02 case), it's not that the 
bootloader is mandating specifically ext2.

> I agree it does beg the question of what is "minimal".
>
> The proposal did explain quite well what Qi aims for: not duplicating
> lots of kernel drivers badly.  If it succeeds in the area of flash
> writing, network drivers, network protocols and so on it would be no
> bad thing.

Thanks.

> One area for potential common ground among bootloaders could be to
> share the code for parsing filesystems.  It'd be great to see that in
> a library shared by GRUB, Qi, U-boot and so on as it's not device
> specific at all and not particularly large, but not so trivial that
> it's good to have lots of clones.

Yeah it's not a bad idea.

> It's possible to boot without parsing filesystems, but that is one
> rather nice feature, and with the right filesystems it can make system
> updates powerfail-safe.

Bootloader is tricky, but actually on this iMX31 device Fedora is used, 
yum update keeps the last 3 kernels around and our kernel package 
follows that.  So it's possible to have backup kernels automatically 
integrated into the bootloader and packaging system.

>> Rationale for not providing a boot menu is you don't want to mess with video
>> init.  I don't think I've actually seen an embedded bootloader that messes
>> with video, they do serial console instead, and you have a screen shot of
>> serial console messages so apparently the serial driver part is there...
>
> In perspective, serial is usually quite simple.  Output only serial is
> even simpler, though :-)

Totally agree!

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Robert Schwebel]

On Wed, Dec 23, 2009 at 08:38:08AM +0000, Andy Green wrote:
> I don't know or care when I get the binary packages from a repo where
> they're already built. The whole point of a distro solution is someone
> did all the work for you. You're only thinking about mass rebuild
> yourself because it's the buildroot mindset, that whole task
> disappears  with a distro basis.

If you don't step into for example toolchain problems or other crazy
things...

> You can emulate "issue 6 monthly rootfs tarball updates" by just
> updating the stable package repo at long intervals with well-tested
> packagesets. At the same time you can offer other repos with newer
> features earlier, get changed packages tested easier, confirm
> patchlevel on test systems, etc.

Yes, that's a valuable option.

> I take your point but actually there's no reason the *bootloader*
> needs that when the bootloader is focussed solely on booting Linux.
> *Linux* might want an equipment list from the board, but then
> typically you would build all the drivers and they can simply probe
> and fail if its not there on the board.

The oftree is currently provided by the bootloader, and much of what it
contains is unprobable peripherals, i.e. the IP cores in the SoC cpus.
For example for i.MX (which we happen to maintain in the mainline),
there is a strong aim for having one kernel that runs on as many devices
as possible. If you want to do this and if you can't probe significant
parts of the hardware, you need an instance outside of the kernel who
tells you what's actually there.

> I'm not sure I managed to give the flavour of a bunch of hardware guys
> half a world away rotating in and out on Military service. Even
> patches internally aren't happening, Mainline isn't an answer.

Well, there are many projects out there which are not so secret that one
cannot expose the kernel drivers. And even if they are, it is possible
to establish a peer-review culture inside of a corporation. But you
won't get the full power of community review. That's the trade off one
has to accept for having secrets :-) But quality is generally a big
issue.

> > > bus driver back into the bootloader. But actually, normal
> > > customers don't care about 200ms on boot either way. They can get
> > > the thing to market quicker and so cheaper and more reliably
> > > without that stuff in the bootloader.
> >
> > That's a matter of the definition of "normal customers" :-)
>
> What I mean by it is for geeks like us, it's interesting to see how
> fast it will go. The actual customer cannot tell 200ms by eye he will
> accept it if it's not passing his threshold of being "too slow". But
> he will like getting it shipping earlier because the bootloader is
> almost invisible in dev effort and in management of production.

We have customers who care about "splash in 0.5 s" vs. "shell runs after
3 s, then qt starts". People may be used to that kind of noticable boot
time in the phone business, but in the industry (where embedded Linux
boxes are even more "devices" than computers) they often are not.

Do you remember the times when we had analog TV? We could zap through 5
channels in under 3 seconds. *That's* performance :-) My sattelite
receiver needs about 10 seconds to boot. Sometimes it feels like
innovation goes backwards.

Cheers,
rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/23/09 08:56, Somebody in the thread at some point said:

Hi -

>> yourself because it's the buildroot mindset, that whole task
>> disappears  with a distro basis.
>
> If you don't step into for example toolchain problems or other crazy
> things...

Again this is buildroot thinking.  The distro provides both the native 
and cross toolchains for you.  You're going to want to use the same 
distro as you normally use on your box so the cross toolchain installs 
as a package there.

If all that's left is the risk of "crazy things" happening well that's a 
risk whatever you do or even if you do nothing :-)

> The oftree is currently provided by the bootloader, and much of what it
> contains is unprobable peripherals, i.e. the IP cores in the SoC cpus.
> For example for i.MX (which we happen to maintain in the mainline),
> there is a strong aim for having one kernel that runs on as many devices
> as possible. If you want to do this and if you can't probe significant
> parts of the hardware, you need an instance outside of the kernel who
> tells you what's actually there.

The only thing I know of that matches "outside the kernel" requirement 
is the machine ID that's passed in on ATAG.  I agree it's generally good 
to have a single build that's multipurpose.

On iMX you have to go read IIM to get device info but actually that's 
not hard.  But that device ID will itself alone tell you the on-SoC 
peripherals since there's an ID per die; it makes no difference if this 
expanded SoC oftree data itself lives in the kernel then.  The non-SoC 
stuff can just be probed.

> We have customers who care about "splash in 0.5 s" vs. "shell runs after
> 3 s, then qt starts". People may be used to that kind of noticable boot

That's fine, they can pay for the extra time to market and work done on 
the crap in their bootloader :-)  Many customers don't care if it acts 
in line with their general expectation and was delivered faster and 
cheaper than they expected.

> Do you remember the times when we had analog TV? We could zap through 5

But your CRT took a while to warm up / "boot" as well...

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Robert Schwebel]

On Wed, Dec 23, 2009 at 09:29:22AM +0000, Andy Green wrote:
> > If you don't step into for example toolchain problems or other crazy
> > things...
>
> Again this is buildroot thinking. The distro provides both the native
> and cross toolchains for you. You're going to want to use the same
> distro as you normally use on your box so the cross toolchain installs
> as a package there.
>
> If all that's left is the risk of "crazy things" happening well that's
> a risk whatever you do or even if you do nothing :-)

What I mean is that in the past we had more than once the case that
we've found showstopper bugs in upstream gcc, binutils etc. (ARM has
much less coverage than x86). It was never a problem in the project,
because we actually have been able to port test cases to the latest gcc,
reproduc, report to upstream, get it fixed, port it back. That at least
implies that you *are* able to rebuild everything (which is possible in
both ways, of course).

> The only thing I know of that matches "outside the kernel" requirement
> is the machine ID that's passed in on ATAG. I agree it's generally
> good to have a single build that's multipurpose.

Machine ID is a good concept, but doesn't work on everything. What's the
machine on a system that consists of a cpu module and a baseboard?
Things move into the oftree direction, and I think bootloaders have to
deal with that, or there has to be a possibility that an early kernel
stage deals with the oftree. But that just moves the hackery-code from
the bootloader into the kernel, which is no gain.

> On iMX you have to go read IIM to get device info but actually that's
> not hard.

... as long as the hardware guys don't decide in their great wisdom to
change the meaning of some bits from chip to chip ...

> But that device ID will itself alone tell you the on-SoC peripherals
> since there's an ID per die; it makes no difference if this expanded
> SoC oftree data itself lives in the kernel then. The non-SoC stuff
> can just be probed.

Off-board devices behind SPI, I2C and chip select busses cannot be
probed.

You are observing a subset of the embedded universe, and you are right
if you limit to it. Bootloaders like barebox or u-boot provide a concept
for *all* use cases.

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Rob Landley]

On Tuesday 22 December 2009 16:23:37 Andy Green wrote:
> On 12/22/09 11:12, Somebody in the thread at some point said:
>
> Hi Robert -
>
> >> (Personally I used Fedora ARM port and RPM, but any distro and
> >> packagesystem like Debian workable on ARM would be fine).
> >
> > Until now, we are using the "build it yourself" approach with ptxdist,
> > basically because of these reasons:
> >
> > - If something goes wrong, we want to be able to fix it, which means
> >    that we must be able to recompile everything. Having the source is no
> >    value by itself, if you are not able to build it.
>
> Fedora provides a whole solution there, with the restriction it's
> designed for native build, not cross.

QEMU: it's not just for breakfast anymore.

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Rob Landley]

On Wednesday 23 December 2009 03:29:22 Andy Green wrote:
> On 12/23/09 08:56, Somebody in the thread at some point said:
>
> Hi -
>
> >> yourself because it's the buildroot mindset, that whole task
> >> disappears  with a distro basis.
> >
> > If you don't step into for example toolchain problems or other crazy
> > things...
>
> Again this is buildroot thinking.  The distro provides both the native
> and cross toolchains for you.  You're going to want to use the same
> distro as you normally use on your box so the cross toolchain installs
> as a package there.

Because boards that use things like uClibc and busybox just aren't interesting 
to you?

Please don't confuse "development environment" with "build environment".  A 
development environment has xterms and IDEs and visual diff tools and a web 
browser and PDF viewer and so on.  A build environment just compiles stuff to 
produce executables.  (Even on x86, your fire breathing SMP build server in the 
back room isn't necessarily something you're going to VNC into and boot a 
desktop on.)

I agree it's nice to have a build environment compatible with your deployment 
environment, and distros certainly have their advantages, but you may not want 
to actually _deploy_ 48 megabytes of /var/lib/apt from Ubuntu in an embedded 
device.

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/27/09 07:17, Somebody in the thread at some point said:

Hi Rob -

>> Fedora provides a whole solution there, with the restriction it's
>> designed for native build, not cross.
>
> QEMU: it's not just for breakfast anymore.

That's right Qemu often requires lunch, teatime and supper too to build 
anything :-)

Newer ARM platforms like Cortex8+ and the Marvell Sheevaplug will 
outstrip emulated performance on a normal PC.  There are 2GHz multi-core 
ARMs coming as well apparently.  So I took the view I should ignore Qemu 
and get an early start on the true native build that will be the future 
of "native build" as opposed to cross due to that.

The point of the distro is you just let them build the bulk of it, just 
installing binary packages.  You're only rebuilding the bits you are 
changing for your application.  For a lot of cases that's a few small 
app packages that are mainly linking against stuff from the distro and 
they're not too bad to do natively.  (In addition my workflow is to edit 
on a host PC and use scripts to teleport a source tree tarball to the 
device where it's built as a package every time and installed together 
with its -devel, so everything is always under package control).

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/27/09 07:27, Somebody in the thread at some point said:

Hi Rob -

>> Again this is buildroot thinking.  The distro provides both the native
>> and cross toolchains for you.  You're going to want to use the same
>> distro as you normally use on your box so the cross toolchain installs
>> as a package there.
>
> Because boards that use things like uClibc and busybox just aren't interesting
> to you?

I used them both before, but I can say with confidence if the platform 
will take glibc and bash, most people will expect more complete, in that 
sense, more reliable, performance from those.

It breaks down at stock distro init because it's painfully slow.  But 
otherwise there are real advantages in having the full-strength versions 
of everything.

> Please don't confuse "development environment" with "build environment".  A

Since I didn't use either term, I don't know why you think I'm confusing 
them.

> development environment has xterms and IDEs and visual diff tools and a web
> browser and PDF viewer and so on.  A build environment just compiles stuff to
> produce executables.  (Even on x86, your fire breathing SMP build server in the
> back room isn't necessarily something you're going to VNC into and boot a
> desktop on.)

As I said in the other reply, my workflow is to edit a package's source 
tree on a host (so you can use any editor on your host not just kate / 
fish:// ) and by host script with scp and ssh get the current tree 
package-built and installed on the device in one step.

So I hope it's clear there is solid separation between what you're 
calling "development environment" and "build environment" to the point 
they have nothing to do with each other except ssh-based script to get 
stuff built.

> I agree it's nice to have a build environment compatible with your deployment
> environment, and distros certainly have their advantages, but you may not want
> to actually _deploy_ 48 megabytes of /var/lib/apt from Ubuntu in an embedded
> device.

I did say in the thread you want ARM11+ basis and you need 100-200MBytes 
rootfs space to get the advantages of the distro basis.  If you have 
something weaker (even ARM9 since stock Fedora is ARMv5+ instruction set 
by default) then you have to do things the old way and recook everything 
yourself one way or another.

Even now there are plenty of suitable platforms that will work with it, 
and over time they will only increase.  Nothing seems to totally die out 
(8051-based micros are still in the market) but each time something new 
comes in at the top it grabs some of the market and the older ones shrink.

It boils down to the point that if you just treat the ARM11+ platforms 
like the previous generation and stick fat bootloaders and buildroot 
blobs on them, you are going to miss out on an epochal simplification 
where embedded Linux largely becomes like desktop Linux in workflow, 
quality and reliability of update mechanisms, and effort needed to bring 
up a box / device.

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Rob Landley]

On Sunday 27 December 2009 03:54:51 Andy Green wrote:
> On 12/27/09 07:17, Somebody in the thread at some point said:
>
> Hi Rob -

Before replying, I note that Mark Miller and I gave a presentation entitled 
"Developing for non-x86 targets using QEMU" Ohio LinuxFest in October.

  http://impactlinux.com/fwl/downloads/presentation.pdf
  http://impactlinux.com/fwl/presentation.html

There's even horribly mangled audio of our rushed 1 hour presentation.  (The 
slides are for a day-long course and we had 55 minutes to give a talk, so we 
skimmed like mad.  Unfortunately, the netbook they used for audio recording 
had the mother of all latency spikes whenever the cache flush did an actual 
flash erase and write, so there are regular audio dropouts the whole way 
through.)  Still, it's somewhere under:

  http://www.archive.org/details/OhioLinuxfest2009

I've also spent the last few years developing a project that produces native 
built environments for various QEMU targets and documents how to bootstrap 
various distros under them:

  http://impactlinux.com/fwl

So I do have some firsthand experience here.

> >> Fedora provides a whole solution there, with the restriction it's
> >> designed for native build, not cross.
> >
> > QEMU: it's not just for breakfast anymore.
>
> That's right Qemu often requires lunch, teatime and supper too to build
> anything :-)

Which is why you hook it up to distcc so it can call out to the cross 
compiler, which speeds up the build and lets you take advantage of SMP.  
(Pages 217-226 of the above PDF.)

That's also why my FWL project uses a statically linked version of busybox, 
because the static linking avoids the extra page retranslations on each exec 
and thus sped up the ./configure stage by 20%.  (Pages 235-236 of PDF.)

There's more things you can do to speed it up if you want to go down that 
rabbit hole (which the presentation does), and there's more work being done in 
qemu.  (TCG was originally a performance hit but has improved since, although 
it varies widely by platform and is its own rabbit hole.  Also switching to 
gigabit NIC emulation with jumbo frames helped distcc a lot.)

But in general, Moore's Law says that qemu on current PC hardware is about the 
speed of current PC hardware seven years ago.  (And obviously nobody ever 
built anything before 2003. :)

> Newer ARM platforms like Cortex8+ and the Marvell Sheevaplug will
> outstrip emulated performance on a normal PC.  There are 2GHz multi-core
> ARMs coming as well apparently.  So I took the view I should ignore Qemu
> and get an early start on the true native build that will be the future
> of "native build" as opposed to cross due to that.

Pages 24-34 of the above PDF go over this.  The first two pages are on the 
advantages of native compiling on real hardware, the next eight pages are on 
the disadvantages.  It can certainly be made to work, especially in a large 
corporation willing to spend a lot of money on hardware as a _prerequisite_ to 
choosing a deployment platform.

For hobbyists, small businesses, and open source developers in general, there 
are significant advantages to emulation.  (Page 208 comes to mind.)  And if you 
_are_ going to throw money at hardware, x86-64 continues to have better 
price/performance ratio, which was always its thing.

> The point of the distro is you just let them build the bulk of it, just
> installing binary packages.  You're only rebuilding the bits you are
> changing for your application.

Pages 68-71.  If your definition of embedded development is using off the shelf 
hardware and installing prebuilt binary packages into it, life becomes a lot 
easier, sure.

> For a lot of cases that's a few small
> app packages that are mainly linking against stuff from the distro and
> they're not too bad to do natively.

Pages 78-84

> (In addition my workflow is to edit
> on a host PC

Pages 178-180

> and use scripts to teleport a source tree tarball to the
> device where it's built as a package every time and installed together
> with its -devel,

Pages 181-202

> so everything is always under package control).

Package control or source control?  (Different page ranges...)

> -Andy

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Rob Landley]

On Sunday 27 December 2009 04:09:23 Andy Green wrote:
> > I agree it's nice to have a build environment compatible with your
> > deployment environment, and distros certainly have their advantages, but
> > you may not want to actually _deploy_ 48 megabytes of /var/lib/apt from
> > Ubuntu in an embedded device.
>
> I did say in the thread you want ARM11+ basis and you need 100-200MBytes
> rootfs space to get the advantages of the distro basis.  If you have
> something weaker (even ARM9 since stock Fedora is ARMv5+ instruction set
> by default) then you have to do things the old way and recook everything
> yourself one way or another.

I started programming on a commodore 64.  By modern standards, that system is 
so far down into "embedded" territory it's barely a computer.  And yet people 
did development on it.

  http://landley.net/history/catfire/wheniwasaboy.mp3

That said, you can follow Moore's Law in two directions: either it makes stuff 
twice as powerful every 18 months or it makes the same amount of power half 
the price.

What really interests me is disposable computing.  Once upon a time swiss 
watches were these amazingly valuable things (which Rolex and friends try to 
cling to even today by gold-plating the suckers), but these days you can get a 
cheap little LCD clock as a happy meal toy.  The cheapest crappiest machines 
capable of running Linux are 32-bit boxes with 4 gigs of ram, which were high-
end server systems circa 1987 that cost maybe about $5k (adjusted for inflation 
anyway).  These days, the cheapest low-end Linux boxes (of the "repurposed 
router" variety) are what, about $35 new?  Moore's Law says that 21 years is 
14 doublings, which would be 1) $2500, 2) $1250, 3) $635, 4) $312, 5) $156, 6) 
$87, 7) $39, 8) $19, 9) $9.76, 10) $4.88, 11) $2.44, 12) $1.22, 13) $0.61, 14) 
$0.31.

So in 2009 that $5000 worth of computing power should actually cost about 30 
cents, and should _be_ disposable.  In reality, the CPU in that router is 
clocked 20 times faster than a Compaq deskpro 386, you get 4 to 8 times the 
memory, they added networking hardware, and so on.  And there are fixed costs 
for a case and power supply that don't care about Moore's Law, plus up-front 
costs to any design that need to be amortized over a large production run to 
become cheap, and so on.

And the real outstanding research problems include ergonomic UI issues for 
tiny portable devices, batteries wearing out after too many cycles, and the 
fact that making "disposable" devices out of materials like cadmium is dubious 
from environmental standpoint.  Oh, and of course there was the decade or two 
companies like Intel lost going up a blind alley by bolting giant heat sinks 
and fans onto abominations like the Pentium 4 and Itanic.  They didn't care 
about power consumption at all until fairly recently, and are still backing 
out of that cul-de-sac even today...

Still, I forsee a day when cereal boxes have a display on the front that 
changes every 30 seconds to attract passerby, driven by the same amount of 
circuitry and battery that makes the "free toy inside" blink an LED today.  (I 
don't know what else that sort of thing will be used for, any more than people 
predicted checking twitter from the iPhone,)

This I'm reluctant to abandond the low-end and say "oh we have more power now, 
only machines with X and Y are interesting".  The mainframe, minicomputer, and 
micro (PC) guys each said that, and today the old PC form factor's getting 
kicked into the server space by the iPhone and such.  I want to follow Moore's 
Law down into disruptive technology territory and find _out_ what it does.

> Even now there are plenty of suitable platforms that will work with it,
> and over time they will only increase.

You must be this tall to ride the computer.

> Nothing seems to totally die out
> (8051-based micros are still in the market)

Mainframes are still on the market too.

> but each time something new
> comes in at the top it grabs some of the market and the older ones shrink.
>
> It boils down to the point that if you just treat the ARM11+ platforms
> like the previous generation and stick fat bootloaders and buildroot
> blobs on them, you are going to miss out on an epochal simplification
> where embedded Linux largely becomes like desktop Linux in workflow,
> quality and reliability of update mechanisms, and effort needed to bring
> up a box / device.

New computing niches will develop new usage patterns.  The iPhone is currently 
doing this, and is unlikely to be the last cycle.

They'll also grow more powerful and expand into old niches the way "blade 
servers" are constructed from laptop components and used for batch processing 
today, but I personally find that less interesting.

> -Andy

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/27/09 23:15, Somebody in the thread at some point said:

Hi Rob -

> I've also spent the last few years developing a project that produces native
> built environments for various QEMU targets and documents how to bootstrap
> various distros under them:
>
>    http://impactlinux.com/fwl
>
> So I do have some firsthand experience here.

I wasn't suggesting you don't have firsthand experience all over the 
place, eg, busybox, I know it.  What I do suggest in the principles I 
have been bothering your Christmas with here can really pay off and are 
in the opposite direction to Qemu, bloated bootloaders, and buildroot 
style distro production.

>>>> Fedora provides a whole solution there, with the restriction it's
>>>> designed for native build, not cross.
>>>
>>> QEMU: it's not just for breakfast anymore.
>>
>> That's right Qemu often requires lunch, teatime and supper too to build
>> anything :-)
>
> Which is why you hook it up to distcc so it can call out to the cross
> compiler, which speeds up the build and lets you take advantage of SMP.
> (Pages 217-226 of the above PDF.)

Or you just do it native and you don't care about extreme rebuild case 
because you're using a distro that built it already.

There's a critical advantage to build in specifically the execution 
environment you do cover in three words on slide 14 of your PDF I expand 
on at the end of the mail.

> That's also why my FWL project uses a statically linked version of busybox,
> because the static linking avoids the extra page retranslations on each exec
> and thus sped up the ./configure stage by 20%.  (Pages 235-236 of PDF.)
>
> There's more things you can do to speed it up if you want to go down that
> rabbit hole (which the presentation does), and there's more work being done in
> qemu.  (TCG was originally a performance hit but has improved since, although
> it varies widely by platform and is its own rabbit hole.  Also switching to
> gigabit NIC emulation with jumbo frames helped distcc a lot.)

Just saying people don't do distribution rebuilds for their desktop or 
server boxes unless they're Gentoo believers.  So the need to consider 
this heavy duty stuff only exists at the distro.  In Fedora case, guys 
at Marvell are running it and have access to lots of fast physical 
hardware they hooked up to Koji (you touch on this in your PDF).  I 
don't think it's like they were waiting to hear about Qemu now they're 
gonna drop that and move to emulation.  They're in the business of 
making fast ARM chips and they're going to outpace Qemu.

> But in general, Moore's Law says that qemu on current PC hardware is about the
> speed of current PC hardware seven years ago.  (And obviously nobody ever
> built anything before 2003. :)

The speed surely depends on the architecture being emulated.  I did try 
Qemu on ARM on a nice desktop box here and it seemed unworkable to me. 
I am perfectly happy to trust Fedora to take care of disribution 
rebuilds for me.

>> Newer ARM platforms like Cortex8+ and the Marvell Sheevaplug will
>> outstrip emulated performance on a normal PC.  There are 2GHz multi-core
>> ARMs coming as well apparently.  So I took the view I should ignore Qemu
>> and get an early start on the true native build that will be the future
>> of "native build" as opposed to cross due to that.
>
> Pages 24-34 of the above PDF go over this.  The first two pages are on the
> advantages of native compiling on real hardware, the next eight pages are on
> the disadvantages.  It can certainly be made to work, especially in a large
> corporation willing to spend a lot of money on hardware as a _prerequisite_ to
> choosing a deployment platform.

This is what I have been calling "buildroot thinking" again.  What do 
you think you will be rebuilding exactly out of Fedora when you put it 
on an ARM11+ system?  Typically, it's going to be zero packages, nothing 
at all.  What you will need to build typically is this:

  - bootloader (using Fedora's cross compiler package)
  - kernel (using Fedora's cross compiler package)
  - your own apps and libs (native build on device against libs there)

In addition you will need to take evasive action around the boot flow, 
but to get started that's just init=/bin/my-init.sh

> For hobbyists, small businesses, and open source developers in general, there
> are significant advantages to emulation.  (Page 208 comes to mind.)  And if you
> _are_ going to throw money at hardware, x86-64 continues to have better
> price/performance ratio, which was always its thing.

The point is you will definitely be throwing lots of money at hardware 
in this game, your ARM device platform.  Since all products start with 
zero users, and we get to be the first, we had best spend all our time 
gaining experience with the hardware we plan to ship.

If there are problems with upstream unless it's something really in the 
plumbing like cache behaviour which is a kernel issue, it can normally 
be shown in the code easy enough.

> Pages 68-71.  If your definition of embedded development is using off the shelf
> hardware and installing prebuilt binary packages into it, life becomes a lot
> easier, sure.

Well that's exactly my point, except the shelf the hardware came off is 
the one with your own prototype ARM devices.  No Qemu, no mass rebuilds, 
no funky "it's embedded so we have to whip ourselves with nettles".

If life's a lot easier blowing away buildroot style and the 
infrastructure for that, why is anyone considering doing it the hard way 
on devices that can handle the faster, more reliable and well understood 
packaged distro rootfs basis?  It really does become like working on a 
desktop Linux box with Fedora.

>> For a lot of cases that's a few small
>> app packages that are mainly linking against stuff from the distro and
>> they're not too bad to do natively.
>
> Pages 78-84
...

I have been actually living these principles the past couple of years, 
consider this a report from a specific frontline rather than a review of 
possibilities.

> Package control or source control?  (Different page ranges...)

Package control is also source control, that is a whole other advantage 
of using a packaged distro, license compliance becomes real easy.  A 
Source RPM is generated with the binary ones, you just need to get the 
SRPMs of the packageset you are shipping (which you can get a list of 
with rpm -qa) and publish it for GPL compliance.  You can also grep the 
packages for licence and automate other tasks that way (such as 
excluding source).


Anyway here's the reason promised earlier that building out of the 
execution environment will bite you, you need to keep the "-devel" stuff 
in sync: includes and libraries.  You can easily install a new version 
of something with a -devel on your actual device and still be building 
against the old version of it on this Qemu or Cross build environment.

Building on the device you are running on, and using packaged builds for 
your own apps removes this issue completely.


This thread was meant to be about merits of Qi, it's kinda gone off into 
embedded with distro rootfs because the philosophy is related.  In both 
cases burden on the developer is intended to be removed and effort 
simplified to get the job done more reliably and quicker.

My experience has been this kind of talk makes people feel jumpy when 
they're invested in the other philosophies.  I hope what has been 
discussed has left people at least wondering if there might be value in 
disallowing the bootloader from doing anything but get you quickly into 
Linux, so the development and support effort can focus just in Linux and 
the bootloader was just something you spent a day tuning months ago and 
never touched again (especially not in the factory).

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/28/09 00:21, Somebody in the thread at some point said:

Hi -

> I started programming on a commodore 64.  By modern standards, that system is
> so far down into "embedded" territory it's barely a computer.  And yet people
> did development on it.

My dear Rob I got started on a PET, I can understand your POV :-)

> kicked into the server space by the iPhone and such.  I want to follow Moore's
> Law down into disruptive technology territory and find _out_ what it does.

The big challenge I see is delivering highly complex Linux devices with 
insufficient developers in a way that won't disappear up its own ass and 
kill the project / customer with delays or failure to perform.

It's nice if the device is efficient with every cycle, but that is a 
geek preoccupation.  Many customers in suits will tell to spend an extra 
$1 to overcome it by hardware and gain back $5 from accelerated time to 
market, so long as they can depend on the high quality of the software 
basis to not kill that logic by delays.

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Peter Korsgaard]

>>>>> "Andy" == Andy Green <andy@warmcat.com> writes:

Hi,

 Andy> This is what I have been calling "buildroot thinking" again.

Thanks for an interesting discussion and all your free buildroot
advertising ;)

(buildroot maintainer)

 Andy> This thread was meant to be about merits of Qi, it's kinda gone
 Andy> off into embedded with distro rootfs because the philosophy is
 Andy> related.  In both cases burden on the developer is intended to be
 Andy> removed and effort simplified to get the job done more reliably
 Andy> and quicker.

Sure, if your embedded device is very PC like (size/functionality) that
makes sense.

-- 
Bye, Peter Korsgaard

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/28/09 19:57, Somebody in the thread at some point said:

Hi Peter -

> Thanks for an interesting discussion and all your free buildroot
> advertising ;)
>
> (buildroot maintainer)

Hey you are welcome... in the situations where you can't play the distro 
game then buildroot is the lifesaver.  Same as U-Boot I have experienced 
in detail the amount of work involved in what buildroot delivers and it 
has to be respected for what it does.

I'm really saying there's a region where buildroot is the right answer 
and a threshold beyond which a "real distro" is the right answer, we 
should not just keep on doing what we have being doing.

(But you want to be careful you don't grow into OpenEmbedded's build 
system.  For me it failed partway through a 1000+ package build on 
trying to build *host* dbus libs for me (yes, host dbus).

I just wanted to package "hello world" on Openmoko.  They did fix it in 
the end so it would build host dbus OK to be fair.  But I felt that was 
a very long way from the point.)

>   Andy>  This thread was meant to be about merits of Qi, it's kinda gone
>   Andy>  off into embedded with distro rootfs because the philosophy is
>   Andy>  related.  In both cases burden on the developer is intended to be
>   Andy>  removed and effort simplified to get the job done more reliably
>   Andy>  and quicker.
>
> Sure, if your embedded device is very PC like (size/functionality) that
> makes sense.

Thanks.  As written the threshold seems to be at ARM11+ and SD 
particularly making it into enough of a "PC" you don't have to worry 
about bash vs ash or where a few 100MB of storage is going to come from.

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Rob Landley]

On Monday 28 December 2009 04:27:04 Andy Green wrote:
> I wasn't suggesting you don't have firsthand experience all over the
> place, eg, busybox, I know it.  What I do suggest in the principles I
> have been bothering your Christmas with here can really pay off and are
> in the opposite direction to Qemu, bloated bootloaders, and buildroot
> style distro production.

In the PDF I spent a dozen pages pointing out that I think buildroot is silly 
and how it diverged from its original goals.  Why do you bring it up here?

Building natively is nice.  I think cross compiling extensive amounts of stuff 
is generally counterproductive, and you don't seem to disagree with that.  
(However, if there weren't cases where cross compiling _does_ make sense I 
wouldn't bother building the second stage cross compilers in my project and 
packaging them up for other people to use.  "I don't recommend it" != "people 
who do this are stupid".)

You're conflating a bunch of issues.  You can boot a full distro under qemu or 
under real hardware, so "use a native distro" is orthogonal to "develop under 
emulation vs develop on target hardware".  And building a hand-rolled system 
or assembling prebuilt distro bits can both be done natively (under either 
kind of "native"), so that's orthogonal too.

I'm really not understanding what argument you're making.  "The one specific 
configuration I've chosen to use is better than any other possible 
configuration"...  Possibly there's a "for your needs" attached somewhere that 
you're missing?   You seem to be confusing "that's not an interesting option 
for me" with "that's not an interesting option".  You're conflating a bunch of 
different issues into a One True Way, which is sad.

> >>>> Fedora provides a whole solution there, with the restriction it's
> >>>> designed for native build, not cross.
> >>>
> >>> QEMU: it's not just for breakfast anymore.
> >>
> >> That's right Qemu often requires lunch, teatime and supper too to build
> >> anything :-)
> >
> > Which is why you hook it up to distcc so it can call out to the cross
> > compiler, which speeds up the build and lets you take advantage of SMP.
> > (Pages 217-226 of the above PDF.)
>
> Or you just do it native and you don't care about extreme rebuild case
> because you're using a distro that built it already.

I find reproducibility to be kind of nice.  (Remember when ubuntu dropped 
Powerpc support?  Or how Red hat Enterprise is finally dropping itanic?  Even 
today can you build a full distro natively on new hardware like microblaze or 
blackfin?)

I also don't see any difference between your "let the distro handle everything" 
with "the vendor supplies a BSP, why would you need anything else?"  That's 
certainly a point of view, and for you it may be fine.  Anything can be 
outsourced.  But "I don't have to do it, therefore nobody does" is a 
questionable position.

> > There's more things you can do to speed it up if you want to go down that
> > rabbit hole (which the presentation does), and there's more work being
> > done in qemu.  (TCG was originally a performance hit but has improved
> > since, although it varies widely by platform and is its own rabbit hole. 
> > Also switching to gigabit NIC emulation with jumbo frames helped distcc a
> > lot.)
>
> Just saying people don't do distribution rebuilds for their desktop or
> server boxes unless they're Gentoo believers.

Linux From Scratch does not exist?  My friend Mark (who inherited a "Slackware 
for PowerPC" cross-build system) is in reality unemployed?  Nobody ever 
actually works _on_ Fedora and needs to do a development build?

It's interesting to see someone who can imply these things with a straight 
face on an embedded development list.

> So the need to consider
> this heavy duty stuff only exists at the distro.  In Fedora case, guys
> at Marvell are running it and have access to lots of fast physical
> hardware they hooked up to Koji (you touch on this in your PDF).  I
> don't think it's like they were waiting to hear about Qemu now they're
> gonna drop that and move to emulation.

You also seem to be unaware that QEMU was only started in 2003 (first commit to 
the repository, Feb 18, 2003), and didn't even work on x86 all that well until 
the end of 2005.  Decent arm support started showing up in 2006 but it was 
darn wonky, and the switch from dyngen to TCG didn't happen until Feb 1, 2008, 
before which you couldn't even build qemu with gcc 4.x:

  http://landley.net/qemu/2008-01-29.html#Feb_1,_2008_-_TCG

Therefore, they didn't "drop" support, QEMU wasn't ready for prime time back 
in 2006 when my then-boss (Manas Saksena) quit Timesys to go launch Fedora for 
Arm.  (The boss I had after that, David Mandala, is now in charge of Ubuntu 
Mobile.  Timesys had some darn impressive alumni, pity the company screwed up 
so badly none of us stayed around.  I myself stayed through four bosses, three 
CEOs, and outlasted 80% of my fellow engineers...)

By the way:

  http://fedoraproject.org/wiki/Architectures/ARM/HowToQemu

You continue to conflate orthogonal issues.

There is a REASON QEMU project hasn't shipped its 1.0 release yet.  Dismissing 
it as irrelevant to the future based on your experiences with it years ago is 
kind of hilarious, really.  (And dismissing emulation in general as of 
interest to nobody...  Sigh.)

> They're in the business of
> making fast ARM chips and they're going to outpace Qemu.

They're not in the same business.  They don't compete.  You don't deploy on 
qemu, and when you go down to Fry's to get a laptop the x86-64 is likely to 
continue to be the main option for the forseeable future.

> > But in general, Moore's Law says that qemu on current PC hardware is
> > about the speed of current PC hardware seven years ago.  (And obviously
> > nobody ever built anything before 2003. :)
>
> The speed surely depends on the architecture being emulated.  I did try
> Qemu on ARM on a nice desktop box here and it seemed unworkable to me.

You couldn't make it work, therefore it can't be made to work?

> I am perfectly happy to trust Fedora to take care of disribution
> rebuilds for me.

Good for you.

Did it ever occur to you that there are people out there who do the bits you 
don't personally do?  That all these packages don't emerge from zeus's 
forehead on a quarterly basis?  That someone, somewhere, is actually 
interested in how this stuff _does_ work?

> >> Newer ARM platforms like Cortex8+ and the Marvell Sheevaplug will
> >> outstrip emulated performance on a normal PC.  There are 2GHz multi-core
> >> ARMs coming as well apparently.  So I took the view I should ignore Qemu
> >> and get an early start on the true native build that will be the future
> >> of "native build" as opposed to cross due to that.
> >
> > Pages 24-34 of the above PDF go over this.  The first two pages are on
> > the advantages of native compiling on real hardware, the next eight pages
> > are on the disadvantages.  It can certainly be made to work, especially
> > in a large corporation willing to spend a lot of money on hardware as a
> > _prerequisite_ to choosing a deployment platform.
>
> This is what I have been calling "buildroot thinking" again.  What do
> you think you will be rebuilding exactly out of Fedora when you put it
> on an ARM11+ system?  Typically, it's going to be zero packages, nothing
> at all.  What you will need to build typically is this:

"All the world's a Cortex A8, and always will be."

Look, go have fun.  I'm sure what you've chosen to do is going to work just 
great for you, and you'll never need to care about anything else.  Personally, 
I no interest in speaking to you ever again about anything, and I have a spam 
filter for situations like this.

I'll even make you happy: "You win!"  There you go.

*plonk*

Rob
-- 
Latency is more important than throughput. It's that simple. - Linus Torvalds

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/29/09 04:25, Somebody in the thread at some point said:

Hi -

(We're talking about ARM11+ case here, just to save a lot of qualifying)

>> I wasn't suggesting you don't have firsthand experience all over the
>> place, eg, busybox, I know it.  What I do suggest in the principles I
>> have been bothering your Christmas with here can really pay off and are
>> in the opposite direction to Qemu, bloated bootloaders, and buildroot
>> style distro production.
>
> In the PDF I spent a dozen pages pointing out that I think buildroot is silly
> and how it diverged from its original goals.  Why do you bring it up here?

Buildroot is the original "select a bunch of projects and I will sit 
building them for you all day and give you a rootfs tarball".  What I 
talked about is "buildroot style distro production", where you sat there 
building things being a bad way if it's possible to use a real distro 
with precooked binaries.

> Building natively is nice.  I think cross compiling extensive amounts of stuff
> is generally counterproductive, and you don't seem to disagree with that.

I completely agree with it.

> (However, if there weren't cases where cross compiling _does_ make sense I
> wouldn't bother building the second stage cross compilers in my project and
> packaging them up for other people to use.  "I don't recommend it" != "people
> who do this are stupid".)

Earlier I pointed out that I use cross for bootloader and kernel.

The problems with cross generally come from having to continually fight 
projects that don't want to build cross and maintaining and building the 
hundreds of projects that make up your rootfs as your own personal 
burden.  Over time it seems to me it will be on a loser when it comes up 
against established distro projects like Fedora that are inheriting the 
quality of native built stuff and delivering binaries for your CPU ready 
cooked.  (Of course this is only true at this ARM11+ point, below that 
cross like buildroot is the lifesaver).

So yes, I would tell people to avoid cross distros if they can find a 
native built one that is workable for them because they can expect less 
issues and more consistent quality.  And they're not gonna be expected 
to build anything themselves.

> You're conflating a bunch of issues.  You can boot a full distro under qemu or
> under real hardware, so "use a native distro" is orthogonal to "develop under
> emulation vs develop on target hardware".  And building a hand-rolled system
> or assembling prebuilt distro bits can both be done natively (under either
> kind of "native"), so that's orthogonal too.

See above.  Qemu itself is an unnecessary burden for the developer, that 
goes triple if the guy follows the way you suggested and makes a giant 
Qemu build farm so he can follow that buildroot style and rebuild his 
whole distro pointlessly with distributed make, instead of simply 
downloading distro-built binaries and getting on with his life.

> I'm really not understanding what argument you're making.  "The one specific
> configuration I've chosen to use is better than any other possible
> configuration"...  Possibly there's a "for your needs" attached somewhere that
> you're missing?   You seem to be confusing "that's not an interesting option
> for me" with "that's not an interesting option".  You're conflating a bunch of
> different issues into a One True Way, which is sad.

Yes, I am sorry if it makes you uncomfortable but some things are 
inherently the wrong direction.  Bootloader bloat is the case in point, 
if there's an OS like Linux on the box it's always preferable to do your 
business in that, not add faux bash to your bootloader.

Sometimes you might get forced into going in the wrong direction, Robert 
was making the case about needing to light up his display super quick at 
boot.  If that happens you may bend to it, but it doesn't change that it 
was the wrong direction and you want to try to avoid it next time.

>> Or you just do it native and you don't care about extreme rebuild case
>> because you're using a distro that built it already.
>
> I find reproducibility to be kind of nice.  (Remember when ubuntu dropped
> Powerpc support?  Or how Red hat Enterprise is finally dropping itanic?  Even
> today can you build a full distro natively on new hardware like microblaze or
> blackfin?)
>
> I also don't see any difference between your "let the distro handle everything"
> with "the vendor supplies a BSP, why would you need anything else?"  That's
> certainly a point of view, and for you it may be fine.  Anything can be
> outsourced.  But "I don't have to do it, therefore nobody does" is a
> questionable position.

Hm, you were telling me I was conflating things earlier.

With RPM in terms of process you can easily rebuild the packages if you 
have the SRPM.  So it's nothing like a BSP blob, rebuild is always 
easily possible package by package, and the packages explain their build 
dependencies so it can be automated.  So "reproducibility" is possible.

My point is that typically on desktop and server distro installs, you do 
not go around rebuilding packages, Gentoo excepted because that's their 
thing.  I've done it to a couple of packages over the years and that's 
it.  The reason for going the distro route is I can rely on and exploit 
the work they already did.  (And in Fedora's case, their quality demands 
are higher than mine so it's an automatic win).

So while reproducibility is possible, it's almost never needed by end 
users.  If they do need it, they're going to have to get some metal that 
allows them to do it, maybe the mass Qemu farm thing or whatever.  But 
not typical users.

>>> There's more things you can do to speed it up if you want to go down that
>>> rabbit hole (which the presentation does), and there's more work being
>>> done in qemu.  (TCG was originally a performance hit but has improved
>>> since, although it varies widely by platform and is its own rabbit hole.
>>> Also switching to gigabit NIC emulation with jumbo frames helped distcc a
>>> lot.)
>>
>> Just saying people don't do distribution rebuilds for their desktop or
>> server boxes unless they're Gentoo believers.
>
> Linux From Scratch does not exist?  My friend Mark (who inherited a "Slackware
> for PowerPC" cross-build system) is in reality unemployed?  Nobody ever
> actually works _on_ Fedora and needs to do a development build?
>
> It's interesting to see someone who can imply these things with a straight
> face on an embedded development list.

You are making straw men here.  Normally, people who are using a distro 
do not do whole distribution rebuilds for their desktop or server boxes, 
it is a stone cold fact.  It must be 99%+ of distro users never 
customized one package let alone dist-rebuild.  They use the binary 
packages coming from the distro, that is the point of having a distro.

Yeah other people have buildroot-style build policies and do other 
things, good luck to them.  I'm suggesting they should be avoided for 
use at this ARM11+ level.

>> So the need to consider
>> this heavy duty stuff only exists at the distro.  In Fedora case, guys
>> at Marvell are running it and have access to lots of fast physical
>> hardware they hooked up to Koji (you touch on this in your PDF).  I
>> don't think it's like they were waiting to hear about Qemu now they're
>> gonna drop that and move to emulation.
>
> You also seem to be unaware that QEMU was only started in 2003 (first commit to
...
> Therefore, they didn't "drop" support, QEMU wasn't ready for prime time back

Please read what I actually wrote above, which has nothing to do with 
Qemu dropping anything.  It seems you would call this kind of thing 
"conflating orthogonal issues".

> By the way:
>
>    http://fedoraproject.org/wiki/Architectures/ARM/HowToQemu
>
> You continue to conflate orthogonal issues.

Again I didn't say it's not possible to use Qemu, it's just slow.  And 
for the reasons of having unified dependencies in your build system, if 
it is possible to build on your execution device that is preferable.

>> They're in the business of
>> making fast ARM chips and they're going to outpace Qemu.
>
> They're not in the same business.  They don't compete.  You don't deploy on
> qemu, and when you go down to Fry's to get a laptop the x86-64 is likely to
> continue to be the main option for the forseeable future.

Developers have to make a decision whether to use the native metal or 
emulation.  Qemu is slow, so much so you suggest having a build farm of 
Qemus to work around that.  Native metal is getting faster.

So I would advise people to use their fast native box if they have one, 
because I think it's going to get faster quicker than Qemu basis over 
the next years.

>>> But in general, Moore's Law says that qemu on current PC hardware is
>>> about the speed of current PC hardware seven years ago.  (And obviously
>>> nobody ever built anything before 2003. :)
>>
>> The speed surely depends on the architecture being emulated.  I did try
>> Qemu on ARM on a nice desktop box here and it seemed unworkable to me.
>
> You couldn't make it work, therefore it can't be made to work?

Oh no: it worked.  It was "unworkable" as a basis for development 
because it was slower than my native hardware.  So I moved to that and 
removed the burden of Qemu and syncing two rootfses from my development 
thinking.

>> I am perfectly happy to trust Fedora to take care of disribution
>> rebuilds for me.
>
> Good for you.
>
> Did it ever occur to you that there are people out there who do the bits you
> don't personally do?  That all these packages don't emerge from zeus's
> forehead on a quarterly basis?  That someone, somewhere, is actually
> interested in how this stuff _does_ work?

Of course.  Do you know about Koji?  It's pretty cool.

http://arm.koji.fedoraproject.org/koji/

That level of infrastructure and investment is done by the distro, not 
the users who can get high quality, immediately usable binaries for 
free.  We shouldn't be advising devs that they need to get into 
reproducing the distro as the price of entry.  (They can get into that 
if they absolutely have to, they can even run their own koji which is 
packaged in Fedora.)

>> This is what I have been calling "buildroot thinking" again.  What do
>> you think you will be rebuilding exactly out of Fedora when you put it
>> on an ARM11+ system?  Typically, it's going to be zero packages, nothing
>> at all.  What you will need to build typically is this:
>
> "All the world's a Cortex A8, and always will be."
>
> Look, go have fun.  I'm sure what you've chosen to do is going to work just
> great for you, and you'll never need to care about anything else.  Personally,
> I no interest in speaking to you ever again about anything, and I have a spam
> filter for situations like this.
>
> I'll even make you happy: "You win!"  There you go.
>
> *plonk*

I see.

Well, unfortunate as it is I now have to address the back of your head 
with your hands clamped over your ears, I would like to thank you for an 
interesting thread.

-Andy

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Jamie Lokier]

Andy Green wrote:
> >No TCP/IP, no TFTP, not even BOOTP (but it's a nice bonus), no command
> >line interpreter (just a GPIO on board to boot into "unbrick me" mode
> >:-), and most strikingly _no_ flash driver for flash chip du jour.
> >
> >To flash it you send a kernel to boot from RAM which is capable of
> >flashing it.
> 
> Sorry I missed where this kernel appears from and the bootloader that 
> spawned it, since both could get trashed.

The kernel appeared from the network/serial/USB, as described.  It's
not on the device.  The point is you don't need a "spare" kernel for
unbrickability (and kernels are quite large); it's enough to be able
to send one in unbricking mode.

> That is actually a conundrum 
> on a lot of systems and some of the solutions (write-once backup 
> bootloader) in the long run lead to other issues.

I agree especially about the write-once backup bootloader.

> True SD Boot does truly deliver unbrickability if you are willing to 
> swap out or reformat the SD card.

With the disadvantage that you need the board space, connector cost
etc. for an SD card.  It's fine if you already have that.

I'd put it into a similar category to network/serial/USB: Using
whatever your board provides for bare-bones communication, bootstrap a
kernel for more advanced stuff involving fancier drivers.

> >>>http://wiki.openmoko.org/wiki/Qi
> >>
> >>Looking at the screen shot there, you've got code to parse ext2 
> >>filesystems.
> >>What is your definition of "minimal"?
> >
> >Ew, ext2 doesn't even satisfy powerfail-during-kernel-upgrade safety.
> 
> It's just misleading (but accurate).  ext2 is the "lowest common 
> denominator" read-only parsing that actually supports ext3 and ext4 if 
> you are careful about the formatting options.  So the actual filesystem 
> is ext3 or ext4 typically (ext3 in GTA02 case), it's not that the 
> bootloader is mandating specifically ext2.

If it reads an ext3/4 filesystem using an ext2 read-only parser, then
it will not be powerfail-safe.  Partially written files and directory
updates will look like corruption on boot.

It's essential that it parses the journal as well.  It does not have
to commit the journal (which would need device writing ability), but
in that case it must allow journal blocks to supercede blocks from
elsewhere in the filesystem.

> >It's possible to boot without parsing filesystems, but that is one
> >rather nice feature, and with the right filesystems it can make system
> >updates powerfail-safe.
> 
> Bootloader is tricky, but actually on this iMX31 device Fedora is used, 
> yum update keeps the last 3 kernels around and our kernel package 
> follows that.  So it's possible to have backup kernels automatically 
> integrated into the bootloader and packaging system.

It's useless to have 3 kernels around if the directory containing them
looks corrupted due to not parsing the journal. :-)  Then you won't see
any of the kernels at boot time.

-- Jamie

Re: [Celinux-dev] CELF Project Proposal- Refactoring Qi, lightweight bootloader

[Andy Green]

On 12/29/09 13:13, Somebody in the thread at some point said:

Hi -

>> Sorry I missed where this kernel appears from and the bootloader that
>> spawned it, since both could get trashed.
>
> The kernel appeared from the network/serial/USB, as described.  It's
> not on the device.  The point is you don't need a "spare" kernel for
> unbrickability (and kernels are quite large); it's enough to be able
> to send one in unbricking mode.

The bootloader that will load this kernel can't be updated or trashed 
itself?

> With the disadvantage that you need the board space, connector cost
> etc. for an SD card.  It's fine if you already have that.

Sure.  Although the connector and pullups needed are really cheap.  You 
also need an SD interface on your CPU, but most of this kind of 
processor have multiple SD interfaces already.

> I'd put it into a similar category to network/serial/USB: Using
> whatever your board provides for bare-bones communication, bootstrap a
> kernel for more advanced stuff involving fancier drivers.

Well my suggestion is SD is in a different category.  Unlike network or 
serial or USB it works for boot without a host to talk to.

>>> Ew, ext2 doesn't even satisfy powerfail-during-kernel-upgrade safety.
>>
>> It's just misleading (but accurate).  ext2 is the "lowest common
>> denominator" read-only parsing that actually supports ext3 and ext4 if
>> you are careful about the formatting options.  So the actual filesystem
>> is ext3 or ext4 typically (ext3 in GTA02 case), it's not that the
>> bootloader is mandating specifically ext2.
>
> If it reads an ext3/4 filesystem using an ext2 read-only parser, then
> it will not be powerfail-safe.  Partially written files and directory
> updates will look like corruption on boot.

Yeah I think that is true.

> It's essential that it parses the journal as well.  It does not have
> to commit the journal (which would need device writing ability), but
> in that case it must allow journal blocks to supercede blocks from
> elsewhere in the filesystem.

Personally I don't use ext2 for the containing the kernels, but VFAT on 
SD.  Qi supports ext2 because that's what was going on GTA02.

>>> It's possible to boot without parsing filesystems, but that is one
>>> rather nice feature, and with the right filesystems it can make system
>>> updates powerfail-safe.
>>
>> Bootloader is tricky, but actually on this iMX31 device Fedora is used,
>> yum update keeps the last 3 kernels around and our kernel package
>> follows that.  So it's possible to have backup kernels automatically
>> integrated into the bootloader and packaging system.
>
> It's useless to have 3 kernels around if the directory containing them
> looks corrupted due to not parsing the journal. :-)  Then you won't see
> any of the kernels at boot time.

You're assuming that those kernels are in ext2/3/4, as mentioned above 
actually I have been using VFAT.  But actually with Qi there's another 
solution even for ext2 "/boot" partition that makes it safe.

Qi has a fallthru scheme for acquiring the kernels, you give a list of 
partitions and filenames and it uses the first one that gets it a good 
kernel image.

Therefore you can have a backup kernel and rootfs in a second partition 
which is not normally updated.  If the normal kernel is broken (plus or 
minus the fs parsing code being good enough it won't infinitely loop or 
whatever when given junk) it will select the backup one from the clean 
partition.

-Andy