Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with kernel packages

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with kernel packages

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

CC: to the OE list

Paul Sokolovsky schreef:
> Hello angstrom-distro-devel,
> 
>   Thanks for providing pre-holiday 20061219 snapshots for testing.
> They don't provide separate zImages, but well, that's ok - they are in
> the images, after all.
> 
>   Well, not for hx4700, for example. Because:
> 
> ======== linux-handhelds-2.6.inc =====
> FILES_kernel-image_hx4700 = ""
> ALLOW_EMPTY_hx4700 = "1"
> FILES_kernel-image_htcuniversal = ""
> ALLOW_EMPTY_htcuniversal = "1"
> ======================================
> 
>   Oops. Sources of such a special treatment of hx4700 and htcuniversal
> would be unclear to outsider, but people who deal with (mis)features
> of "ipaqs" know, that hx4700 has one (sic!, even if the most popular)
> bootloader which doesn't use zImage in the image, but instead requires
> adhoc means to install kernel, ditto for Universal, though bootloader
> is another.
> 
>   So, what is tried here is to "save space". But there're few issues:

Those lines should go, but not before we have a way to exclude 'kernel-image' from the
rootfs *and* kernel-module-foo doesn't depend kernel-image anymore.

regards,

Koen

> 1. See concern in the beginning of the message.
> 2. It's pretty weird to do such hack on a package level. Kernel
> package supposed to contain kernel, period. If there's some pressure
> to not have kernel in image, that's from where it should be removed.
> 3. Keywords in the description why this being done above are "a
> bootloader" and "currently". Sooner or later all this adhockery will
> be gone, and existing generic, machine-independent solutions, like
> LAB, will be adopted, which boot from the filesystem, and thus adhere
> to idea of complete system maintenance via packaged install
> (including, for kernel). For the multimachine distribution like
> Angstrom, supporting hacks like above is already pain in the neck, as
> the case above shows.
> 
> 
>   So, my proposal is to remove the lines quoted above. People who
> under pressure to have an extra MB and don't shudder to steal it from
> the kernel, can do that manually ;-).
> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFFlAPyMkyGM64RGpERAm9wAJ9zlE192Y6AZ6wT1jQbq/P3Cg6S5wCghO1f
KIQjN4BTSU+eRiWa2Aga/qM=
=EOjP
-----END PGP SIGNATURE-----

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with kernel packages

[Richard Purdie]

On Thu, 2006-12-28 at 18:50 +0100, Koen Kooi wrote:
> Paul Sokolovsky schreef:
> > Hello angstrom-distro-devel,
> > 
> >   Thanks for providing pre-holiday 20061219 snapshots for testing.
> > They don't provide separate zImages, but well, that's ok - they are in
> > the images, after all.
> > 
> >   Well, not for hx4700, for example. Because:
> > 
> > ======== linux-handhelds-2.6.inc =====
> > FILES_kernel-image_hx4700 = ""
> > ALLOW_EMPTY_hx4700 = "1"
> > FILES_kernel-image_htcuniversal = ""
> > ALLOW_EMPTY_htcuniversal = "1"
> > ======================================
> > 
> >   Oops. Sources of such a special treatment of hx4700 and htcuniversal
> > would be unclear to outsider, but people who deal with (mis)features
> > of "ipaqs" know, that hx4700 has one (sic!, even if the most popular)
> > bootloader which doesn't use zImage in the image, but instead requires
> > adhoc means to install kernel, ditto for Universal, though bootloader
> > is another.
> > 
> >   So, what is tried here is to "save space". But there're few issues:
> 
> Those lines should go, but not before we have a way to exclude 'kernel-image' from the
> rootfs *and* kernel-module-foo doesn't depend kernel-image anymore.

The accepted approach is to make the kernel-image file empty which
solves both those problems. This is the same as the Zaurus which also
requires an external zImage file (and this code is handled in
linux-rp.inc in the same way as the above but for all machines).

Arguably, the machine files should perhaps set this...

The only other solution probably involves a nasty anonymous function
acting on a setting of EXTERNAL_KERNEL_IMAGE = "1" in kernel.bbclass.

Richard

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with kernel packages

[Paul Sokolovsky]

Hello Richard,

Thursday, December 28, 2006, 10:32:26 PM, you wrote:

> On Thu, 2006-12-28 at 18:50 +0100, Koen Kooi wrote:
>> Paul Sokolovsky schreef:
>> > Hello angstrom-distro-devel,
>> > 
>> >   Thanks for providing pre-holiday 20061219 snapshots for testing.
>> > They don't provide separate zImages, but well, that's ok - they are in
>> > the images, after all.
>> > 
>> >   Well, not for hx4700, for example. Because:
>> > 
>> > ======== linux-handhelds-2.6.inc =====
>> > FILES_kernel-image_hx4700 = ""
>> > ALLOW_EMPTY_hx4700 = "1"
>> > FILES_kernel-image_htcuniversal = ""
>> > ALLOW_EMPTY_htcuniversal = "1"
>> > ======================================
>> > 
>> >   Oops. Sources of such a special treatment of hx4700 and htcuniversal
>> > would be unclear to outsider, but people who deal with (mis)features
>> > of "ipaqs" know, that hx4700 has one (sic!, even if the most popular)
>> > bootloader which doesn't use zImage in the image, but instead requires
>> > adhoc means to install kernel, ditto for Universal, though bootloader
>> > is another.
>> > 
>> >   So, what is tried here is to "save space". But there're few issues:
>> 
>> Those lines should go, but not before we have a way to exclude 'kernel-image' from the
>> rootfs *and* kernel-module-foo doesn't depend kernel-image anymore.

> The accepted approach is to make the kernel-image file empty which
> solves both those problems.

  Richard, is this response to Koen or to me? If to Koen, that's ok,
but my original mail (
http://lists.linuxtogo.org/pipermail/angstrom-distro-devel/2006-December/000058.html )
started with observation that there're few things wrong withe exactly
this approach.

> This is the same as the Zaurus which also
> requires an external zImage file (and this code is handled in
> linux-rp.inc in the same way as the above but for all machines).

  Well, looking into linux-rp.inc, it becomes clear what is actual
problem:

---------
# Compensate for sucky bootloader on all Sharp Zaurus models
#
FILES_kernel-image = ""
ALLOW_EMPTY = "1"
---------

  And PocketPC have exactly the same problem of having sucky
bootloaders. But I wouldn't like to accept that as the eternal state of
affairs, but tackle it to get some generic and reusable solution. The
path I see here is:

1) Stop issuing broken kernel-image packages which don't actually
have kernel image; instead, offer other means to achieve the
result of not having zImage in rootfs, for the cases when it is
needed.

2) Argue to Angstrom maintainers to stop killing zImage from rootfs at
all - consistently for all machines.

3) Move towards adopting 3-level bootloading with a tool like LAB (Linux as
Bootloader) with the following stages:
  a) 1st level bootloader bootstraps 2nd level bootloader out of NAND
  and such - in most cases supplied by device vendor.
  b) 2nd level bootloader boots LAB out of flash (what we have now is
  that 2nd level bootloader boots actual production kernel). 2nd level
  bootloader is vendor's stuff, or small bootloader like u-boot or
  HH.org bootldr.
  c) LAB, 3rd stage bootloader, boots actual kernel out of real
  rootfs, like it is usually done on standard desktop systems.


  Package management of the kernel is fully ensured. LAB, being
essentially a full Linux kernel itself, offers amazing capabilities
for diagnostics and recover in the case of problems - it can easily
boot from secondary storage, NFS, whatever.

  I already heard ideas in that direction from Koen, and well, knowing
that he worked on LAb himself, I assume he would have close picture in
mind to what's written above anyway.

  So, I'd like to start initial small steps in that direction. I don't
pledge for them to apply to anything else but PocketPC (hh.org)
devices, and just hope that Angstrom maintainers will support further
steps outlined above.

  Of course, it's clear that it's a case of common "choose 2 of 3"
compromise, but I really think that general-purpose distro like
Angstrom has good reasons to value "consist and general support across
all (many) devices" over "could hack a way to save a megabyte".

> Arguably, the machine files should perhaps set this...

  Well, in my initial days I was told many times what should be
within machine's discretion, and what's not, and I think I got idea
;-). So, I'd argue that distro config should set it for machines it
supports. This is based on the ideas outlined above - suckiness of
a bootloader is not unalienable property of machine, you can easily add
a 1Mb level to abstract away that suckiness. And in this regard, it is a
distro policy, if it wants to abstract away at some expense, or deal
with dirty details striving to save some space.

> The only other solution probably involves a nasty anonymous function
> acting on a setting of EXTERNAL_KERNEL_IMAGE = "1" in kernel.bbclass.

  The solution I propose,

ROOTFS_POSTPROCESS_COMMAND += "remove_zimages; "

is not nastier than what we already have and use.

> Richard



-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with kernel packages

[Richard Purdie]

On Fri, 2006-12-29 at 04:05 +0200, Paul Sokolovsky wrote:
>   Richard, is this response to Koen or to me? If to Koen, that's ok,
> but my original mail (
> http://lists.linuxtogo.org/pipermail/angstrom-distro-devel/2006-December/000058.html )
> started with observation that there're few things wrong withe exactly
> this approach.

Its to both of you. Your original mail seems to be to basically say you
don't like the idea as its "unclear" and "weird". These are more
documentation issues.

>   And PocketPC have exactly the same problem of having sucky
> bootloaders. But I wouldn't like to accept that as the eternal state of
> affairs, but tackle it to get some generic and reusable solution. The
> path I see here is:
> 
> 1) Stop issuing broken kernel-image packages which don't actually
> have kernel image; instead, offer other means to achieve the
> result of not having zImage in rootfs, for the cases when it is
> needed.

Ok, so you remove the kernel-image package. What do modules then focus
on to make sure they're all in sync? The kernel-image file serves two
purposes - the first to add a kernel image file if needed, the second to
ensure the modules all have a common versioned dependency package.
Please don't suggest we need another empty package for this.

> 2) Argue to Angstrom maintainers to stop killing zImage from rootfs at
> all - consistently for all machines.

So we start to waste 1.2MB disk space on at least 50% of the machines
Angstrom supports? Thats a lot of space on a machine with a 32MB root
partition.

Also, it creates a user support problem. "I upgraded the zImage file on
my rootfs but my device still uses my old kernel?".

> 3) Move towards adopting 3-level bootloading with a tool like LAB (Linux as
> Bootloader) with the following stages:
>   a) 1st level bootloader bootstraps 2nd level bootloader out of NAND
>   and such - in most cases supplied by device vendor.
>   b) 2nd level bootloader boots LAB out of flash (what we have now is
>   that 2nd level bootloader boots actual production kernel). 2nd level
>   bootloader is vendor's stuff, or small bootloader like u-boot or
>   HH.org bootldr.
>   c) LAB, 3rd stage bootloader, boots actual kernel out of real
>   rootfs, like it is usually done on standard desktop systems.

Its a nice idea but its complex. Nobody has ever looked at LAB on the
Zaurus for example and there are nasty issues to do with the proprietary
flash translation layer which means we can't easily read from flash
partitions under boot loader control (e.g. where the kernel is flashed).

Also, on the Zaurus it means wasting a couple of megabytes of flash for
the extra copies of kernels you'll need.

> > Arguably, the machine files should perhaps set this...
> 
>   Well, in my initial days I was told many times what should be
> within machine's discretion, and what's not, and I think I got idea
> ;-). So, I'd argue that distro config should set it for machines it
> supports. This is based on the ideas outlined above - suckiness of
> a bootloader is not unalienable property of machine, you can easily add
> a 1Mb level to abstract away that suckiness. And in this regard, it is a
> distro policy, if it wants to abstract away at some expense, or deal
> with dirty details striving to save some space.

Machine config files set information that is specific to the machine.
Whether of not a given machine has a sucky bootloader is a property of
the machine. Whether the DISTRO wants to allow zImage removal from the
images is a distro policy. It can't enforce that policy without knowing
which machines need to do what though so you need some machine specific
information.

> > The only other solution probably involves a nasty anonymous function
> > acting on a setting of EXTERNAL_KERNEL_IMAGE = "1" in kernel.bbclass.
> 
>   The solution I propose,
> 
> ROOTFS_POSTPROCESS_COMMAND += "remove_zimages; "
> 
> is not nastier than what we already have and use.

Yes, it is. Try upgrading kernel-image and you have a user support
nightmare.

Basically, the Zauruii at least are going to be stuck with the kernel
flashed outside the rootfs. Even if we write a better reflashing util,
we'll probably stick to using that flash space for the kernel. The
Zaurus machines are therefore going to need support for external kernels
from OE. If things are unclear, we need to document them better.

Richard

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with [linux-handhelds-2.6] kernel packages

[Paul Sokolovsky]

Hello Richard, Marcin,

      [Sorry for delay with response, due to holiday.]


Friday, December 29, 2006, 1:34:04 PM, you wrote:

> On Fri, 2006-12-29 at 04:05 +0200, Paul Sokolovsky wrote:
>>   Richard, is this response to Koen or to me? If to Koen, that's ok,
>> but my original mail (
>> http://lists.linuxtogo.org/pipermail/angstrom-distro-devel/2006-December/000058.html )
>> started with observation that there're few things wrong withe exactly
>> this approach.

> Its to both of you. Your original mail seems to be to basically say you
> don't like the idea as its "unclear" and "weird". These are more
> documentation issues.

  No. My original mail said that it lead to broken snapshots released
for hx4700, so I RFCed its removal for it, and for another PocketPC
machine, htcuniversal, because I'm pretty sure it can live without it.
As additional argumentation, I reviewed how it functions and hopefully
showed that it is hack at best anyway. This has nothing to do with me
thinking it's '"unclear" and "weird"'.

  My still remains the same - clean up PocketPC machines (and only
them!) from this feature/hack. I don't pledge for anything else but
removing 4 lines mentioned in the original mail, from
linux-handhelds-2.6 recipe. This whole thread continued with Koen's
request for an alternative solution. So yes, I tried to propose one,
but well, I wouldn't think it would be the fully general solution, and
responses clearly show it's not suitable for Zaurus, and few other
machines. But that doesn't mean that there's no need for alternative
solution, even if it partial/hack/workaround, because
FILES_kernel-image = "" (essentially having Zaurus roots) is the
workaround just the same. So, until we have fully generic solution, it
would be nice to at least have alternatives even with partial
coverage.

>>   And PocketPC have exactly the same problem of having sucky
>> bootloaders. But I wouldn't like to accept that as the eternal state of
>> affairs, but tackle it to get some generic and reusable solution. The
>> path I see here is:
>> 
>> 1) Stop issuing broken kernel-image packages which don't actually
>> have kernel image; instead, offer other means to achieve the
>> result of not having zImage in rootfs, for the cases when it is
>> needed.

> Ok, so you remove the kernel-image package.

  Well, there must be a misunderstanding - I don't suggest removing
kernel-image package. I suggest not to hack it into such shape that
it, instead of containing zImage, is empty.

>  What do modules then focus
> on to make sure they're all in sync? The kernel-image file serves two
> purposes - the first to add a kernel image file if needed, the second to
> ensure the modules all have a common versioned dependency package.
> Please don't suggest we need another empty package for this.

  Well, if there're two so different purposes, and that causes
confusion and tangled handling (no matter how you document it), then
it may be just a good idea to exactly separate them.

>> 2) Argue to Angstrom maintainers to stop killing zImage from rootfs at
>> all - consistently for all machines.

> So we start to waste 1.2MB disk space on at least 50% of the machines
> Angstrom supports? Thats a lot of space on a machine with a 32MB root
> partition.

  It's not going to be waste, as it is supposed to be used by LAB,
eventually. Also, while 1.2MB seems like a figure, here's the paradox -
the difference of what you can fit into 32MB and 30.8MB is pretty
small. Or to be more exact, the probability that a user will be
satisfied with what you provide in 30.8MB (or alternatively,
percentage of users which will be satisfied with what is provided in
30.8MB) would be almost equal to what you provide in 32MB - after all,
that's indeed not big enough, and what is put in that volume is a
matter of compromise and convention, and subject for users' tweaking
anyway,

  In this regard, why not adopt another convention - instead of
fitting yet another cool app, why not provide solid
upgrade/bootloading/troubleshooting/diagnostics solution by default,
and teach users that a kernel upgrade is piece of cake, you can't
brick a device by doing that, and generally, Linux on PDA foundation
is as solid as on desktop?


  Note that this is all relates to the future step 2 of the masterplan,
it's nothing but a thing to ponder about for now...

> Also, it creates a user support problem. "I upgraded the zImage file on
> my rootfs but my device still uses my old kernel?".

  Well, that's the crucial point. Could you gentlemen describe how you
handle kernel upgrade on Zaurus now? I imagine it's shipping a separate
zImage file together with some upgrade script, or rely on native
bootloader to install it somehow.

  With my proposal to use full kernel-image packages it would
be instead:

1. One does "ipkg install kernel-image"
2. One prays and with shaking hands runs
"adhoc_kernel_update_script.sh /boot/zImage"
3. Optionally, rm "/boot/*"

  Note that we could automate it if we bother (so it's probably better
to make user responsible for possible bricking, and make one do the
above manually).

  As for "user support problem", how that differs from what you have
now? User installs empty kernel-image package and also doesn't get
upgraded kernel. You likely solved this for Zaurus by telling users
that installing kernel-image doesn't really upgrade their kernel.
Well, "Zaurus" appears to be the keyword here.

>> 3) Move towards adopting 3-level bootloading with a tool like LAB (Linux as
>> Bootloader) with the following stages:
>>   a) 1st level bootloader bootstraps 2nd level bootloader out of NAND
>>   and such - in most cases supplied by device vendor.
>>   b) 2nd level bootloader boots LAB out of flash (what we have now is
>>   that 2nd level bootloader boots actual production kernel). 2nd level
>>   bootloader is vendor's stuff, or small bootloader like u-boot or
>>   HH.org bootldr.
>>   c) LAB, 3rd stage bootloader, boots actual kernel out of real
>>   rootfs, like it is usually done on standard desktop systems.

> Its a nice idea but its complex. Nobody has ever looked at LAB on the
> Zaurus for example and there are nasty issues to do with the proprietary
> flash translation layer which means we can't easily read from flash
> partitions under boot loader control (e.g. where the kernel is flashed).

> Also, on the Zaurus it means wasting a couple of megabytes of flash for
> the extra copies of kernels you'll need.

  Well, just in case, let me repeat again - I don't pledge for
converting every machine to this scheme, especially that it is still
more of tentative work in progress. I wouldn't touch Zaurus with any
such ideas at all. All I probe, and ask, for, is allowing to make
such experiment on PocketPC devices, and for Angstrom maintainers,
to be favorable towards this.

  As for LAB, that's exactly its strength - if you have support for
some device (like funky flash) in kernel, LAB will be able to use it,
as it's just kind of module for the kernel itself. The price of this
is that LAB is indeed too big as for bootloader - ~1Mb instead of
usual 256Mb/128Mb. So well, if you have good bootloading scheme for
Zauruses, you don't need it. But for PocketPCs, which come in dozens,
if not hundreds, looking for adhoc optimized solution simply doesn't
scale, the current situation is a good example of that - after so many
years of porting, there're only 2 devices which can be called well
supported. So, PocketPCs need some general solution do leave dark
ages. Optimization can be left to happy folks who have time for that.


>> > Arguably, the machine files should perhaps set this...
>> 
>>   Well, in my initial days I was told many times what should be
>> within machine's discretion, and what's not, and I think I got idea
>> ;-). So, I'd argue that distro config should set it for machines it
>> supports. This is based on the ideas outlined above - suckiness of
>> a bootloader is not unalienable property of machine, you can easily add
>> a 1Mb level to abstract away that suckiness. And in this regard, it is a
>> distro policy, if it wants to abstract away at some expense, or deal
>> with dirty details striving to save some space.

> Machine config files set information that is specific to the machine.
> Whether of not a given machine has a sucky bootloader is a property of
> the machine.

  Point of view. It's all just point of view. Many-many people would
consider you crazy for wanting to install something else, handmade,
instead of vendor-carved-in stuff. But you smile and do just that. And
yet you tell that /home partition and bootloader are carved in stone.
But someone else will flash them away just as easily.

  That of course doesn't mean I disagree with the current
state-of-affairs of mostly adhoc bootloaders to be used, and need to
account for that somehow. It's just one day it may become a limitation
to have it in machine config.

> Whether the DISTRO wants to allow zImage removal from the
> images is a distro policy. It can't enforce that policy without knowing
> which machines need to do what though so you need some machine specific
> information.

>> > The only other solution probably involves a nasty anonymous function
>> > acting on a setting of EXTERNAL_KERNEL_IMAGE = "1" in kernel.bbclass.
>> 
>>   The solution I propose,
>> 
>> ROOTFS_POSTPROCESS_COMMAND += "remove_zimages; "
>> 
>> is not nastier than what we already have and use.

> Yes, it is. Try upgrading kernel-image and you have a user support
> nightmare.

  As argued, on the order of magnitude you'd have been had already.
But err..., I don't ask Zaurus to switch. Just to let PocketPCs try it,
and preferably, consistently. We already have user support nightmare
just because there're few adhoc kernel install/upgrade methods for each of
the few well supported models, and lack of any booting for the rest.

> Basically, the Zauruii at least are going to be stuck with the kernel
> flashed outside the rootfs. Even if we write a better reflashing util,
> we'll probably stick to using that flash space for the kernel. The
> Zaurus machines are therefore going to need support for external kernels
> from OE. If things are unclear, we need to document them better.

  Ok, so it's clear that Zaurus machines need to keep using existing
methods, and those methods should not be touched. On the other hand, I
hope I was able to argue that other machines might have other
requirements. In this regard, I'll still need to argue to machine
mentors of (PocketPC!) machines in question to drop
FILES_kernel-image_MACH = "".

  But anyway, I think that ROOTFS_POSTPROCESS_COMMAND +=  "remove_zimages; "
thing is still useful. It doesn't mean that machines should switch to
it, but it offers nice, potentially machine-independent, way to remove
zImage file from the image. So, besides alternative way to produce
real rootfs image without zImage which will be still possible to
upgrade with ipkg (even if semi-manually) - and that's considered to
be a feature, not breakage, it can be useful for other things, like
initrd creation.


> Richard




-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with [linux-handhelds-2.6] kernel packages

[Richard Purdie]

On Thu, 2007-01-04 at 21:03 +0200, Paul Sokolovsky wrote:
>   No. My original mail said that it lead to broken snapshots released
> for hx4700, so I RFCed its removal for it, and for another PocketPC
> machine, htcuniversal, because I'm pretty sure it can live without it.

Why were the snapshots broken? Was the problem not the total lack of a
zImage file. The fact that doesn't automatically deploy atm is actually
a bitbake trunk issue as it doesn't see any dependency to trigger that.
If those devices don't boot from a kernel within the image and need a
separate one, this would appear to be an unrelated problem as most users
are not going to extract the zImage from the image file.

> > Machine config files set information that is specific to the machine.
> > Whether of not a given machine has a sucky bootloader is a property of
> > the machine.
> 
>   Point of view. It's all just point of view. Many-many people would
> consider you crazy for wanting to install something else, handmade,
> instead of vendor-carved-in stuff. But you smile and do just that. And
> yet you tell that /home partition and bootloader are carved in stone.
> But someone else will flash them away just as easily.

I was thinking of the Zaurus case where the separate zImage is a
property of the machine at this time as there are no practical
alternatives to having a kernel outside the rootfs. The hh.org case is
admittedly less clear cut. There are documented standard methods of
using these handhelds with Linux and its machine specific whether a
separate kernel is required or not. Having this documented on a per
machine basis therefore makes more sense than hardcoding it into the
linux-*.inc files. The example would be when I try a Zaurus kernel from
hh.org. If its set in the machine files, the hh.org kernel would stand a
better chance of working.

At the end of the day its the machine maintainer's choice. You put RFC
in the subject and you got comments. You don't have to agree with them!

>   As argued, on the order of magnitude you'd have been had already.
> But err..., I don't ask Zaurus to switch. Just to let PocketPCs try it,
> and preferably, consistently. We already have user support nightmare
> just because there're few adhoc kernel install/upgrade methods for each of
> the few well supported models, and lack of any booting for the rest.

I said this reply was to both you and Koen. Koen called for one unified
approach which implies the Zaurus would have to follow your proposal. I
took issue with that for the reasons I outlined. 

>   Ok, so it's clear that Zaurus machines need to keep using existing
> methods, and those methods should not be touched. On the other hand, I
> hope I was able to argue that other machines might have other
> requirements. In this regard, I'll still need to argue to machine
> mentors of (PocketPC!) machines in question to drop
> FILES_kernel-image_MACH = "".

You've shown your position, I've not seen other PocketPC machine
maintainers comment. You won't change my view regarding the hx2000
either though ;-).

>   But anyway, I think that ROOTFS_POSTPROCESS_COMMAND +=  "remove_zimages; "
> thing is still useful. It doesn't mean that machines should switch to
> it, but it offers nice, potentially machine-independent, way to remove
> zImage file from the image. So, besides alternative way to produce
> real rootfs image without zImage which will be still possible to
> upgrade with ipkg (even if semi-manually) - and that's considered to
> be a feature, not breakage, it can be useful for other things, like
> initrd creation.

In one breath you complain about the zimage packages being a "hack at
best". You then propose hacking the generated images ;-).

Richard

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with [linux-handhelds-2.6] kernel packages

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
> On Thu, 2007-01-04 at 21:03 +0200, Paul Sokolovsky wrote:
>>   No. My original mail said that it lead to broken snapshots released
>> for hx4700, so I RFCed its removal for it, and for another PocketPC
>> machine, htcuniversal, because I'm pretty sure it can live without it.
> 
> Why were the snapshots broken? Was the problem not the total lack of a
> zImage file. The fact that doesn't automatically deploy atm is actually
> a bitbake trunk issue as it doesn't see any dependency to trigger that.

Worked around in the buildscripts by doing 'bitbake virtual/kernel' before the image.

regards,

Koen

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFFnXAAMkyGM64RGpERAlmrAJ9EnziALoKwLdjf5lELigVz7QaGfACdF4mP
JacO3VQYsKJjMQPxnvvRGfs=
=sQTs
-----END PGP SIGNATURE-----

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with [linux-handhelds-2.6] kernel packages

[Richard Purdie]

On Thu, 2007-01-04 at 22:22 +0100, Koen Kooi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Richard Purdie schreef:
> > On Thu, 2007-01-04 at 21:03 +0200, Paul Sokolovsky wrote:
> >>   No. My original mail said that it lead to broken snapshots released
> >> for hx4700, so I RFCed its removal for it, and for another PocketPC
> >> machine, htcuniversal, because I'm pretty sure it can live without it.
> > 
> > Why were the snapshots broken? Was the problem not the total lack of a
> > zImage file. The fact that doesn't automatically deploy atm is actually
> > a bitbake trunk issue as it doesn't see any dependency to trigger that.
> 
> Worked around in the buildscripts by doing 'bitbake virtual/kernel' before the image.

the proper fix might be something like:

do_rootfs[recrdeptas] = "do_package_write do_deploy" in image.bbclass

but that could also blow up. I'll think a bit more about it :)

Cheers,

Richard

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with [linux-handhelds-2.6] kernel packages

[Paul Sokolovsky]

Hello Richard,

Thursday, January 4, 2007, 10:55:18 PM, you wrote:

> On Thu, 2007-01-04 at 21:03 +0200, Paul Sokolovsky wrote:
>>   No. My original mail said that it lead to broken snapshots released
>> for hx4700, so I RFCed its removal for it, and for another PocketPC
>> machine, htcuniversal, because I'm pretty sure it can live without it.

> Why were the snapshots broken? Was the problem not the total lack of a
> zImage file. The fact that doesn't automatically deploy atm is actually
> a bitbake trunk issue as it doesn't see any dependency to trigger that.

  It was quite simple: while usually snapshots include standalone
zImage, 1219 lacked it. It's not issue for the most devices, because
zImage can be taken from rootfs, but hx4700 for example lacks it,
rendering snapshot for it useless.

  Obviously, it's small release mistake, but mistakes like this will
come up again and again.

> If those devices don't boot from a kernel within the image and need a
> separate one, this would appear to be an unrelated problem as most users
> are not going to extract the zImage from the image file.

  It's a matter of habit. I personally find it quite natural to have
kernel in /boot, because that's what I have with all other Linuxes I
run around.


>> > Machine config files set information that is specific to the machine.
>> > Whether of not a given machine has a sucky bootloader is a property of
>> > the machine.
>> 
>>   Point of view. It's all just point of view. Many-many people would
>> consider you crazy for wanting to install something else, handmade,
>> instead of vendor-carved-in stuff. But you smile and do just that. And
>> yet you tell that /home partition and bootloader are carved in stone.
>> But someone else will flash them away just as easily.

> I was thinking of the Zaurus case where the separate zImage is a
> property of the machine at this time as there are no practical
> alternatives to having a kernel outside the rootfs. The hh.org case is
> admittedly less clear cut. There are documented standard methods of
> using these handhelds with Linux and its machine specific whether a
> separate kernel is required or not. Having this documented on a per
> machine basis therefore makes more sense than hardcoding it into the
> linux-*.inc files. The example would be when I try a Zaurus kernel from
> hh.org. If its set in the machine files, the hh.org kernel would stand a
> better chance of working.

  Yes, machine config is much better than kernel recipes. And as long
as machine choices can be overridden by distro's, that's just perfect
solution too.

> At the end of the day its the machine maintainer's choice. You put RFC
> in the subject and you got comments. You don't have to agree with them!

  Well, I didn't expect that to be easy, that's why I decided to raise
this topic early ;-)

>>   As argued, on the order of magnitude you'd have been had already.
>> But err..., I don't ask Zaurus to switch. Just to let PocketPCs try it,
>> and preferably, consistently. We already have user support nightmare
>> just because there're few adhoc kernel install/upgrade methods for each of
>> the few well supported models, and lack of any booting for the rest.

> I said this reply was to both you and Koen. Koen called for one unified
> approach which implies the Zaurus would have to follow your proposal. I
> took issue with that for the reasons I outlined. 

>>   Ok, so it's clear that Zaurus machines need to keep using existing
>> methods, and those methods should not be touched. On the other hand, I
>> hope I was able to argue that other machines might have other
>> requirements. In this regard, I'll still need to argue to machine
>> mentors of (PocketPC!) machines in question to drop
>> FILES_kernel-image_MACH = "".

> You've shown your position, I've not seen other PocketPC machine
> maintainers comment. You won't change my view regarding the hx2000
> either though ;-).

>>   But anyway, I think that ROOTFS_POSTPROCESS_COMMAND +=  "remove_zimages; "
>> thing is still useful. It doesn't mean that machines should switch to
>> it, but it offers nice, potentially machine-independent, way to remove
>> zImage file from the image. So, besides alternative way to produce
>> real rootfs image without zImage which will be still possible to
>> upgrade with ipkg (even if semi-manually) - and that's considered to
>> be a feature, not breakage, it can be useful for other things, like
>> initrd creation.

> In one breath you complain about the zimage packages being a "hack at
> best". You then propose hacking the generated images ;-).

  Because packages and images are two different things. Packages can
be used for different purposes - at lease for initial rootfs creation
and for following upgrades, whereas image's purpose is more specific
and constrained. So as usual, the question is not to do or not to do,
but how (in particular, in what place). I'll raise question of initrd
creation later in separate topic.


> Richard





-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with [linux-handhelds-2.6] kernel packages

[Marcin Juszkiewicz]

Dnia czwartek, 4 stycznia 2007 20:03, Paul Sokolovsky napisał:

> >> 2) Argue to Angstrom maintainers to stop killing zImage from rootfs
> >> at all - consistently for all machines.
> >
> > So we start to waste 1.2MB disk space on at least 50% of the machines
> > Angstrom supports? Thats a lot of space on a machine with a 32MB root
> > partition.
>
>   It's not going to be waste, as it is supposed to be used by LAB,
> eventually. Also, while 1.2MB seems like a figure, here's the paradox -
> the difference of what you can fit into 32MB and 30.8MB is pretty
> small. 

I can see difference in fitting in 14.4MB which collie offers (or 21-23M 
of poodle, tosa, c700, c750). Keeping not needed files in rootfs hurts. 
We currently have problems with generating rootfs small enough.

> In this regard, why not adopt another convention - instead of
> fitting yet another cool app, why not provide solid
> upgrade/bootloading/troubleshooting/diagnostics solution by default,
> and teach users that a kernel upgrade is piece of cake, you can't
> brick a device by doing that, and generally, Linux on PDA foundation
> is as solid as on desktop?

pdaX people tried to switch to u-boot on Zaurus line. Look into OESF 
forums to check how many users failed to follow 'simple' 2 step reflash.

> > Also, it creates a user support problem. "I upgraded the zImage file
> > on my rootfs but my device still uses my old kernel?".
>
>   Well, that's the crucial point. Could you gentlemen describe how you
> handle kernel upgrade on Zaurus now? I imagine it's shipping a separate
> zImage file together with some upgrade script, or rely on native
> bootloader to install it somehow.

On PXA models you flash two files: kernel and rootfs. Any of them can be 
flashed without second one. There is a 'encrypted' script built by 
zaurus-updater recipe which do it under rescue 2.4 system (kernel + 
rootfs) which is stored in 2 others flash partitions.

>   With my proposal to use full kernel-image packages it would
> be instead:
>
> 1. One does "ipkg install kernel-image"
> 2. One prays and with shaking hands runs
> "adhoc_kernel_update_script.sh /boot/zImage"

This step is impossible under 2.6 on Zaurus due to limitations of Sharp 
bootloader.

>   As for "user support problem", how that differs from what you have
> now? User installs empty kernel-image package and also doesn't get
> upgraded kernel. You likely solved this for Zaurus by telling users
> that installing kernel-image doesn't really upgrade their kernel.
> Well, "Zaurus" appears to be the keyword here.

Under OpenZaurus user gets message on boot that he run incorrect kernel. 
I do not know will Angstrom adopt it or not.

> > Also, on the Zaurus it means wasting a couple of megabytes of flash
> > for the extra copies of kernels you'll need.

>   As for LAB, that's exactly its strength - if you have support for
> some device (like funky flash) in kernel, LAB will be able to use it,
> as it's just kind of module for the kernel itself. The price of this
> is that LAB is indeed too big as for bootloader - ~1Mb instead of
> usual 256Mb/128Mb.

So why not u-boot or Apex? They are much smaller but maybe does not 
provide some stuff which LAB do.

> > Machine config files set information that is specific to the machine.
> > Whether of not a given machine has a sucky bootloader is a property
> > of the machine.

>   Point of view. It's all just point of view. Many-many people would
> consider you crazy for wanting to install something else, handmade,
> instead of vendor-carved-in stuff. But you smile and do just that. And
> yet you tell that /home partition and bootloader are carved in stone.
> But someone else will flash them away just as easily.
>
>   That of course doesn't mean I disagree with the current
> state-of-affairs of mostly adhoc bootloaders to be used, and need to
> account for that somehow. It's just one day it may become a limitation
> to have it in machine config.

OE support machines in a way which maintainers want. If some Joe Dev 
decided to change his machine to work in other way he is free to create 
own variations of configs/recipes. Flash resizing needs changing kernel 
CMDLINE which some bootloaders (like Zaurus one) does not support.

>   Ok, so it's clear that Zaurus machines need to keep using existing
> methods, and those methods should not be touched. On the other hand, I
> hope I was able to argue that other machines might have other
> requirements. In this regard, I'll still need to argue to machine
> mentors of (PocketPC!) machines in question to drop
> FILES_kernel-image_MACH = "".

I'm open to changes for machines which CAN support it. I do not like to 
have machine which do:

1. 1st stage bootloader (small, usually vendor provided)
2. 2nd stage bootloader (big LAB)
3. kernel (also big)

Think about booting time...

>   But anyway, I think that ROOTFS_POSTPROCESS_COMMAND += 
> "remove_zimages; " thing is still useful. It doesn't mean that machines
> should switch to it, but it offers nice, potentially
> machine-independent, way to remove zImage file from the image. 

Which will get back when user update kernel-image package. Which is BAD.

-- 
JID: hrw-jabber.org
OpenEmbedded developer/consultant

      catholic god himself invented autotools just for amusement
      first there was the great flood, then the plague, now autotools

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with [linux-handhelds-2.6] kernel packages

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marcin Juszkiewicz schreef:
> Dnia czwartek, 4 stycznia 2007 20:03, Paul Sokolovsky napisał:

>>   But anyway, I think that ROOTFS_POSTPROCESS_COMMAND += 
>> "remove_zimages; " thing is still useful. It doesn't mean that machines
>> should switch to it, but it offers nice, potentially
>> machine-independent, way to remove zImage file from the image. 
> 
> Which will get back when user update kernel-image package. Which is BAD.

I think the easiest solution is to remove empty kernel-image* files from the feeds.

regards,

Koen


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFFnhdHMkyGM64RGpERAh01AJ92+KNMVx1wPw6XYb7zqOgaRPEMRwCfSyRn
O/XMT8vBywOXSt84pf9jPzY=
=Fyhh
-----END PGP SIGNATURE-----

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with [linux-handhelds-2.6] kernel packages

[pHilipp Zabel]

On 1/5/07, Koen Kooi <koen@dominion.kabel.utwente.nl> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Marcin Juszkiewicz schreef:
> > Dnia czwartek, 4 stycznia 2007 20:03, Paul Sokolovsky napisał:
>
> >>   But anyway, I think that ROOTFS_POSTPROCESS_COMMAND +=
> >> "remove_zimages; " thing is still useful. It doesn't mean that machines
> >> should switch to it, but it offers nice, potentially
> >> machine-independent, way to remove zImage file from the image.
> >
> > Which will get back when user update kernel-image package. Which is BAD.
>
> I think the easiest solution is to remove empty kernel-image* files from the feeds.

But we then somehow have to provide users with the empty kernel-image packages
together with the zImage so that they can update their module packages
from the feeds
before flashing a new kernel.

regards
Philipp

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with [linux-handhelds-2.6] kernel packages

[Paul Sokolovsky]

Hello Marcin,

Friday, January 5, 2007, 10:41:45 AM, you wrote:

> Dnia czwartek, 4 stycznia 2007 20:03, Paul Sokolovsky napisał:

>> >> 2) Argue to Angstrom maintainers to stop killing zImage from rootfs
>> >> at all - consistently for all machines.
>> >
>> > So we start to waste 1.2MB disk space on at least 50% of the machines
>> > Angstrom supports? Thats a lot of space on a machine with a 32MB root
>> > partition.
>>
>>   It's not going to be waste, as it is supposed to be used by LAB,
>> eventually. Also, while 1.2MB seems like a figure, here's the paradox -
>> the difference of what you can fit into 32MB and 30.8MB is pretty
>> small. 

> I can see difference in fitting in 14.4MB which collie offers (or 21-23M
> of poodle, tosa, c700, c750). Keeping not needed files in rootfs hurts.
> We currently have problems with generating rootfs small enough.

  Ok. But machines for which that was intended, neither have such
flash limits, nor files would be not needed.

[]
> pdaX people tried to switch to u-boot on Zaurus line. Look into OESF
> forums to check how many users failed to follow 'simple' 2 step reflash.

  Don't tell me, I know how many users can read two words in row.

[]

> On PXA models you flash two files: kernel and rootfs. Any of them can be
> flashed without second one. There is a 'encrypted' script built by 
> zaurus-updater recipe which do it under rescue 2.4 system (kernel + 
> rootfs) which is stored in 2 others flash partitions.

  I see. Thanks.

[]

>>   As for LAB, that's exactly its strength - if you have support for
>> some device (like funky flash) in kernel, LAB will be able to use it,
>> as it's just kind of module for the kernel itself. The price of this
>> is that LAB is indeed too big as for bootloader - ~1Mb instead of
>> usual 256Mb/128Mb.

> So why not u-boot or Apex? They are much smaller but maybe does not 
> provide some stuff which LAB do.

  LAB is essentially a Linux kernel. People spent years adding support
for all the funky CF, SD, network, etc controllers to the kernel. But now
LAB can use all that as the boot/access sources for free. If instead
people will spend their time porting all that crap to u-boot, to the time
they finish, their already old PDAs will drop to the floor, they will
buy new PDA instead, and will have to follow all the multi-year cycle
again.

  Taking into account that multi-year thing was followed by bunch of
people already, it's nice idea to try something fresh ;-).

[]

> OE support machines in a way which maintainers want. If some Joe Dev 
> decided to change his machine to work in other way he is free to create
> own variations of configs/recipes. Flash resizing needs changing kernel
> CMDLINE which some bootloaders (like Zaurus one) does not support.

  Joe doesn't even have to use OE for all that coolness. So I hope, OE
will concentrate on best practices, one of them being clear separation
of recipes, machines, in distros. That will help Joe too, btw.

>>   Ok, so it's clear that Zaurus machines need to keep using existing
>> methods, and those methods should not be touched. On the other hand, I
>> hope I was able to argue that other machines might have other
>> requirements. In this regard, I'll still need to argue to machine
>> mentors of (PocketPC!) machines in question to drop
>> FILES_kernel-image_MACH = "".

> I'm open to changes for machines which CAN support it. I do not like to
> have machine which do:

> 1. 1st stage bootloader (small, usually vendor provided)
> 2. 2nd stage bootloader (big LAB)
> 3. kernel (also big)

> Think about booting time...

  Current boot time in 95% depends on runtime of bunch of inefficient
shell scripts, 5sec kernel boot time is nil comparing to that. Also,
as I hinted above, for many machines it's not a question of liking vs
not liking, but of having vs not having.

   By all means I think that people should optimize boot procedure for
each device, and OE should support that. But at the same time, it
would be nice to find maybe not so optimal, but generic solution.

>>   But anyway, I think that ROOTFS_POSTPROCESS_COMMAND += 
>> "remove_zimages; " thing is still useful. It doesn't mean that machines
>> should switch to it, but it offers nice, potentially
>> machine-independent, way to remove zImage file from the image. 

> Which will get back when user update kernel-image package. Which is BAD.

  No, that's feature, as was described above. And besides, not every
images needs to be package-managed at all, mentioned initrd's are
example. In this regard, a machine may want to have zImage in rootfs,
but not having zImage in initrd, and obviously, you can't achieve this
by having either hacked or not hacked kernel-image package.




-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [Angstrom-devel] [RFC] Get rid of adhoc machine-specific messing with kernel packages

[Paul Sokolovsky]

Hello Koen,

Thursday, December 28, 2006, 7:50:42 PM, you wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1

> CC: to the OE list

> Paul Sokolovsky schreef:
>> Hello angstrom-distro-devel,
>> 
>>   Thanks for providing pre-holiday 20061219 snapshots for testing.
>> They don't provide separate zImages, but well, that's ok - they are in
>> the images, after all.
>> 
>>   Well, not for hx4700, for example. Because:
>> 
>> ======== linux-handhelds-2.6.inc =====
>> FILES_kernel-image_hx4700 = ""
>> ALLOW_EMPTY_hx4700 = "1"
>> FILES_kernel-image_htcuniversal = ""
>> ALLOW_EMPTY_htcuniversal = "1"
>> ======================================
>> 
>>   Oops. Sources of such a special treatment of hx4700 and htcuniversal
>> would be unclear to outsider, but people who deal with (mis)features
>> of "ipaqs" know, that hx4700 has one (sic!, even if the most popular)
>> bootloader which doesn't use zImage in the image, but instead requires
>> adhoc means to install kernel, ditto for Universal, though bootloader
>> is another.
>> 
>>   So, what is tried here is to "save space". But there're few issues:

> Those lines should go, but not before we have a way to exclude 'kernel-image' from the
> rootfs *and* kernel-module-foo doesn't depend kernel-image anymore.

  Well, so am I right with assumption above that the only reason for
that is to "save space"? If yes, then we should fix it the right
place. So, if we want to be able to remove zImage file (not
kernel-image package, actually, as indeed modules depend on it, and do
that rightfully) from rootfs, we should do just that. We already
[have ability to] postprocess rootfs in a number of ways, like: remove
root password, make sure that /sbin/init is a real file, not a link,
make sure a /boot/zImage is a relative symlink. So, in that vein, I've
made a patch, submitted as http://bugs.openembedded.org/show_bug.cgi?id=1719
. Tested. Please review.

> regards,

> Koen

[]


-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com