[RFC] Setting default IMAGE_FSTYPES to tar.gz in bitbake.conf

[RFC] Setting default IMAGE_FSTYPES to tar.gz in bitbake.conf

[Paul Sokolovsky]

Hello,

Currently, we have in bitbake.conf:

IMAGE_FSTYPES ?= "jffs2"

jffs2 is a special filesystem format, suitable for limited number of
hardware on the global scale. So I would like to propose to replace
that default with some neutral and well-know format, say tar.gz.


Oh, and btw, I have a bunch of standalone filesystem conversion scripts
which I can't find time to RFC for addition to contrib/. While OE has
ability to generate images in different formats, sometimes there's a
need/easier to convert one existing flexible format. Few usecases I
personally had:

1. I don't have jffs2 in IMAGE_FSTYPES, as I don't flash stuff, so
having that would increase image generation runs and wasted disk space.
But when I needed to play with LiveRamdisks, which use jffs2, I just
converted tar.gz to jffs2.

2. For Angstrom loopback installs, I run scripts which convert
specific tar.gz to ext2 right on the download server, instead of
downloading, converting and uploading back, of overloading autobuilder
with generating ext2 for each image.


-- 
Best regards,
 Paul                          mailto:pmiscml@gmail.com

Re: [RFC] Setting default IMAGE_FSTYPES to tar.gz in bitbake.conf

[Richard Purdie]

On Tue, 2008-01-29 at 01:41 +0200, Paul Sokolovsky wrote:
> Currently, we have in bitbake.conf:
> 
> IMAGE_FSTYPES ?= "jffs2"
> 
> jffs2 is a special filesystem format, suitable for limited number of
> hardware on the global scale. So I would like to propose to replace
> that default with some neutral and well-know format, say tar.gz.

This has been discussed before and I'd like to see that changed. The
complication is that "bitbake image" is meant to give a sane default
image for a given machine without any tweaking by the user and a load of
machines depend on the jffs2 default. 

I'm not an ipaq expert but I'd guess a number of them use jffs2, as
probably do some of the htcs, the at91 board and the motorola phones.
Someone needs to go through and check the machines. Since this is pretty
daunting to fix, I've just gone through and I think the following files
need that line adding as they appear to use jffs2 in some form:

at32stk1000.conf
at91sam9260ek.conf
at91sam9261ek.conf
at91sam9263ek.conf
bd-neon.conf
fic-gta01.conf
eseries.conf
h1910.conf
h2200.conf
h3600.conf
h5000.conf
h6300.conf
hx4700.conf
gesbc-9302.conf
./include/motorola-ezx-base.inc (hence e680.conf rokre2.conf rokre6.conf
a1200.conf a780.conf)
./include/gumstix.inc (hence gumstix-connex.conf gumstix-verdex.conf)
omap1510inn.conf
omap1610h2.conf
omap1710h3.conf
omap2420h4.conf
omap2430sdp.conf
omap5912osk.conf
palmtt.conf
palmz71.conf
shark.conf
simpad.conf
triton.conf
xxs1500.conf

I'm unsure of the following machines:

acern30.conf
adsp-bf537-stamp.conf
adzs-bf538f-ezlite.conf
adzs-bf548-ezlite.conf
asus620.conf
asus730.conf
atngw100.conf
aximx50.conf
aximx50v.conf
colinux.conf
davinci-dvevm.conf
davinci-sffsdr.conf
db1200.conf
devkitidp-pxa255.conf
dht-walnut.conf
efika.conf
em-x270.conf
ep93xx.conf
er0100.conf
eteng500.conf
geodegx.conf
geodelx.conf
ghi270.conf
guinness.conf
h1940.conf
h3800.conf
h3900.conf
h4000.conf
htcalpine.conf
htcapache.conf
htcblueangel.conf
htchimalaya.conf
htcsable.conf
htctornado.conf
htcuniversal.conf
htcwallaby.conf
jornada56x.conf
jornada6xx.conf
jornada7xx.conf
kb9202.conf
ks8695.conf
logicpd-pxa270.conf
looxc550.conf
magicbox.conf
magician.conf
mainstone.conf
mnci.conf
mpc8313e-rdb.conf
mpc8323e-rdb.conf
mtx-1.conf
mtx-2.conf
mtx-3a.conf
mtx-3.conf
mx21ads.conf
navman-icn330.conf
netbook-pro.conf
netvista.conf
palmld.conf
palmt650.conf
palmt680.conf
palmtc.conf
palmtt2.conf
palmtt3.conf
palmtt5.conf
palmtx.conf
palmz31.conf
palmz72.conf
rb500.conf
rt3000.conf
rx1950.conf
rx3000.conf
titan.conf
ts72xx.conf
wl500g.conf
wrap.conf
xilinx-ml403.conf
xilinx-ml410.conf

I've ruled the rest out. If someone who knows a load of the above
machines could confirm or deny that they use jffs2 images that would be
great...

> Oh, and btw, I have a bunch of standalone filesystem conversion scripts
> which I can't find time to RFC for addition to contrib/. While OE has
> ability to generate images in different formats, sometimes there's a
> need/easier to convert one existing flexible format. 

I'd say feel free to add them...

Regards,

Richard

[ann] Utils for converting rootfs images to other formats

[Paul Sokolovsky]

Hello,

[]

> > Oh, and btw, I have a bunch of standalone filesystem conversion
> > scripts which I can't find time to RFC for addition to contrib/.
> > While OE has ability to generate images in different formats,
> > sometimes there's a need/easier to convert one existing flexible
> > format. 
> 
> I'd say feel free to add them...

Added as contrib/image-convert/ . Of course, they mostly include
ability to convert generic archive formats (cpio/tar) to something ;-).

> 
> Regards,
> 
> Richard
> 
> 


-- 
Best regards,
 Paul                          mailto:pmiscml@gmail.com

Re: [RFC] Setting default IMAGE_FSTYPES to tar.gz in bitbake.conf

[Paul Sokolovsky]

Hello,

On Tue, 29 Jan 2008 00:07:05 +0000
Richard Purdie <rpurdie@rpsys.net> wrote:

> On Tue, 2008-01-29 at 01:41 +0200, Paul Sokolovsky wrote:
> > Currently, we have in bitbake.conf:
> > 
> > IMAGE_FSTYPES ?= "jffs2"
> > 
> > jffs2 is a special filesystem format, suitable for limited number of
> > hardware on the global scale. So I would like to propose to replace
> > that default with some neutral and well-know format, say tar.gz.
> 
> This has been discussed before and I'd like to see that changed. The
> complication is that "bitbake image" is meant to give a sane default
> image for a given machine without any tweaking by the user and a load
> of machines depend on the jffs2 default. 

Unfortunately, this is not achievable and not sustainable goal. It's
not achievable because there's no single criteria for "sane", it
depends first of all on user's needs. So, trying to make adhoc choices
instead of user would mean that for some group of users they will be
unpredictable and thus insane. So, it would make more sense to use "not
insane" criteria instead and use some neutral, well-known format, which
is why this RFC is being made.

> 
> I'm not an ipaq expert but I'd guess a number of them use jffs2, as
> probably do some of the htcs, the at91 board and the motorola phones.
> Someone needs to go through and check the machines. Since this is
> pretty daunting to fix, I've just gone through and I think the
> following files need that line adding as they appear to use jffs2 in
> some form:

[]

> 
> I'm unsure of the following machines:

And those above is why it's not sustainable - now it comes that to
achieve that goal it takes to be a machine expert, etc. etc. Of course,
noone is expert in all machines, some people are not experts in the
machine they add/hack on, all people can not know something, make
mistakes and wrong choices, etc. In the end and overall, presence of
jffs2 in specific machine's config turns out to be rather random
matter. We may want to rethink that to be based on more firm grounds.
One possible way:

1. IMAGE_FSTYPES is not a machine config variable.
2. IMAGE_FSTYPES is a top-level parameter specified by user. User needs
specific format, user requests it. There's a default though offering
least surprise as integrated over user base - i.e. neutral, well-known
format.
3. Alternatively, a *distro* may constrain a way in which images are
used, and thus force a particular image format. So, for a dumb headless
router machine, one distro will try to follow vendor's way and generate
images in an adhoc format. Yet another for the same machine will assume
that bootloader was replaced with some generic one and will use generic
flash image format like jffs2. And yet another will be intended for the
people who soldered few wires at the appropriate place and use bit-bang
to boot from SD.


Well, as usual, thoughts aloud. I'm going to add 
IMAGE_FSTYPES ?= "jffs2" to the machines from your 1st list to get
matter going.


[]
> 
> Regards,
> 
> Richard
> 



-- 
Best regards,
 Paul                          mailto:pmiscml@gmail.com

Re: [RFC] Setting default IMAGE_FSTYPES to tar.gz in bitbake.conf

[Richard Purdie]

Hi Paul,

On Sat, 2008-02-09 at 13:48 +0200, Paul Sokolovsky wrote:
> > On Tue, 2008-01-29 at 01:41 +0200, Paul Sokolovsky wrote:
> > This has been discussed before and I'd like to see that changed. The
> > complication is that "bitbake image" is meant to give a sane default
> > image for a given machine without any tweaking by the user and a load
> > of machines depend on the jffs2 default. 
> 
> Unfortunately, this is not achievable and not sustainable goal. It's
> not achievable because there's no single criteria for "sane", it
> depends first of all on user's needs. So, trying to make adhoc choices
> instead of user would mean that for some group of users they will be
> unpredictable and thus insane. So, it would make more sense to use "not
> insane" criteria instead and use some neutral, well-known format, which
> is why this RFC is being made.

Often there is an obvious format which makes sense for a given machine.
For qemu, its ext2, at least in Poky where the scripts to run the images
expect that. For my c7x0 its jffs2 for spitz its a tgz etc.

I agree that not everyone is going to want these formats in every case,
thats why its a "soft" ?= setting and not a hard set one.

> And those above is why it's not sustainable - now it comes that to
> achieve that goal it takes to be a machine expert, etc. etc. Of course,
> noone is expert in all machines, some people are not experts in the
> machine they add/hack on, all people can not know something, make
> mistakes and wrong choices, etc. In the end and overall, presence of
> jffs2 in specific machine's config turns out to be rather random
> matter. 

No, its not random. jffs2 is the default image format that makes sense
on say a c7x0 or akita since thats what its reflash utils accept and
what the Angstrom install notes assume is used on that device.

> We may want to rethink that to be based on more firm grounds.
> One possible way:
> 
> 1. IMAGE_FSTYPES is not a machine config variable.

No, I want it to have a sane value for machines where it makes sense. It
doesn't make sense in all cases but where it does, it hurts nothing to
set it.

> 2. IMAGE_FSTYPES is a top-level parameter specified by user. User needs
> specific format, user requests it. There's a default though offering
> least surprise as integrated over user base - i.e. neutral, well-known
> format.

This "neutral" offering being a tgz? Thats useless to boot my c7x0. I
want OE to produce something usable, I don't want to have to change my
IMAGE_FSTYPES every time I change machine.

> 3. Alternatively, a *distro* may constrain a way in which images are
> used, and thus force a particular image format. So, for a dumb headless
> router machine, one distro will try to follow vendor's way and generate
> images in an adhoc format. Yet another for the same machine will assume
> that bootloader was replaced with some generic one and will use generic
> flash image format like jffs2. And yet another will be intended for the
> people who soldered few wires at the appropriate place and use bit-bang
> to boot from SD.

The DISTRO can override the machine if it wishes now with no change to
OE, that is up to the distros concerned. That doesn't stop machines from
having sane defaults.

> Well, as usual, thoughts aloud. I'm going to add 
> IMAGE_FSTYPES ?= "jffs2" to the machines from your 1st list to get
> matter going.

ok.

Cheers,

Richard

Re: [RFC] Setting default IMAGE_FSTYPES to tar.gz in bitbake.conf

[Paul Sokolovsky]

Hello,

On Sat, 09 Feb 2008 18:55:33 +0000
Richard Purdie <rpurdie@rpsys.net> wrote:

[]

> The DISTRO can override the machine if it wishes now with no change to
> OE, that is up to the distros concerned. That doesn't stop machines
> from having sane defaults.
> 
> > Well, as usual, thoughts aloud. I'm going to add 
> > IMAGE_FSTYPES ?= "jffs2" to the machines from your 1st list to get
> > matter going.
> 
> ok.
> 

Done, so are we ready for this patch now?


@@ -541,7 +545,7 @@ DL_DIR ?= "${TMPDIR}/downloads" 
 ################################################################## 
  
 DL_DIR ?= "${TMPDIR}/downloads" 
-IMAGE_FSTYPES ?= "jffs2" 
+IMAGE_FSTYPES ?= "tar.gz" 
 PCMCIA_MANAGER ?= "pcmcia-cs" 
 DEFAULT_TASK_PROVIDER ?= "task-base" 
 MACHINE_TASK_PROVIDER ?= "${DEFAULT_TASK_PROVIDER}" 


> Cheers,
> 
> Richard



-- 
Best regards,
 Paul                          mailto:pmiscml@gmail.com

Re: [RFC] Setting default IMAGE_FSTYPES to tar.gz in bitbake.conf

[Mike (mwester)]

Paul Sokolovsky wrote:
> Hello,
>
> On Sat, 09 Feb 2008 18:55:33 +0000
> Richard Purdie <rpurdie@rpsys.net> wrote:
>
> []
>
>   
>> The DISTRO can override the machine if it wishes now with no change to
>> OE, that is up to the distros concerned. That doesn't stop machines
>> from having sane defaults.
>>
>>     
>>> Well, as usual, thoughts aloud. I'm going to add 
>>> IMAGE_FSTYPES ?= "jffs2" to the machines from your 1st list to get
>>> matter going.
>>>       
>> ok.
>>
>>     
>
> Done, so are we ready for this patch now?
>
>
> @@ -541,7 +545,7 @@ DL_DIR ?= "${TMPDIR}/downloads" 
>  ################################################################## 
>   
>  DL_DIR ?= "${TMPDIR}/downloads" 
> -IMAGE_FSTYPES ?= "jffs2" 
> +IMAGE_FSTYPES ?= "tar.gz" 
>   
Why exactly are we simply replacing a jffs2 default with a new default?  
Just to make some other parties happier?  Wouldn't it be simply better, 
if this is such a big deal, to do:

+IMAGE_FSTYPES ?= "you-must-define-an-image-fstype-that-is-suitable-for-you"

That way we ensure that *nobody* relies on defaults, and everything is 
correctly defined in every case, and if someone fails to define it, it 
should be pretty obvious what they did wrong.

>  PCMCIA_MANAGER ?= "pcmcia-cs" 
>  DEFAULT_TASK_PROVIDER ?= "task-base" 
>  MACHINE_TASK_PROVIDER ?= "${DEFAULT_TASK_PROVIDER}" 
>
>
>   
>> Cheers,
>>
>> Richard
>>     
> Mike (mwester)
>
>
>   

Re: [RFC] Setting default IMAGE_FSTYPES to tar.gz in bitbake.conf

[Paul Sokolovsky]

Hello,

On Sun, 10 Feb 2008 12:43:24 -0600
"Mike (mwester)" <mwester@dls.net> wrote:

> Paul Sokolovsky wrote:
> > Hello,
> >
> > On Sat, 09 Feb 2008 18:55:33 +0000
> > Richard Purdie <rpurdie@rpsys.net> wrote:
> >
> > []
> >
> >   
> >> The DISTRO can override the machine if it wishes now with no
> >> change to OE, that is up to the distros concerned. That doesn't
> >> stop machines from having sane defaults.
> >>
> >>     
> >>> Well, as usual, thoughts aloud. I'm going to add 
> >>> IMAGE_FSTYPES ?= "jffs2" to the machines from your 1st list to get
> >>> matter going.
> >>>       
> >> ok.
> >>
> >>     
> >
> > Done, so are we ready for this patch now?
> >
> >
> > @@ -541,7 +545,7 @@ DL_DIR ?= "${TMPDIR}/downloads" 
> >  ################################################################## 
> >   
> >  DL_DIR ?= "${TMPDIR}/downloads" 
> > -IMAGE_FSTYPES ?= "jffs2" 
> > +IMAGE_FSTYPES ?= "tar.gz" 
> >   
> Why exactly are we simply replacing a jffs2 default with a new
> default? Just to make some other parties happier?  

No, to make sure that probability of someone being unhappy and confused
is minimized. jffs2 is an adhoc, niche format, while tar.gz is
something well known. Getting tar.gz by default could be not exactly
what someone wants, but at least not something completely strange for
some people (consider x86 for example).

> Wouldn't it be
> simply better, if this is such a big deal, to do:
> 
> +IMAGE_FSTYPES ?=
> "you-must-define-an-image-fstype-that-is-suitable-for-you"
> 
> That way we ensure that *nobody* relies on defaults, and everything
> is correctly defined in every case, and if someone fails to define
> it, it should be pretty obvious what they did wrong.

This is exactly the opposite direction to what I'd like to move.
Configuring OE is already beyond capabilities of mere human, so many
people simply can't surpass initial setup curve of it, and other,
trying to help them, invent different adhoc solutions like
meta-makefiles. Our recent discussion exposed the fact that this is
pressing issue understood by many people, yet no good, well-fitting
solution was proposed. In this regard, I'm trying to follow my idea
that OE itself should change, first of all paradigmatically, to let
people use it "out of the box", without "extraordinary" effort required
to setup. For me, that means that OE should be usable immediately
after checkout of bitbake and itself, and adding bitbake to PATH. I've
been having such config for several months, and spooling my changes to
OE now.

So, with this idea, we want to have good, "not insane" defaults for
options, with the idea to offer our users easy learning curve on OE,
starting with it producing something easily visible and recognizable
by default, and then easy steps to tweak setup to get exactly what they
want.

[]
> >>     
> > Mike (mwester)
> >



-- 
Best regards,
 Paul                          mailto:pmiscml@gmail.com