[U-Boot-Users] RFC: New U-boot image format

[U-Boot-Users] RFC: New U-boot image format

[Marian Balakowicz]

Hello,

New format for U-boot images has been on the list few times already.
There were different ideas and features discussed but no final
conclusion has been made.

I've put together a set requirements that aims to assist further
discussion. They are quite generic at this point of time and do not
cover all details, use cases, features.

So far they got briefly review by Wolfgang but I hope to hear more
comments, suggestions, criticism, opinions, etc.

All feedback is welcome.

Thanks,
Marian
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: uboot_new_image.reqs.txt
Url: http://lists.denx.de/pipermail/u-boot/attachments/20071211/16eac4de/attachment.txt 

[U-Boot-Users] RFC: New U-boot image format - open issues

[Marian Balakowicz]

Marian Balakowicz wrote:
> 
> I've put together a set requirements that aims to assist further
> discussion. They are quite generic at this point of time and do not
> cover all details, use cases, features.

I would also like to collect questions, open issues, ideas, etc.
Attached is a initial list of such items.

Cheers,
Marian
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: uboot_new_image.issues.txt
Url: http://lists.denx.de/pipermail/u-boot/attachments/20071211/2afffe39/attachment.txt 

[U-Boot-Users] RFC: New U-boot image format

[Jerry Van Baren]

Marian Balakowicz wrote:
> Hello,
> 
> New format for U-boot images has been on the list few times already.
> There were different ideas and features discussed but no final
> conclusion has been made.
> 
> I've put together a set requirements that aims to assist further
> discussion. They are quite generic at this point of time and do not
> cover all details, use cases, features.
> 
> So far they got briefly review by Wolfgang but I hope to hear more
> comments, suggestions, criticism, opinions, etc.
> 
> All feedback is welcome.
> 
> Thanks,
> Marian

Very interesting!  Thanks for putting in the time and thought.

> Extending U-Boot image format requirements
> ------------------------------------------
> 
> 1. Preserve old image format, cleanup and re-factor old image handling code

[snip]

> 2. Develop a 'new image' format
> 
>   (a) 'New image' format shall be based on a Flat Device Tree (FDT) structure

Trivia: Flattened (not flat) Device Tree.

[snip]

> 3. 'New image' format must support the following features:

[snip]

> 	- 'container' image blob shall include 'component' images'
> 	  data, which means direct data embedding  - as opposed to
> 	  having only references

Q for Jon L: This would require an extension to the dtc to "include" a 
raw file into the blob?  I'm presuming that we don't want to take a 
binary (ELF) file, turn it into ASCII bytes, include it into a dts, and 
then use dtc to compile it back into binary.

Am I missing something that is already available?  Do you see any 
problems with extending dtc to support this?

>   (b) 'container' source file (.dts)
> 
> 	- the following bindings and properties shall be defined for a device
> 	  tree source file (.dts) that is corresponding to a 'container'
> 	  image blob:
> 
> 		- root node of the DTS shall represent a 'container' node
> 		- 'container' node shall support:
> 			- label property
> 			- timestamp property
> 		- 'container' node shall support multiple 'component' subnodes
> 
> 		- 'component' subnode shall support:
> 			- label property
> 			- type property
> 			- hash properties (crc32, md5, sha1, etc.)

Would hash be two entries (type and value), or would it be just the type 
and use conventions for where the value is stored (i.e. last /n/ bytes 
of the image)?  I would vote for (type and value) if this were a democracy.

Are image hashes to validate what is stored in the blob (compressed) or 
to validate what is in memory after decompressing?  (Ability to support 
both options would be very good IMHO.)

> 			- data compression type property (compressions 
> 			  currently supported by U-boot)
> 			- data size property
> 			- timestamps property
> 
> 			- properties corresponding to remaining header
> 			  fields from the old image format:
> 
> 				os type, cpu architecture properties
> 				data load address
> 				entry points for executable images
> 
> 	  Note: the above list is considered *draft* and open to discussion

Which properties are new inventions?  Data compression, timestamps, OS 
Type, CPU arch, data load address, and entry point?  Not a bad thing, 
just trying to understand how much we are extending.

> 	- properties/bindings, their format and definition and the way they
> 	  are accessed and processed in U-boot shall allow for easy and
> 	  flexible extensions; that is, adding new 'component' type,
> 	  new compression type, etc. shall be possible in a clean and
> 	  compact way.
> 
> 	- detailed image bindings description shall be provided as a separate
> 	  document (e.g. wiki web page)

I vote for capturing it git as a text document equivalent to 
booting_without_of.txt (fdt_images.txt?). (Do we need a [Dd]ocumentation 
subdirectory?)

> 4. Extend mkimage tool to support new image handling operations:

[ack-snip]

> 5. Add 'new image' handling to U-Boot
> 
>   (a) support for 'new image' shall be added to related U-boot commands: bootm,
>       imls, iminfo, imxtract, doc, fdcboot, fpga, ide, nboot, scsiboot, usbboot

I presume...
imls     == image list (overlaps with "fdt l /", no?)
iminfo   == image info?  What does it do more than imls?
imxtract == image extract?
    * Does what?
    * Somebody buy that man a vowl ;-)

It is probably an urban legend, but I recall somebody once asked Brian 
Kernighan (Dennis Ritchie?) what his greatest regret in creating the C 
language was, and he replied "Leaving the 'e' off create."

> 	- command syntax (especially 'bootm') will need to be extended to
> 	include 'new image' format related uses cases

Should we create a new command rather than trying to create yet another 
bootm extension (YABE), or is that too disruptive?  If not, bootfdt?

[ack-snip]

Thanks again,
gvb

[U-Boot-Users] RFC: New U-boot image format

[Wolfgang Denk]

In message <475EE3AE.6060102@ge.com> you wrote:
>
> Q for Jon L: This would require an extension to the dtc to "include" a 
> raw file into the blob?  I'm presuming that we don't want to take a 
> binary (ELF) file, turn it into ASCII bytes, include it into a dts, and 
> then use dtc to compile it back into binary.

Correct. We want a binary include feature here :-)

> > 		- 'component' subnode shall support:
> > 			- label property
> > 			- type property
> > 			- hash properties (crc32, md5, sha1, etc.)
> 
> Would hash be two entries (type and value), or would it be just the type 
> and use conventions for where the value is stored (i.e. last /n/ bytes 
> of the image)?  I would vote for (type and value) if this were a democracy.

Me too :-) 

> Are image hashes to validate what is stored in the blob (compressed) or 
> to validate what is in memory after decompressing?  (Ability to support 
> both options would be very good IMHO.)

Good point.

So far, we always verify the (compressed) blob -  both  for  security
(don't  even  load  a  compromised  image)  and  speed - checking the
compressed image is much faster, obviously.

But your request is perfectly reasonable.

> > 				os type, cpu architecture properties
> > 				data load address
> > 				entry points for executable images
> > 
> > 	  Note: the above list is considered *draft* and open to discussion
> 
> Which properties are new inventions?  Data compression, timestamps, OS 
> Type, CPU arch, data load address, and entry point?  Not a bad thing, 
> just trying to understand how much we are extending.

All of this already exists, but in a static way.

New is: ability to chose checksum method; ability to compine  several
blobs into one image in a structured way, so building and booting the
equivalent  of  a  multifile  image  with  any combination of kernel,
ramdisk and dtb blobs can be done in a clean way.

> I presume...
> imls     == image list (overlaps with "fdt l /", no?)
> iminfo   == image info?  What does it do more than imls?

	It lists any image the address of which you pass it. "imls"
	only does a limited scan, checking sector boundaries in NOR
	flash only.

> imxtract == image extract?
>     * Does what?

	Extract a blob from an image?

>     * Somebody buy that man a vowl ;-)

:-)

> > 	- command syntax (especially 'bootm') will need to be extended to
> > 	include 'new image' format related uses cases
> 
> Should we create a new command rather than trying to create yet another 
> bootm extension (YABE), or is that too disruptive?  If not, bootfdt?

My preference would be to add it to bootm ... (I've been typing this
for too long now).

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Looks clean and obviously correct to me, but then _everything_ I
write always looks obviously correct to me.  - Linus Torvalds in
<Pine.LNX.4.10.10012090054360.791-100000@penguin.transmeta.com>

[U-Boot-Users] RFC: New U-boot image format

[Jerry Van Baren]

Wolfgang Denk wrote:
> In message <475EE3AE.6060102@ge.com> you wrote:

[snip]

>>> 				os type, cpu architecture properties
>>> 				data load address
>>> 				entry points for executable images
>>>
>>> 	  Note: the above list is considered *draft* and open to discussion
>> Which properties are new inventions?  Data compression, timestamps, OS 
>> Type, CPU arch, data load address, and entry point?  Not a bad thing, 
>> just trying to understand how much we are extending.
> 
> All of this already exists, but in a static way.
> 
> New is: ability to chose checksum method; ability to compine  several
> blobs into one image in a structured way, so building and booting the
> equivalent  of  a  multifile  image  with  any combination of kernel,
> ramdisk and dtb blobs can be done in a clean way.

Ahh, my question wasn't clear.  The intended question was, which of 
these properties are new OF (IEEE-1275) or "existing linux kernel" 
*properties* (I didn't intend to ask which are new capabilities).

For instance, how to specify an address already exists, obviously, but I 
suspect a property "data load address" is new.

The intent behind the question was thinking about how big of a 
documentation job this will be.

[snip]

> Best regards,
> 
> Wolfgang Denk

Thanks,
gvb

[U-Boot-Users] RFC: New U-boot image format

[Marian Balakowicz]

Hi Jerry,

Thanks for your comments, see my replies below.

Jerry Van Baren wrote:
[...]
>> 3. 'New image' format must support the following features:
> 
> [snip]
> 
>> 	- 'container' image blob shall include 'component' images'
>> 	  data, which means direct data embedding  - as opposed to
>> 	  having only references
> 
> Q for Jon L: This would require an extension to the dtc to "include" a 
> raw file into the blob?  I'm presuming that we don't want to take a 
> binary (ELF) file, turn it into ASCII bytes, include it into a dts, and 
> then use dtc to compile it back into binary.
> 
> Am I missing something that is already available?  Do you see any 
> problems with extending dtc to support this?

AFAIK dtc currently has no support for data includes. I've seen such
feature on a dtc wish list though, so adding it should not be troublesome.


>>   (b) 'container' source file (.dts)
>>
>> 	- the following bindings and properties shall be defined for a device
>> 	  tree source file (.dts) that is corresponding to a 'container'
>> 	  image blob:
>>
>> 		- root node of the DTS shall represent a 'container' node
>> 		- 'container' node shall support:
>> 			- label property
>> 			- timestamp property
>> 		- 'container' node shall support multiple 'component' subnodes
>>
>> 		- 'component' subnode shall support:
>> 			- label property
>> 			- type property
>> 			- hash properties (crc32, md5, sha1, etc.)
> 
> Would hash be two entries (type and value), or would it be just the type 
> and use conventions for where the value is stored (i.e. last /n/ bytes 
> of the image)?  I would vote for (type and value) if this were a democracy.
>
> Are image hashes to validate what is stored in the blob (compressed) or 
> to validate what is in memory after decompressing?  (Ability to support 
> both options would be very good IMHO.)

Agree, entries are much more flexible, e.g. we can easily add third
entry which will distinguish compressed/uncompressed data hashes.

>>
>> 	- detailed image bindings description shall be provided as a separate
>> 	  document (e.g. wiki web page)
> 
> I vote for capturing it git as a text document equivalent to 
> booting_without_of.txt (fdt_images.txt?).

I mentioned wiki as it's easier to update and work with, but I am fine
with the git txt as well.

> (Do we need a [Dd]ocumentation subdirectory?)

Not sure, there is a doc directory which contains board README files.
And there is a (quite large) README file that documents a lot of U-boot
internals. I guess we may want to hear Wolfgang's opinion?

Cheers,
Marian

[U-Boot-Users] RFC: New U-boot image format

[Marian Balakowicz]

Jerry Van Baren wrote:
> Wolfgang Denk wrote:
>> In message <475EE3AE.6060102@ge.com> you wrote:
> [snip]
> 
>>>> 				os type, cpu architecture properties
>>>> 				data load address
>>>> 				entry points for executable images
>>>>
>>>> 	  Note: the above list is considered *draft* and open to discussion
>>> Which properties are new inventions?  Data compression, timestamps, OS 
>>> Type, CPU arch, data load address, and entry point?  Not a bad thing, 
>>> just trying to understand how much we are extending.
>> All of this already exists, but in a static way.
>>
>> New is: ability to chose checksum method; ability to compine  several
>> blobs into one image in a structured way, so building and booting the
>> equivalent  of  a  multifile  image  with  any combination of kernel,
>> ramdisk and dtb blobs can be done in a clean way.
> 
> Ahh, my question wasn't clear.  The intended question was, which of 
> these properties are new OF (IEEE-1275) or "existing linux kernel" 
> *properties* (I didn't intend to ask which are new capabilities).
> 
> For instance, how to specify an address already exists, obviously, but I 
> suspect a property "data load address" is new.
> 
> The intent behind the question was thinking about how big of a 
> documentation job this will be.

We may want to try to use some of the already defined OF bindings, but I
tend to think that this kind of reuse may become evil. We would be using
properties defined for kernel device tree for purpose that has noting to
do with the real device tree, and that may bring unnecessary confusion.
So, I'd rather opt for a independent specification that introduces a
clear separation, especially that many of the properties will be new anyway.

Cheers,
Marian

[U-Boot-Users] RFC: New U-boot image format

[Wolfgang Denk]

In message <4761B528.3080601@semihalf.com> you wrote:
> 
> > I vote for capturing it git as a text document equivalent to 
> > booting_without_of.txt (fdt_images.txt?).
> 
> I mentioned wiki as it's easier to update and work with, but I am fine
> with the git txt as well.
> 
> > (Do we need a [Dd]ocumentation subdirectory?)
> 
> Not sure, there is a doc directory which contains board README files.
> And there is a (quite large) README file that documents a lot of U-boot
> internals. I guess we may want to hear Wolfgang's opinion?

My immediate response was to saet up a wiki page; actually I  already
started  converting  your  proposal  to such a page, but I dropped it
when I saw the request for a file that goes with the source tree,  as
this makes sense for me, too.

In other words: I can  live  with  both  options.  In  the  end,  the
document  should be available with the source tree, so we may as well
start this way.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Mirrors should reflect a little before throwing back images.
- Jean Cocteau

[U-Boot-Users] RFC: New U-boot image format

[T Ziomek]

On Thu, 13 Dec 2007, Marian Balakowicz wrote:
>
> Hi Jerry,
>
> Thanks for your comments, see my replies below.
>
> Jerry Van Baren wrote:
  . . .
>> (Do we need a [Dd]ocumentation subdirectory?)
>
> Not sure, there is a doc directory which contains board README files.
> And there is a (quite large) README file that documents a lot of U-boot
> internals. I guess we may want to hear Wolfgang's opinion?

Note that some of the READMEs in the doc directory are not board-specific.
NetConsole is one example.

-- 
   /"\  ASCII Ribbon Campaign   |
   \ /                          |   Email to user 'CTZ001'
    X        Against HTML       |             at 'email.mot.com'
   / \     in e-mail & news     |

[U-Boot-Users] RFC: New U-boot image format

[Timur Tabi]

Marian Balakowicz wrote:
> Hello,
> 
> New format for U-boot images has been on the list few times already.
> There were different ideas and features discussed but no final
> conclusion has been made.

I have a request for a new feature.  I think we need image-format plug-ins. 
That is, we need a way for a plug-in to register itself with the main format 
processing code.  When the processing code sees a blob that it doesn't 
understand, it calls the plug-in to handle it.

This would be a handy way to handle stuff like the QE firmware binary blob 
format (see the thread titled "[PATCH] 85xx: add ability to upload QE 
firmware").  The format of the QE firmware blob has already been decided, so all 
I would need from the multi-image format is:

1) A way to package my blob.
2) A way to pass the address of the blob to the QE code

For option #2, setting an environment variable would be the easiest.  To do 
that, the QE code could register a call-back that says, "If you see a QE 
firmware blob, call this function and pass the address of the blob".

-- 
Timur Tabi
Linux kernel developer at Freescale

[U-Boot-Users] RFC: New U-boot image format

[Jerry Van Baren]

Timur Tabi wrote:
> Marian Balakowicz wrote:
>> Hello,
>>
>> New format for U-boot images has been on the list few times already.
>> There were different ideas and features discussed but no final
>> conclusion has been made.
> 
> I have a request for a new feature.  I think we need image-format plug-ins. 
> That is, we need a way for a plug-in to register itself with the main format 
> processing code.  When the processing code sees a blob that it doesn't 
> understand, it calls the plug-in to handle it.
> 
> This would be a handy way to handle stuff like the QE firmware binary blob 
> format (see the thread titled "[PATCH] 85xx: add ability to upload QE 
> firmware").  The format of the QE firmware blob has already been decided, so all 
> I would need from the multi-image format is:
> 
> 1) A way to package my blob.
> 2) A way to pass the address of the blob to the QE code
> 
> For option #2, setting an environment variable would be the easiest.  To do 
> that, the QE code could register a call-back that says, "If you see a QE 
> firmware blob, call this function and pass the address of the blob".

Hi Timur,

#1 is already in there in the form of properties:
<http://article.gmane.org/gmane.comp.boot-loaders.u-boot/34055>
- 'component' subnode shall support:
     - label property
     - type property
     - hash properties (crc32, md5, sha1, etc.)
     - data compression type property (compressions
       currently supported by U-boot)
     - data size property
     - timestamps property

     - properties corresponding to remaining header
       fields from the old image format:

         os type, cpu architecture properties
         data load address
         entry points for executable images

I think you have #2 backwards.  I envision the board/CPU specific code 
querying the multi-image blob for specific QE firmware (for instance, 
some boards may want serial fixup firmware rather than ethernet 
enhancement firmware, both of which could be in the multi-image) and 
loading it.  I envision this as part of the board/CPU QE handling code 
(the part-of-u-boot GPL code you wrote).

Since libfdt can find the QE firmware and the properties associated with 
that firmware, including the necessary address(es), your "qe" command 
(and the C function corresponding to it) would not need an address at 
all (I would make it optional rather than removing it, so the 
multi-image value could be overridden).

Best regards,
gvb

[U-Boot-Users] RFC: New U-boot image format

[Timur Tabi]

Jerry Van Baren wrote:

> Since libfdt can find the QE firmware and the properties associated with 
> that firmware, including the necessary address(es), your "qe" command 
> (and the C function corresponding to it) would not need an address at 
> all (I would make it optional rather than removing it, so the 
> multi-image value could be overridden).

That works.

-- 
Timur Tabi
Linux kernel developer at Freescale

[U-Boot-Users] RFC: New U-boot image format

[Wolfgang Denk]

In message <4767E487.8020103@freescale.com> you wrote:
> 
> I have a request for a new feature.  I think we need image-format plug-ins. 
> That is, we need a way for a plug-in to register itself with the main format 
> processing code.  When the processing code sees a blob that it doesn't 
> understand, it calls the plug-in to handle it.

This sounds like a good idea as it allows the code to deal with new or
customized image formats.

> I would need from the multi-image format is:
> 
> 1) A way to package my blob.

OK.

> 2) A way to pass the address of the blob to the QE code
> 
> For option #2, setting an environment variable would be the easiest.  To do 

NAK. Using an envuironment variable is definitely not  a  good  idea,
and  probably  not  sufficient.  Assume  the situation where you have
"autostart" set to "yes" and use  DHCP  to  download  and  start  the
image,  or  where  you use "tftp" followed by a "bootm" command to do
the same. In both cases there is no direct "hook" to do what you  ask
for  - ther eis no sane place where the envrionment variable would be
set (in the network code?) nor where it would be taken into account.

Your initial comment about  image-specific  plugins  (that  could  be
specified  for  and  run with specific commands like iminfo or bootm)
makes much more sense to me.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Contrary to popular belief, thinking does not cause brain damage.

[U-Boot-Users] RFC: New U-boot image format

[Timur Tabi]

Wolfgang Denk wrote:

> NAK. Using an envuironment variable is definitely not  a  good  idea,
> and  probably  not  sufficient.  Assume  the situation where you have
> "autostart" set to "yes" and use  DHCP  to  download  and  start  the
> image,  or  where  you use "tftp" followed by a "bootm" command to do
> the same. In both cases there is no direct "hook" to do what you  ask
> for  - ther eis no sane place where the envrionment variable would be
> set (in the network code?) nor where it would be taken into account.

If I have a plug-in for QE firmware blobs, couldn't the plug-in code just set 
the environment variable anyway?

-- 
Timur Tabi
Linux kernel developer at Freescale

[U-Boot-Users] RFC: New U-boot image format

[Wolfgang Denk]

In message <4767F28C.3030304@freescale.com> you wrote:
> 
> > NAK. Using an envuironment variable is definitely not  a  good  idea,
> > and  probably  not  sufficient.  Assume  the situation where you have
> > "autostart" set to "yes" and use  DHCP  to  download  and  start  the
> > image,  or  where  you use "tftp" followed by a "bootm" command to do
> > the same. In both cases there is no direct "hook" to do what you  ask
> > for  - ther eis no sane place where the envrionment variable would be
> > set (in the network code?) nor where it would be taken into account.
> 
> If I have a plug-in for QE firmware blobs, couldn't the plug-in code just set 
> the environment variable anyway?

What good would this do to you? If you  run  a  "dhcp"  command  with
autostart  enabled,  the image will immediately be autobooted - there
will be no place for you to issue a command to do anything with  your
environment variable.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Horses just naturally have mohawk haircuts.

[U-Boot-Users] RFC: New U-boot image format

[Timur Tabi]

Wolfgang Denk wrote:

> What good would this do to you? If you  run  a  "dhcp"  command  with
> autostart  enabled,  the image will immediately be autobooted - there
> will be no place for you to issue a command to do anything with  your
> environment variable.

Wouldn't it be better to run the 'bootcmd' variable?  That's how U-Boot has 
always autobooted.  I'm sure there are boards out there that need to run some 
commands before booting, and they use bootcmd to do that.

-- 
Timur Tabi
Linux kernel developer at Freescale

[U-Boot-Users] RFC: New U-boot image format

[Wolfgang Denk]

In message <476823D1.6040104@freescale.com> you wrote:
> Wolfgang Denk wrote:
> 
> > What good would this do to you? If you  run  a  "dhcp"  command  with
> > autostart  enabled,  the image will immediately be autobooted - there
> > will be no place for you to issue a command to do anything with  your
> > environment variable.
> 
> Wouldn't it be better to run the 'bootcmd' variable?  That's how U-Boot has 
> always autobooted.  I'm sure there are boards out there that need to run some 
> commands before booting, and they use bootcmd to do that.

Maybe that's how U-Boot has beeen booted for you in your  setup.  But
it's  just  one  of a plethora of options, including alternative boot
commands for  example  selected  by  failed  POST  or  exceeded  boot
counters, etc.

And btw: that's what I'm describing all the time. Consider the
situation where your environment has the following two variable
settings:

	autostart=yes
	bootcmd=dhcp

Which part of the code  would  you  expect  to  set  the  environment
variable  (i.  e.  in  which  part  would  you place the hook for the
plugin), and where would you add any code  to  take  notice  of  this
variable setting?


To mee it seems logical to add the hooks to the "bootm" command code,
which will have to figure out what to do with the individual parts of
the multifile image, and then setting a  variable  is  obviously  not
sufficient - you must run some code then.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Either one of us, by himself, is expendable.  Both of us are not.
	-- Kirk, "The Devil in the Dark", stardate 3196.1

[U-Boot-Users] RFC: New U-boot image format

[Timur Tabi]

Wolfgang Denk wrote:

> To mee it seems logical to add the hooks to the "bootm" command code,
> which will have to figure out what to do with the individual parts of
> the multifile image, and then setting a  variable  is  obviously  not
> sufficient - you must run some code then.

Ok, I get it.

-- 
Timur Tabi
Linux kernel developer at Freescale

[U-Boot-Users] RFC: New U-boot image format

[Marian Balakowicz]

Timur Tabi wrote:
> Marian Balakowicz wrote:
>>
>> New format for U-boot images has been on the list few times already.
>> There were different ideas and features discussed but no final
>> conclusion has been made.
> 
> I have a request for a new feature.  I think we need image-format
> plug-ins. That is, we need a way for a plug-in to register itself with
> the main format processing code.  When the processing code sees a blob
> that it doesn't understand, it calls the plug-in to handle it.

The hooks idea sounds reasonable but, the problem
is that there isn't really any such thing like
"main fromat processing code".

The 'bootm' command code does a lot of image
related manipulation but it's not generic or main
format processing code. It is rather a specialized
command that handles one thing, starting
executables, like kernels or standalone
applications. And it's a command, it will not do
anything unless directly called.

So, having hooks in bootm, you would need to call
bootm and then your 'qe fw' command which is sort
of wired.

There are also other commands that deal with the
images, these are other boot commands like
scsiboot, usbboot, etc, and also imls, imxtract.
And all of them process images directly, finding
necessary information on their own.

Your command seem to be of the same type, it needs
information from the uImage (soon from the New
uImage) and it knows best what it needs.
IMHO, reasonable is to make it get this
information from New uImage directly.

Your current syntax is:
> qe fw <addr>

Extended syntax could be:
qe fw 200000:fw_image at 1

Where 200000 is the address of New uImage and
"fw_image at 1" is a example node name of your blob
data. The New uImage API will allow, in simple way
(one, two calls),  to get other details for a
given image, so you would be able to get the blob
data start address.

More on the New uImage format soon, stay tuned.

> This would be a handy way to handle stuff like the QE firmware binary
> blob format (see the thread titled "[PATCH] 85xx: add ability to upload
> QE firmware").  The format of the QE firmware blob has already been
> decided, so all I would need from the multi-image format is:
> 
> 1) A way to package my blob.
> 2) A way to pass the address of the blob to the QE code
> 
> For option #2, setting an environment variable would be the easiest.  To
> do that, the QE code could register a call-back that says, "If you see a
> QE firmware blob, call this function and pass the address of the blob".

Cheers,
Marian

[U-Boot-Users] RFC: New U-boot image format

[Marian Balakowicz]

Marian Balakowicz wrote:
[...]
>>> 3. 'New image' format must support the following features:
>> [snip]
>>
>>> 	- 'container' image blob shall include 'component' images'
>>> 	  data, which means direct data embedding  - as opposed to
>>> 	  having only references
>> Q for Jon L: This would require an extension to the dtc to "include" a 
>> raw file into the blob?  I'm presuming that we don't want to take a 
>> binary (ELF) file, turn it into ASCII bytes, include it into a dts, and 
>> then use dtc to compile it back into binary.
>>
>> Am I missing something that is already available?  Do you see any 
>> problems with extending dtc to support this?
> 
> AFAIK dtc currently has no support for data includes. I've seen such
> feature on a dtc wish list though, so adding it should not be troublesome.

Jon, David,

Are you aware if anyone is working on a binary includes for dtc?
This feature will be needed for U-boot New uImage format implementation,
so I wonder what is the actual status of this dtc wish list item.

Thanks,
Marian

[U-Boot-Users] RFC: New U-boot image format

[David Gibson]

On Wed, Dec 19, 2007 at 03:26:11PM +0100, Marian Balakowicz wrote:
> Marian Balakowicz wrote:
> [...]
> >>> 3. 'New image' format must support the following features:
> >> [snip]
> >>
> >>> 	- 'container' image blob shall include 'component' images'
> >>> 	  data, which means direct data embedding  - as opposed to
> >>> 	  having only references
> >> Q for Jon L: This would require an extension to the dtc to "include" a 
> >> raw file into the blob?  I'm presuming that we don't want to take a 
> >> binary (ELF) file, turn it into ASCII bytes, include it into a dts, and 
> >> then use dtc to compile it back into binary.
> >>
> >> Am I missing something that is already available?  Do you see any 
> >> problems with extending dtc to support this?
> > 
> > AFAIK dtc currently has no support for data includes. I've seen such
> > feature on a dtc wish list though, so adding it should not be troublesome.
> 
> Jon, David,
> 
> Are you aware if anyone is working on a binary includes for dtc?
> This feature will be needed for U-boot New uImage format implementation,
> so I wonder what is the actual status of this dtc wish list item.

I don't believe anyone's looked at it so far.  It really shouldn't
take more than a couple of days once Jon or I gets to thinking about
it.  The trickiest bit is just thinking of a good syntax for it.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[U-Boot-Users] RFC: New U-boot image format

[Scott Wood]

David Gibson wrote:
> On Wed, Dec 19, 2007 at 03:26:11PM +0100, Marian Balakowicz wrote:
>> Are you aware if anyone is working on a binary includes for dtc? 
>> This feature will be needed for U-boot New uImage format
>> implementation, so I wonder what is the actual status of this dtc
>> wish list item.
> 
> I don't believe anyone's looked at it so far.

I've got a patch that I'll post later today.

> It really shouldn't take more than a couple of days once Jon or I
> gets to thinking about it.  The trickiest bit is just thinking of a
> good syntax for it.

I went with this:
	prop = /bin-include/ "/path/name";


I'm open to other suggestions, of course.

-Scott

[U-Boot-Users] RFC: New U-boot image format

[Wolfgang Denk]

In message <4769258D.6090709@semihalf.com> you wrote:
>
> The hooks idea sounds reasonable but, the problem
> is that there isn't really any such thing like
> "main fromat processing code".
> 
> The 'bootm' command code does a lot of image
> related manipulation but it's not generic or main
> format processing code. It is rather a specialized

But it's *the* generic command for booting a Linux kernel.

> command that handles one thing, starting
> executables, like kernels or standalone
> applications. And it's a command, it will not do
> anything unless directly called.

Correct.

> So, having hooks in bootm, you would need to call
> bootm and then your 'qe fw' command which is sort
> of wired.

MY current understandign is, that when  running  "bootm",  the  bootm
command  would detect if there are registred hook routines for any of
the contained sub-images, and run that code automagically.

> There are also other commands that deal with the
> images, these are other boot commands like
> scsiboot, usbboot, etc, and also imls, imxtract.

scsiboot, usbboot etc. are just means to load an image into RAM,
similar to tftpboot or dhcpboot. If they also boot the loaded image
(because autostart is set to yes), then they do so by calling bootm
internally. So everything is fine.

imls would probably have it's ownset of image type hooks - while  you
would  register some "boot me" type of handlers with bootm, you would
register some "display me" type of handlers with imls.

imxtract will have to be changed completely in any case.

> And all of them process images directly, finding
> necessary information on their own.

This is no argument for not  creating  some  more  centralized  image
processing code, or is it?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
By the way, ALL software projects are done by iterative  prototyping.
Some companies call their prototypes "releases", that's all.

[U-Boot-Users] RFC: New U-boot image format

[Marian Balakowicz]

Scott Wood wrote:
> David Gibson wrote:
>> On Wed, Dec 19, 2007 at 03:26:11PM +0100, Marian Balakowicz wrote:
>>> Are you aware if anyone is working on a binary includes for dtc? This
>>> feature will be needed for U-boot New uImage format
>>> implementation, so I wonder what is the actual status of this dtc
>>> wish list item.
>>
>> I don't believe anyone's looked at it so far.
> 
> I've got a patch that I'll post later today.

Good news!

> 
>> It really shouldn't take more than a couple of days once Jon or I
>> gets to thinking about it.  The trickiest bit is just thinking of a
>> good syntax for it.
> 
> I went with this:
>     prop = /bin-include/ "/path/name";
> 
> 
> I'm open to other suggestions, of course.

I was thinking of a syntax that would be similar to regular data
sections which are enclosed in [] braces. Something like:

prop = {/path/to/file};
or
prop = [file:/path/to/file];

Thanks,
Marian

[U-Boot-Users] RFC: New U-boot image format

[Scott Wood]

Marian Balakowicz wrote:
> I was thinking of a syntax that would be similar to regular data
> sections which are enclosed in [] braces. Something like:
> 
> prop = {/path/to/file};

This one overloads {} based primarily on whether there's an equal sign 
before it... a bit icky.  Plus, sooner or later we'll run out of special 
characters to assign meaning to, and we'll become Perlish in the 
process. :-)

I went with text-between slashes as it's an already-established keyword 
marker.

> or prop = [file:/path/to/file];

This is ambiguous; file: will be scanned as a label unless we give 
precedence to it as a keyword, and I don't think we want to go down the 
path of reserved words when we have other mechanisms available to 
separate the namespaces.

-Scott

[U-Boot-Users] RFC: New U-boot image format

[David Gibson]

On Thu, Dec 20, 2007 at 04:39:38PM -0600, Scott Wood wrote:
> Marian Balakowicz wrote:
>> I was thinking of a syntax that would be similar to regular data
>> sections which are enclosed in [] braces. Something like:
>> prop = {/path/to/file};
>
> This one overloads {} based primarily on whether there's an equal sign 
> before it... a bit icky.  Plus, sooner or later we'll run out of special 
> characters to assign meaning to, and we'll become Perlish in the process. 
> :-)

Indeed.  Plus I'd liked to keep { } consistently for delimiting
lexical contexts which will accept node or property names without
quoting.

Plus you can't even count on the { being after an =, because you could have:

	prop = "a string followed by", {/a/file};

> I went with text-between slashes as it's an already-established keyword 
> marker.
>
>> or prop = [file:/path/to/file];
>
> This is ambiguous; file: will be scanned as a label unless we give 
> precedence to it as a keyword, and I don't think we want to go down the 
> path of reserved words when we have other mechanisms available to separate 
> the namespaces.

And it's gratuitously similar to bytestring notation.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[U-Boot-Users] RFC: New U-boot image format

[Marian Balakowicz]

Wolfgang Denk wrote:
> In message <4769258D.6090709@semihalf.com> you wrote:
>> The hooks idea sounds reasonable but, the problem
>> is that there isn't really any such thing like
>> "main fromat processing code".
>>
>> The 'bootm' command code does a lot of image
>> related manipulation but it's not generic or main
>> format processing code. It is rather a specialized
> 
> But it's *the* generic command for booting a Linux kernel.

Right.

[...]
> imls would probably have it's ownset of image type hooks - while  you
> would  register some "boot me" type of handlers with bootm, you would
> register some "display me" type of handlers with imls.
> 
> imxtract will have to be changed completely in any case.
> 
>> And all of them process images directly, finding
>> necessary information on their own.
> 
> This is no argument for not  creating  some  more  centralized  image
> processing code, or is it?

No, it's not. But I have doubts whether bootm is a good place for it.
Maybe we should rather go for new generic command, something like
'imrun' or similar. And maybe even, move standalone application
execution from bootm to a dedicated hook that'll be available from
'imrun'. This way, bootm would become strict OS booting command,
potentially also available from 'imrun' via a hook.

How that sounds?

Cheers,
Marian

[U-Boot-Users] RFC: New U-boot image format

[Wolfgang Denk]

In message <476BD5E9.3030601@semihalf.com> you wrote:
>
> > This is no argument for not  creating  some  more  centralized  image
> > processing code, or is it?
> 
> No, it's not. But I have doubts whether bootm is a good place for it.
> Maybe we should rather go for new generic command, something like
> 'imrun' or similar. And maybe even, move standalone application

It certainly makes sense to factor out this code, but I'm not sure yet
if a separate command will be needed.

> execution from bootm to a dedicated hook that'll be available from
> 'imrun'. This way, bootm would become strict OS booting command,

But "bootm" already *is* a strict OS booting command.

I'm not sure what  youre  reference  to  standalone  applications  is
supposed  to  mean  in  this contexzt - but SA apps are NOT booted by
bootm, they get started using the "go" command.

> potentially also available from 'imrun' via a hook.

...or vice versa.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
But the only way of discovering the limits  of  the  possible  is  to
venture a little way past them into the impossible.
                         - _Profiles of the Future_ (1962; rev. 1973)
                  ``Hazards of Prophecy: The Failure of Imagination''

[U-Boot-Users] RFC: New U-boot image format

[Marian Balakowicz]

Wolfgang Denk wrote:
> In message <476BD5E9.3030601@semihalf.com> you wrote:
>>> This is no argument for not  creating  some  more  centralized  image
>>> processing code, or is it?
>> No, it's not. But I have doubts whether bootm is a good place for it.
>> Maybe we should rather go for new generic command, something like
>> 'imrun' or similar. And maybe even, move standalone application
> 
> It certainly makes sense to factor out this code, but I'm not sure yet
> if a separate command will be needed.
> 
>> execution from bootm to a dedicated hook that'll be available from
>> 'imrun'. This way, bootm would become strict OS booting command,
> 
> But "bootm" already *is* a strict OS booting command.
> 
> I'm not sure what  youre  reference  to  standalone  applications  is
> supposed  to  mean  in  this contexzt - but SA apps are NOT booted by
> bootm, they get started using the "go" command.

What I meant was that bootm accepts also uImages of type
IH_TYPE_STANDALONE, which corresponds to standalone applications. If
such uImage is passed to bootm and autostart is not set to 'no', bootm
will execute it. That kind of image processing would perfectly fit
hooks framework.

Cheers,
Marian