[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Michael Schwingen]

Hi,

to conclude the discussion in the thread "Re: [U-Boot] Update and Cut down
mach types", I tried a short patch that demonstrates how to automatically
generate the mach-types.h file from a database dump (from
http://www.arm.linux.org.uk/developer/machines/?action=new).

This has multiple advantages:
 - pulling in new machine types is easier (drop in a new downloaded database
   dump), and produces a much smaller diff.
 - adding new machines is decoupled from the time they appear in Linux.
 - boards that are not in mainline Linux will not be break due to removal of
   their mach types from the Linux headers.

The AWK and Makefile fragment script is taken verbatim from Linux 2.6.38.3. 
I think the AWK script is simple enough that it will not require big
maintenance efforts (unless the machine database format changes).

The patch is edited down - I removed the diff for the deletion of the old
mach-types.h file, and shortened the new mach-types file to a few entries
just to show the concept - otherwise, the patch would be much too big for
the list.

Is this an acceptable solution? Should I go on and produce a full-fledged
patch?

cu
Michael

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Mike Frysinger]

On Thu, May 5, 2011 at 17:48, Michael Schwingen wrote:
> --- a/Makefile
> +++ b/Makefile
> @@ -469,7 +469,7 @@ $(obj)System.map: ? $(obj)u-boot
> ?# This target actually generates 2 files; autoconf.mk and autoconf.mk.dep.
> ?# the dep file is only include in this top level makefile to determine when
> ?# to regenerate the autoconf.mk file.
> -$(obj)include/autoconf.mk.dep: $(obj)include/config.h include/common.h
> +$(obj)include/autoconf.mk.dep: $(obj)include/config.h include/common.h $(obj)include/asm/mach-types.h
> ? ? ? ?@$(XECHO) Generating $@ ; \
> ? ? ? ?set -e ; \
> ? ? ? ?: Generate the dependancies ; \
> @@ -530,13 +530,18 @@ unconfig:
> ? ? ? ? ? ? ? ?$(obj)board/*/config.tmp $(obj)board/*/*/config.tmp \
> ? ? ? ? ? ? ? ?$(obj)include/autoconf.mk $(obj)include/autoconf.mk.dep
>
> -%_config:: ? ? unconfig
> +%_config:: ? ? unconfig $(obj)include/asm/mach-types.h
> ? ? ? ?@$(MKCONFIG) -A $(@:_config=)
>
> ?sinclude $(obj).boards.depend
> ?$(obj).boards.depend: ?boards.cfg
> ? ? ? ?awk '(NF && $$1 !~ /^#/) { print $$1 ": " $$1 "_config; $$(MAKE)" }' $< > $@
>
> +
> +$(obj)include/asm/mach-types.h: arch/arm/tools/gen-mach-types arch/arm/tools/mach-types
> + ? ? ? @mkdir -p $(obj)include/asm
> + ? ? ? awk -f $^ > $@ || { rm -f $@; /bin/false; }
> +
> ?#
> ?# Functions to generate common board directory names
> ?#

this all belongs in arch/arm/config.mk and not the toplevel makefile

also, dont hardcode full paths to things.  there's no reason for it.

might want to add an "update-mach-types" target so people can type
`make update-mach-types` and it'll automatically wget the right file
to the right place ...
-mike

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Wolfgang Denk]

Dear Michael Schwingen,

In message <20110505214836.GA5313@discworld.dascon.de> you wrote:
> 
> to conclude the discussion in the thread "Re: [U-Boot] Update and Cut down
> mach types", I tried a short patch that demonstrates how to automatically
> generate the mach-types.h file from a database dump (from
> http://www.arm.linux.org.uk/developer/machines/?action=new).
> 
> This has multiple advantages:
>  - pulling in new machine types is easier (drop in a new downloaded database
>    dump), and produces a much smaller diff.
>  - adding new machines is decoupled from the time they appear in Linux.
>  - boards that are not in mainline Linux will not be break due to removal of
>    their mach types from the Linux headers.
> 
> The AWK and Makefile fragment script is taken verbatim from Linux 2.6.38.3. 
> I think the AWK script is simple enough that it will not require big
> maintenance efforts (unless the machine database format changes).
> 
> The patch is edited down - I removed the diff for the deletion of the old
> mach-types.h file, and shortened the new mach-types file to a few entries
> just to show the concept - otherwise, the patch would be much too big for
> the list.
> 
> Is this an acceptable solution? Should I go on and produce a full-fledged
> patch?

I agree with the comments alreay made by Mike Frysinger.

In addition, I would like to point out that I consider it mandatory
thatthe normal build process does NOT involve any network download.
Instead, it must be run completely based on files checked into the git
repository only.

That means, that the "update-mach-types" target Mike mentions is
something that the ARM maintainer can use to update the in-tree copy
of the mach-types.h file, which then gets commited into the git
repository.  Also users can run this if they want to use the latest
version, but it MUST NOT be used for a regular build.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
The only way to learn a new programming language is by  writing  pro-
grams in it.                                        - Brian Kernighan

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Michael Schwingen]

Am 05/06/2011 07:09 AM, schrieb Mike Frysinger:
> On Thu, May 5, 2011 at 17:48, Michael Schwingen wrote:
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -469,7 +469,7 @@ $(obj)System.map:   $(obj)u-boot
>>  # This target actually generates 2 files; autoconf.mk and autoconf.mk.dep.
>>  # the dep file is only include in this top level makefile to determine when
>>  # to regenerate the autoconf.mk file.
>> -$(obj)include/autoconf.mk.dep: $(obj)include/config.h include/common.h
>> +$(obj)include/autoconf.mk.dep: $(obj)include/config.h include/common.h $(obj)include/asm/mach-types.h
>>        @$(XECHO) Generating $@ ; \
>>        set -e ; \
>>        : Generate the dependancies ; \
>> @@ -530,13 +530,18 @@ unconfig:
>>                $(obj)board/*/config.tmp $(obj)board/*/*/config.tmp \
>>                $(obj)include/autoconf.mk $(obj)include/autoconf.mk.dep
>>
>> -%_config::     unconfig
>> +%_config::     unconfig $(obj)include/asm/mach-types.h
>>        @$(MKCONFIG) -A $(@:_config=)
>>
>>  sinclude $(obj).boards.depend
>>  $(obj).boards.depend:  boards.cfg
>>        awk '(NF && $$1 !~ /^#/) { print $$1 ": " $$1 "_config; $$(MAKE)" }' $< > $@
>>
>> +
>> +$(obj)include/asm/mach-types.h: arch/arm/tools/gen-mach-types arch/arm/tools/mach-types
>> +       @mkdir -p $(obj)include/asm
>> +       awk -f $^ > $@ || { rm -f $@; /bin/false; }
>> +
>>  #
>>  # Functions to generate common board directory names
>>  #
> this all belongs in arch/arm/config.mk and not the toplevel makefile
Fine with me, however, I am hitting a big problem with the Makefile
structure:

mach-types.h needs to be built before autoconf.mk can be generated, and
the rules for autoconf.mk are in the top-level Makefile.

If I put rules in arch/arm/config.mk, then the first of these rules
becomes the default rule which is executed in subdir makes (like "make
-C arch"), which breaks compilation completely, since config.mk is
included before the rules are defined in the subdir Makefiles.

It seems the current scheme allows only variable definitions in
config.mk files, which is not sufficient here.

rules.mk would be fine, however, there is no provision to include rules
from lower directories, since all the building in subdirectories is
handled by recursively calling make (this is one of the problems that
arise by recursively calling make for each directory, but that is a
different topic).

Any ideas? Using the current Makefile structure, I see no other solution
than defining the mach-types.h generation rules in the toplevel Makefile.

> also, dont hardcode full paths to things.  there's no reason for it.
Which of these can be omitted?

When putting the rules in arch/arm/config.mk, I can make them trigger
only when compiling for ARM, and I can use $(ARCH) instead of  arm/, but
apart from that, I think I need to specify the paths, no?

> might want to add an "update-mach-types" target so people can type
> `make update-mach-types` and it'll automatically wget the right file
> to the right place ...
Good idea.
I agree with Wolfgang that this is intended to be used by the maintainer
mainly, so the mach-types file should be included in the source so that
a normal user does not need to download the file.

cu
Michael

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Albert ARIBAUD]

Hi Wolfgang,

Le 06/05/2011 12:14, Wolfgang Denk a ?crit :
> Dear Michael Schwingen,
>
> In message<20110505214836.GA5313@discworld.dascon.de>  you wrote:
>>
>> to conclude the discussion in the thread "Re: [U-Boot] Update and Cut down
>> mach types", I tried a short patch that demonstrates how to automatically
>> generate the mach-types.h file from a database dump (from
>> http://www.arm.linux.org.uk/developer/machines/?action=new).
>>
>> This has multiple advantages:
>>   - pulling in new machine types is easier (drop in a new downloaded database
>>     dump), and produces a much smaller diff.
>>   - adding new machines is decoupled from the time they appear in Linux.
>>   - boards that are not in mainline Linux will not be break due to removal of
>>     their mach types from the Linux headers.
>>
>> The AWK and Makefile fragment script is taken verbatim from Linux 2.6.38.3.
>> I think the AWK script is simple enough that it will not require big
>> maintenance efforts (unless the machine database format changes).
>>
>> The patch is edited down - I removed the diff for the deletion of the old
>> mach-types.h file, and shortened the new mach-types file to a few entries
>> just to show the concept - otherwise, the patch would be much too big for
>> the list.
>>
>> Is this an acceptable solution? Should I go on and produce a full-fledged
>> patch?
>
> I agree with the comments alreay made by Mike Frysinger.
>
> In addition, I would like to point out that I consider it mandatory
> thatthe normal build process does NOT involve any network download.
> Instead, it must be run completely based on files checked into the git
> repository only.
>
> That means, that the "update-mach-types" target Mike mentions is
> something that the ARM maintainer can use to update the in-tree copy
> of the mach-types.h file, which then gets commited into the git
> repository.  Also users can run this if they want to use the latest
> version, but it MUST NOT be used for a regular build.

Agreed on all points.

> Best regards,
>
> Wolfgang Denk

Amicalement,
-- 
Albert.

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Wolfgang Denk]

Dear Michael Schwingen,

In message <4DC5259C.7040208@discworld.dascon.de> you wrote:
>
> If I put rules in arch/arm/config.mk, then the first of these rules
> becomes the default rule which is executed in subdir makes (like "make
> -C arch"), which breaks compilation completely, since config.mk is
> included before the rules are defined in the subdir Makefiles.

You are not suppoed to put any make rules in config.mk files.

> It seems the current scheme allows only variable definitions in
> config.mk files, which is not sufficient here.

As the name implies, these files contain configuration (= variable)
settings. Nothing else.

> rules.mk would be fine, however, there is no provision to include rules
> from lower directories, since all the building in subdirectories is
> handled by recursively calling make (this is one of the problems that
> arise by recursively calling make for each directory, but that is a
> different topic).

But lower level directories inherit all settings from the top level
Makefile?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
It's a small world, but I wouldn't want to paint it.

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Michael Schwingen]

Am 05/09/2011 07:37 AM, schrieb Wolfgang Denk:
> Dear Michael Schwingen,
>
> In message <4DC5259C.7040208@discworld.dascon.de> you wrote:
>> If I put rules in arch/arm/config.mk, then the first of these rules
>> becomes the default rule which is executed in subdir makes (like "make
>> -C arch"), which breaks compilation completely, since config.mk is
>> included before the rules are defined in the subdir Makefiles.
> You are not suppoed to put any make rules in config.mk files.
>
>> It seems the current scheme allows only variable definitions in
>> config.mk files, which is not sufficient here.
> As the name implies, these files contain configuration (= variable)
> settings. Nothing else.
>
>> rules.mk would be fine, however, there is no provision to include rules
>> from lower directories, since all the building in subdirectories is
>> handled by recursively calling make (this is one of the problems that
>> arise by recursively calling make for each directory, but that is a
>> different topic).
> But lower level directories inherit all settings from the top level
> Makefile?
Yes.
However, the rule to generate autoconf.mk is in the toplevel Makefile,
and I need the rule to generate mach-types.h at the same level, as
autoconf.mk depends on it. The lower-level *Makefiles* are executed too
late to generate anything that is needed for autoconf.mk.

cu
Michael

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Mike Frysinger]

On Saturday, May 07, 2011 06:57:32 Michael Schwingen wrote:
> mach-types.h needs to be built before autoconf.mk can be generated, and
> the rules for autoconf.mk are in the top-level Makefile.
> 
> If I put rules in arch/arm/config.mk, then the first of these rules
> becomes the default rule which is executed in subdir makes (like "make
> -C arch"), which breaks compilation completely, since config.mk is
> included before the rules are defined in the subdir Makefiles.
> 
> It seems the current scheme allows only variable definitions in
> config.mk files, which is not sufficient here.
> 
> rules.mk would be fine, however, there is no provision to include rules
> from lower directories, since all the building in subdirectories is
> handled by recursively calling make (this is one of the problems that
> arise by recursively calling make for each directory, but that is a
> different topic).
> 
> Any ideas? Using the current Makefile structure, I see no other solution
> than defining the mach-types.h generation rules in the toplevel Makefile.

do you need the mach-types file for anything else ?  if not, dont keep that in 
git, keep the generated header.

> > also, dont hardcode full paths to things.  there's no reason for it.
> 
> Which of these can be omitted?

look for "/bin/" or "/sbin/".  if your patch contains those, it is wrong.

> > might want to add an "update-mach-types" target so people can type
> > `make update-mach-types` and it'll automatically wget the right file
> > to the right place ...
> 
> Good idea.
> I agree with Wolfgang that this is intended to be used by the maintainer
> mainly, so the mach-types file should be included in the source so that
> a normal user does not need to download the file.

i didnt suggest this in the first place
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20110509/22259402/attachment.pgp 

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Michael Schwingen]

Mike Frysinger wrote:
> On Saturday, May 07, 2011 06:57:32 Michael Schwingen wrote:
>   
>> mach-types.h needs to be built before autoconf.mk can be generated, and
>> the rules for autoconf.mk are in the top-level Makefile.
>>
>> If I put rules in arch/arm/config.mk, then the first of these rules
>> becomes the default rule which is executed in subdir makes (like "make
>> -C arch"), which breaks compilation completely, since config.mk is
>> included before the rules are defined in the subdir Makefiles.
>>
>> It seems the current scheme allows only variable definitions in
>> config.mk files, which is not sufficient here.
>>
>> rules.mk would be fine, however, there is no provision to include rules
>> from lower directories, since all the building in subdirectories is
>> handled by recursively calling make (this is one of the problems that
>> arise by recursively calling make for each directory, but that is a
>> different topic).
>>
>> Any ideas? Using the current Makefile structure, I see no other solution
>> than defining the mach-types.h generation rules in the toplevel Makefile.
>>     
>
> do you need the mach-types file for anything else ?  if not, dont keep that in 
> git, keep the generated header.
>   
Then we are back at basically the current state.
If I do this, we need no Makefile support at all - we just need a script 
that is run by the maintainer that downloads the current mach-types from 
the web, and generates mach-types.h, which is then checked in.

cu
Michael

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Mike Frysinger]

On Monday, May 09, 2011 11:54:15 Michael Schwingen wrote:
> Then we are back at basically the current state.
> If I do this, we need no Makefile support at all - we just need a script
> that is run by the maintainer that downloads the current mach-types from
> the web, and generates mach-types.h, which is then checked in.

i'm ok with that, but i'm not an arm maintainer ;)
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20110509/da447596/attachment.pgp 

[U-Boot] RFC: auto-generate ARM mach-types.h file from ARM machine database

[Albert ARIBAUD]

Hi all,

Le 09/05/2011 18:16, Mike Frysinger a ?crit :
> On Monday, May 09, 2011 11:54:15 Michael Schwingen wrote:
>> Then we are back at basically the current state.
>> If I do this, we need no Makefile support at all - we just need a script
>> that is run by the maintainer that downloads the current mach-types from
>> the web, and generates mach-types.h, which is then checked in.
>
> i'm ok with that, but i'm not an arm maintainer ;)
> -mike

I'm the ARM maintainer -- custodian, actually -- and I approve this 
message. :)

Amicalement,
-- 
Albert.