[U-Boot] [RFC] Build errors in u-boot mainline and daily builds

[U-Boot] [RFC] Build errors in u-boot mainline and daily builds

[Remy Bohmer]

Hello All,

I started a server to do only daily builds of all U-boot board
configurations. (currently on a old slow box, but it works, it takes
about 2 hours to build all boards for ARM only)

The primary goal was to find early compile regressions in the
u-boot-USB tree that I maintain, before pushing patches to mainline.
To find a regression compared to u-boot mainline, the u-boot mainline
(master branch) also needs to build, so I build that tree also daily
from now.

Currently I only build for ARM based boards, but there is no limit on
architectures, if I have all the cross-compilers.
I use the Denx ELDK for ARM as cross-compile toolchain.

First, I discovered some build failures of which the logging is listed below.
What can we do about these errors? Are these boards still maintained?
Should these be fixed? Who wants to fix them?
I currently run the 'MAKEALL arm' script to build all these boards, so
either the boards should be fixed, or if no longer maintained removed
(at least from the MAKEALL script)

Further, is there any interest to make the daily build results public?
* Is it appreciated if I post the build results in a daily post to the
mailinglist, _if_ there is a build failure? (No news is good news)
What format is preferred?
* Or is it preferred to post the results on a website?
* Or ...

Working it up from here to make it public will take me some time, so I
will only do that if there is any serious interest.


Kind Regards,

Remy


Here are the build errors of last night build:
------------------------------------------------------------
Building U-boot:
* branch: master
* src-dir: /home/remy/nightbuild/git/u-boot
------------------------------------------------------------
Configuring for m501sk board...
cpu/arm920t/at91rm9200/libat91rm9200.a(lowlevel_init.o): In function `SMRDATA':
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `MC_PUIA_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `MC_PUP_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `MC_PUER_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `MC_ASR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `MC_AASR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `EBI_CFGR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SMC_CSR0_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `PLLAR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `PLLBR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `MCKR_VAL'
cpu/arm920t/at91rm9200/libat91rm9200.a(lowlevel_init.o): In function `SMRDATA1':
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `PIOC_ASR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `PIOC_BSR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `PIOC_PDR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `EBI_CSA_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRC_CR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRC_MR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRC_MR_VAL1'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRC_MR_VAL2'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM1'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRC_TR_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRC_MR_VAL3'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM'
/home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
undefined reference to `SDRAM_VAL'
make: *** [/home/remy/nightbuild/scratch/build/u-boot] Error 1


Configuring for actux4 board...
actux4.c: In function 'board_init':
actux4.c:83: warning: left shift count >= width of type
actux4.c:83: warning: left shift count >= width of type
   text	   data	    bss	    dec	    hex	filename
 222576	   6692	 550812	 780080	
be730	/home/remy/nightbuild/scratch/build/u-boot


Configuring for ixdp425 board...
make[1]: *** No rule to make target
`/home/remy/nightbuild/scratch/build/cpu/ixp/npe/IxNpeMicrocode.o',
needed by `/home/remy/nightbuild/scratch/build/cpu/ixp/npe/libnpe.a'.
Stop.
make: *** [/home/remy/nightbuild/scratch/build/cpu/ixp/npe/libnpe.a] Error 2


Configuring for ixdpg425 board...
make[1]: *** No rule to make target
`/home/remy/nightbuild/scratch/build/cpu/ixp/npe/IxNpeMicrocode.o',
needed by `/home/remy/nightbuild/scratch/build/cpu/ixp/npe/libnpe.a'.
Stop.
make: *** [/home/remy/nightbuild/scratch/build/cpu/ixp/npe/libnpe.a] Error 2


Configuring for pdnb3 board...
make[1]: *** No rule to make target
`/home/remy/nightbuild/scratch/build/cpu/ixp/npe/IxNpeMicrocode.o',
needed by `/home/remy/nightbuild/scratch/build/cpu/ixp/npe/libnpe.a'.
Stop.
make: *** [/home/remy/nightbuild/scratch/build/cpu/ixp/npe/libnpe.a] Error 2
... on SCPU board variant


Configuring for pdnb3 board...
make[1]: *** No rule to make target
`/home/remy/nightbuild/scratch/build/cpu/ixp/npe/IxNpeMicrocode.o',
needed by `/home/remy/nightbuild/scratch/build/cpu/ixp/npe/libnpe.a'.
Stop.
make: *** [/home/remy/nightbuild/scratch/build/cpu/ixp/npe/libnpe.a] Error 2

[U-Boot] [RFC] Build errors in u-boot mainline and daily builds

[Mike Frysinger]

On Saturday 03 January 2009 06:10:24 Remy Bohmer wrote:
> Currently I only build for ARM based boards, but there is no limit on
> architectures, if I have all the cross-compilers.

the Blackfin tree is already compile tested daily, so i wouldnt waste cycles 
on doing those ... failures get published to the Blackfin u-boot development 
list.

> Further, is there any interest to make the daily build results public?
> * Is it appreciated if I post the build results in a daily post to the
> mailinglist, _if_ there is a build failure? (No news is good news)
> What format is preferred?
> * Or is it preferred to post the results on a website?
> * Or ...

having them public is good, but i'd find posting them to the normal u-boot 
mailing list annoying ... perhaps we should start up another u-boot mailing 
list just to publish these kind of results ?
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20090103/a3514589/attachment.pgp 

[U-Boot] [RFC] Build errors in u-boot mainline and daily builds

[Wolfgang Denk]

Dear Remy,

In message <3efb10970901030310r7cfc1245g97ac0db4375ddc33@mail.gmail.com> you wrote:
> 
> I started a server to do only daily builds of all U-boot board
> configurations. (currently on a old slow box, but it works, it takes
> about 2 hours to build all boards for ARM only)

I appreciate your efforts. Actually this is  something  everybody  is
supposed  to  do  before  submitting  a  patch,  and  especially  the
custodians before submitting a pull request.

Unfortually, that's theory only :-(

> Currently I only build for ARM based boards, but there is no limit on
> architectures, if I have all the cross-compilers.
> I use the Denx ELDK for ARM as cross-compile toolchain.
> 
> First, I discovered some build failures of which the logging is listed below.
> What can we do about these errors? Are these boards still maintained?
> Should these be fixed? Who wants to fix them?

These are questions for the ARM custodian to answer.

The problems are well known - I reported these several times before.

> Further, is there any interest to make the daily build results public?
> * Is it appreciated if I post the build results in a daily post to the
> mailinglist, _if_ there is a build failure? (No news is good news)
> What format is preferred?

Please do not post such results here.

> * Or is it preferred to post the results on a website?

Yes, that would be beter, IMO.


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"Text processing has made it possible to right-justify any idea, even
one which cannot be justified on any other grounds."
                                                 -- J. Finnegan, USC.

[U-Boot] [RFC] Build errors in u-boot mainline and daily builds

[Jean-Christophe PLAGNIOL-VILLARD]

On 12:10 Sat 03 Jan     , Remy Bohmer wrote:
> Hello All,
> 
> I started a server to do only daily builds of all U-boot board
> configurations. (currently on a old slow box, but it works, it takes
> about 2 hours to build all boards for ARM only)
> 
> The primary goal was to find early compile regressions in the
> u-boot-USB tree that I maintain, before pushing patches to mainline.
> To find a regression compared to u-boot mainline, the u-boot mainline
> (master branch) also needs to build, so I build that tree also daily
> from now.
> 
> Currently I only build for ARM based boards, but there is no limit on
> architectures, if I have all the cross-compilers.
> I use the Denx ELDK for ARM as cross-compile toolchain.
> 
> First, I discovered some build failures of which the logging is listed below.
> What can we do about these errors? Are these boards still maintained?
> Should these be fixed? Who wants to fix them?
> I currently run the 'MAKEALL arm' script to build all these boards, so
> either the boards should be fixed, or if no longer maintained removed
> (at least from the MAKEALL script)
I do the same weekly and on patch commit on arm and other arch as ppc, mips, sh4

for ARM I use different toolchains

> 
> Further, is there any interest to make the daily build results public?
> * Is it appreciated if I post the build results in a daily post to the
> mailinglist, _if_ there is a build failure? (No news is good news)
> What format is preferred?
> * Or is it preferred to post the results on a website?
> * Or ...
a mail notice to the custodian will nice on new build issue
and a website

> ------------------------------------------------------------
> Building U-boot:
> * branch: master
> * src-dir: /home/remy/nightbuild/git/u-boot
> ------------------------------------------------------------
> Configuring for m501sk board...
> cpu/arm920t/at91rm9200/libat91rm9200.a(lowlevel_init.o): In function `SMRDATA':
> /home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
> undefined reference to `MC_PUIA_VAL'
> /home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
> undefined reference to `MC_PUP_VAL'
> /home/remy/nightbuild/git/u-boot/cpu/arm920t/at91rm9200/lowlevel_init.S:132:
> undefined reference to `MC_PUER_VAL'
The problem are already known and I've send patch to fix it just before my vacation

 
> 
> Configuring for actux4 board...
> actux4.c: In function 'board_init':
> actux4.c:83: warning: left shift count >= width of type
> actux4.c:83: warning: left shift count >= width of type
>    text	   data	    bss	    dec	    hex	filename
>  222576	   6692	 550812	 780080	
> be730	/home/remy/nightbuild/scratch/build/u-boot
Known too
> 
> 
> Configuring for ixdp425 board...
> make[1]: *** No rule to make target
> `/home/remy/nightbuild/scratch/build/cpu/ixp/npe/IxNpeMicrocode.o',
> needed by `/home/remy/nightbuild/scratch/build/cpu/ixp/npe/libnpe.a'.
> Stop.
> make: *** [/home/remy/nightbuild/scratch/build/cpu/ixp/npe/libnpe.a] Error 2

we can not add the IxNpeMicrocode.c due to licencing issue
a new way has be commit to allow the FW to be load from flash instead of
built-in

Best Regards,
J.

[U-Boot] [RFC] Build errors in u-boot mainline and daily builds

[Jerry Van Baren]

Wolfgang Denk wrote:
> Dear Remy,
> 
> In message <3efb10970901030310r7cfc1245g97ac0db4375ddc33@mail.gmail.com> you wrote:
>> I started a server to do only daily builds of all U-boot board
>> configurations. (currently on a old slow box, but it works, it takes
>> about 2 hours to build all boards for ARM only)

That's still not bad.

> I appreciate your efforts. Actually this is  something  everybody  is
> supposed  to  do  before  submitting  a  patch,  and  especially  the
> custodians before submitting a pull request.
> 
> Unfortually, that's theory only :-(

An unenforceable demand. :-(  A build machine is the pragmatic solution, 
but it puts the burden on the build machine maintainer.  :-/  Thanks for 
volunteering, Remy. :-)

Side note: I've been looking at BuildBot but haven't done anything 
useful.  I've looked at CruiseControl a few times and then I see how it 
does the control logic in (haxtended) XML.  Bleah!

[snip]

>> Further, is there any interest to make the daily build results public?
>> * Is it appreciated if I post the build results in a daily post to the
>> mailinglist, _if_ there is a build failure? (No news is good news)
>> What format is preferred?
> 
> Please do not post such results here.

Wolfgang: I would suggest another mailman list.  Interested parties can 
subscribe to it, disinterested parties can look in the list archives if 
they need to.  There probably isn't a need for long term archives, say 6 
months (~2-3 releases).

>> * Or is it preferred to post the results on a website?
> 
> Yes, that would be beter, IMO.

Web sites are awfully easy to ignore/forget. :-/

> Best regards,
> 
> Wolfgang Denk

Ditto,
gvb

[U-Boot] [RFC] Build errors in u-boot mainline and daily builds

[Remy Bohmer]

Hello All,

>> I appreciate your efforts. Actually this is  something  everybody  is
>> supposed  to  do  before  submitting  a  patch,  and  especially  the
>> custodians before submitting a pull request.
>> Unfortually, that's theory only :-(
> An unenforceable demand. :-(  A build machine is the pragmatic solution, but
> it puts the burden on the build machine maintainer.  :-/  Thanks for
> volunteering, Remy. :-)

As mentioned, I already needed a build machine for the u-boot-USB
tree, so why not making it public while I have it anyway :-)
And in my opinion, having a daily build does not mean anybody can skip
the build testing part ;-)

> Side note: I've been looking at BuildBot but haven't done anything useful.
>  I've looked at CruiseControl a few times and then I see how it does the
> control logic in (haxtended) XML.  Bleah!

I currently do it by means of a plain and simple shell script and a
crontab, and I (=crond) sent myself an email if the build fails, with
the buildlog zipped attached.
I can put the build results online as is (in plain text) and
increasing the list of email-addresses is also very easy.

I also started looking at CruiseControl.rb (some colleagues of me are
using it) which seems to have git support on board, the web interface
looks nice, but only reading the manuals, installing the stuff and so
on, already took me more time than writing the shell script :-) And
every tool that cost me more time to maintain than this shell script
is a bad tool by design ;-))
I will investigate some more on tools to use (e.g. buildbot), and if I
cannot find anything suitable, I will stick to the shell script for
the time being :-) (being pragmatic)

>>> Further, is there any interest to make the daily build results public?
>>> * Is it appreciated if I post the build results in a daily post to the
>>> mailinglist, _if_ there is a build failure? (No news is good news)
>>> What format is preferred?
>> Please do not post such results here.
> Wolfgang: I would suggest another mailman list.  Interested parties can
> subscribe to it, disinterested parties can look in the list archives if they
> need to.  There probably isn't a need for long term archives, say 6 months
> (~2-3 releases).
>>> * Or is it preferred to post the results on a website?
>> Yes, that would be beter, IMO.
> Web sites are awfully easy to ignore/forget. :-/

That was my idea also, if I put it on a website, people need to poll
for the status and will forget its existence if the build goes well
for a while, and the benefits are gone. On the other hand, spamming
the mailinglist with daily build failure messages is also not wanted,
and I would not like that either!

But (just thinking out loud), build failures are not allowed to
happen, and when it happens at a certain time, is it so bad to sent it
to the (or any) mailinglist?
So, for example:
1. build broken -> sent mail (goto step 2, or 3)
2. build more broken-> sent another mail (goto step 2, or 3)
3. build repaired->sent mail...  (goto step 4)
4. and then (forever) silence... until step 1 happens again.

The advantage of using a mailinglist is that if a build gets broken,
everybody gets informed immediately so actions can be taken fast, soon
after the problem has been introduced, and the patches are still fresh
in memory of the people who wrote them.
Another advantage of the mailinglist is (let's be mean here) that it
is public to see who breaks the build regularly, so we can bring that
person to justice in public (gna gna) :-)))))))

I also thought about sending only emails to people from whom patches
are integrated since the last successful build.
That would be a more ideal like situation, not bothering anyone who
did not break anything...
That kind of information could be extracted from the git log :-)

Just TOL, keeping the discussion going ;-)


Kind Regards,

Remy

[U-Boot] [RFC] Build errors in u-boot mainline and daily builds

[Wolfgang Denk]

Dear Jerry Van Baren,

In message <495FA045.6050909@gmail.com> you wrote:
>
> >> * Is it appreciated if I post the build results in a daily post to the
> >> mailinglist, _if_ there is a build failure? (No news is good news)
> >> What format is preferred?
> > 
> > Please do not post such results here.
> 
> Wolfgang: I would suggest another mailman list.  Interested parties can 
> subscribe to it, disinterested parties can look in the list archives if 
> they need to.  There probably isn't a need for long term archives, say 6 
> months (~2-3 releases).

Setting up a mailing list is trivial - but will it be really useful /
used? Speaking for myself, I have to  admit  that  I  would  probably
store the messages in a folder - and soon start to forget them there.

> >> * Or is it preferred to post the results on a website?
> > 
> > Yes, that would be beter, IMO.
> 
> Web sites are awfully easy to ignore/forget. :-/

Hm... So are e-mail messages once they are filed away. A web site
wastes the storage only once ;-)

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
I must follow the people.  Am I not their leader? - Benjamin Disraeli

[U-Boot] [RFC] Build errors in u-boot mainline and daily builds

[Wolfgang Denk]

Dear Remy,

In message <3efb10970901031205p5a54c743y2fdf4b7a82949e9f@mail.gmail.com> you wrote:
> 
> I will investigate some more on tools to use (e.g. buildbot), and if I
> cannot find anything suitable, I will stick to the shell script for
> the time being :-) (being pragmatic)

DUTS (http://www.denx.de/wiki/DUTS/DUTSDocs) makes such an approach,
too - but it goes one step further and adds automatic regression
testing, too.

> But (just thinking out loud), build failures are not allowed to
> happen, and when it happens at a certain time, is it so bad to sent it
> to the (or any) mailinglist?

Certain failures are known, and exist for a long time.

> So, for example:
> 1. build broken -> sent mail (goto step 2, or 3)

Nobody will remember this any more two weeks later.

> 2. build more broken-> sent another mail (goto step 2, or 3)
> 3. build repaired->sent mail...  (goto step 4)
> 4. and then (forever) silence... until step 1 happens again.

It's a tightrope walk. If you send a build failure report just  once,
it  may  be  easily  forgotten, so you have to send reminders. If you
send reminders too often, messages will be ignored quickly.

IMHO the only thing we can do is to use a bug tracking  system  where
broken  builds  are flagged, and where responsibility can be flagged,
too.

Unfortunately our gnats expert ran away leaving the last 10%  of  the
task of setting up bugs.denx.de undone.  [Any volunteers here?]


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
The first 90% of a project takes 90% of the time, the last 10%  takes
the other 90% of the time.