[U-Boot] Fix for a build ?

[U-Boot] Fix for a build ?

[Sylvain Lamontagne]

Hi everybody,

We have recently installed an automated build system in our company
and occasionally there is a problem when U-Boot is built.
One build will work the other will not without any changes to the sources
(it is not: 1 work, then other fail... it is more like 3 may work the 4 will
not but the 5 will). Our build system guy says that the problem is probably
a race condition with a dependency in the U-Boot make.

Here is the error that trigger the failure.
*[09:23:58]:* */home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boot-1.3.4/cpu/mpc5xxx/.depend:73:
*** multiple target patterns. Stop.*
*[09:23:58]:* *make: ***
[/home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boot-1.3.4/cpu/mpc5xxx/start.o]
Error 2*
*[09:23:58]:* */home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boot-1.3.4/cpu/mpc5xxx/.depend:73:
*** multiple target patterns. Stop.*
*[09:23:58]:* *make: *** Waiting for unfinished jobs....*
*[09:23:58]:* *make: ***
[/home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boot-1.3.4/cpu/mpc5xxx/libmpc5xxx.a]
Error 2*
*[09:24:01]:* *Process exited with code 2*

It is not extremely problematic because everything is automated and next
build after the failing one will work (when it is this error), but it's only
that we get a notification email when a build fail and we are afraid that
our developers will take the failure as a "Boy Who Cried Wolf"... if you
know what I mean.

Is there a way to fix this ? It could be putting some "sleep" somewhere...

Any suggestion ?

Sylvain

* Also, upgrading U-Boot is not an option... since we don't have the need
nor resources to port a new version of U-Boot to this product. So please
don't suggest that.

[U-Boot] Fix for a build ?

[Mike Frysinger]

On Monday 03 May 2010 09:51:29 Sylvain Lamontagne wrote:
> We have recently installed an automated build system in our company
> and occasionally there is a problem when U-Boot is built.
> One build will work the other will not without any changes to the sources
> (it is not: 1 work, then other fail... it is more like 3 may work the 4
> will not but the 5 will). Our build system guy says that the problem is
> probably a race condition with a dependency in the U-Boot make.
> 
> Here is the error that trigger the failure.
> *[09:23:58]:*
> */home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boo
> t-1.3.4/cpu/mpc5xxx/.depend:73: *** multiple target patterns. Stop.*

this should already be fixed in the latest version
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20100503/5d2f62b3/attachment.pgp 

[U-Boot] Fix for a build ?

[Sylvain Lamontagne]

And do you know what was the fix so that I could apply it to our version ?

On Mon, May 3, 2010 at 2:52 PM, Mike Frysinger <vapier@gentoo.org> wrote:

> On Monday 03 May 2010 09:51:29 Sylvain Lamontagne wrote:
> > We have recently installed an automated build system in our company
> > and occasionally there is a problem when U-Boot is built.
> > One build will work the other will not without any changes to the sources
> > (it is not: 1 work, then other fail... it is more like 3 may work the 4
> > will not but the 5 will). Our build system guy says that the problem is
> > probably a race condition with a dependency in the U-Boot make.
> >
> > Here is the error that trigger the failure.
> > *[09:23:58]:*
> >
> */home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boo
> > t-1.3.4/cpu/mpc5xxx/.depend:73: *** multiple target patterns. Stop.*
>
> this should already be fixed in the latest version
> -mike
>

[U-Boot] Fix for a build ?

[Wolfgang Denk]

Dear Sylvain Lamontagne,

please do not top-post / full quote. Please read
http://www.netmeister.org/news/learn2quote.html

In message <t2x3e45db431005031314s2a2ef407oa2eb9eb256c7b2df@mail.gmail.com> you wrote:
>
> And do you know what was the fix so that I could apply it to our version ?

You can try and run git bisect on the tree to find out yourself. But I
guess the fix is buried in a ton of other changes, so you will find a
backport is not as trivial a thing as it might seem.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Nobody trips over mountains. It is the small pebble that  causes  you
to  stumble.  Pass all the pebbles in your path and you will find you
have crossed the mountain.

[U-Boot] Fix for a build ?

[Wolfgang Denk]

Dear Sylvain Lamontagne,

In message <h2y3e45db431005030651sacba80cz14740e8d79c4d178@mail.gmail.com> you wrote:
>
> We have recently installed an automated build system in our company
> and occasionally there is a problem when U-Boot is built.
> One build will work the other will not without any changes to the sources
> (it is not: 1 work, then other fail... it is more like 3 may work the 4 will
> not but the 5 will). Our build system guy says that the problem is probably
> a race condition with a dependency in the U-Boot make.
...
> *[09:23:58]:* *make: ***
> [/home/slamon/server/TeamCity/buildAgent/work/19b40828dc62d35d/build/u-boot-1.3.4/cpu/mpc5xxx/start.o]
-----------------------------------------------------------------------^^^^^^^^^^^^

Ah, U-Boot release 1.3.4.

That's nearly two years old, which means it's ancient.

Yes, there may be race conditions during building, especially on SMP
machines.

> Is there a way to fix this ? It could be putting some "sleep" somewhere...

Please upgrade to a recent version.

> * Also, upgrading U-Boot is not an option... since we don't have the need
> nor resources to port a new version of U-Boot to this product. So please
> don't suggest that.

Heh. So what do you want? A solution, but no changes?

We will not fix for free any bugs in old versions of the code. You
have the following options:

- stick with the obsolete code you have and stop complaining
- update to the current release and ask again if there should still be
  problems (which I doubt)
- fix the problems yourself
- hire somebody else to fix the problems (and most probably the most
  efficient way to fix the problem will be an update to the current
  release)

For your next project you can try and learn a lesson from this issue:
the amount of effort saved by not pushing your port upstream into
mainline is often smaller than the added maintenance cost for an
out-of-tree port.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
You never have the time to do things right,  but you will always have
the time to do them twice.

[U-Boot] Fix for a build ?

[Mike Frysinger]

+1 ;)
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20100503/c7973ea7/attachment.pgp 

[U-Boot] Fix for a build ?

[Sylvain Lamontagne]

> > Is there a way to fix this ? It could be putting some "sleep" somewhere...
>
> Please upgrade to a recent version.

Thank you for suggesting ...

>
> > * Also, upgrading U-Boot is not an option... since we don't have the need
> > nor resources to port a new version of U-Boot to this product. So please
> > don't suggest that.
>
> Heh. So what do you want? A solution, but no changes?

I was asking in case someone remembered and could have been an easy fix.
Something like:
Yes, Sylvain If I remember there was a race condition with MakefileX
in Y?specially on an SMP machine.
That would have been a perfectly correct answer.

>
> We will not fix for free any bugs in old versions of the code.


That was absolutely not what I was asking for ... I did not want
anybody of you to fix this for me. I was asking for pointers and it
was a quick question that should have get a quick answer, nothing
more.

> - stick with the obsolete code you have and stop complaining
Again, I was not complaining. Reading the archive of this mailing
list, it seems that you, Wolfgang, tend to often take what people ask
as a complaint and I suggest that you should work on this aspect if
you want people to contribute by their free will.

> - update to the current release and ask again if there should still be
> ?problems (which I doubt)
> - fix the problems yourself
> - hire somebody else to fix the problems (and most probably the most
> ?efficient way to fix the problem will be an update to the current
> ?release)

You may have never been in this situation, but I work for a company
that is big enough to not let me change something that work fine only
for some failure from time to time related to a race condition on a
new automated build server.
I did try to update U-Boot in the past few month but I always end-up
with problem related to variables that changes names (CFG vs CONFIG)
or to bizarre memory config created by my predecessor  that make the
new version unworkable. I am not an electronician and there is still
aspect of embedded development that I don't yet understand. I have an
IT degree and I'm currently learning embedded "in the field" since my
predecessor was laid off. If giving chance to people is not in your
nature, then I'm sorry to have been taking your time.

Make it easy to upgrade and people may then upgrade more often. The
Linux kernel 2.4 is still supported in some way for people that have
very old hardware that work only with it, so why wouldn't it be
legitimate to keep a tested version of U-Boot on a running product
only because it's 2 year old ?

Anyway thx for answering even if the answer was a bit rude...

Sylvain

[U-Boot] Fix for a build ?

[Mike Frysinger]

On Monday 03 May 2010 18:02:26 Sylvain Lamontagne wrote:
> Make it easy to upgrade and people may then upgrade more often.

as Wolfgang said, this is your fault.  integrated board ports get updated 
automatically by other people.  so if you had integrated your board port, the 
CFG vs CONFIG and related changes wouldnt have been an issue as other people 
would have fixed it for you.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20100503/b4896980/attachment.pgp 

[U-Boot] Fix for a build ?

[Wolfgang Denk]

Dear Sylvain Lamontagne,

In message <s2j3e45db431005031502zb199821fndee7e8f220853489@mail.gmail.com> you wrote:
>
> > Heh. So what do you want? A solution, but no changes?
>
> I was asking in case someone remembered and could have been an easy fix.
> Something like:
> Yes, Sylvain If I remember there was a race condition with MakefileX
> in Y specially on an SMP machine.
> That would have been a perfectly correct answer.

We have about 650 Makefiles in the current U-Boot source tree; the top
level Makefile alone has seen 371 commits since v1.3.4, including
major reworks like changes to the directory structure, architecture
renames, etc. There have been a number of plain bug fixes to the U-Boot
build system, and there have been a number of work-arounds for broken
tool chains like the broken GNU make 3.80 or for some broken ARM
compilers.

In total there must be way over 1000 commits to Makefiles, plus
several hundred to config.mk and other make related files.

I am not in a position to remember each and every of these changes, or
their potential impact on building in SMP configurations.

> That was absolutely not what I was asking for ... I did not want
> anybody of you to fix this for me. I was asking for pointers and it
> was a quick question that should have get a quick answer, nothing
> more.

I gave you  quick answer: use git bisect to find out which commit
makes a difference for your build system. Of course this requires that
you have some build target that is supported in mainline, and that
shows the same problem like your out-of-tree port.

> You may have never been in this situation, but I work for a company
> that is big enough to not let me change something that work fine only
> for some failure from time to time related to a race condition on a
> new automated build server.

I have. More than once. And of course we have customers in such
situations, too.

> I did try to update U-Boot in the past few month but I always end-up
> with problem related to variables that changes names (CFG vs CONFIG)
> or to bizarre memory config created by my predecessor  that make the
> new version unworkable. I am not an electronician and there is still

As Mike already pointed out this is a well known problem, and it is
important that you understand that it's a completely self-made
purgatory. U-Boot develops continuously and quickly, so any
out-of-tree port is obsolete as soon as you release it, and
maintaining it over more than a few months costs way more effort than
pushing your changes upstream into the mainline repositories.

You can now waste efforts on trying to back-port fix after fix from
mainline to your old code, or you can invest the effort in updating
your code base and pushing it upstream. This is your (or your
manager's) decision. We don't actually care about this, but it seems
only fair to mention that from our experience the one-time effort to
push things upstream is smaller than the repeating costs of
maintaining an out-of-tree port.

> aspect of embedded development that I don't yet understand. I have an
> IT degree and I'm currently learning embedded "in the field" since my
> predecessor was laid off. If giving chance to people is not in your
> nature, then I'm sorry to have been taking your time.
> Make it easy to upgrade and people may then upgrade more often. The

You seem to fail to understand how a free software project like U-Boot
works. U-Boot is very easy to upgrade - we take great care not to
break support for any of the supported boards, and we accept even
exotic boards and include these into the mainline tree, even if there
is most likely no other user ever.

So if you want to be able to upgrade easily just make sure your code
is part of the mainline distribution.

By not pushing your code upstream you may save a certain amount of
effort, but you also miss the chance of getting a free code review by
a number of highly qualified experts that may improve the quality of
your code considerably, and you accept that you will have to spend
maintenance efforts all yourself.

This is *your* choice.  It is nothing we can do for you, or refrain
from doing for you.

> Linux kernel 2.4 is still supported in some way for people that have
> very old hardware that work only with it, so why wouldn't it be
> legitimate to keep a tested version of U-Boot on a running product
> only because it's 2 year old ?

U-Boot takes really, really lots of efforts to keep even ancient
boards working - in mainline, that it.

The responsibility for any out-of-tree port like yours is completely
in your own hands.

> Anyway thx for answering even if the answer was a bit rude...

I did not intend to be rude, but I have to admit that your attitude
is not exactly in line how community projects like this work.

I don't have the time to answer requests like yours in long prosa -
the build system is a pretty complex issue and there are tons of
factors that have impact. I already mentioned tool chain issues like
broken ARM compilers or things like the white-space madness in GNU
make 3.80, but there are tons more. Is your source directory mounted
over NFS? Depending on the NFS version you may see problems (like
https://bugzilla.redhat.com/show_bug.cgi?id=530625 resp.
http://bugs.gentoo.org/show_bug.cgi?id=289022). Are you using any
sort of caching? Fscache has it's onw collection of bugs, ccache has
some known issues, and so on and on. You did not reveal any
information about you build environment, so you should not expect too
much useful feedback either. Maybe you want to read
http://catb.org/esr/faqs/smart-questions.html

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Computers are not intelligent.  They only think they are.

[U-Boot] Fix for a build ?

[Sylvain Lamontagne]

Dear Wolfgang Denk,

> We have about 650 Makefiles in the current U-Boot source tree; the top
> level Makefile alone has seen 371 commits since v1.3.4
[snip]
> I am not in a position to remember each and every of these changes, or
> their potential impact on building in SMP configurations.
>
I understand that one person cannot remember the awesome lot of work
that have been done since 1.3.4 and that many fundamental changes
occurred during these two years.

> I gave you ?quick answer: use git bisect to find out which commit
> makes a difference for your build system. Of course this requires that
> you have some build target that is supported in mainline, and that
> shows the same problem like your out-of-tree port.
I'm sorry I did not see the email related to this before sending the
other one. This is a perfect suggestion and I'll probably setup a
build configuration for the lite5200 board that is on my desk to see
if my problem show itself.

[snip]

> You seem to fail to understand how a free software project like U-Boot
> works. U-Boot is very easy to upgrade - we take great care not to
> break support for any of the supported boards, and we accept even
> exotic boards and include these into the mainline tree, even if there
> is most likely no other user ever.
>
> So if you want to be able to upgrade easily just make sure your code
> is part of the mainline distribution.
Ok, I am now in the seat of someone that could push the idea of
putting the next platform we develop into the mainline. But other
MBA/Finance/Manager persons that really take decisions will surely ask
my department why I would like to give internal informations of our
product in a way that any of our competitors could get it easily. How
could I give them an answer that would be aligned with the philosophy
?

I understand how free software project works, I have participate in
some and even worked to revive an old dead project
(http://datavibe.net/~essiene/ale/) into something up-to-date while I
was at University. (http://sonia.etsmtl.ca/index.php?id=553)
I've also participate in some kernel janitor task in my free time
because I care about open source and I'm passionate by technology. I
was a quite active member in the french forum of gentoo to help new
comers and find solutions to problems. I care about people and I want
the people around me to improve as best as they can.

> I did not intend to be rude, but I have to admit that your attitude
> is not exactly in line how community projects like this work.
Then I'm sorry, but for me this project sounds like it is directed by
a bunch of elitist who would not accept someone that don't already
know how the project work. Criticizing and judging questions asked by
people that may never have work on or with  something like U-Boot
before. If you are afraid of getting to much dumb questions then this
means that the documentation and the FAQ from the U-Boot website could
be improve in a way that new comers would find easily the informations
they need.

> I don't have the time to answer requests like yours in long prosa -
Then you can simply skip the question, this mailing list have surely
more than hundred persons that can answer if they think they could
help. I don't think anybody ask that you answer to all emails, this is
surely an extremely huge time consuming task.

>  Maybe you want to read
> http://catb.org/esr/faqs/smart-questions.html
And maybe you want to read
http://www.reddit.com/r/linux/comments/9y9de/eric_raymonds_famous_how_to_ask_questions_the/

Have a nice day

Sylvain

[U-Boot] Fix for a build ?

[Wolfgang Denk]

Dear Sylvain Lamontagne,

In message <n2m3e45db431005040739m7ff0eb77x712f3e39681f44f8@mail.gmail.com> you wrote:
> 
> Ok, I am now in the seat of someone that could push the idea of
> putting the next platform we develop into the mainline. But other
> MBA/Finance/Manager persons that really take decisions will surely ask
> my department why I would like to give internal informations of our
> product in a way that any of our competitors could get it easily. How
> could I give them an answer that would be aligned with the philosophy
> ?

First, U-Boot is covered by the GPL, so you will have to give the
full sources anyway to any of your customers who asks for it - even
if it's one of your direct competitors.

Second, do you really think there is any intellectual property in a
port of U-Boot that is worth such protection?  If so, you better write
your own proprietary boot loader from scratch.

Third: according to Ohloh (http://www.ohloh.net/p/u-boot) it would
cost you (or your company) some 270+ man-years or $_15,000,000 to
develop something like U-Boot from scratch. You got all this code
completely for free, so it seems only just and equitable that you
return the little you added to the community, too.

> > I did not intend to be rude, but I have to admit that your attitude
> > is not exactly in line how community projects like this work.
> Then I'm sorry, but for me this project sounds like it is directed by
> a bunch of elitist who would not accept someone that don't already
> know how the project work. Criticizing and judging questions asked by
> people that may never have work on or with  something like U-Boot
> before. If you are afraid of getting to much dumb questions then this
> means that the documentation and the FAQ from the U-Boot website could
> be improve in a way that new comers would find easily the informations
> they need.

All the documentation is in a wiki - please feel free to add any
parts you consider missing.


Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
People seldom know what they want until you give them what  they  ask
for.

[U-Boot] Fix for a build ?

[Igor Luri]

Dear Sylvain Lamontagne,

We have had a similir issue building U-Boot (1.1.6) in a server with 2
cuadcore processors that is responsible of the automated build system. 

GNU Make has the posibility to run several jobs simultaneously, passing
the argument "-j N", where N is the number of jobs executed
simultaneously. Since our server has 8 processors, we use "-j 9"
arguments in the automated build (the documentation recomends using N=
number of cores +1). This way, U-Boot and other software projects,
sometimes build ok and sometimes fail.

We solve this problem building U-Boot and failing projects passing "-j
1" argument to Make (without running simultaneous jobs), and the rest of
software with "-j 9".

I have seen that you are running simultaneous jobs:

*[09:23:58]:* *make: *** Waiting for unfinished jobs....*



Hope this helps.

Peace.



-- 
Igor Luri
R&D Software Department
Fagor Aotek S. Coop.
P. O. Box 144
E-20500 Mondrag?n-Arrasate
Tel.    ++34 943 71 92 00 
        ++34 943 71 92 01 (Ext. 44268)
Fax.    ++34 943 79 92 03
www.aotek.es

[U-Boot] Fix for a build ?

[Sylvain Lamontagne]

Hi Igor Luri,

> We solve this problem building U-Boot and failing projects passing "-j
> 1" argument to Make (without running simultaneous jobs), and the rest of
> software with "-j 9".
>
Your suggestion solved the problem for us, thank you very much.

> Hope this helps.
It did :) again thank you
>
> Peace.
Have a wonderful day

Sylvain