Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Rob Landley]

On Wednesday 30 April 2008 14:11:49 David Woodhouse wrote:
> > There are simply already far too many of them and they make the
> > kernel harder and harder to change.
>
> I agree. And if we do want to pay attention to pure code size, there are
> other approaches -- like --gc-sections and/or building with '--combine
> -fwhole-program' which I was playing with for OLPC a while back. I must
> dust that off now that the GCC fixes should mostly have made it into
> current distributions.

I like simplicity.  I like _simple_.  Half my attraction to embedded systems 
is that they're simpler than big desktop systems where you have to figure out 
how udev interacts with dbus to figure out why knetworkmanager convinced 
Konqueror that your network doesn't exist when you can wget from the command 
line just fine...

I submitted a patch to remove the use of perl to build the linux kernel (which 
HPA added in 2.6.25) not because it affected the result, but because it 
unnecessarily complicates the build system.  (And perl tends to metasticize.  
Nothing says "maintainability" like perl...)

Yes, Perl was used for some debugging tools before, things like the 
check-patch script.  But until 2.6.25 you never needed Perl to configure and 
build a kernel...

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Leon Woestenberg]

Hello,

On Wed, May 28, 2008 at 11:52 PM, Rob Landley <rob@landley.net> wrote:
> On Wednesday 30 April 2008 14:11:49 David Woodhouse wrote:
>> I agree. And if we do want to pay attention to pure code size, there are
>> other approaches -- like --gc-sections and/or building with '--combine
>> -fwhole-program' which I was playing with for OLPC a while back. I must
>> dust that off now that the GCC fixes should mostly have made it into
>> current distributions.
>
> I like simplicity.  I like _simple_.  Half my attraction to embedded systems
>
Same here.

> I submitted a patch to remove the use of perl to build the linux kernel (which
> HPA added in 2.6.25) not because it affected the result, but because it
> unnecessarily complicates the build system.  (And perl tends to metasticize.

Thanks, I hope it is or gets accepted.

In general I think one of the aspects of embedded Linux is about
minimizing the amount of bloat dependencies. Especially, when each
dependency can explode in a hurd of new dependencies.

> "One of my most productive days was throwing away 1000 lines of code."
>  - Ken Thompson.
>
How appropriate.

Regards,
-- 
Leon

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Rob Landley]

On Monday 09 June 2008 16:27:29 Leon Woestenberg wrote:
> > I submitted a patch to remove the use of perl to build the linux kernel
> > (which HPA added in 2.6.25) not because it affected the result, but
> > because it unnecessarily complicates the build system.  (And perl tends
> > to metasticize.
>
> Thanks, I hope it is or gets accepted.

Nope, Peter Anvin shot it down (saying what I was doing was a strange, purely 
academic exercise), and Sam Ravnborg announced his vague intention to rewrite 
large chunks of kbuild in perl in the future.  (Because nothing 
says "maintainability" like perl...)

For some reason they wanted to mail me about it off-list rather than cc:ing 
the list about this plan.

A number of embedded people emailed me about it off list, but nobody ever 
replied to my post _on_ the list to say I wasn't alone in this, or acked the 
patch to say it worked for them, or anything like that.  So there was a 
perception of zero support, and I gave up trying to follow up on it for 
2.6.25.  I've got it working for me, and if more perl shows up I'll come up 
with more patches to remove it for my own personal build environment.

I might try to submit again in a few months, but in the meantime my patches 
are at
http://landley.net/hg/firmware/file/tip/sources/patches/ (see the 
linux-*noperl*.patch one).

Maybe I'll try linux-kernel again.  I submitted my miniconfig patches 
something like three times.  But if the guys on the list didn't want it, and 
the guys in the embedded world poking me to do something about it weren't 
willing to even ack the darn patch on the list, and the people who want 
changes made to it aren't willing to change it from what I'm happy with to 
what they want and expect me to do it for them when I'm already happy with 
it...

My todo list is long.  This goes somewhere after resorting the contents of my 
bookshelves.

I've found that posting to linux-kernel tends to attract "belling the cat" 
ideas.  For example, I use kconfig in toybox and I posted patches to make it 
easier to use kconfig in other, non-kernel projects.  The resulting thread 
turned into a grandiose plan about a separate kconfig that would be 
maintained as a spearate project and get installed on your system ala make, 
for use by any source package that wants that functionality.  To which my 
response was "feel free, I have no interest in doing that, did you want my 
patches or not?"  Improvements in the UI to use miniconfigs were held up by 
people objecting to how they were generated, which is separate from how 
they're used.  And one of peter's objections to the perl removal patch (which 
basically checked the precalculated header file into the tree) was that some 
out of tree mips patch that never got merged might want to let you enter the 
clock speed as an input field, so you can't possibly just rely on 
precalculated values and people who want to check in code using a new value 
checking in an updated header with that value.  (Even though that makes every 
existing architecture happy, last I checked.)

I'm a hobbyist.  I made it work for me.  If nobody else is interested in what 
I did, I'm ok with that...

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Sam Ravnborg]

On Mon, Jun 09, 2008 at 10:53:11PM -0500, Rob Landley wrote:
> On Monday 09 June 2008 16:27:29 Leon Woestenberg wrote:
> > > I submitted a patch to remove the use of perl to build the linux kernel
> > > (which HPA added in 2.6.25) not because it affected the result, but
> > > because it unnecessarily complicates the build system.  (And perl tends
> > > to metasticize.
> >
> > Thanks, I hope it is or gets accepted.
> 
> Nope, Peter Anvin shot it down (saying what I was doing was a strange, purely 
> academic exercise), and Sam Ravnborg announced his vague intention to rewrite 
> large chunks of kbuild in perl in the future.  (Because nothing 
> says "maintainability" like perl...)
> 
> For some reason they wanted to mail me about it off-list rather than cc:ing 
> the list about this plan.
Rob - please reread the mails. It was your broken mailer or something
that caused it to be off-list.
When I hit reply-to-all in mutt I expect it to be sent to the list -
and in your case it did not not hit the list.

> 
> A number of embedded people emailed me about it off list, but nobody ever 
> replied to my post _on_ the list to say I wasn't alone in this, or acked the 
> patch to say it worked for them, or anything like that.  So there was a 
> perception of zero support, and I gave up trying to follow up on it for 
> 2.6.25.  I've got it working for me, and if more perl shows up I'll come up 
> with more patches to remove it for my own personal build environment.
> 
> I might try to submit again in a few months
Then please try to address the comments first.

	Sam

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Rob Landley]

On Monday 09 June 2008 23:30:20 Sam Ravnborg wrote:
> Rob - please reread the mails. It was your broken mailer or something
> that caused it to be off-list.
> When I hit reply-to-all in mutt I expect it to be sent to the list -
> and in your case it did not not hit the list.

Could easily be the case.  Kmail's gotten increasingly weird ever since 
Kontact ate it, and I do tend to break stuff anyway...

> > A number of embedded people emailed me about it off list, but nobody ever
> > replied to my post _on_ the list to say I wasn't alone in this, or acked
> > the patch to say it worked for them, or anything like that.  So there was
> > a perception of zero support, and I gave up trying to follow up on it for
> > 2.6.25.  I've got it working for me, and if more perl shows up I'll come
> > up with more patches to remove it for my own personal build environment.
> >
> > I might try to submit again in a few months
>
> Then please try to address the comments first.

The comments I remember (it's been a while) were that it was futile to fight 
against the addition of perl because A) nobody else was bothered by it, B) 
plans were in place to add more perl to the build in future so it was only a 
matter of time anyway, C) some out of tree patch might add an architecture 
variant that wanted to define arbitrary clock values.  My response to C 
was "I can't find this patch, show me", my response to B) was "I'll address 
that when it happens", and my response to A) was "yeah, everybody else who 
seemed to care about this has decided to shut up and bog off, haven't they?"

I keep meaning to write a much improved patch that records a comment at the 
start of the .h file with the complete command line the perl thing was run 
as, so re-running the header generator is a question of cut and paste and 
then adding the extra value you want before hitting enter.  And then ripping 
all the logic out of the makefile that tries to rebuild the thing itself 
because only a patch to Kconfig files can ever require a new value, and that 
patch should change the .h file so it can build the result.  If they forget 
to check in the .h file, the build should fail noisily.

The result is simpler, has many fewer lines, and doesn't need perl to compile 
(just to add new clock values during development), but since nobody else in 
the embedded community cared to speak up for it, I kind of lost interest in 
trying to get it merged.  (Maybe I _am_ the only person who still cares about 
building on a host without perl.  If I wasn't, somebody else would have acked 
the patch...)

*shrug*  If I go back to revisit this issue I'll dig up the old thread and 
reread it, but at the moment I have no immediate plans to.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Sam Ravnborg]

> (Maybe I _am_ the only person who still cares about 
> building on a host without perl.  If I wasn't, somebody else would have acked 
> the patch...)

perl is pretty standard and I fail to see the benefits of avoiding it.
For embedded development I see even less benefits as I assume
any sane embedded development environment are based on a
cross-toolchain so you do the build on a high perfomance box.

Building everything for my arm board on the arm board would be a disater
for example.

	Sam

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Wolfgang Denk]

In message <20080610075432.GB776@uranus.ravnborg.org> you wrote:
> > (Maybe I _am_ the only person who still cares about 
> > building on a host without perl.  If I wasn't, somebody else would have acked 
> > the patch...)
> 
> perl is pretty standard and I fail to see the benefits of avoiding it.
> For embedded development I see even less benefits as I assume
> any sane embedded development environment are based on a
> cross-toolchain so you do the build on a high perfomance box.
> 
> Building everything for my arm board on the arm board would be a disater
> for example.

Well, compiling the Linux kernel on the native system with the root
file system mounted over NFS has always been a really good regression
test for us. It exercises a *lot* of kernel code - tasks, memory,
network, ...

Being unable to do this just because we now also would need a  native
Perl is indeed a PITA...

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
You don't have to worry about me. I might have been born yesterday...
but I stayed up all night.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Jamie Lokier]

My only comment is I remember when Eric Raymond submitted a smart
config thing (before Kconfig, which copied Eric's best ideas).

The main objection to Eric's patch was that it was written in Python,
causing kernel builds to depend Python.

When did this policy change, so that it's now acceptable to depend on
Perl, which is roughly equivalent as a tool dependency?

-- Jamie

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Adrian Bunk]

On Tue, Jun 10, 2008 at 11:20:18AM +0100, Jamie Lokier wrote:
> My only comment is I remember when Eric Raymond submitted a smart
> config thing (before Kconfig, which copied Eric's best ideas).
> 
> The main objection to Eric's patch was that it was written in Python,
> causing kernel builds to depend Python.
>...

Although the usage of Python was brought as a reason against it the 
main objection against CML2 was ESRs interaction with the kernel 
community, like the fact that he was not willing to split his
changes into appropriate pieces but only offered one big patch.

> -- Jamie

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Sam Ravnborg]

> 
> When did this policy change, so that it's now acceptable to depend on
> Perl, which is roughly equivalent as a tool dependency?

We have perl as a mandatory part of the kernel build in several places
for various architectures.
And I do not recall anyone submitting a bug that they could not build
a kernel due to the perl dependency.
But I am obviously well aware of that we use it for the time stuff.

For the headers_* targets I will shortly introduce yet another
perl dependency - but only if these targets are used - so less of an issue.

o We shall try to keep the dependencies low for the common things
o but for the more exoctic things a wides dependency is OK.

Which is also why I'm happy to apply a patch that remove
the mandatory dependency of perl we have today - if and only if
that patch meet normal patch acceptance criterias.

Rob's initial patch had some issues and neither Rob nor I have
fixed these and therefore it has not been applied.

Rob seems to put much more into this (private reply accustions etc)
for reasons unknown to me. And doing so does not help to get me interested.

So try to get the facts correct here - there is noone against removing
the mandatory perl dependency. But it is lower on my priority list
than many other things which explain why I do not do it myself.
But if someone submit a patch to do so then if the patch is OK it
will be applied.

And if we have a policy that say no-go to perl then it is new to me.
I hope one day to rewrite part of kbuild and perl seems to be the
best candidate around. But that day may never come.

	Sam

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Jamie Lokier]

Wolfgang Denk wrote:
> Being unable to do this just because we now also would need a  native
> Perl is indeed a PITA...

You can run the Perl bit with "ssh remote perl", and still do the rest
of the compile natively.  It's not pretty, but workable.

-- Jamie

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Will Newton]

On Tue, Jun 10, 2008 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
> Wolfgang Denk wrote:
>> Being unable to do this just because we now also would need a  native
>> Perl is indeed a PITA...
>
> You can run the Perl bit with "ssh remote perl", and still do the rest
> of the compile natively.  It's not pretty, but workable.

I'm not convinced it matters at all. Self hosting on an embedded
architecture is, as has been mentioned, pretty pointless.

Using a kernel compile as a test isn't such a great idea. Stress tests
of that kind are not particularly useful for pinning down bugs - so
your kernel compile failed, what now? Far better to use LTP tests or
similar that are designed to be reproduceable and tunable for your
system. For example I don't think I'll ever be able to self host a
kernel build on a board with only 32Mb of on-board RAM.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[David Woodhouse]

On Tue, 2008-06-10 at 14:25 +0100, Will Newton wrote:
> On Tue, Jun 10, 2008 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
> > Wolfgang Denk wrote:
> >> Being unable to do this just because we now also would need a  native
> >> Perl is indeed a PITA...
> >
> > You can run the Perl bit with "ssh remote perl", and still do the rest
> > of the compile natively.  It's not pretty, but workable.
> 
> I'm not convinced it matters at all. Self hosting on an embedded
> architecture is, as has been mentioned, pretty pointless.
> 
> Using a kernel compile as a test isn't such a great idea. Stress tests
> of that kind are not particularly useful for pinning down bugs - so
> your kernel compile failed, what now? Far better to use LTP tests or
> similar that are designed to be reproduceable and tunable for your
> system. For example I don't think I'll ever be able to self host a
> kernel build on a board with only 32Mb of on-board RAM.

Actually, cross-building on NFS does tend to find a _lot_ of issues
which crop up with board ports; especially PCI arbitration, DMA
coherency, cache and MMU issues. LTP often doesn't catch the same
problems.

I agree that it's not so easy on a board with 32Mb of RAM, since that's
only 4,000,000 bytes -- but 32MiB ought to be _perfectly_ sufficient :)

-- 
dwmw2

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Wolfgang Denk]

In message <20080610131236.GC28565@shareable.org> you wrote:
>
> You can run the Perl bit with "ssh remote perl", and still do the rest
> of the compile natively.  It's not pretty, but workable.

This may or may not work. If, for  example,  perl  is  running  on  a
remote  little  endian host (like a standard x86 PC) and we depend on
native bin endian byte order, there may be trouble...

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Quote from a recent meeting:   "We are going to continue having these
meetings everyday until I find out why no work is getting done."

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Will Newton]

On Tue, Jun 10, 2008 at 2:33 PM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Tue, 2008-06-10 at 14:25 +0100, Will Newton wrote:
>> On Tue, Jun 10, 2008 at 2:12 PM, Jamie Lokier <jamie@shareable.org> wrote:
>> > Wolfgang Denk wrote:
>> >> Being unable to do this just because we now also would need a  native
>> >> Perl is indeed a PITA...
>> >
>> > You can run the Perl bit with "ssh remote perl", and still do the rest
>> > of the compile natively.  It's not pretty, but workable.
>>
>> I'm not convinced it matters at all. Self hosting on an embedded
>> architecture is, as has been mentioned, pretty pointless.
>>
>> Using a kernel compile as a test isn't such a great idea. Stress tests
>> of that kind are not particularly useful for pinning down bugs - so
>> your kernel compile failed, what now? Far better to use LTP tests or
>> similar that are designed to be reproduceable and tunable for your
>> system. For example I don't think I'll ever be able to self host a
>> kernel build on a board with only 32Mb of on-board RAM.
>
> Actually, cross-building on NFS does tend to find a _lot_ of issues
> which crop up with board ports; especially PCI arbitration, DMA
> coherency, cache and MMU issues. LTP often doesn't catch the same
> problems.

It may trigger a number of bugs, I don't disagree, but as a test it is
a blunt instrument. It's likely to be hard to reproduce and have an
inconsistent failure mode. If you're really serious about testing it's
not the best solution. It's like using gcc instead of memtest86 to
test your memory. Eventually it might go wrong but you won't be much
the wiser about why, or have any way to trim your testcase down so you
can run it on an in-circuit emulator or pass it to your silicon
vendor.

> I agree that it's not so easy on a board with 32Mb of RAM, since that's
> only 4,000,000 bytes -- but 32MiB ought to be _perfectly_ sufficient :)

I would be surprised if it was possible to compile Linux with gcc 4.2
with 32MiB of total system memory.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Wolfgang Denk]

In message <87a5b0800806100625m5a6d20dao47b884bff663c24c@mail.gmail.com> you wrote:
>
> I'm not convinced it matters at all. Self hosting on an embedded
> architecture is, as has been mentioned, pretty pointless.

YMMV...

> system. For example I don't think I'll ever be able to self host a
> kernel build on a board with only 32Mb of on-board RAM.

It depends. We've done this on board with as little as 16 MB RAM.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"Confound these ancestors.... They've stolen our best ideas!"
- Ben Jonson

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[David Woodhouse]

On Tue, 2008-06-10 at 14:47 +0100, Will Newton wrote:
> On Tue, Jun 10, 2008 at 2:33 PM, David Woodhouse <dwmw2@infradead.org> wrote:
> > On Tue, 2008-06-10 at 14:25 +0100, Will Newton wrote:
> >> Using a kernel compile as a test isn't such a great idea. Stress tests
> >> of that kind are not particularly useful for pinning down bugs - so
> >> your kernel compile failed, what now? Far better to use LTP tests or
> >> similar that are designed to be reproduceable and tunable for your
> >> system. For example I don't think I'll ever be able to self host a
> >> kernel build on a board with only 32Mb of on-board RAM.
> >
> > Actually, cross-building on NFS does tend to find a _lot_ of issues
> > which crop up with board ports; especially PCI arbitration, DMA
> > coherency, cache and MMU issues. LTP often doesn't catch the same
> > problems.
> 
> It may trigger a number of bugs, I don't disagree, but as a test it is
> a blunt instrument.

Yes, it's a blunt instrument, but blunt instruments are often effective.

I disagree with your claim that using it as a test isn't a good idea.
I would, however, grant you that using it as your _only_ test is a bad
idea :)

-- 
dwmw2

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Grant Likely]

On Tue, Jun 10, 2008 at 7:53 AM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Tue, 2008-06-10 at 14:47 +0100, Will Newton wrote:
>> On Tue, Jun 10, 2008 at 2:33 PM, David Woodhouse <dwmw2@infradead.org> wrote:
>> > On Tue, 2008-06-10 at 14:25 +0100, Will Newton wrote:
>> >> Using a kernel compile as a test isn't such a great idea. Stress tests
>> >> of that kind are not particularly useful for pinning down bugs - so
>> >> your kernel compile failed, what now? Far better to use LTP tests or
>> >> similar that are designed to be reproduceable and tunable for your
>> >> system. For example I don't think I'll ever be able to self host a
>> >> kernel build on a board with only 32Mb of on-board RAM.
>> >
>> > Actually, cross-building on NFS does tend to find a _lot_ of issues
>> > which crop up with board ports; especially PCI arbitration, DMA
>> > coherency, cache and MMU issues. LTP often doesn't catch the same
>> > problems.
>>
>> It may trigger a number of bugs, I don't disagree, but as a test it is
>> a blunt instrument.
>
> Yes, it's a blunt instrument, but blunt instruments are often effective.
>
> I disagree with your claim that using it as a test isn't a good idea.
> I would, however, grant you that using it as your _only_ test is a bad
> idea :)

Just to add my voice to the chorus; I fully agree.  Brute force
testing is useful.  It can expose corner cases that haven't been
considered in formal test suites.

Cheers,
g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Wolfgang Denk]

In message <87a5b0800806100647r178e54c0qd34cbf26f6ce24d@mail.gmail.com> you wrote:
>
> I would be surprised if it was possible to compile Linux with gcc 4.2
> with 32MiB of total system memory.

Hint: if memory really gets tight, you can use swap space. Either  to
a  local  drive  (either through PCI or PCMCIA/PCCard or USB or FW or
...), or over the network. This just adds  another  level  of  stress
testing  to  areas in the kernel that are not so well covered by some
other tests.

Guess who found out that swap on MPC8xx was broken for  a  long,  long
time, and how it was detected?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
You Don't Have To Be 'Damned' To Work Here, But It Helps!!!
                                             - Terry Pratchett, _Eric_

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Jamie Lokier]

Wolfgang Denk wrote:
> In message <87a5b0800806100647r178e54c0qd34cbf26f6ce24d@mail.gmail.com> you wrote:
> >
> > I would be surprised if it was possible to compile Linux with gcc 4.2
> > with 32MiB of total system memory.
> 
> Hint: if memory really gets tight, you can use swap space. Either  to
> a  local  drive  (either through PCI or PCMCIA/PCCard or USB or FW or
> ...), or over the network. This just adds  another  level  of  stress
> testing  to  areas in the kernel that are not so well covered by some
> other tests.

Great, I'm looking forward to your implementation of swap on no-MMU!

Thanks,
-- Jamie

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Tim Bird]

Sam Ravnborg wrote:
>> (Maybe I _am_ the only person who still cares about 
>> building on a host without perl.  If I wasn't, somebody else would have acked 
>> the patch...)
> 
> perl is pretty standard and I fail to see the benefits of avoiding it.
> For embedded development I see even less benefits as I assume
> any sane embedded development environment are based on a
> cross-toolchain so you do the build on a high perfomance box.
> 
> Building everything for my arm board on the arm board would be a disater
> for example.

It is true that perl is pervasive and standard, in most scenarios.
However, it is not uncommon in the embedded world for people
to strive to containerize the build process for a whole distro (kernel
included) in order to reduce problems with configure scripts that
(incorrectly) probe the host environment.  I know of at least 3 distros
or distro-building methods that do this.  Working to reduce the
complexity of these constrained environments is a valid goal.

My own reasons for wanting to avoid using perl, however, aren't
based on its availability.

I don't want to start a language war, but in general I think adding
"knowledge of perl" to the list of things someone needs to understand
in order to understand how to build the kernel, would be a bad thing.
There's already a huge mountain to climb for someone trying to learn
enough to modify the kernel (or it's build system).  Adding perl
doesn't IMHO contribute enough value to warrant this extension in
required knowledge.

Also (again IMHO), perl is harder to maintain than other alternatives,
if a full-service scripting language is really needed.

I don't care so much if perl is used in some non-essential role,
like python is used for bloat-o-meter, for example.  But it would
be nice, IMHO, to minimize the external dependencies for required
kernel build infrastructure.

Having said all this, I can't object too strenuously, since I
have some code, out of tree, that uses perl as a pre-processor.
One of the reasons I haven't submitted it is that I find its
use of perl somewhat distasteful, for the reasons mentioned
above.
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Rob Landley]

On Tuesday 10 June 2008 02:54:32 Sam Ravnborg wrote:
> > (Maybe I _am_ the only person who still cares about
> > building on a host without perl.  If I wasn't, somebody else would have
> > acked the patch...)
>
> perl is pretty standard

An implementation is not the same thing as a standard.  If you mean "there is 
one implementation everybody uses, ala excel and Word, and even the Perl guys 
can't reproduce it from scratch as parrot showed", then you're using a 
different definition of the word "standard" than I am.

Or do you mean it comes preinstalled on most modern systems, the way Windows 
does, and who could object to that?

I know from experience that it's an _amazing_ pain to try to cross compile the 
sucker...

> and I fail to see the benefits of avoiding it. 
> For embedded development I see even less benefits as I assume
> any sane embedded development environment are based on a
> cross-toolchain so you do the build on a high perfomance box.
>
> Building everything for my arm board on the arm board would be a disater
> for example.

I build everything for my arm board natively, on an arm system running under 
qemu, calling out to the cross compiler via distcc to accelerate the limited 
parts of the process that cross compiling doesn't actually break.  To get to 
that point, I cross compile just enough to get a native development 
environment, and then I avoid cross compiling from then on because it's an 
enormous source of complexity and random breakage.

I did this because throwing hardware at the problem is cheaper than throwing 
engineering time at the problem, because Moore's Law is on my side, and 
because I find native compiling (where possible) more straightforward and 
conceptually cleaner.

Before 2.6.25, I had a complete self-bootstrapping environment with seven 
packages (busybox, uClibc, make, gcc, binutils, bash, and linux).  Building 
glibc needs perl, if you're using a uClibc based system you may _never_ need 
Perl, and could go up through Linux From Scratch and beyond to a fairly 
powerful system.

I can probably natively build perl under the mini-native environment running 
inside qemu, but it wasn't needed before and there really wasn't a good 
reason to add it.  The task being performed and the build dependency brought 
in to perform it are hugely disproportionate.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Rob Landley]

On Tuesday 10 June 2008 08:49:32 Wolfgang Denk wrote:
> In message <87a5b0800806100625m5a6d20dao47b884bff663c24c@mail.gmail.com> you 
wrote:
> > I'm not convinced it matters at all. Self hosting on an embedded
> > architecture is, as has been mentioned, pretty pointless.
>
> YMMV...
>
> > system. For example I don't think I'll ever be able to self host a
> > kernel build on a board with only 32Mb of on-board RAM.
>
> It depends. We've done this on board with as little as 16 MB RAM.

There's also qemu.  You can native build under emulation.

In addition, if you have a cross compiler but don't want to spend all your 
time lying to ./configure, preventing gcc from linking against the host's 
zlib or grabbing stuff out of /usr/include that your target hasn't got, or 
trying to figure out why on EARTH the perl build decided to use x86 signal 
numbers when you built it for mips, you can build natively inside the 
emulator but use distcc to call out to the cross compiler through the virtual 
network.  This uses the compiler for the heavy lifting of compilation 
_and_nothing_else_.  (make runs natively inside the emulator, ./configure 
runs inside the emulator, headers are #included inside the emulator, 
libraries are linked inside the emulator, anything that wants to look 
at /proc or sysinfo does it natively inside the emulator...)

And the thing is, QEMU is running on fast cheap x86 hardware with buckets of 
memory and disk space, and Moore's Law is doubling the power of it every 18 
months (whereas it's making a lot of embedded stuff cheaper and have longer 
battery life instead, at least until we get to fully disposable computers).  
So yay using x86 as your build envionment, but building natively under 
emulation is now an alternative to trying to make cross compiling scale.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Paul Mundt]

On Tue, Jun 10, 2008 at 12:50:34PM +0200, Sam Ravnborg wrote:
> > 
> > When did this policy change, so that it's now acceptable to depend on
> > Perl, which is roughly equivalent as a tool dependency?
> 
> We have perl as a mandatory part of the kernel build in several places
> for various architectures.
> And I do not recall anyone submitting a bug that they could not build
> a kernel due to the perl dependency.
> But I am obviously well aware of that we use it for the time stuff.
> 
And plenty of places in scripts/ have dependencies on either perl or
python already (and have for some time). Both are pretty ubiquitous these
days, whether people like it or not. There's not much point in trying to
keep the build limping along for people who don't want to set up these
tools on their platforms, since they're going to lose out on half of the
functionality anyways (checkstack, bloat-o-meter, etc.).

Building natively is a good stress tester, and does find a lot of bugs.
For that same reason, if you can build the kernel, you can build python
and perl natively on your platform, too. Some of us have already been
doing this for ages and have no idea what the fuss is about.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Rob Landley]

On Tuesday 10 June 2008 08:47:20 Will Newton wrote:
> On Tue, Jun 10, 2008 at 2:33 PM, David Woodhouse <dwmw2@infradead.org> 
wrote:
> > On Tue, 2008-06-10 at 14:25 +0100, Will Newton wrote:
> >> On Tue, Jun 10, 2008 at 2:12 PM, Jamie Lokier <jamie@shareable.org> 
wrote:
> >> > Wolfgang Denk wrote:
> >> >> Being unable to do this just because we now also would need a  native
> >> >> Perl is indeed a PITA...
> >> >
> >> > You can run the Perl bit with "ssh remote perl", and still do the rest
> >> > of the compile natively.  It's not pretty, but workable.
> >>
> >> I'm not convinced it matters at all. Self hosting on an embedded
> >> architecture is, as has been mentioned, pretty pointless.
> >>
> >> Using a kernel compile as a test isn't such a great idea. Stress tests
> >> of that kind are not particularly useful for pinning down bugs - so
> >> your kernel compile failed, what now? Far better to use LTP tests or
> >> similar that are designed to be reproduceable and tunable for your
> >> system. For example I don't think I'll ever be able to self host a
> >> kernel build on a board with only 32Mb of on-board RAM.
> >
> > Actually, cross-building on NFS does tend to find a _lot_ of issues
> > which crop up with board ports; especially PCI arbitration, DMA
> > coherency, cache and MMU issues. LTP often doesn't catch the same
> > problems.
>
> It may trigger a number of bugs, I don't disagree, but as a test it is
> a blunt instrument. It's likely to be hard to reproduce and have an
> inconsistent failure mode. If you're really serious about testing it's
> not the best solution. It's like using gcc instead of memtest86 to
> test your memory. Eventually it might go wrong but you won't be much
> the wiser about why, or have any way to trim your testcase down so you
> can run it on an in-circuit emulator or pass it to your silicon
> vendor.
>
> > I agree that it's not so easy on a board with 32Mb of RAM, since that's
> > only 4,000,000 bytes -- but 32MiB ought to be _perfectly_ sufficient :)
>
> I would be surprised if it was possible to compile Linux with gcc 4.2
> with 32MiB of total system memory.

Haven't tried, but I generally run emulated builds in 128 megs of ram (on 32 
bit hosts), and I use this:

# This tells gcc to aggressively garbage collect its internal data
# structures.  Without this, gcc triggers the OOM killer trying to rebuild
# itself in 128 megs of ram, which is the QEMU default size.
export CFLAGS="--param ggc-min-expand=0 --param ggc-min-heapsize=8192"

Don't do that on a 64 bit host or your build will slow to a crawl, of course.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Greg Ungerer]

Hi Rob,

Rob Landley wrote:
> On Tuesday 10 June 2008 02:54:32 Sam Ravnborg wrote:
>>> (Maybe I _am_ the only person who still cares about
>>> building on a host without perl.  If I wasn't, somebody else would have
>>> acked the patch...)
>> perl is pretty standard
> 
> An implementation is not the same thing as a standard.  If you mean "there is 
> one implementation everybody uses, ala excel and Word, and even the Perl guys 
> can't reproduce it from scratch as parrot showed", then you're using a 
> different definition of the word "standard" than I am.
> 
> Or do you mean it comes preinstalled on most modern systems, the way Windows 
> does, and who could object to that?
> 
> I know from experience that it's an _amazing_ pain to try to cross compile the 
> sucker...

Ain't that the truth!


>> and I fail to see the benefits of avoiding it. 
>> For embedded development I see even less benefits as I assume
>> any sane embedded development environment are based on a
>> cross-toolchain so you do the build on a high perfomance box.
>>
>> Building everything for my arm board on the arm board would be a disater
>> for example.
> 
> I build everything for my arm board natively, on an arm system running under 
> qemu, calling out to the cross compiler via distcc to accelerate the limited 
> parts of the process that cross compiling doesn't actually break.  To get to 
> that point, I cross compile just enough to get a native development 
> environment, and then I avoid cross compiling from then on because it's an 
> enormous source of complexity and random breakage.

Random breakage?


> I did this because throwing hardware at the problem is cheaper than throwing 
> engineering time at the problem, because Moore's Law is on my side, and 

Are you sure about that?
How well does qemu do SMP?  At all?

Modern x86 and friends are getting most new performance from
more cores. A cross compile today can take advantage of those
for the most part. Your emulated system probably can't.

An order of magnitude compile time (or more) for a native build
is quite a high price to pay :-)

Regards
Greg



> because I find native compiling (where possible) more straightforward and 
> conceptually cleaner.
> 
> Before 2.6.25, I had a complete self-bootstrapping environment with seven 
> packages (busybox, uClibc, make, gcc, binutils, bash, and linux).  Building 
> glibc needs perl, if you're using a uClibc based system you may _never_ need 
> Perl, and could go up through Linux From Scratch and beyond to a fairly 
> powerful system.
> 
> I can probably natively build perl under the mini-native environment running 
> inside qemu, but it wasn't needed before and there really wasn't a good 
> reason to add it.  The task being performed and the build dependency brought 
> in to perform it are hugely disproportionate.
> 
> Rob

-- 
------------------------------------------------------------------------
Greg Ungerer  --  Chief Software Dude       EMAIL:     gerg@snapgear.com
Secure Computing Corporation                PHONE:       +61 7 3435 2888
825 Stanley St,                             FAX:         +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia         WEB: http://www.SnapGear.com

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Rob Landley]

On Wednesday 11 June 2008 00:47:36 Greg Ungerer wrote:
> Hi Rob,
>
> Rob Landley wrote:
> > On Tuesday 10 June 2008 02:54:32 Sam Ravnborg wrote:
> >>> (Maybe I _am_ the only person who still cares about
> >>> building on a host without perl.  If I wasn't, somebody else would have
> >>> acked the patch...)
> >>
> >> perl is pretty standard
> >
> > An implementation is not the same thing as a standard.  If you mean
> > "there is one implementation everybody uses, ala excel and Word, and even
> > the Perl guys can't reproduce it from scratch as parrot showed", then
> > you're using a different definition of the word "standard" than I am.
> >
> > Or do you mean it comes preinstalled on most modern systems, the way
> > Windows does, and who could object to that?
> >
> > I know from experience that it's an _amazing_ pain to try to cross
> > compile the sucker...
>
> Ain't that the truth!
>
> >> and I fail to see the benefits of avoiding it.
> >> For embedded development I see even less benefits as I assume
> >> any sane embedded development environment are based on a
> >> cross-toolchain so you do the build on a high perfomance box.
> >>
> >> Building everything for my arm board on the arm board would be a disater
> >> for example.
> >
> > I build everything for my arm board natively, on an arm system running
> > under qemu, calling out to the cross compiler via distcc to accelerate
> > the limited parts of the process that cross compiling doesn't actually
> > break.  To get to that point, I cross compile just enough to get a native
> > development environment, and then I avoid cross compiling from then on
> > because it's an enormous source of complexity and random breakage.
>
> Random breakage?

Cross compiling breaks stuff, yes.

Most packages don't cross compile at all.  Debian has somewhere north of 
30,000 packages.  Every project that does large scale cross compiling 
(buildroot, gentoo embedded, timesys making fedora cross compile, etc) tends 
to have about 200 packages that cross compile more or less easily, another 
400 or so that can be made to cross compile with _lot_ of effort and a large 
enough rock, and then the project stalls at about that size.

> > I did this because throwing hardware at the problem is cheaper than
> > throwing engineering time at the problem, because Moore's Law is on my
> > side, and
>
> Are you sure about that?
> How well does qemu do SMP?  At all?

QEMU does not currently do SMP at all.  There are proposals to make qemu 
multi-threaded and handle SMP that way, but they're at least a year away from 
anybody actually trying to implement them.  (The big destabilization right 
now is the new code generator, and that's plenty for the moment.  Anybody who 
wanted to wave money at codesourcery could probably get it implemented on a 
deadline for their platforms of interest, but in the absence of that it's not 
really a priority right now.)

Distcc can take advantage of smp, but that won't help the ./configure stage 
and I need to do some work on distcc to teach it to understand more gcc 
command lines.  (For one thing, distcc can't break "compile and link" 
commands ala "gcc hello.c" into separate "compile" and "link" stages, this it 
can't distribute those.  This takes out pretty much the entire uClibc build, 
for example.  But the gcc build parallelizes quite well.)

> Modern x86 and friends are getting most new performance from
> more cores. A cross compile today can take advantage of those
> for the most part. Your emulated system probably can't.

It can if you're using distcc to call out to the cross compiler, which my 
scripts do (./emulator-build.sh does it with the build/* stuff 
and ./run-with-distcc.sh does it for the shipped system-image tarballs).

Some of the other build systems out there hook qemu application emulation up 
to the kernel's misc binary support so a ./configure that builds arm 
executables can run them.  (Openembedded, was it?  Open moko?  Something like 
that.)  But this only solves one of about a dozen major problems cross 
compiling is prone to.  Building natively solves all of 'em, except for 
speed.

> An order of magnitude compile time (or more) for a native build
> is quite a high price to pay :-)

Yup.  Agreed.  No argument there.

However, having one or more full-time engineers devoted to debugging 
cross-compile issues is quite a high price to pay too.  Moore's law really 
doesn't help that one.

I'm not saying either solution is perfect, I'm just saying the "build under 
emulation" approach is a viable alternative that gets more attractive as time 
passes, both because of ongoing development on emulators and because of 
Moore's law on the hardware.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Jamie Lokier]

Rob Landley wrote:
> Most packages don't cross compile at all.  Debian has somewhere north of 
> 30,000 packages.  Every project that does large scale cross compiling 
> (buildroot, gentoo embedded, timesys making fedora cross compile, etc) tends 
> to have about 200 packages that cross compile more or less easily, another 
> 400 or so that can be made to cross compile with _lot_ of effort and a large 
> enough rock, and then the project stalls at about that size.

+1.

I spent several months fixing up cross-compile issues on Gentoo
Embedded a few years ago, for a specific application - only a small
subset of packages needed.  The majority of packages I needed failed
to compile out of the box, one way or another - Glibc dependencies,
arch dependencies, scripts which depend on the host environment, or
invoke the host compiler, or the host Perl (and then depend on it's
byte order). etc.

More recently, I've been compiling a build which is _intended_ for
cross-compilation - it's an old uClinux kit, patched by a third party.
Even that fails to build on newer GNU/Linuxes, as the syntax of GNU
Make has changed, and Bash has changed.  Also GCC 2.old doesn't
compile on current GNU/Linux with GCC 4.new.

Fortunately the latter were a few small, easy to fix issues.  But I
understand now why some find it important to have a replicatable build
environment when you need to get an old distribution out of the closet
to update firmware for some 5 year old device.

Virtual machines ought to be great for that.  They are.  But even
those are surprisingly changable - images that worked on a VM a few
years ago no longer work on the current version of the VM host.

-- Jamie

cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Tim Bird]

Rob Landley wrote:
> However, having one or more full-time engineers devoted to debugging 
> cross-compile issues is quite a high price to pay too.  Moore's law really 
> doesn't help that one.
> 
> I'm not saying either solution is perfect, I'm just saying the "build under 
> emulation" approach is a viable alternative that gets more attractive as time 
> passes, both because of ongoing development on emulators and because of 
> Moore's law on the hardware.

I agree with much that you have said, Rob, and I understand the argument
for getting the most gain from the least resources, but I have a philosophical
problem with working around the cross-compilation problems instead of fixing
them in the upstream packages (or in the autoconf system itself).

Once someone fixes the cross-compilation issues for a package, they usually
stay fixed, if the fixes are mainlined.
 -- Tim


=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[David Woodhouse]

On Thu, 2008-06-12 at 08:23 -0700, Tim Bird wrote:
> Rob Landley wrote:
> > However, having one or more full-time engineers devoted to debugging 
> > cross-compile issues is quite a high price to pay too.  Moore's law really 
> > doesn't help that one.
> > 
> > I'm not saying either solution is perfect, I'm just saying the "build under 
> > emulation" approach is a viable alternative that gets more attractive as time 
> > passes, both because of ongoing development on emulators and because of 
> > Moore's law on the hardware.
> 
> I agree with much that you have said, Rob, and I understand the argument
> for getting the most gain from the least resources, but I have a philosophical
> problem with working around the cross-compilation problems instead of fixing
> them in the upstream packages (or in the autoconf system itself).
> 
> Once someone fixes the cross-compilation issues for a package, they usually
> stay fixed, if the fixes are mainlined.

I don't think that's true, unfortunately. Autoconf makes it _easy_ to do
the wrong thing, and people will often introduce new problems.

If we just made people write portable code and proper Makefiles, it
would be less of an issue :)

-- 
dwmw2

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Mike Frysinger]

On Thu, Jun 12, 2008 at 11:50 AM, David Woodhouse wrote:
> On Thu, 2008-06-12 at 08:23 -0700, Tim Bird wrote:
>> Rob Landley wrote:
>> > However, having one or more full-time engineers devoted to debugging
>> > cross-compile issues is quite a high price to pay too.  Moore's law really
>> > doesn't help that one.
>> >
>> > I'm not saying either solution is perfect, I'm just saying the "build under
>> > emulation" approach is a viable alternative that gets more attractive as time
>> > passes, both because of ongoing development on emulators and because of
>> > Moore's law on the hardware.
>>
>> I agree with much that you have said, Rob, and I understand the argument
>> for getting the most gain from the least resources, but I have a philosophical
>> problem with working around the cross-compilation problems instead of fixing
>> them in the upstream packages (or in the autoconf system itself).
>>
>> Once someone fixes the cross-compilation issues for a package, they usually
>> stay fixed, if the fixes are mainlined.
>
> I don't think that's true, unfortunately. Autoconf makes it _easy_ to do
> the wrong thing, and people will often introduce new problems.
>
> If we just made people write portable code and proper Makefiles, it
> would be less of an issue :)

people cant even write proper *native* makefiles.  mtd-utils for example ;).
-mike

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Paul Mundt]

On Thu, Jun 12, 2008 at 04:50:31PM +0100, David Woodhouse wrote:
> On Thu, 2008-06-12 at 08:23 -0700, Tim Bird wrote:
> > Rob Landley wrote:
> > > However, having one or more full-time engineers devoted to debugging 
> > > cross-compile issues is quite a high price to pay too.  Moore's law really 
> > > doesn't help that one.
> > > 
> > > I'm not saying either solution is perfect, I'm just saying the "build under 
> > > emulation" approach is a viable alternative that gets more attractive as time 
> > > passes, both because of ongoing development on emulators and because of 
> > > Moore's law on the hardware.
> > 
> > I agree with much that you have said, Rob, and I understand the argument
> > for getting the most gain from the least resources, but I have a philosophical
> > problem with working around the cross-compilation problems instead of fixing
> > them in the upstream packages (or in the autoconf system itself).
> > 
> > Once someone fixes the cross-compilation issues for a package, they usually
> > stay fixed, if the fixes are mainlined.
> 
> I don't think that's true, unfortunately. Autoconf makes it _easy_ to do
> the wrong thing, and people will often introduce new problems.
> 
> If we just made people write portable code and proper Makefiles, it
> would be less of an issue :)
> 
The other issue is that people that are working in this space tend to do
very little beyond solving their immediate troubles. Since perl was
mentioned, it also makes a good example. Embedded distros have been
cross-compiling perl for pretty much the last decade, yet even today
people are having the exact same issues and acting as this is some sort
of surprise that needs to be worked arond, rather than treating it as a
solved problem. If you opt to cross-compile, having to deal with those
sorts of things is the price you pay.

Holding the kernel hostage to this kind of brain-damage is simply beyond
ridiculous. There are a lot of things outside of the kernel that have a
dependency on perl too, how much time do people want to spend trying to
fix the build system to match their environment before they realize their
environment needs to scale to match the build environment that everyone
else is already using?

Building under qemu in a "native" environment is one way to work around
this problem, and a lot of companies have opted for that approach rather
than trying to fix the problematic packages. If you aren't building
natively, your options are effectively limited by convention. You can
either try to fix the packages in question, convince the package
developers to rip out the parts that cause trouble for your environment,
fix your own build environment to meet the needs of the packages, or
whine about it on a mailing list. Empirically we already know which one
of those options is going to win out. ;-)

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[David Woodhouse]

On Thu, 2008-06-12 at 12:05 -0400, Mike Frysinger wrote:
> people cant even write proper *native* makefiles.  mtd-utils for
> example ;).

Criticism in 'diff -u' form preferred :)

-- 
dwmw2

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Robert P. J. Day]

On Thu, 12 Jun 2008, Mike Frysinger wrote:

> On Thu, Jun 12, 2008 at 11:50 AM, David Woodhouse wrote:

> > If we just made people write portable code and proper Makefiles,
> > it would be less of an issue :)
>
> people cant even write proper *native* makefiles.  mtd-utils for
> example ;).

meooowww!  :-)  but at the risk of dragging this even further
off-topic, i am *constantly* asked by people how to set up makefiles
for their software project, and what would be nice is a small
collection of examples of a makefile (or makefiles) done *right*.  as
in, properly recursive, supports cross-compiling without having to
remove your left nad, etc, etc.

so ... would anyone like to recommend a software package or two
somewhere whose makefile(s) is/are, ITHO, done well?  that i/we could
just point at and say, "do it like that!"  anyone?  anyone?  bueller?

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
    Have classroom, will lecture.

http://crashcourse.ca                          Waterloo, Ontario, CANADA
========================================================================

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Mike Frysinger]

On Thu, Jun 12, 2008 at 12:08 PM, David Woodhouse wrote:
> On Thu, 2008-06-12 at 12:05 -0400, Mike Frysinger wrote:
>> people cant even write proper *native* makefiles.  mtd-utils for
>> example ;).
>
> Criticism in 'diff -u' form preferred :)

i sent via `git-send-email` yesterday to the mtd lists ... either it
got silently dropped, or someone hates me
-mike

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Tim Bird]

David Woodhouse wrote:
>>... fixing
>> them in the upstream packages (or in the autoconf system itself).
>>
>> Once someone fixes the cross-compilation issues for a package, they usually
>> stay fixed, if the fixes are mainlined.
> 
> I don't think that's true, unfortunately. Autoconf makes it _easy_ to do
> the wrong thing, and people will often introduce new problems.

If autoconf is the problem (and I think it is), then that's what
should be fixed (see my original post).  At a minimum, it would be
nice if it had more built-in detection and warning of techniques
that were dangerous for cross-compilation.

While I was writing the above,

Paul Mundt wrote:
> You can
> either try to fix the packages in question, convince the package
> developers to rip out the parts that cause trouble for your environment,
> fix your own build environment to meet the needs of the packages, or
> whine about it on a mailing list. Empirically we already know which one
> of those options is going to win out. ;-)

LOL.  Well, at least Rob has put his money where his mouth is (so to
speak) with Firmware Linux.  The chance that I'll do anything but whine
about autoconf is slim indeed...  I'll shut up now!
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bill Gatliff]

Guys:

> If you opt to cross-compile, having to deal with those
> sorts of things is the price you pay.


If the build system derives from autoconf, then a hacked-up config.cache (or
equivalent command-line args) often solves problems for me.  Just give the cache
the answers that it would otherwise have to get by running code on the target
machine.

That's how emdebian is doing a bunch of their stuff, and I have to admit that it
works pretty darned well.  It's also handy for configuration management, since
the cache file itself is plaintext and therefore svn/git/bzr/cvs/...-friendly.



b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Paul Mundt]

On Thu, Jun 12, 2008 at 11:28:10AM -0500, Bill Gatliff wrote:
> Guys:
> 
> > If you opt to cross-compile, having to deal with those
> > sorts of things is the price you pay.
> 
> 
> If the build system derives from autoconf, then a hacked-up config.cache (or
> equivalent command-line args) often solves problems for me.  Just give the cache
> the answers that it would otherwise have to get by running code on the target
> machine.
> 
> That's how emdebian is doing a bunch of their stuff, and I have to admit that it
> works pretty darned well.  It's also handy for configuration management, since
> the cache file itself is plaintext and therefore svn/git/bzr/cvs/...-friendly.
> 
Yes, that's the easy case. It's things like perl that are the corner
cases, and my objection comes from the fact that people think we ought to
not have the kernel depend on perl rather than just fixing the package
itself. Autoconf/libtool damage is an entirely different problem :-)

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[David Woodhouse]

On Thu, 2008-06-12 at 11:28 -0500, Bill Gatliff wrote:
> Guys:
> 
> > If you opt to cross-compile, having to deal with those
> > sorts of things is the price you pay.
> 
> 
> If the build system derives from autoconf, then a hacked-up config.cache (or
> equivalent command-line args) often solves problems for me.  Just give the cache
> the answers that it would otherwise have to get by running code on the target
> machine.
> 
> That's how emdebian is doing a bunch of their stuff, and I have to admit that it
> works pretty darned well.  It's also handy for configuration management, since
> the cache file itself is plaintext and therefore svn/git/bzr/cvs/...-friendly.

Yeah, I was building Red Hat Linux packages for sh3 many years ago,
using tricks like that. But there was always _something_ else going
wrong, however much you hacked around it. And a lot of it would only
turn up at runtime, not build time. I would never consider shipping a
product with a large number of userspace packages cross-compiled.

For minimal file systems with a select handful of tools which can be
tested exhaustively, it's not so bad. But for any 'full-featured'
userspace, I think cross-compilation is completely insane.

-- 
dwmw2

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Mike Frysinger]

On Thu, Jun 12, 2008 at 12:31 PM, Paul Mundt wrote:
> On Thu, Jun 12, 2008 at 11:28:10AM -0500, Bill Gatliff wrote:
>> > If you opt to cross-compile, having to deal with those
>> > sorts of things is the price you pay.
>>
>>
>> If the build system derives from autoconf, then a hacked-up config.cache (or
>> equivalent command-line args) often solves problems for me.  Just give the cache
>> the answers that it would otherwise have to get by running code on the target
>> machine.
>>
>> That's how emdebian is doing a bunch of their stuff, and I have to admit that it
>> works pretty darned well.  It's also handy for configuration management, since
>> the cache file itself is plaintext and therefore svn/git/bzr/cvs/...-friendly.
>
> Yes, that's the easy case. It's things like perl that are the corner
> cases, and my objection comes from the fact that people think we ought to
> not have the kernel depend on perl rather than just fixing the package
> itself. Autoconf/libtool damage is an entirely different problem :-)

of the core packages, perl and openssl tend to be heavily damaged.
openssl because it depends on perl instead of a real build system.
-mike

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Adrian Bunk]

On Thu, Jun 12, 2008 at 05:37:35PM +0100, David Woodhouse wrote:
> On Thu, 2008-06-12 at 11:28 -0500, Bill Gatliff wrote:
> > Guys:
> > 
> > > If you opt to cross-compile, having to deal with those
> > > sorts of things is the price you pay.
> > 
> > 
> > If the build system derives from autoconf, then a hacked-up config.cache (or
> > equivalent command-line args) often solves problems for me.  Just give the cache
> > the answers that it would otherwise have to get by running code on the target
> > machine.
> > 
> > That's how emdebian is doing a bunch of their stuff, and I have to admit that it
> > works pretty darned well.  It's also handy for configuration management, since
> > the cache file itself is plaintext and therefore svn/git/bzr/cvs/...-friendly.
> 
> Yeah, I was building Red Hat Linux packages for sh3 many years ago,
> using tricks like that. But there was always _something_ else going
> wrong, however much you hacked around it. And a lot of it would only
> turn up at runtime, not build time. I would never consider shipping a
> product with a large number of userspace packages cross-compiled.
> 
> For minimal file systems with a select handful of tools which can be
> tested exhaustively, it's not so bad. But for any 'full-featured'
> userspace, I think cross-compilation is completely insane.

There's also the middle way taken by e.g. Scratchbox of using 
cross-compilers and other tools from the machine the compilation
is done on, but emulating a native build for the software being
compiled.

This works quite good in practice.

> dwmw2

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bill Gatliff]

Paul Mundt wrote:

> Yes, that's the easy case. It's things like perl that are the corner
> cases, and my objection comes from the fact that people think we ought to
> not have the kernel depend on perl rather than just fixing the package
> itself. Autoconf/libtool damage is an entirely different problem :-)
> 

At first glance, it seems like checkincludes.pl could be duplicated by egrep |
uniq | wc vs. egrep | wc.  Not quite sure what checkversion.pl is trying to do.

The namespace.pl script looks optional, as does export_report.pl.

So maybe we could _reduce_ dependency on perl, if there's any advantage to gain
by doing so.  But the kernel build machinery isn't dependent on very many other
systems (just tcl, bash and gcc-core), so I don't really see the point unless
you could completely eliminate perl.  And I don't see how you might do *that*
without dragging in a bunch of stuff to replace it, thereby increasing the
number of dependencies.

Any machine beefy enough to build the kernel is more than adequate to host perl
as well.  So maybe the "corner case" is situations where you're trying to build
the kernel on an overly-minimalist host--- perhaps one that you constructed
yourself using linux-from-scratch?

Not trying to fan the flames any, of course.  I'm just sayin'...

The worst for cross-building so far for me has been openssh.  And even that's
tolerable once you find your way around the autoconf stuff.  But this would be
the wrong list for that.  :)


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bill Gatliff]

David Woodhouse wrote:
> Yeah, I was building Red Hat Linux packages for sh3 many years ago,
> using tricks like that. But there was always _something_ else going
> wrong, however much you hacked around it. And a lot of it would only
> turn up at runtime, not build time. I would never consider shipping a
> product with a large number of userspace packages cross-compiled.

I take it on a case-by-case basis.  But yea, validation is definitely a pain.
Especially with all the silent failures (i.e. getting endianness wrong) that the
configure script creates for you.

> For minimal file systems with a select handful of tools which can be
> tested exhaustively, it's not so bad. But for any 'full-featured'
> userspace, I think cross-compilation is completely insane.

We both agree that it'll *drive* you insane, at least.  And I'm 90% with you on
the rest.  :)


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bill Gatliff]

Adrian Bunk wrote:
> There's also the middle way taken by e.g. Scratchbox of using 
> cross-compilers and other tools from the machine the compilation
> is done on, but emulating a native build for the software being
> compiled.

I suppose.  But I've never gotten Scratchbox to work for me, at all.

Whether the problem is me or Scratchbox, I don't really want to know.  :)


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Mike Frysinger]

On Thu, Jun 12, 2008 at 1:14 PM, Bill Gatliff wrote:
> Paul Mundt wrote:
>> Yes, that's the easy case. It's things like perl that are the corner
>> cases, and my objection comes from the fact that people think we ought to
>> not have the kernel depend on perl rather than just fixing the package
>> itself. Autoconf/libtool damage is an entirely different problem :-)
>
> At first glance, it seems like checkincludes.pl could be duplicated by egrep |
> uniq | wc vs. egrep | wc.  Not quite sure what checkversion.pl is trying to do.
>
> The namespace.pl script looks optional, as does export_report.pl.
>
> So maybe we could _reduce_ dependency on perl, if there's any advantage to gain
> by doing so.  But the kernel build machinery isn't dependent on very many other
> systems (just tcl, bash and gcc-core), so I don't really see the point unless
> you could completely eliminate perl.  And I don't see how you might do *that*
> without dragging in a bunch of stuff to replace it, thereby increasing the
> number of dependencies.

the idea is to keep perl from being required to build the kernel
itself.  tcl is not required in this case either.  if you want to use
crazy languages to optional things above and beyond, then no one is
going to complain (and if they do, well screw them).
-mike

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Sam Ravnborg]

On Thu, Jun 12, 2008 at 12:14:32PM -0500, Bill Gatliff wrote:
> Paul Mundt wrote:
> 
> > Yes, that's the easy case. It's things like perl that are the corner
> > cases, and my objection comes from the fact that people think we ought to
> > not have the kernel depend on perl rather than just fixing the package
> > itself. Autoconf/libtool damage is an entirely different problem :-)
> > 
> 
> At first glance, it seems like checkincludes.pl could be duplicated by egrep |
> uniq | wc vs. egrep | wc.  Not quite sure what checkversion.pl is trying to do.
> 
> The namespace.pl script looks optional, as does export_report.pl.
> 
> So maybe we could _reduce_ dependency on perl, if there's any advantage to gain
> by doing so.  But the kernel build machinery isn't dependent on very many other
> systems (just tcl, bash and gcc-core), so I don't really see the point unless
> you could completely eliminate perl.  And I don't see how you might do *that*
> without dragging in a bunch of stuff to replace it, thereby increasing the
> number of dependencies.

All the noise about the perl dependency of the kernel build are
rooted in two things:

1) That we now have a mandatory part of the build that uses perl (see kernel/Makefile)
2) That I told that I consider rewriting the core of the build system in perl

As for 1) I have seen a patch submitted once. And I do not hear many complaint either,
albeit Rob Landley is a bit loud here (and he was also the one submitting the patch).
That patch was not acceptable as is - and noone has updated it.

As for 2) then let see if that ever happens ;-)

	Sam

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Enrico Weigelt]

* Rob Landley <rob@landley.net> schrieb:

Hi,


> There's also qemu.  You can native build under emulation.

did you ever consider that crosscompiling is not only good for 
some other arch, but a few more things ?
 
> In addition, if you have a cross compiler but don't want to spend all your 
> time lying to ./configure, preventing gcc from linking against the host's 
> zlib or grabbing stuff out of /usr/include that your target hasn't got, or 

#1: use a proper (sysroot'ed) toolchain
#2: fix broken configure.in's (and feed back to upstream or OSS-QM)
#3: replace libtool by unitool

> trying to figure out why on EARTH the perl build decided to use x86 signal 

Perl never was crosscompile-capable.
I've rewrote much of the buildscripts, but not finished yet.

Just in case that anyone's interested in it, let me know.

> libraries are linked inside the emulator, anything that wants to look 
> at /proc or sysinfo does it natively inside the emulator...)

Only crap sw looks at /proc at build time.
Yes, there's *much* crap sw out there :(


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Enrico Weigelt]

* Rob Landley <rob@landley.net> schrieb:

> Cross compiling breaks stuff, yes.
> 
> Most packages don't cross compile at all.  Debian has somewhere north of 
> 30,000 packages.  Every project that does large scale cross compiling 
> (buildroot, gentoo embedded, timesys making fedora cross compile, etc) tends 
> to have about 200 packages that cross compile more or less easily, another 
> 400 or so that can be made to cross compile with _lot_ of effort and a large 
> enough rock, and then the project stalls at about that size.

The problem is: most embedded projects don't make really general-purpose
fixes (instead strange things like hacking up autogenerated files), so 
they can't feed back to upstream.

IMHO, a huge waste of working time.

Did someone ever hear of the OSS-QM project ?

> Some of the other build systems out there hook qemu application emulation up 
> to the kernel's misc binary support so a ./configure that builds arm 
> executables can run them.

WTF ?!


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Josh Boyer]

On Thu, Jun 12, 2008 at 12:05 PM, Mike Frysinger <vapier.adi@gmail.com> wrote:
> On Thu, Jun 12, 2008 at 11:50 AM, David Woodhouse wrote:
>> On Thu, 2008-06-12 at 08:23 -0700, Tim Bird wrote:
>>> Rob Landley wrote:
>>> > However, having one or more full-time engineers devoted to debugging
>>> > cross-compile issues is quite a high price to pay too.  Moore's law really
>>> > doesn't help that one.
>>> >
>>> > I'm not saying either solution is perfect, I'm just saying the "build under
>>> > emulation" approach is a viable alternative that gets more attractive as time
>>> > passes, both because of ongoing development on emulators and because of
>>> > Moore's law on the hardware.
>>>
>>> I agree with much that you have said, Rob, and I understand the argument
>>> for getting the most gain from the least resources, but I have a philosophical
>>> problem with working around the cross-compilation problems instead of fixing
>>> them in the upstream packages (or in the autoconf system itself).
>>>
>>> Once someone fixes the cross-compilation issues for a package, they usually
>>> stay fixed, if the fixes are mainlined.
>>
>> I don't think that's true, unfortunately. Autoconf makes it _easy_ to do
>> the wrong thing, and people will often introduce new problems.
>>
>> If we just made people write portable code and proper Makefiles, it
>> would be less of an issue :)
>
> people cant even write proper *native* makefiles.  mtd-utils for example ;).

What's wrong with it?  I'll fix it.

josh

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Paul Mundt <lethal@linux-sh.org> schrieb:

> The other issue is that people that are working in this space tend to do
> very little beyond solving their immediate troubles. 

Sad but true :(

> Since perl was mentioned, it also makes a good example. Embedded distros 
> have been cross-compiling perl for pretty much the last decade, yet even 

Now I'm curious: where can I find a patch ?

Upstream's build scripts are really far from being cross'able.

> You can either try to fix the packages in question, convince the package
> developers to rip out the parts that cause trouble for your environment,
> fix your own build environment to meet the needs of the packages, or
> whine about it on a mailing list. Empirically we already know which one
> of those options is going to win out. ;-)

I, personally, prefer to *solve* problems instead of trying to work 
around somehow. In longer terms, this worked quite good for me.

Of course, fixing requires that you find out what's the problem
really is before ;-P


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Bill Gatliff <bgat@billgatliff.com> schrieb:

> If the build system derives from autoconf, then a hacked-up 
> config.cache (or equivalent command-line args) often solves 
> problems for me. 

Only if you're working on *one specific* target for a long time.
I, for example, have to support lots of different targets, so your
approach does not work for me. Ah, and it's not *solving* any problem,
just deferring it to some other day.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Tim Bird <tim.bird@am.sony.com> schrieb:

> If autoconf is the problem (and I think it is), then that's what
> should be fixed (see my original post).  

ACK. One of the first things I did was replacing libtool and
kicked off broken-by-design macros like AC_TRY_RUN. This solved
at >80% of the problems for me.

> At a minimum, it would be nice if it had more built-in detection 
> and warning of techniques that were dangerous for cross-compilation.

Meanwhile, autoconf warns on some broken macros like AC_TRY_RUN.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bernhard Fischer]

On Thu, Jun 12, 2008 at 12:38:42PM -0400, Mike Frysinger wrote:
>On Thu, Jun 12, 2008 at 12:31 PM, Paul Mundt wrote:
>> On Thu, Jun 12, 2008 at 11:28:10AM -0500, Bill Gatliff wrote:

>>> That's how emdebian is doing a bunch of their stuff, and I have to admit that it
>>> works pretty darned well.  It's also handy for configuration management, since
>>> the cache file itself is plaintext and therefore svn/git/bzr/cvs/...-friendly.
>>
>> Yes, that's the easy case. It's things like perl that are the corner
>> cases, and my objection comes from the fact that people think we ought to
>> not have the kernel depend on perl rather than just fixing the package
>> itself. Autoconf/libtool damage is an entirely different problem :-)
>
>of the core packages, perl and openssl tend to be heavily damaged.
>openssl because it depends on perl instead of a real build system.

It's also a pity that automake doesn't work with microperl. And even for
the most simple .pm you have to do perlward^Wawkward stuff like

ifeq ($(BR2_PACKAGE_AUTOMAKE),y)
        # we need to build a perl for the host just for Errno.pm
        (cd $(MICROPERL_DIR); ./Configure -de; \
         $(MAKE) CC="$(HOSTCC)"; \
         $(SHELL) ext/util/make_ext nonxs Errno MAKE="$(firstword $(MAKE))"; \
        )
endif
        (cd $(MICROPERL_DIR); chmod u+w uconfig.h; . ./uconfig.sh; \
         $(MAKE) -f Makefile.micro regen_uconfig; \
[snip rest of the misery]

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Wolfgang Denk]

In message <20080612181806.GA7423@nibiru.local> you wrote:
> 
> Perl never was crosscompile-capable.
> I've rewrote much of the buildscripts, but not finished yet.

Note that Perl is included with our ELDK, and we do cross-compile it.
It's not exactly trivial, but not really difficult either.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
I have a theory that it's impossible to prove anything, but  I  can't
prove it.

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Wolfgang Denk]

In message <20080612183057.GC7423@nibiru.local> you wrote:
> 
> > Since perl was mentioned, it also makes a good example. Embedded distros 
> > have been cross-compiling perl for pretty much the last decade, yet even 
> 
> Now I'm curious: where can I find a patch ?

See for example:

http://git.denx.de/?p=eldk/build.git;a=tree;f=target_rpms/perl;h=fd830dd2e6bf86703fab84b83922eea8c10f0310;hb=3e5c010c91e17e819048937eb4105d762c7e1497


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Nothing is easier than to denounce  the  evildoer;  nothing  is  more
difficult than to understand him.                 - Fyodor Dostoevski

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bill Gatliff]

Enrico Weigelt wrote:
> * Bill Gatliff <bgat@billgatliff.com> schrieb:
> 
>> If the build system derives from autoconf, then a hacked-up 
>> config.cache (or equivalent command-line args) often solves 
>> problems for me. 
> 
> Only if you're working on *one specific* target for a long time.
> I, for example, have to support lots of different targets, so your
> approach does not work for me.

I'm not exactly a one-specific-target kinda guy, either.  I routinely move
between MIPS, ARM, PPC and x86.

Yea, that means target-specific config.cache files.  But that works for me.

> Ah, and it's not *solving* any problem,
> just deferring it to some other day.

If "solving" means helping the tool get the answer it needs, even when running
in an environment/configuration it wasn't intended for, without breaking
compatibility with everyone else out there still using the same tool, then I
disagree.

But hey, I never said you had to do it my way.  :)


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Mike Frysinger]

On Thu, Jun 12, 2008 at 2:29 PM, Josh Boyer wrote:
> On Thu, Jun 12, 2008 at 12:05 PM, Mike Frysinger wrote:
>> On Thu, Jun 12, 2008 at 11:50 AM, David Woodhouse wrote:
>>> On Thu, 2008-06-12 at 08:23 -0700, Tim Bird wrote:
>>>> Rob Landley wrote:
>>>> > However, having one or more full-time engineers devoted to debugging
>>>> > cross-compile issues is quite a high price to pay too.  Moore's law really
>>>> > doesn't help that one.
>>>> >
>>>> > I'm not saying either solution is perfect, I'm just saying the "build under
>>>> > emulation" approach is a viable alternative that gets more attractive as time
>>>> > passes, both because of ongoing development on emulators and because of
>>>> > Moore's law on the hardware.
>>>>
>>>> I agree with much that you have said, Rob, and I understand the argument
>>>> for getting the most gain from the least resources, but I have a philosophical
>>>> problem with working around the cross-compilation problems instead of fixing
>>>> them in the upstream packages (or in the autoconf system itself).
>>>>
>>>> Once someone fixes the cross-compilation issues for a package, they usually
>>>> stay fixed, if the fixes are mainlined.
>>>
>>> I don't think that's true, unfortunately. Autoconf makes it _easy_ to do
>>> the wrong thing, and people will often introduce new problems.
>>>
>>> If we just made people write portable code and proper Makefiles, it
>>> would be less of an issue :)
>>
>> people cant even write proper *native* makefiles.  mtd-utils for example ;).
>
> What's wrong with it?  I'll fix it.

is linux-mtd@lists.infradead.org not the place to post ?  that's where
i sent the first fix yesterday ... not that i'm subscribed since i
dont have a direct interest in mtd development ...
-mike

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Enrico Weigelt]

* Wolfgang Denk <wd@denx.de> schrieb:
> In message <20080612181806.GA7423@nibiru.local> you wrote:
> > 
> > Perl never was crosscompile-capable.
> > I've rewrote much of the buildscripts, but not finished yet.
> 
> Note that Perl is included with our ELDK, and we do cross-compile it.
> It's not exactly trivial, but not really difficult either.

URL ?


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[David VomLehn]

Enrico Weigelt wrote:
> * Rob Landley <rob@landley.net> schrieb:
>   
>> Cross compiling breaks stuff, yes.
>>
>> Most packages don't cross compile at all.  Debian has somewhere north of 
>> 30,000 packages.  Every project that does large scale cross compiling 
>> (buildroot, gentoo embedded, timesys making fedora cross compile, etc) tends 
>> to have about 200 packages that cross compile more or less easily, another 
>> 400 or so that can be made to cross compile with _lot_ of effort and a large 
>> enough rock, and then the project stalls at about that size.
>>     
>
> The problem is: most embedded projects don't make really general-purpose
> fixes (instead strange things like hacking up autogenerated files), so 
> they can't feed back to upstream.
>
> IMHO, a huge waste of working time.
>
>   
Amen, brother. I'm fortunate in that I work for an organization that is 
quite good about enforcing code reviews, specifically, the QA 
organization is empowered to reject changes that do not have code review 
notes. I also have a fairly broad scope, so I'm in on code reviews for a 
number of open source components. At each such review, one of my 
criteria is whether the change is suitable for pushing back to the 
appropriate community. This is not necessarily a short-term way to make 
friends, but the long-term effects will be good both for the company and 
for the open source community in general.

Now, if we can only get the time to actually push all the backlogged 
fixes out...
--
David VomLehn, dvomlehn@cisco.com
The opinions expressed herein are likely mine, but might not be my 
employer's...

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[James Chapman]

David VomLehn wrote:
> Enrico Weigelt wrote:
>> * Rob Landley <rob@landley.net> schrieb:
>>  
>>> Cross compiling breaks stuff, yes.
>>>
>>> Most packages don't cross compile at all.  Debian has somewhere north 
>>> of 30,000 packages.  Every project that does large scale cross 
>>> compiling (buildroot, gentoo embedded, timesys making fedora cross 
>>> compile, etc) tends to have about 200 packages that cross compile 
>>> more or less easily, another 400 or so that can be made to cross 
>>> compile with _lot_ of effort and a large enough rock, and then the 
>>> project stalls at about that size.
>>>     
>>
>> The problem is: most embedded projects don't make really general-purpose
>> fixes (instead strange things like hacking up autogenerated files), so 
>> they can't feed back to upstream.
>>
>> IMHO, a huge waste of working time.
>>
>>   
> Amen, brother. I'm fortunate in that I work for an organization that is 
> quite good about enforcing code reviews, specifically, the QA 
> organization is empowered to reject changes that do not have code review 
> notes. I also have a fairly broad scope, so I'm in on code reviews for a 
> number of open source components. At each such review, one of my 
> criteria is whether the change is suitable for pushing back to the 
> appropriate community. This is not necessarily a short-term way to make 
> friends, but the long-term effects will be good both for the company and 
> for the open source community in general.
> 
> Now, if we can only get the time to actually push all the backlogged 
> fixes out...

Er, is that GPL or LGPL code that you're modifying? If so, you *have* to 
push those code changes out (make them available to others), whether you 
think people will be interested or not!

> -- 
> David VomLehn, dvomlehn@cisco.com
> The opinions expressed herein are likely mine, but might not be my 
> employer's...

-- 
James Chapman
Katalix Systems Ltd
http://www.katalix.com
Catalysts for your Embedded Linux software development

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Mike Frysinger]

On Thu, Jun 12, 2008 at 5:42 PM, James Chapman wrote:
> David VomLehn wrote:
>> Amen, brother. I'm fortunate in that I work for an organization that is
>> quite good about enforcing code reviews, specifically, the QA organization
>> is empowered to reject changes that do not have code review notes. I also
>> have a fairly broad scope, so I'm in on code reviews for a number of open
>> source components. At each such review, one of my criteria is whether the
>> change is suitable for pushing back to the appropriate community. This is
>> not necessarily a short-term way to make friends, but the long-term effects
>> will be good both for the company and for the open source community in
>> general.
>>
>> Now, if we can only get the time to actually push all the backlogged fixes
>> out...
>
> Er, is that GPL or LGPL code that you're modifying? If so, you *have* to
> push those code changes out (make them available to others), whether you
> think people will be interested or not!

umm, not really.  only if (1) he gives a binary to someone and (2)
they ask him for the source.  if he doesnt distribute or no one asks,
he doesnt have to do squat.
-mike

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Tim Bird]

Mike Frysinger wrote:
>> Er, is that GPL or LGPL code that you're modifying? If so, you *have* to
>> push those code changes out (make them available to others), whether you
>> think people will be interested or not!
> 
> umm, not really.  only if (1) he gives a binary to someone and (2)
> they ask him for the source.  if he doesnt distribute or no one asks,
> he doesnt have to do squat.
This is closer to correct, but missing some important details.

Start the GPL compliance tutorial/flameware in 3, 2, 1...

Luckily this in only on the linux-embedded list.  If it were on LKML
the fun would really begin. :-)

 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Mike Frysinger]

On Thu, Jun 12, 2008 at 5:53 PM, Tim Bird wrote:
> Mike Frysinger wrote:
>>> Er, is that GPL or LGPL code that you're modifying? If so, you *have* to
>>> push those code changes out (make them available to others), whether you
>>> think people will be interested or not!
>>
>> umm, not really.  only if (1) he gives a binary to someone and (2)
>> they ask him for the source.  if he doesnt distribute or no one asks,
>> he doesnt have to do squat.
>
> This is closer to correct, but missing some important details.
>
> Start the GPL compliance tutorial/flameware in 3, 2, 1...

yeah, i really dont think licensing things belong here.  sorry for following up.

how about this policy: if you want to make a statement, go pay a
lawyer.  but that statement still shouldnt be made here ;).
-mike

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Jim Freeman]

On Thu, Jun 12, 2008 at 05:46:42PM -0400, Mike Frysinger wrote:
> On Thu, Jun 12, 2008 at 5:42 PM, James Chapman wrote:
> > David VomLehn wrote:
> >> Amen, brother. I'm fortunate in that I work for an organization that is
> >> quite good about enforcing code reviews, specifically, the QA organization
> >> is empowered to reject changes that do not have code review notes. I also
> >> have a fairly broad scope, so I'm in on code reviews for a number of open
> >> source components. At each such review, one of my criteria is whether the
> >> change is suitable for pushing back to the appropriate community. This is
> >> not necessarily a short-term way to make friends, but the long-term effects
> >> will be good both for the company and for the open source community in
> >> general.
> >>
> >> Now, if we can only get the time to actually push all the backlogged fixes
> >> out...
> >
> > Er, is that GPL or LGPL code that you're modifying? If so, you *have* to
> > push those code changes out (make them available to others), whether you
> > think people will be interested or not!
> 
> umm, not really.  only if (1) he gives a binary to someone and (2)
> they ask him for the source.  if he doesnt distribute or no one asks,
> he doesnt have to do squat.
> -mike

And I'm just betting that when he said "push ... fixes ... out"
he meant "work to get them incorporated back upstream", not just
make them available to requesters.

Most vendors these days have finally gotten the clue that sources/changes
have to be made available to downstream requesters, but far fewer
are sufficiently self-enlightened to figure out that changes need to
be accepted upstream for them to keep flowing back.  And to make that
happen, vendors have to take on substantially higher overhead to win
acceptance of patches/changes upstream, an undertaking often sadly
fraught with hassle, uncertainty, and even peril.  So they mostly
don't bother.  To their (and their customers, and our) long-term
detriment.

And Cisco has probably learned by now (and by sad experience) to do
the Right (tm) thing.

...jfree

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Rob Landley]

On Thursday 12 June 2008 11:12:13 Robert P. J. Day wrote:
> On Thu, 12 Jun 2008, Mike Frysinger wrote:
> > On Thu, Jun 12, 2008 at 11:50 AM, David Woodhouse wrote:
> > > If we just made people write portable code and proper Makefiles,
> > > it would be less of an issue :)
> >
> > people cant even write proper *native* makefiles.  mtd-utils for
> > example ;).
>
> meooowww!  :-)  but at the risk of dragging this even further
> off-topic, i am *constantly* asked by people how to set up makefiles
> for their software project, and what would be nice is a small
> collection of examples of a makefile (or makefiles) done *right*.  as
> in,

Make doesn't scale.

99% of the builds in the open source world are "make all", and most of the 
smaller projects build natively on modern dual processor 2ghz laptops in 
under 10 seconds anyway.

The larger projects with significant build times usually find that make 
doesn't suit their needs, so that they write some other build system.  
Sometimes they do it on top of make, such as the kernel's kbuild.  Sometimes 
they use another language like apache's ANT.  Sometimes they roll their own 
in C (anybody remember X11's imake?)  KDE switched to cmake: 
http://lwn.net/Articles/188693/

Current compilers have a "build at once" mode where they suck the whole 
project in and run the optimizer on it at once, resulting in noticeably 
smaller and faster output at the expense of needing buckets of memory to hold 
all the source code and intermediate structures in memory at once.  The main 
roadblock to making use of this?  Ripping out the existing makefiles and 
replacing them with a very small shell script that does something similar 
to "gcc *.c".

The first question you should be asking when doing a new build system from 
scratch is probably "should I really be using make"?

> properly recursive, 

Recursive make considered harmful:
  http://aegis.sourceforge.net/auug97.pdf

How is needing to call make recursively _not_ just another way of sayng "the 
dependency checking make does, which was the central idea behind its design, 
is a lost cause and we need to jettison it to do builds"?

I just did a "make distclean" on a qemu tree I had lying around.  On my 1.7 
ghz 64 bit laptop, it took 9.2 seconds to figure out it had nothing to do, 
just because it had to recurse into so many subdirectories to do it.

> supports cross-compiling without having to 
> remove your left nad, etc, etc.

Can of worms.  Answering that question would be a longer post than the rest of 
this combined, but I've pointed out that I don't think going there is worth 
it anymore for most packages.

> so ... would anyone like to recommend a software package or two
> somewhere whose makefile(s) is/are, ITHO, done well?  that i/we could
> just point at and say, "do it like that!"  anyone?  anyone?  bueller?

A software package with makefiles done well is like asking for a _good_ 
example of cobol programming.

The one and only advantage make has right now is that it's ubiquitous.  So 
was "ed", which dates from the same general timeframe.  That's not an 
endorsement.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bill Gatliff]

Rob Landley wrote:

> I just did a "make distclean" on a qemu tree I had lying around.  On my 1.7 
> ghz 64 bit laptop, it took 9.2 seconds to figure out it had nothing to do, 
> just because it had to recurse into so many subdirectories to do it.

Which is precisely why recursive makes are considered bad.


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Rob Landley]

On Thursday 12 June 2008 11:23:19 Tim Bird wrote:
> David Woodhouse wrote:
> > I don't think that's true, unfortunately. Autoconf makes it _easy_ to do
> > the wrong thing, and people will often introduce new problems.
>
> If autoconf is the problem (and I think it is), then that's what
> should be fixed (see my original post).  At a minimum, it would be
> nice if it had more built-in detection and warning of techniques
> that were dangerous for cross-compilation.

If autoconf is currently too complicated for people to use correctly, 
complicating it more probably isn't the answer.  (Down that path lies C++.)

> Paul Mundt wrote:
> > You can
> > either try to fix the packages in question, convince the package
> > developers to rip out the parts that cause trouble for your environment,
> > fix your own build environment to meet the needs of the packages, or
> > whine about it on a mailing list. Empirically we already know which one
> > of those options is going to win out. ;-)
>
> LOL.  Well, at least Rob has put his money where his mouth is (so to
> speak) with Firmware Linux.  The chance that I'll do anything but whine
> about autoconf is slim indeed...  I'll shut up now!

Um, actually Eric Raymond and I have had long talks about this, and if we're 
ever in the same state for more than 3 days we may finally get to do a 
serious research project about the successors to ./configure and make.

He recently converted Battle for Wesnoth to use something called "scons" as 
its build system, and apparently the resulting make stuff was 1/17th the size 
of the original.  You'd have to ask him for details, though...

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Robert P. J. Day]

On Thu, 12 Jun 2008, Rob Landley wrote:

> He recently converted Battle for Wesnoth to use something called
> "scons" as its build system, and apparently the resulting make stuff
> was 1/17th the size of the original.  You'd have to ask him for
> details, though...

scons is definitely wicked cool.  every time i start a new project, i
am seriously tempted to give it a shot.

rday
--

========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry:
    Have classroom, will lecture.

http://crashcourse.ca                          Waterloo, Ontario, CANADA
========================================================================

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Mike Frysinger]

On Thu, Jun 12, 2008 at 9:25 PM, Rob Landley wrote:
> He recently converted Battle for Wesnoth to use something called "scons" as
> its build system, and apparently the resulting make stuff was 1/17th the size
> of the original.

probably because scons has ~1/17th the functionality of autotools.
seriously, it's terrible.  funny you should mention it in a thread
about cross-compiling seeing as how it doesnt have a chance of
supporting it (has hard enough time supporting native builds).
-mike

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Sam Ravnborg]

> 
> Um, actually Eric Raymond and I have had long talks about this, and if we're 
> ever in the same state for more than 3 days we may finally get to do a 
> serious research project about the successors to ./configure and make.

Tom has started a nice project which he named: quagmire.
See: http://code.google.com/p/quagmire/

From the website:
quagmire is intended to replace automake and libtool. Unlike these tools 
it requires GNU make and is written solely using GNU make
features -- no preprocessing is required.

	Sam

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Friday 13 June 2008 03:29:52 Mike Frysinger wrote:
> On Thu, Jun 12, 2008 at 9:25 PM, Rob Landley wrote:
> > He recently converted Battle for Wesnoth to use something called "scons"
> > as its build system, 

Battle of Wesnoth is currently converted to both Scons and CMake, and in the 
end they will decide about the winner.
(since Eric is good at arguing I guess it will be scons).

> > and apparently the resulting make stuff was 1/17th
> > the size of the original.
>
> probably because scons has ~1/17th the functionality of autotools.
> seriously, it's terrible.  

I saw the presentation about scons by the main scons developer at FOSDEM this 
year and it felt more like a library which you can use to create a 
buildsystem, not like a ready-to-use buildsystem itself.

Alex
-- 
TU-Kaiserslautern
Lehrstuhl für Echtzeitsysteme
Postfach 3049
D-67653 Kaiserslautern
Germany

Tel: +49 (0)631 205 3644
Fax: +49 (0)631 205 4199
--
To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Thursday 12 June 2008 17:50:31 you wrote:
> On Thu, 2008-06-12 at 08:23 -0700, Tim Bird wrote:
> > Rob Landley wrote:
> > > However, having one or more full-time engineers devoted to debugging
> > > cross-compile issues is quite a high price to pay too.  Moore's law
> > > really doesn't help that one.
> > >
> > > I'm not saying either solution is perfect, I'm just saying the "build
> > > under emulation" approach is a viable alternative that gets more
> > > attractive as time passes, both because of ongoing development on
> > > emulators and because of Moore's law on the hardware.
> >
> > I agree with much that you have said, Rob, and I understand the argument
> > for getting the most gain from the least resources, but I have a
> > philosophical problem with working around the cross-compilation problems
> > instead of fixing them in the upstream packages (or in the autoconf
> > system itself).
> >
> > Once someone fixes the cross-compilation issues for a package, they
> > usually stay fixed, if the fixes are mainlined.
>
> I don't think that's true, unfortunately. Autoconf makes it _easy_ to do
> the wrong thing, and people will often introduce new problems.
>
> If we just made people write portable code and proper Makefiles, it
> would be less of an issue :)

Well, IMO this makes it sound too easy.
If you write portable software, you have to do platform checks.
Basically they can be done by
-checking for the existence of files
-checking if something builds
-checking the output of running something you just built

The last one is the problem for cross compiling.
Example: detecting the size of ints

Easy way:

...
printf("%d\n", sizeof(long));
...

and then building and running that code and checking the output. Converting 
checks like this, which seem perfectly fine for native builds, to something 
which doesn't need to run an executable is not always trivial (but the way to 
go if you want to make your package cross-compilable).
(and yes, we converted this test in cmake 2.6 to a build-only test)

Alex

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Friday 13 June 2008 02:25:34 you wrote:
> On Thursday 12 June 2008 11:12:13 Robert P. J. Day wrote:
> > On Thu, 12 Jun 2008, Mike Frysinger wrote:
> > > On Thu, Jun 12, 2008 at 11:50 AM, David Woodhouse wrote:
> > > > If we just made people write portable code and proper Makefiles,
> > > > it would be less of an issue :)
> > >
> > > people cant even write proper *native* makefiles.  mtd-utils for
> > > example ;).
> >
> > meooowww!  :-)  but at the risk of dragging this even further
> > off-topic, i am *constantly* asked by people how to set up makefiles
> > for their software project, and what would be nice is a small
> > collection of examples of a makefile (or makefiles) done *right*.  as
> > in,
>
> Make doesn't scale.
>
> 99% of the builds in the open source world are "make all", and most of the
> smaller projects build natively on modern dual processor 2ghz laptops in
> under 10 seconds anyway.
>
> The larger projects with significant build times usually find that make
> doesn't suit their needs, so that they write some other build system.
> Sometimes they do it on top of make, such as the kernel's kbuild. 
> Sometimes they use another language like apache's ANT.  Sometimes they roll
> their own in C (anybody remember X11's imake?)  KDE switched to cmake:
> http://lwn.net/Articles/188693/

CMake can create Makefiles on all platforms (including complete dependencies, 
i.e. if you build an app which depends even indirectly on some lib then that 
lib will be rebuilt before), additionally XCode projects and MSVC projects 
(and Eclipse, KDevelop3 and CodeBlocks projects, but these use makefiles 
too).
So, we at KDE are still using make mainly, I don't really see how it doesn't 
scale (at least it scales much better than scons).

> Current compilers have a "build at once" mode where they suck the whole
> project in and run the optimizer on it at once, resulting in noticeably
> smaller and faster output at the expense of needing buckets of memory to
> hold all the source code and intermediate structures in memory at once. 
> The main roadblock to making use of this?  Ripping out the existing
> makefiles and replacing them with a very small shell script that does
> something similar to "gcc *.c".

We support that in KDE since a long time, with KDE 2.x to 3.x 
this "include-all" file was created via autotools, now with KDE 4 it's 
created by cmake. It makes compilation of the whole project faster, certainly 
not if you are building only a few files.
So, there is not really a roadblock to this. 

> The first question you should be asking when doing a new build system from
> scratch is probably "should I really be using make"?
>
> > properly recursive,
>
> Recursive make considered harmful:
>   http://aegis.sourceforge.net/auug97.pdf
>
> How is needing to call make recursively _not_ just another way of sayng
> "the dependency checking make does, which was the central idea behind its
> design, is a lost cause and we need to jettison it to do builds"?
>
> I just did a "make distclean" on a qemu tree I had lying around.  On my 1.7
> ghz 64 bit laptop, it took 9.2 seconds to figure out it had nothing to do,
> just because it had to recurse into so many subdirectories to do it.

I'm quite sure for a sufficiently large project this will always be the case. 
There are thousands of files which have to be checked whether they are 
up-to-date, so this takes some time.

> > supports cross-compiling without having to
> > remove your left nad, etc, etc.
>
> Can of worms.  Answering that question would be a longer post than the rest
> of this combined, but I've pointed out that I don't think going there is
> worth it anymore for most packages.
>
> > so ... would anyone like to recommend a software package or two
> > somewhere whose makefile(s) is/are, ITHO, done well?  that i/we could
> > just point at and say, "do it like that!"  anyone?  anyone?  bueller?

Use CMake >= 2.6.0 (http://www.cmake.org).
It supports cross compiling (... and of course everything else you might ever 
need ;-)
In general, cross compiling is not hard. You just have to call the cross 
toolchain, give it the correct parameters, search files in the right location 
and ... make sure you don't test stuff by running programs.
Basically only the last one is problematic (but IMO still less problematic 
than running a virtual system on the host just to build something).

How does cmake support that for cross compiling ?
If it detects a try_run() (which is the command for building a test executable 
and then running it) in cross compiling mode (i.e. when it was told what the 
target platform is instead of just assuming it's the same as the build host), 
then it 
-issues a warning
-stores the created executable in a place so the developer can find it
-running some existing executable and checking its output
-creates a file where the developer can enter the results which running the 
executable on the target host would have brought (includes comments where 
that command was found, the arguments, etc.,). The developer can then run the 
test executable manually on the target and enter the results. This seems to 
be similar to the config.cache somebody mentioned for debian.

So really the hard part is getting rid of running stuff to check the platform. 
If the project does that, all the build tool can do is support the developer 
in finding these spots and caring for them.


Did I mention that getting started with cmake is easy ?
A file CMakeLists.txt containing the following command is all you need for 
a "hello world":
add_executable(hello main.c)

Alex

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[David Woodhouse]

On Thu, 2008-06-12 at 19:25 -0500, Rob Landley wrote:
> Make doesn't scale.

Make scales just fine. The only real problem with make is that it's a
complete pain to debug.

-- 
dwmw2

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bernd Petrovitsch]

On Fre, 2008-06-13 at 08:43 +0200, Alexander Neundorf wrote:
> On Thursday 12 June 2008 17:50:31 you wrote:
> > On Thu, 2008-06-12 at 08:23 -0700, Tim Bird wrote:
> > > Rob Landley wrote:
> > > > However, having one or more full-time engineers devoted to debugging
> > > > cross-compile issues is quite a high price to pay too.  Moore's law
> > > > really doesn't help that one.
> > > >
> > > > I'm not saying either solution is perfect, I'm just saying the "build
> > > > under emulation" approach is a viable alternative that gets more
> > > > attractive as time passes, both because of ongoing development on
> > > > emulators and because of Moore's law on the hardware.
> > >
> > > I agree with much that you have said, Rob, and I understand the argument
> > > for getting the most gain from the least resources, but I have a
> > > philosophical problem with working around the cross-compilation problems
> > > instead of fixing them in the upstream packages (or in the autoconf
> > > system itself).
> > >
> > > Once someone fixes the cross-compilation issues for a package, they
> > > usually stay fixed, if the fixes are mainlined.
> >
> > I don't think that's true, unfortunately. Autoconf makes it _easy_ to do
> > the wrong thing, and people will often introduce new problems.
> >
> > If we just made people write portable code and proper Makefiles, it
> > would be less of an issue :)

ACK. And proper build time tools.

> Well, IMO this makes it sound too easy.
> If you write portable software, you have to do platform checks.
> Basically they can be done by
> -checking for the existence of files

That can be done as - sooner or later - one must install the compiled
stuff anyway. So one has root directory somewhere and one can tell the
tools.

> -checking if something builds
> -checking the output of running something you just built

And the above are not really a big problem - embedded people usually
know such details and can tell the autoconf tools.
Even worse is (or at least were) tools like pkg_config and libtool,
which generate directories to the build time library.

The only simple solution so far (without diving into the implementation
and searching for root causes) were AFAICS:
- do not use libtool for linking (as the link line as such without
  libtool works as expected)
- rewrite generated pkg_config files after generation.
Yes, that's pretty ugly.
But perhaps I was just too dumb to find the correct solutions.

> The last one is the problem for cross compiling.
> Example: detecting the size of ints

Why on earth does someone need this explicitly during the build?
If you have portable software, all of that should be hidden in the code
and use "sizeof(int)".

IMHO the code (or whatever piece uses it) should be fixed and the build
time stuff removed.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[James Chapman]

Mike Frysinger wrote:
> On Thu, Jun 12, 2008 at 5:53 PM, Tim Bird wrote:
>> Mike Frysinger wrote:
>>>> Er, is that GPL or LGPL code that you're modifying? If so, you *have* to
>>>> push those code changes out (make them available to others), whether you
>>>> think people will be interested or not!
>>> umm, not really.  only if (1) he gives a binary to someone and (2)
>>> they ask him for the source.  if he doesnt distribute or no one asks,
>>> he doesnt have to do squat.
>> This is closer to correct, but missing some important details.
>>
>> Start the GPL compliance tutorial/flameware in 3, 2, 1...
> 
> yeah, i really dont think licensing things belong here.  sorry for following up.
> 
> how about this policy: if you want to make a statement, go pay a
> lawyer.  but that statement still shouldnt be made here ;).
> -mike

Sorry, I didn't mean to provoke a GPL flame war. The point I was trying 
to make (badly as it turns out) is that if a company really wants to see 
its changes taken upstream, it could simply publish the work on its 
website and let each relevant community know that it's there. A diff of 
the changes would be ideal. This is above and beyond what they have to 
do under GPL terms of course. There's no need for a company to filter 
out changes that it thinks others won't be interested in.

-- 
James Chapman
Katalix Systems Ltd
http://www.katalix.com
Catalysts for your Embedded Linux software development

Re: cross-compiling alternatives

[Bernd Petrovitsch]

On Don, 2008-06-12 at 19:25 -0500, Rob Landley wrote:
> On Thursday 12 June 2008 11:12:13 Robert P. J. Day wrote:
> > On Thu, 12 Jun 2008, Mike Frysinger wrote:
> > > On Thu, Jun 12, 2008 at 11:50 AM, David Woodhouse wrote:
> > > > If we just made people write portable code and proper Makefiles,
> > > > it would be less of an issue :)
> > >
> > > people cant even write proper *native* makefiles.  mtd-utils for
> > > example ;).
> >
> > meooowww!  :-)  but at the risk of dragging this even further
> > off-topic, i am *constantly* asked by people how to set up makefiles
> > for their software project, and what would be nice is a small
> > collection of examples of a makefile (or makefiles) done *right*.  as
> > in,
> 
> Make doesn't scale.
> 
> 99% of the builds in the open source world are "make all", and most of the 
> smaller projects build natively on modern dual processor 2ghz laptops in 
> under 10 seconds anyway.
>
> The larger projects with significant build times usually find that make 
> doesn't suit their needs, so that they write some other build system.  
> Sometimes they do it on top of make, such as the kernel's kbuild.  Sometimes 
> they use another language like apache's ANT.  Sometimes they roll their own 

"ant" is also only "make reimplemented in Java" (or did I miss
something). I see no win here.

> in C (anybody remember X11's imake?)  KDE switched to cmake: 

That generated "only" a Makefile IIRC.

[...]
> Current compilers have a "build at once" mode where they suck the whole 
> project in and run the optimizer on it at once, resulting in noticeably 
> smaller and faster output at the expense of needing buckets of memory to hold 
> all the source code and intermediate structures in memory at once.  The main 
> roadblock to making use of this?  Ripping out the existing makefiles and 
> replacing them with a very small shell script that does something similar 
> to "gcc *.c".
> 
> The first question you should be asking when doing a new build system from 
> scratch is probably "should I really be using make"?
> 
> > properly recursive, 
> 
> Recursive make considered harmful:
>   http://aegis.sourceforge.net/auug97.pdf

ACK.

> How is needing to call make recursively _not_ just another way of sayng "the 
> dependency checking make does, which was the central idea behind its design, 
> is a lost cause and we need to jettison it to do builds"?

The problem is that build systems have (at least) two layers:
- the lower are the usual apps and libs and kernel ... bringing their
  working Makefile's with them.
- the upper layer needs to build the kernel, libs and apps.
  This needs usually a defined sequence and set of consistent
  parameters.
But the lower layer doesn't "export" it's local available rule base so
the `make` (or shell script) on the upper layer can't use it and one
must therefore `make -C $tooldir` - even if there is absolutely nothing
to do.
Of course the upper layer may remember if an lib/app has been build to
avoid 60% of all "obviously" useless `make -C` calls. But that doesn't
solve any problem really.

> I just did a "make distclean" on a qemu tree I had lying around.  On my 1.7 
> ghz 64 bit laptop, it took 9.2 seconds to figure out it had nothing to do, 
> just because it had to recurse into so many subdirectories to do it.

"Recursive make considered harmful"
Or you need more RAM and faster disks;-)

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Daniel THOMPSON]

James Chapman wrote:
> Mike Frysinger wrote:
>> On Thu, Jun 12, 2008 at 5:53 PM, Tim Bird wrote:
>>> Mike Frysinger wrote:
>>>>> Er, is that GPL or LGPL code that you're modifying? If so, you
>>>>> *have* to
>>>>> push those code changes out (make them available to others),
>>>>> whether you
>>>>> think people will be interested or not!
>>>> umm, not really.  only if (1) he gives a binary to someone and (2)
>>>> they ask him for the source.  if he doesnt distribute or no one asks,
>>>> he doesnt have to do squat.
>>> This is closer to correct, but missing some important details.
>>>
>>> Start the GPL compliance tutorial/flameware in 3, 2, 1...
>>
>> yeah, i really dont think licensing things belong here.  sorry for
>> following up.
>>
>> how about this policy: if you want to make a statement, go pay a
>> lawyer.  but that statement still shouldnt be made here ;).
>> -mike
> 
> Sorry, I didn't mean to provoke a GPL flame war. The point I was trying
> to make (badly as it turns out) is that if a company really wants to see
> its changes taken upstream, it could simply publish the work on its
> website and let each relevant community know that it's there.

Isn't this a lot of the problem with the way embedded companies and
developers interact with upstream.

In some cases it is in the embedded developers interests to see their
code adopted upstream (i.e. so they don't have to maintain it).

Just tossing some code over the wall will, in almost all circumstances,
result in the code being ignored.

> A diff of
> the changes would be ideal. This is above and beyond what they have to
> do under GPL terms of course. There's no need for a company to filter
> out changes that it thinks others won't be interested in.

This is a legal concern, not a practical one. All companies *should*
comply with the law. Really it is more useful to convince people that it
is in their own self-interest to do more than this.


-- 
Daniel Thompson (STMicroelectronics) <daniel.thompson@st.com>
1000 Aztec West, Almondsbury, Bristol, BS32 4SQ. 01454 462659

If a car is a horseless carriage then is a motorcycle a horseless horse?

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Friday 13 June 2008 10:38:36 you wrote:
> On Fre, 2008-06-13 at 08:43 +0200, Alexander Neundorf wrote:
...
> > Well, IMO this makes it sound too easy.
> > If you write portable software, you have to do platform checks.
> > Basically they can be done by
> > -checking for the existence of files
>
> That can be done as - sooner or later - one must install the compiled
> stuff anyway. So one has root directory somewhere and one can tell the
> tools.

Yes.

> > -checking if something builds
> > -checking the output of running something you just built
>
> And the above are not really a big problem - 

"checking if something builds" is no problem, this just works. Running 
something is a problem, as in "it doesn't just work" (...because you cannot 
run it).

> embedded people usually know such details and can tell the autoconf tools.

Basically yes. But if you have a big number of packages (or a huge package) 
which you didn't write yourself, there will be tests which run executables. 
Figuring out what all the tests are supposed to test in a complex unknown 
software project is not trivial.

> Even worse is (or at least were) tools like pkg_config and libtool,
> which generate directories to the build time library.

What do you mean with "generate directories" ? RPATH ?

> The only simple solution so far (without diving into the implementation
> and searching for root causes) were AFAICS:
> - do not use libtool for linking (as the link line as such without
>   libtool works as expected)

Yes, libtool sucks, it's the wrong solution to the problem.
(and CMake doesn't use it).

> - rewrite generated pkg_config files after generation.
> Yes, that's pretty ugly.
> But perhaps I was just too dumb to find the correct solutions.

Can you please explain ? How do the generated pkg_config files look like ?
Ahh, you mean they contain e.g -L/my/build/host/target/environment/opt/foo/lib 
instead of just -L/opt/foo/lib ?

> > The last one is the problem for cross compiling.
> > Example: detecting the size of ints
>
> Why on earth does someone need this explicitly during the build?
> If you have portable software, all of that should be hidden in the code
> and use "sizeof(int)".

From the "developer of a buildsystem" POV: there will be users who will need 
it. 
But this was not the point. My point was: testing something by running an 
executable can be _a lot_ easier than testing the same without running 
something.

Alex

Re: cross-compiling alternatives

[Alexander Neundorf]

On Friday 13 June 2008 10:50:28 you wrote:
> On Don, 2008-06-12 at 19:25 -0500, Rob Landley wrote:
...
> > in C (anybody remember X11's imake?)  KDE switched to cmake:
>
> That generated "only" a Makefile IIRC.

Yes, cmake doesn't actually build the stuff itself, it generates input files 
for the actual build tool you want to use.
For KDE4 all buildtools supported by cmake are working: the various makes, 
XCode, MSVC projects.

Alex

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[David Woodhouse]

On Fri, 2008-06-13 at 11:06 +0200, Alexander Neundorf wrote:
> > Why on earth does someone need this explicitly during the build?
> > If you have portable software, all of that should be hidden in the code
> > and use "sizeof(int)".
> 
> From the "developer of a buildsystem" POV: there will be users who will need 
> it. 

I think that epitomises what's wrong with autoconf. Sometimes, the best
thing to do is tell your users that they _don't_ need whatever it is
they're asking you for.

-- 
dwmw2

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Friday 13 June 2008 11:12:00 you wrote:
> On Fri, 2008-06-13 at 11:06 +0200, Alexander Neundorf wrote:
> > > Why on earth does someone need this explicitly during the build?
> > > If you have portable software, all of that should be hidden in the code
> > > and use "sizeof(int)".
> >
> > From the "developer of a buildsystem" POV: there will be users who will
> > need it.
>
> I think that epitomises what's wrong with autoconf. Sometimes, the best

Actually I think autoconf itself is not that bad. What is bad is "autotools", 
i.e. that you get a combination of several tools which have to work together, 
all huge shell scripts, all using different syntax, etc.

> thing to do is tell your users that they _don't_ need whatever it is
> they're asking you for.

We agree that if possible, tests which run something should be avoided. 
But seriously, sometimes this is really very hard.
I don't dare to say impossible, but I'm tempted. 
E.g. in python there are tests which call functions and check their result to 
see if we are currently on a platform where that function is broken (I think 
there was such a test for poll() and some other functions).

Alex

Re: cross-compiling alternatives

[Bernd Petrovitsch]

On Fre, 2008-06-13 at 11:06 +0200, Alexander Neundorf wrote:
> On Friday 13 June 2008 10:38:36 you wrote:
> > On Fre, 2008-06-13 at 08:43 +0200, Alexander Neundorf wrote:
> ...
> > > Well, IMO this makes it sound too easy.
> > > If you write portable software, you have to do platform checks.
> > > Basically they can be done by
> > > -checking for the existence of files
> >
> > That can be done as - sooner or later - one must install the compiled
> > stuff anyway. So one has root directory somewhere and one can tell the
> > tools.
> 
> Yes.
> 
> > > -checking if something builds
> > > -checking the output of running something you just built
> >
> > And the above are not really a big problem - 
> 
> "checking if something builds" is no problem, this just works. Running 
> something is a problem, as in "it doesn't just work" (...because you cannot 
> run it).

ACK. AC_TRY_RUN() must die completely.

> > embedded people usually know such details and can tell the autoconf tools.
> 
> Basically yes. But if you have a big number of packages (or a huge package) 
> which you didn't write yourself, there will be tests which run executables. 
> Figuring out what all the tests are supposed to test in a complex unknown 
> software project is not trivial.

Yes, you get used to find the relevant lines in config.log and similar
with `grep` and similar tools;-)
But most embedded projects haven't that much number of "large tools" -
mainly because the space is limited.

> > Even worse is (or at least were) tools like pkg_config and libtool,
> > which generate directories to the build time library.
> 
> What do you mean with "generate directories" ? RPATH ?

pkg-config generated (and generates? - I didn't check recently)
references to libraries including the full absolute path (which is the
one at build time. And at run-time there is usually
no /home/bernd/src/... or where some build may just run).

[...]
> > - rewrite generated pkg_config files after generation.
> > Yes, that's pretty ugly.
> > But perhaps I was just too dumb to find the correct solutions.
> 
> Can you please explain ? How do the generated pkg_config files look like ?
> Ahh, you mean they contain e.g -L/my/build/host/target/environment/opt/foo/lib 
> instead of just -L/opt/foo/lib ?

Yes. And even worse the compiled lib "foo" had explicit dependencies (on
lib "bar") on
"/my/build/host/target/environment/opt/bar/lib/libbar.so.1.2.3.4". And
that is not trivially overridable at run-time AFAIK so that ld-linux
finds "/opt/bar/lib/libbar.so.1.2.3.4" instead.
For real world names: glib is pretty commonly used by other libs. Voila,
an indirect dependency.

And BTW pkg-config didn't support the concept of a "DESTDIR" variable
(and I don't care about the name of that variable).

> > > The last one is the problem for cross compiling.
> > > Example: detecting the size of ints
> >
> > Why on earth does someone need this explicitly during the build?
> > If you have portable software, all of that should be hidden in the code
> > and use "sizeof(int)".
> 
> From the "developer of a buildsystem" POV: there will be users who will need 
> it. 

If there is at least one valid technical reason: Yes.
If the only reasons are "we had it since 10 years with the old system"
or "we don't want to fix the code because it takes us too much time":
well, tough decision.


> But this was not the point. My point was: testing something by running an 
> executable can be _a lot_ easier than testing the same without running 
> something.

Of course. But *that's* in general possible for cross-compiling. And
having a 100% binary compatible qemu installation for every ARM and MIPS
core out there is IMHO also not feasible.

Actually the size of ints (or any other type) can be easily deduced
without running a (for the target) compiled binary:
- compile the binary (for the target) with an initialized variable with
  that value.
- use cross nm (or a similar tool) to read it from there.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Geert Uytterhoeven]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1542 bytes --]

On Thu, 12 Jun 2008, Robert P. J. Day wrote:
> On Thu, 12 Jun 2008, Mike Frysinger wrote:
> > On Thu, Jun 12, 2008 at 11:50 AM, David Woodhouse wrote:
> > > If we just made people write portable code and proper Makefiles,
> > > it would be less of an issue :)
> >
> > people cant even write proper *native* makefiles.  mtd-utils for
> > example ;).
> 
> meooowww!  :-)  but at the risk of dragging this even further
> off-topic, i am *constantly* asked by people how to set up makefiles
> for their software project, and what would be nice is a small
> collection of examples of a makefile (or makefiles) done *right*.  as
> in, properly recursive, supports cross-compiling without having to
      ^^^^^^^^^^^^^^^^^^
      Ugh?

> remove your left nad, etc, etc.
> 
> so ... would anyone like to recommend a software package or two
> somewhere whose makefile(s) is/are, ITHO, done well?  that i/we could
> just point at and say, "do it like that!"  anyone?  anyone?  bueller?

The kernel, except that it's still recursive?

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

Sony Technology and Software Centre Europe
A division of Sony Service Centre (Europe) N.V.
Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium
VAT BE 0413.825.160 · RPR Brussels
Fortis 293-0376800-10 GEBA-BE-BB

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Geert Uytterhoeven]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1845 bytes --]

On Thu, 12 Jun 2008, David Woodhouse wrote:
> On Thu, 2008-06-12 at 11:28 -0500, Bill Gatliff wrote:
> > > If you opt to cross-compile, having to deal with those
> > > sorts of things is the price you pay.
> > 
> > If the build system derives from autoconf, then a hacked-up config.cache (or
> > equivalent command-line args) often solves problems for me.  Just give the cache
> > the answers that it would otherwise have to get by running code on the target
> > machine.
> > 
> > That's how emdebian is doing a bunch of their stuff, and I have to admit that it
> > works pretty darned well.  It's also handy for configuration management, since
> > the cache file itself is plaintext and therefore svn/git/bzr/cvs/...-friendly.
> 
> Yeah, I was building Red Hat Linux packages for sh3 many years ago,
> using tricks like that. But there was always _something_ else going
> wrong, however much you hacked around it. And a lot of it would only
> turn up at runtime, not build time. I would never consider shipping a
> product with a large number of userspace packages cross-compiled.
> 
> For minimal file systems with a select handful of tools which can be
> tested exhaustively, it's not so bad. But for any 'full-featured'
> userspace, I think cross-compilation is completely insane.

So, how does OpenWRT manage to survive?

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

Sony Technology and Software Centre Europe
A division of Sony Service Centre (Europe) N.V.
Registered office: Technologielaan 7 · B-1840 Londerzeel · Belgium
VAT BE 0413.825.160 · RPR Brussels
Fortis 293-0376800-10 GEBA-BE-BB

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[David Woodhouse]

On Fri, 2008-06-13 at 13:15 +0200, Geert Uytterhoeven wrote:
> > For minimal file systems with a select handful of tools which can be
> > tested exhaustively, it's not so bad. But for any 'full-featured'
> > userspace, I think cross-compilation is completely insane.
> 
> So, how does OpenWRT manage to survive?

I don't consider that a 'full-featured' userspace.

Or particularly sane, for that matter :)

-- 
dwmw2

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bart Van Assche]

On Thu, Jun 12, 2008 at 6:12 PM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
...
> meooowww!  :-)  but at the risk of dragging this even further
> off-topic, i am *constantly* asked by people how to set up makefiles
> for their software project, and what would be nice is a small
> collection of examples of a makefile (or makefiles) done *right*.  as
> in, properly recursive, supports cross-compiling without having to
> remove your left nad, etc, etc.
...

Please read the essay "Recursive Make Considered Harmful"
(http://miller.emu.id.au/pmiller/books/rmch/).

Bart.

Re: cross-compiling alternatives

[Alexander Neundorf]

On Friday 13 June 2008 12:03:53 you wrote:
> On Fre, 2008-06-13 at 11:06 +0200, Alexander Neundorf wrote:
...
> > > - rewrite generated pkg_config files after generation.
> > > Yes, that's pretty ugly.
> > > But perhaps I was just too dumb to find the correct solutions.
> >
> > Can you please explain ? How do the generated pkg_config files look like
> > ? Ahh, you mean they contain e.g
> > -L/my/build/host/target/environment/opt/foo/lib instead of just
> > -L/opt/foo/lib ?
>
> Yes. And even worse the compiled lib "foo" had explicit dependencies (on
> lib "bar") on
> "/my/build/host/target/environment/opt/bar/lib/libbar.so.1.2.3.4". And

You mean the RPATH in the installed executable was wrong ?

> that is not trivially overridable at run-time AFAIK so that ld-linux

If RUNPATH is available in the binary (exists since a few years, so it should 
be in your toolchain), then RPATH is ignored. RUNPATH can be overridden using 
LD_LIBRARY_PATH. It can be enabled with the linker flag --enable-new-dtags I 
think.
If there is RPATH but not RUNPATH, then it can't be overridden. But still if 
it doesn't find the library in that place, it will continue searching in the 
other directories, e.g. LD_LIBRARY_PATH.

There is also a tool "chrpath", which you can use to change the RPATH entry in 
an existing executable.
It's a bit hard to find, you can get it e.g. from an older revision of kdesvn:
http://websvn.kde.org/trunk/kdesupport/chrpath/?pathrev=808076
(it was removed again in revision 808077).
It may have issues with endianness.

Alex

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[James Chapman]

Daniel THOMPSON wrote:
> James Chapman wrote:
>> Mike Frysinger wrote:
>>> On Thu, Jun 12, 2008 at 5:53 PM, Tim Bird wrote:
>>>> Mike Frysinger wrote:
>>>>>> Er, is that GPL or LGPL code that you're modifying? If so, you
>>>>>> *have* to
>>>>>> push those code changes out (make them available to others),
>>>>>> whether you
>>>>>> think people will be interested or not!
>>>>> umm, not really.  only if (1) he gives a binary to someone and (2)
>>>>> they ask him for the source.  if he doesnt distribute or no one asks,
>>>>> he doesnt have to do squat.
>>>> This is closer to correct, but missing some important details.
>>>>
>>>> Start the GPL compliance tutorial/flameware in 3, 2, 1...
>>> yeah, i really dont think licensing things belong here.  sorry for
>>> following up.
>>>
>>> how about this policy: if you want to make a statement, go pay a
>>> lawyer.  but that statement still shouldnt be made here ;).
>>> -mike
>> Sorry, I didn't mean to provoke a GPL flame war. The point I was trying
>> to make (badly as it turns out) is that if a company really wants to see
>> its changes taken upstream, it could simply publish the work on its
>> website and let each relevant community know that it's there.
> 
> Isn't this a lot of the problem with the way embedded companies and
> developers interact with upstream.
> 
> In some cases it is in the embedded developers interests to see their
> code adopted upstream (i.e. so they don't have to maintain it).

Totally agree! And the best chance of having code accepted upstream is 
to work with the community _while_ developing it, i.e. discussing the 
code during implementation, rather than presenting it to the community 
when it's done. All too often, companies get frustrated by feedback from 
the community because changes are requested to code that the original 
authors have spent time testing etc. Had early versions been submitted 
for feedback, changes could be made with less chance of wasted effort.

> Just tossing some code over the wall will, in almost all circumstances,
> result in the code being ignored.

It depends. But it stands a better chance of being adopted than holding 
on to the work until someone asks for it. There could be lots of 
embedded developers out there who would be willing to take some code 
from cisco, modify it and work with the community to have it adopted 
upstream.


-- 
James Chapman
Katalix Systems Ltd
http://www.katalix.com
Catalysts for your Embedded Linux software development

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Samuel Robb]

On Thu, 2008-06-12 at 16:02 -0600, Jim Freeman wrote:
> Most vendors these days have finally gotten the clue that sources/changes
> have to be made available to downstream requesters, but far fewer
> are sufficiently self-enlightened to figure out that changes need to
> be accepted upstream for them to keep flowing back.  And to make that
> happen, vendors have to take on substantially higher overhead to win
> acceptance of patches/changes upstream, an undertaking often sadly
> fraught with hassle, uncertainty, and even peril.  So they mostly
> don't bother.  To their (and their customers, and our) long-term
> detriment.

So - how do you reduce that overhead, then?  Keep in mind that while
pushing kernel changes upstream is significant, there are other projects
as well.  Most embedded developers are working not just with the kernel,
but with a constellation of packages related to their projects.  In
order to "push changes upstream", they may end up having to work with
several different communities... each with their own model of
interaction, their own model of patch submission, and their own release
schedules.  Figuring out how to deal with just one community (kernel)
doesn't help them in dealing with another (say, samba).

When you have a one- or two-line fix, and face Yet Another round of
finding the right mailing list, identifying the right maintainers,
figuring out the right way to submit a bug and a patch, and then have to
spend the next 3 weeks explaining how no, you're not interested in being
the PPC maintainer for libfoo... is it any wonder that developers (not
to mention their management) eventually just gives up on the idea of
"giving back to the community?"

One possible solution would be to provide a clearing house for these
sorts of changes, maybe under the auspices of CELF or a similar
organization.  Instead of submitting patches to individual projects,
submit them to the clearing house, and let interested individuals either
gather together and push related patches upstream in individual
projects, or give project maintainers a place to go and find embedded
systems patches related to their projects.

-Samrobb

Re: cross-compiling alternatives

[Jamie Lokier]

Bernd Petrovitsch wrote:
> Actually the size of ints (or any other type) can be easily deduced
> without running a (for the target) compiled binary:
> - compile the binary (for the target) with an initialized variable with
>   that value.
> - use cross nm (or a similar tool) to read it from there.

Or the method autoconf uses - binary search, using a compile-time
numeric comparison which resolves to a successful or failed compile.

That seems more portable to me.

Relying on 'nm 'finding the variable, and not accidentally matching
another variable with the wrong value, does not work for all C
environments.  E.g. some compile to compressed executables; some
produce intermediate objects with incomplete or lazy compiles or
symbolic, to be finished at link stage, and some are even more
abstract.

And it requires the 'nm' tool, which you might not have for
cross-compilation, or might not find the right one.

-- Jamie

Re: cross-compiling alternatives

[Bernd Petrovitsch]

On Fre, 2008-06-13 at 14:17 +0100, Jamie Lokier wrote:
> Bernd Petrovitsch wrote:
> > Actually the size of ints (or any other type) can be easily deduced
> > without running a (for the target) compiled binary:
> > - compile the binary (for the target) with an initialized variable with
> >   that value.
> > - use cross nm (or a similar tool) to read it from there.
> 
> Or the method autoconf uses - binary search, using a compile-time
> numeric comparison which resolves to a successful or failed compile.

Good, I didn't know that.

> That seems more portable to me.

Yes, just using the compiler is better.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Josh Boyer]

On Thu, Jun 12, 2008 at 3:02 PM, Mike Frysinger <vapier.adi@gmail.com> wrote:
>>> people cant even write proper *native* makefiles.  mtd-utils for example ;).
>>
>> What's wrong with it?  I'll fix it.
>
> is linux-mtd@lists.infradead.org not the place to post ?  that's where
> i sent the first fix yesterday ... not that i'm subscribed since i
> dont have a direct interest in mtd development ...

It is.  I just missed it.  I'll look closer today.  Thanks.

josh

Re: cross-compiling alternatives

[Alexander Neundorf]

On Friday 13 June 2008 15:17:54 you wrote:
> Bernd Petrovitsch wrote:
> > Actually the size of ints (or any other type) can be easily deduced
> > without running a (for the target) compiled binary:
> > - compile the binary (for the target) with an initialized variable with
> >   that value.
> > - use cross nm (or a similar tool) to read it from there.
>
> Or the method autoconf uses - binary search, using a compile-time
> numeric comparison which resolves to a successful or failed compile.

How does it do that compile-time numeric comparison ?

Alex

Re: cross-compiling alternatives

[Matthieu CASTET]

Alexander Neundorf wrote:
> On Friday 13 June 2008 15:17:54 you wrote:
>> Bernd Petrovitsch wrote:
>>> Actually the size of ints (or any other type) can be easily deduced
>>> without running a (for the target) compiled binary:
>>> - compile the binary (for the target) with an initialized variable with
>>>   that value.
>>> - use cross nm (or a similar tool) to read it from there.
>> Or the method autoconf uses - binary search, using a compile-time
>> numeric comparison which resolves to a successful or failed compile.
> 
> How does it do that compile-time numeric comparison ?
> 
for example you could do

int test[my comparaison];

if my comparaison < 0, the compilation should abort.


Matthieu

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Josh Boyer]

On Thu, Jun 12, 2008 at 3:02 PM, Mike Frysinger <vapier.adi@gmail.com> wrote:
>>> people cant even write proper *native* makefiles.  mtd-utils for example ;).
>>
>> What's wrong with it?  I'll fix it.
>
> is linux-mtd@lists.infradead.org not the place to post ?  that's where
> i sent the first fix yesterday ... not that i'm subscribed since i
> dont have a direct interest in mtd development ...

It is.  I just missed it.  I'll look closer today.  Thanks.

josh

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Enrico Weigelt]

* Jim Freeman <jfree@sovereign.org> schrieb:

> And I'm just betting that when he said "push ... fixes ... out"
> he meant "work to get them incorporated back upstream", not just
> make them available to requesters.

Exactly.
That's the essence of opensource development: 
Working together, instead of just taking someone else's work.

BTW: everyone here should know that maintaining own branches
(this actually happens if you have your own patches not fed
back to upstream) is an quite work intensive and sometimes
complex task. I doubt anyone here's really eager on doing that ;-P

I understand that not everyone can talk with every project's 
upstream. Simply too much load. And not every project wants to
merge in your patches asap (think of how long it took until
expat team took in my really trivial $DESTDIR patch ;-o).

That's why I founded the OSS-QM project: it a kind of "overlay"
which provides fixes for a lot of packages in a strictly normalized
namespace (so 100% automatic applying is easy).

http://oss-qm.metux.de/

So, eg. if you've made some local changes, please at least feed 
them to us :)


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Enrico Weigelt]

* Samuel Robb <sam@oneparticularharbor.net> schrieb:

> When you have a one- or two-line fix, and face Yet Another round of
> finding the right mailing list, identifying the right maintainers,
> figuring out the right way to submit a bug and a patch, and then have to
> spend the next 3 weeks explaining how no, you're not interested in being
> the PPC maintainer for libfoo... is it any wonder that developers (not
> to mention their management) eventually just gives up on the idea of
> "giving back to the community?"

ACK.

> One possible solution would be to provide a clearing house for these
> sorts of changes, maybe under the auspices of CELF or a similar
> organization.  Instead of submitting patches to individual projects,
> submit them to the clearing house, and let interested individuals either
> gather together and push related patches upstream in individual
> projects, or give project maintainers a place to go and find embedded
> systems patches related to their projects.

See my last mail: that's exactly what the oss-qm project is for:
http://oss-qm.metux.de/

With OSS-QM and related tools you often even don't need your own
patching infrastructure - oss-qm provides complete patches against
virtually any package/release.

Perhaps a few words on the infrastructure:

* single patches are collected per package, each vendor may get his
  own namespace/subdir, where he can feed in his patches.
* the single patches are automatically pulled together based on 
  "listfiles" - per package+release there is an simple text file 
  which just lists the single patches to be pulled together.
* each vendor may get it's own namespace for the listfiles.

In other words: if you fear somebody else breaks your already 
tested/approved packages, just use your own (listfile) namespace.


BTW: CSDB does a similar thing for retrieving source tarballs.
Just query the DB, never ever care about individual URLs in
you local build system. 


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives

[Enrico Weigelt]

* Matthieu CASTET <matthieu.castet@parrot.com> schrieb:

> >How does it do that compile-time numeric comparison ?
> >
> for example you could do
> 
> int test[my comparaison];
> 
> if my comparaison < 0, the compilation should abort.

Cool trick :)

So, eg if you find out wether some type has an specific size:

int foo[(sizeof(int) == 4)?1:-1];


But: the question is whether you'll need such a test at all
or if just using sizeof() at the right place won't do the trick ;-P
(yes, I've came around lots of packages where this was the case!)

cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives

[Jamie Lokier]

Enrico Weigelt wrote:
> But: the question is whether you'll need such a test at all
> or if just using sizeof() at the right place won't do the trick ;-P

It's best to do that if you can, and nearly always possible.  There
are a few coding techniques - especially performance sensitive - where
that's not feasible.

-- Jamie

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Rob Landley <rob@landley.net> schrieb:

> Current compilers have a "build at once" mode where they suck the whole 
> project in and run the optimizer on it at once, resulting in noticeably 
> smaller and faster output at the expense of needing buckets of memory to hold 
> all the source code and intermediate structures in memory at once.  The main 
> roadblock to making use of this?  Ripping out the existing makefiles and 
> replacing them with a very small shell script that does something similar 
> to "gcc *.c".

Yep, that's a fundamental problem with make that it works only on an
per-file basis (which is fine for many projects, but sometimes not).

That's why I prefer an purely descriptive paragidm (= subset of 
delcarative, but practically no logic): a buildfile should only 
describe the package's structure (eg.: "i have some executable foo 
which consists of source [...] and imports libs [...]), so the 
buildtool (and user's config) can cope with it all.

TreeBuild works this way:	http://treebuild.metux.de/


BTW: does anyone known Plan9's mk ?

cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives

[Enrico Weigelt]

* Jamie Lokier <jamie@shareable.org> schrieb:
> Enrico Weigelt wrote:
> > But: the question is whether you'll need such a test at all
> > or if just using sizeof() at the right place won't do the trick ;-P
> 
> It's best to do that if you can, and nearly always possible.  There
> are a few coding techniques - especially performance sensitive - where
> that's not feasible.

Example ?


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives

[Enrico Weigelt]

* Jamie Lokier <jamie@shareable.org> schrieb:

> Relying on 'nm 'finding the variable, and not accidentally matching
> another variable with the wrong value, does not work for all C
> environments.  E.g. some compile to compressed executables; some
> produce intermediate objects with incomplete or lazy compiles or
> symbolic, to be finished at link stage, and some are even more
> abstract.

ACK. And also think of preprocessor symbols (see glibc ;-P)
IIRC perl's buildsystem goes this way and so tends to fail.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[linux-embedded-owner]

Hi David,

Do you wanna set some breakpoints and inspect variables in makefiles?
Have a look at a simple makefile debugger (written in make):
http://newsletter.embedded.com/shared/printableArticle.jhtml?articleID=197003517

Regards,

Robert

---
Robert Berger
Embedded Software Specialist

Reliable Embedded Systems
Consulting Training Engineering
Tel.: (+30) 697 593 3428
Fax.:(+30 210) 684 7881
URL: www.reliableembeddedsystems.com


Quoting David Woodhouse <dwmw2@infradead.org>:

> On Thu, 2008-06-12 at 19:25 -0500, Rob Landley wrote:
>> Make doesn't scale.
>
> Make scales just fine. The only real problem with make is that it's a
> complete pain to debug.
>
> --
> dwmw2
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:

> In general, cross compiling is not hard. You just have to call the cross 
> toolchain, give it the correct parameters, search files in the right location 
> and ... make sure you don't test stuff by running programs.

Same with carefully written makefiles ;-P

> Basically only the last one is problematic (but IMO still less problematic 
> than running a virtual system on the host just to build something).

Yes, and that should be completely prohibited.

> How does cmake support that for cross compiling ?
> If it detects a try_run() (which is the command for building a test executable 
> and then running it) in cross compiling mode (i.e. when it was told what the 
> target platform is instead of just assuming it's the same as the build host), 
> then it 
> -issues a warning
> -stores the created executable in a place so the developer can find it
> -running some existing executable and checking its output
> -creates a file where the developer can enter the results which running the 
> executable on the target host would have brought (includes comments where 
> that command was found, the arguments, etc.,). The developer can then run the 
> test executable manually on the target and enter the results. This seems to 
> be similar to the config.cache somebody mentioned for debian.

Yes, introducing the problems as with autoconf. You'll have to maintain one 
cachefile per package - there is *NO* standard on it's variables. And exactly 
that's the key problem !

Therefore unitool has an generic platform database, where such generic 
platform parameters can be set. These fields have to be standardized,
(adding new ones for some specific package is definitively the wrong way).

> So really the hard part is getting rid of running stuff to check the platform. 

Isnt that hard. In all these years I had no case where it couldn't be 
circumvented.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives

[Enrico Weigelt]

* Bernd Petrovitsch <bernd@firmix.at> schrieb:

> > Basically yes. But if you have a big number of packages (or a huge package) 
> > which you didn't write yourself, there will be tests which run executables. 
> > Figuring out what all the tests are supposed to test in a complex unknown 
> > software project is not trivial.
> 
> Yes, you get used to find the relevant lines in config.log and similar
> with `grep` and similar tools;-)

Which are different on each package. So you have to configure each package
for each target manually, which leads the whole point of autoconf
ad absurdum ;-o

> But most embedded projects haven't that much number of "large tools" -
> mainly because the space is limited.

I'm probably not quite representative, but I'm using crosscompile
techniques also for building to the same arch - mainly for QM reasons.

> pkg-config generated (and generates? - I didn't check recently)
> references to libraries including the full absolute path (which is the
> one at build time. And at run-time there is usually
> no /home/bernd/src/... or where some build may just run).

Recent pkg-config supports sysroot.
So you simply build your .pc files as usual (w/o sysroot prefix) and
set the sysroot prefix via env on the pkg-config call.

> > Can you please explain ? How do the generated pkg_config files look like ?
> > Ahh, you mean they contain e.g -L/my/build/host/target/environment/opt/foo/lib 
> > instead of just -L/opt/foo/lib ?

Then you've got a broken .pc file ;-P

> Yes. And even worse the compiled lib "foo" had explicit dependencies (on
> lib "bar") on
> "/my/build/host/target/environment/opt/bar/lib/libbar.so.1.2.3.4". 

And that's even more broken.

> And BTW pkg-config didn't support the concept of a "DESTDIR" variable
> (and I don't care about the name of that variable).

No, why should it ?! It does not install anything.
Probably you're looking for sysroot ?


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:

> Well, IMO this makes it sound too easy.
> If you write portable software, you have to do platform checks.
> Basically they can be done by
> -checking for the existence of files
> -checking if something builds
> -checking the output of running something you just built

Some concerete examples where you really *need* that ?

> The last one is the problem for cross compiling.
> Example: detecting the size of ints
> 
> Easy way:
> 
> ...
> printf("%d\n", sizeof(long));
> ...

Why not directly using sizeof() in your code ?


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:

> E.g. in python there are tests which call functions and check 
> their result to see if we are currently on a platform where 
> that function is broken (I think there was such a test for 
> poll() and some other functions).

IMHO, that's broken sw engineering from ground up.

BTW: python is one of the packages didn't pass a single one of 
my QM constraints (hey, not completely corrent, IMHO, install
prefix was passed ;-O)


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[David Woodhouse]

On Fri, 2008-06-13 at 10:02 -0500,
linux-embedded-owner@reliableembeddedsystems.com wrote:
> Do you wanna set some breakpoints and inspect variables in makefiles?
> Have a look at a simple makefile debugger (written in make):
> http://newsletter.embedded.com/shared/printableArticle.jhtml?articleID=197003517

404.

You got me all excited there...

-- 
dwmw2

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bill Traynor]

> On Fri, 2008-06-13 at 10:02 -0500,
> linux-embedded-owner@reliableembeddedsystems.com wrote:
>> Do you wanna set some breakpoints and inspect variables in makefiles?
>> Have a look at a simple makefile debugger (written in make):
>> http://newsletter.embedded.com/shared/printableArticle.jhtml?articleID=197003517
>
> 404.

Is this what the link was supposed to be:

http://www.embedded.com/columns/technicalinsights/197003517?printable=true

or in non-printable format:

http://www.embedded.com/columns/technicalinsights/197003517?_requestid=145614

Man, I must be bored today ;)

>
> You got me all excited there...
>
> --
> dwmw2
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Makefile debugger

[linux-embedded-owner]

Sorry, the link did not work.
It wants to pass by my webpage;)

Use this:  
http://www.reliableembeddedsystems.com/newsflashes/newsflash/debugging-makefiles.html

  Regards,

  Robert

  ---
  Robert Berger
  Embedded Software Specialist

  Reliable Embedded Systems
  Consulting Training Engineering
  Tel.: (+30) 697 593 3428
  Fax.:(+30 210) 684 7881
  URL: www.reliableembeddedsystems.com

Quoting linux-embedded-owner@reliableembeddedsystems.com:

> Hi David,
>
> Do you wanna set some breakpoints and inspect variables in makefiles?
> Have a look at a simple makefile debugger (written in make):
> http://newsletter.embedded.com/shared/printableArticle.jhtml?articleID=197003517
>
> Regards,
>
> Robert
>
> ---
> Robert Berger
> Embedded Software Specialist
>
> Reliable Embedded Systems
> Consulting Training Engineering
> Tel.: (+30) 697 593 3428
> Fax.:(+30 210) 684 7881
> URL: www.reliableembeddedsystems.com
>
>
> Quoting David Woodhouse <dwmw2@infradead.org>:
>
>> On Thu, 2008-06-12 at 19:25 -0500, Rob Landley wrote:
>>> Make doesn't scale.
>>
>> Make scales just fine. The only real problem with make is that it's a
>> complete pain to debug.
>>
>> --
>> dwmw2
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Rob Landley]

[-- Attachment #1: Type: text/plain, Size: 3389 bytes --]

On Thursday 12 June 2008 12:14:32 Bill Gatliff wrote:
> Paul Mundt wrote:
> > Yes, that's the easy case. It's things like perl that are the corner
> > cases, and my objection comes from the fact that people think we ought to
> > not have the kernel depend on perl rather than just fixing the package
> > itself. Autoconf/libtool damage is an entirely different problem :-)
>
> At first glance, it seems like checkincludes.pl could be duplicated by
> egrep | uniq | wc vs. egrep | wc.  Not quite sure what checkversion.pl is
> trying to do.

There's a difference between "this is a development tool used while modifying 
source code" and "this is needed to build".

There are situations where it's ok to have a dependency on X11/qt/gtk, such 
as "make xconfig".  This is _not_ the same as adding such dependency to "make 
modules".

So far, none of the perl dependencies prevent you from building the kernel on 
a system that didn't have perl (or didn't have the right version of perl).

> So maybe we could _reduce_ dependency on perl, if there's any advantage to
> gain by doing so.  But the kernel build machinery isn't dependent on very
> many other systems (just tcl, bash and gcc-core),

There's no tcl dependency in the build.  (Yes, I actually know this.)

Part of my FWL work involves getting the system to rebuild itself under 
itself.  (The packages you need to make a minimal self-bootstrapping system 
are gcc, binutils, make, bash, uClibc, linux, and busybox/toybox).  I'm seven 
commands away from doing this.

I know this because I made a horrible little wrapper (attached, it really is 
sad) which touched a file with the name it was called as and then exec()ed 
the actual executable out of another directory.  Then I populated a directory 
with symlinks to every executable in $PATH (for i in `echo $PATH | 
sed 's/:/ /g'`;do for j in `ls $i`; do ln -s $i/$j $j; done; done), and 
another directory of similar symlinks to my wrapper.  I then ran my build 
with that wrapper directory at the start of $PATH and let the wrapper 
populate a directory with all the executables that actually got called during 
the build.  Then I filled up a directory with those executables, tried to run 
the build, and figured out why it broke.  (The above approach won't find 
calls to /bin/bash and a few other things, but it's a good start.)

Most of the point of my ./host-tools.sh wrapper in the FWL build is to 
populate a directory with the command line utilities mini-native will have in 
it (specifically the busybox/toybox versions, not the ones in the host 
system), and set $PATH equal to that directory and only that directory.  This 
way I know the system will build under itself because that's how it's 
building in the first place.

Currently, I need to grab the following out of the host system:

for i in ar as nm cc gcc make ld   bzip2 find install od sort diff wget
do
  [ ! -f "${HOSTTOOLS}/$i" ] && (ln -s `which $i` "${HOSTTOOLS}/$i" || dienow)
done

The first seven are the needed bits of the host toolchain (you'd think "strip" 
would be in there, but it turns out those packages only ever use 
$TARGET-strip).  The last seven are the ones that are either missing or had 
various bugs in the version of busybox I'm using that prevented the build 
from working right.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

[-- Attachment #2: wrappy.c --]
[-- Type: text/x-csrc, Size: 909 bytes --]

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#include <fcntl.h>

char blah[65536];

#define ROOTPATH "/home/landley/firmware/firmware"

int main(int argc, char *argv[], char *env[])
{
  char *p, *p2;

  int i, fd;

  p2 = strrchr(*argv, '/');
  if (!p2) p2=*argv;
  else p2++;

  p=blah + sprintf(blah, "%s ",p2);
  for (i=1; i<argc; i++) {
    p += sprintf(p, "\"%s\" ",argv[i]);
  }
  p[-1]='\n';

  // Log the command line

  fd=open(ROOTPATH "/loggy.txt",O_WRONLY|O_CREAT|O_APPEND,0777);
  write(fd, blah, strlen(blah));
  close(fd);

  // Touch the file that got used.

  sprintf(blah,ROOTPATH "/used/%s", p2);
  close(open(blah, O_WRONLY|O_CREAT, 0777));

  // Hand off control to the real executable
  sprintf(blah, ROOTPATH "/handoff/%s", p2);
  execve(blah, argv, env);

  // Should never happen, means handoff dir is set up wrong.
  dprintf(2,"Didn't find %s\n", *argv);
  exit(1);
}

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Tim Bird]

Bill Traynor wrote:
>>> Do you wanna set some breakpoints and inspect variables in makefiles?
>>> Have a look at a simple makefile debugger (written in make):
> http://www.embedded.com/columns/technicalinsights/197003517?printable=true

The article above shows some macros you can add to your Makefile to show
you the values of certain things.  It is helpful, but not an interactive
debugger.

However, the guy who wrote that article did indeed write a full
Makefile debugger.  I took it for a spin, and found that it worked as
advertised on a simple Makefile.  I tried it on the kernel Makefile, but
without success (but I didn't try very hard).

YMMV.  I put some of the resources and info I found at:
http://elinux.org/Debugging_Makefiles

The link from the package's README doesn't work, but here's a link
I found to an article about the debugger:
http://www.cmcrossroads.com/content/view/6479/120/

 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: cross-compiling alternatives

[Bernd Petrovitsch]

On Fre, 2008-06-13 at 17:16 +0200, Enrico Weigelt wrote:
> * Bernd Petrovitsch <bernd@firmix.at> schrieb:
> 
> > > Basically yes. But if you have a big number of packages (or a huge package) 
> > > which you didn't write yourself, there will be tests which run executables. 
> > > Figuring out what all the tests are supposed to test in a complex unknown 
> > > software project is not trivial.
> > 
> > Yes, you get used to find the relevant lines in config.log and similar
> > with `grep` and similar tools;-)
> 
> Which are different on each package. So you have to configure each package

ACK.

> for each target manually, which leads the whole point of autoconf
> ad absurdum ;-o

Yup.

[...]
> > pkg-config generated (and generates? - I didn't check recently)
> > references to libraries including the full absolute path (which is the
> > one at build time. And at run-time there is usually
> > no /home/bernd/src/... or where some build may just run).
> 
> Recent pkg-config supports sysroot.

FC-6 has "only" 0.21.

> So you simply build your .pc files as usual (w/o sysroot prefix) and
> set the sysroot prefix via env on the pkg-config call.

From a quick glance over the man page of 0.23, yes.

> > > Can you please explain ? How do the generated pkg_config files look like ?
> > > Ahh, you mean they contain e.g -L/my/build/host/target/environment/opt/foo/lib 
> > > instead of just -L/opt/foo/lib ?
> 
> Then you've got a broken .pc file ;-P

The problem is that the build-time (cross-)linker needs to find the
(cross-compiled) lib under /my/build/host/target/environment/opt/foo/lib
at link time and the shared linker under /opt/foo/lib at run-time.
Hmm, after digging into that old project, it seems that libtool and
the .la files were the problem.

> > Yes. And even worse the compiled lib "foo" had explicit dependencies (on
> > lib "bar") on
> > "/my/build/host/target/environment/opt/bar/lib/libbar.so.1.2.3.4". 
> 
> And that's even more broken.

Yup. Maybe it was a result of my trial to make libtool work somehow ..

> > And BTW pkg-config didn't support the concept of a "DESTDIR" variable
> > (and I don't care about the name of that variable).
> 
> No, why should it ?! It does not install anything.

But it may "use" installed files.

> Probably you're looking for sysroot ?

Yes, very probably.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Robert Schwebel]

On Thu, Jun 12, 2008 at 08:37:35PM +0200, Enrico Weigelt wrote:
> ACK. One of the first things I did was replacing libtool and
> kicked off broken-by-design macros like AC_TRY_RUN. This solved
> at >80% of the problems for me.

Instead of hacking around and inventing new things, you should have
spent your time for improving libtool ...

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Robert Schwebel]

On Fri, Jun 13, 2008 at 05:11:04AM +0200, Sam Ravnborg wrote:
> Tom has started a nice project which he named: quagmire.
> See: http://code.google.com/p/quagmire/
> 
> From the website:
> quagmire is intended to replace automake and libtool. Unlike these tools 
> it requires GNU make and is written solely using GNU make
> features -- no preprocessing is required.

Last time I looked it had *at least* 0.1% of the autotools features
supported :-)

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Robert Schwebel]

On Fri, Jun 13, 2008 at 08:30:52AM +0200, Alexander Neundorf wrote:
> Battle of Wesnoth is currently converted to both Scons and CMake, and
> in the end they will decide about the winner. (since Eric is good at
> arguing I guess it will be scons).

The thing is that 'configure && make && make install' plus the usuall
--enable-foo / --disable-foo / --with-bla=blub semantics is simply *the*
standard way of configuring stuff for unix systems. You don't need fancy
tools, you get cross compiling almost for free and unix people simply
know how to use it.

All the cool kids out there who think they know everything better
usually start with "I hate autotools", then invent something which
solves 0.1% of the problems (including their very special problem) and
tell the rest of the world that their shiny new tools is *soooo cooool*.

Been there, seen that. I maintain > 500 packets in PTXdist and guess
which ones make 90% of the problems. Hint: they are not related to
autotools ...

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Sam Ravnborg]

On Fri, Jun 13, 2008 at 11:44:54AM -0700, Tim Bird wrote:
> Bill Traynor wrote:
> >>> Do you wanna set some breakpoints and inspect variables in makefiles?
> >>> Have a look at a simple makefile debugger (written in make):
> > http://www.embedded.com/columns/technicalinsights/197003517?printable=true
> 
> The article above shows some macros you can add to your Makefile to show
> you the values of certain things.  It is helpful, but not an interactive
> debugger.
> 
> However, the guy who wrote that article did indeed write a full
> Makefile debugger.  I took it for a spin, and found that it worked as
> advertised on a simple Makefile.  I tried it on the kernel Makefile, but
> without success (but I didn't try very hard).
> 
> YMMV.  I put some of the resources and info I found at:
> http://elinux.org/Debugging_Makefiles
> 
> The link from the package's README doesn't work, but here's a link
> I found to an article about the debugger:
> http://www.cmcrossroads.com/content/view/6479/120/

Quite impressive what he got out of the make features.
The kernel does a few unusual tricks so I do understand
why it does not work there.

Speaking about the kernel...
Try "make V=2" one day to see why a target are build. It is quite useful
when a small change results in much more being build than you expected.
But this is kernel specific and not a general feature.

	Sam

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bill Traynor]

> Bill Traynor wrote:
>>>> Do you wanna set some breakpoints and inspect variables in makefiles?
>>>> Have a look at a simple makefile debugger (written in make):
>> http://www.embedded.com/columns/technicalinsights/197003517?printable=true
>
> The article above shows some macros you can add to your Makefile to show
> you the values of certain things.  It is helpful, but not an interactive
> debugger.
>
> However, the guy who wrote that article did indeed write a full
> Makefile debugger.  I took it for a spin, and found that it worked as
> advertised on a simple Makefile.  I tried it on the kernel Makefile, but
> without success (but I didn't try very hard).
>
> YMMV.  I put some of the resources and info I found at:
> http://elinux.org/Debugging_Makefiles
>
> The link from the package's README doesn't work, but here's a link
> I found to an article about the debugger:
> http://www.cmcrossroads.com/content/view/6479/120/

The links in the README worked fine for me.  FWIW, the authors homepage is
here: http://gmd.sf.net He's written a self-published book on GNU Make, it
would seem.

>
>  -- Tim
>
> =============================
> Tim Bird
> Architecture Group Chair, CE Linux Forum
> Senior Staff Engineer, Sony Corporation of America
> =============================
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: cross-compiling alternatives

[Robert Schwebel]

On Fri, Jun 13, 2008 at 08:45:09PM +0200, Bernd Petrovitsch wrote:
> > Recent pkg-config supports sysroot.
> 
> FC-6 has "only" 0.21.

Solved in ptxdist by building the "right" pkgconfig as a host tool.

> > So you simply build your .pc files as usual (w/o sysroot prefix) and
> > set the sysroot prefix via env on the pkg-config call.

Ah, great, must be a new feature. Will have to integrate that into
ptxdist... at the moment we fix it with a wrapper script.

> The problem is that the build-time (cross-)linker needs to find the
> (cross-compiled) lib under
> /my/build/host/target/environment/opt/foo/lib at link time and the
> shared linker under /opt/foo/lib at run-time.

pkg-config is fixable (our wrapper in PTXdist does the right thing wrt.
buildtime vs. runtime paths).

> Hmm, after digging into that old project, it seems that libtool and
> the .la files were the problem.

Right, libtool's paths are critical: libtools isn't sysroot aware, and
it seems to be not easily fixable.

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Rob Landley]

On Friday 13 June 2008 04:06:18 Alexander Neundorf wrote:
> > And the above are not really a big problem -
>
> "checking if something builds" is no problem, this just works. Running
> something is a problem, as in "it doesn't just work" (...because you cannot
> run it).

Noticing 2 weeks after deployment that signal handling in the mips version of 
perl is using the x86 signal numbers and they're not the same: priceless.

> > The only simple solution so far (without diving into the implementation
> > and searching for root causes) were AFAICS:
> > - do not use libtool for linking (as the link line as such without
> >   libtool works as expected)
>
> Yes, libtool sucks, it's the wrong solution to the problem.
> (and CMake doesn't use it).

Nothing on Linux really _uses_ libtool.  It's supposed to act as a NOP wrapper 
around the linker on any Linux target.  (It's there for things like Sparc and 
HPUX).

The fact that libtool manages to do its nothing _wrong_ so often would be 
hilarious if it wasn't such a pain.  Just uninstall libtool before trying to 
build for a Linux target, this should never cause any problems and will save 
you lots of headaches.

> > Why on earth does someone need this explicitly during the build?
> > If you have portable software, all of that should be hidden in the code
> > and use "sizeof(int)".

According to the LP64 standard which pretty much all modern Unixes adhere to 
(including both Linux and MacOS X) sizeof(int) is always 4.  Guaranteed.

The LP64 standard:
  http://www.unix.org/whitepapers/64bit.html

The LP64 rationale:
  http://www.unix.org/version2/whatsnew/lp64_wp.html

The insane legacy reasons windows doesn't do this, but in fact sizeof(int) 
will still be 4 there anyway:
  http://blogs.msdn.com/oldnewthing/archive/2005/01/31/363790.aspx

Just FYI. :)

> But this was not the point. My point was: testing something by running an
> executable can be _a lot_ easier than testing the same without running
> something.

I think building natively under qemu is the easy way, yes. :)

> Alex
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Johannes Stezenbach]

On Fri, Jun 13, 2008, Tim Bird wrote:
> 
> YMMV.  I put some of the resources and info I found at:
> http://elinux.org/Debugging_Makefiles

There is also "remake", which is "A patched GNU make with a debuger,
better tracing and error reporting" (based on GNU make 3.80).
Development seems to have stopped, though.
http://sourceforge.net/projects/bashdb/

Johannes

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[David VomLehn]

James Chapman wrote:
> David VomLehn wrote:
>> Enrico Weigelt wrote:
>>> * Rob Landley <rob@landley.net> schrieb:
>>>  
>>>> Cross compiling breaks stuff, yes.
>>>>
>>>> Most packages don't cross compile at all.  Debian has somewhere 
>>>> north of 30,000 packages.  Every project that does large scale 
>>>> cross compiling (buildroot, gentoo embedded, timesys making fedora 
>>>> cross compile, etc) tends to have about 200 packages that cross 
>>>> compile more or less easily, another 400 or so that can be made to 
>>>> cross compile with _lot_ of effort and a large enough rock, and 
>>>> then the project stalls at about that size.
>>>>     
>>>
>>> The problem is: most embedded projects don't make really 
>>> general-purpose
>>> fixes (instead strange things like hacking up autogenerated files), 
>>> so they can't feed back to upstream.
>>>
>>> IMHO, a huge waste of working time.
>>>   
>> Amen, brother. I'm fortunate in that I work for an organization that 
>> is quite good about enforcing code reviews, specifically, the QA 
>> organization is empowered to reject changes that do not have code 
>> review notes. I also have a fairly broad scope, so I'm in on code 
>> reviews for a number of open source components. At each such review, 
>> one of my criteria is whether the change is suitable for pushing back 
>> to the appropriate community. This is not necessarily a short-term 
>> way to make friends, but the long-term effects will be good both for 
>> the company and for the open source community in general.
>>
>> Now, if we can only get the time to actually push all the backlogged 
>> fixes out...
>
> Er, is that GPL or LGPL code that you're modifying? If so, you *have* 
> to push those code changes out (make them available to others), 
> whether you think people will be interested or not!
>
I guess I'm making a distinction that wasn't clear. We *have* to make
the code available, and I can assure you that Cisco is very aware of our
obligations in this area and I spend a fair amount of my time trying to
ensure they are met.

I used the term "push" to mean getting patches ready, posting them to the 
appropriate mailing lists, revising them in light of comments, and doing 
everything else necessary to get them incorporated into the kernel source base. 
"Pushing" is a lot more work than just making source available, but also yield 
much more productive long term results for everyone.

-- 
David VomLehn, dvomlehn@cisco.com
The opinions expressed herein are likely mine, but might not be my employer's...






     - - - - -                              Cisco                            - - - - -         
This e-mail and any attachments may contain information which is confidential, 
proprietary, privileged or otherwise protected by law. The information is solely 
intended for the named addressee (or a person responsible for delivering it to 
the addressee). If you are not the intended recipient of this message, you are 
not authorized to read, print, retain, copy or disseminate this message or any 
part of it. If you have received this e-mail in error, please notify the sender 
immediately by return e-mail and delete it from your computer.

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Jamie Lokier]

Robert Schwebel wrote:
> On Fri, Jun 13, 2008 at 08:30:52AM +0200, Alexander Neundorf wrote:
> > Battle of Wesnoth is currently converted to both Scons and CMake, and
> > in the end they will decide about the winner. (since Eric is good at
> > arguing I guess it will be scons).
> 
> The thing is that 'configure && make && make install' plus the usuall
> --enable-foo / --disable-foo / --with-bla=blub semantics is simply *the*
> standard way of configuring stuff for unix systems. You don't need fancy
> tools, you get cross compiling almost for free and unix people simply
> know how to use it.

A trouble with that is some packages have hundreds of user-selectable
options - or even thousands.  It is unfeasible to use --enable-foo
options for all of those when configuring then.

Some other packages _should_ have more options, but don't because it's
too unwieldy to make them highly configurable with Autoconf.  Imho,
Kconfig would be good for more programs than it's currently used for,
and could be made to work with those --enable/--with options: you'd be
able to configure them entirely on the command line, or interactively
with "./configure --menu" (runs menuconfig), or with a config file.

The "make" / "make install" part is easy to retain even with other
build systems, using a trivial Makefile which calls the other tools.
Lots of packages need special tools or other software installed to
build them; this is no different.

Perhaps it might even be possible to write a very small, portable,
specialised alternative to Make which is small enough to ship with
packages that use it?

-- Jamie

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Robert Schwebel]

On Fri, Jun 13, 2008 at 11:25:23PM +0100, Jamie Lokier wrote:
> A trouble with that is some packages have hundreds of user-selectable
> options - or even thousands.

I've not seen a package with thousands of user selectable options. It's
not even desirable, because the more options you have, the more
difficult it becomes to test all the combinations.

> Some other packages _should_ have more options, but don't because it's
> too unwieldy to make them highly configurable with Autoconf.  Imho,
> Kconfig would be good for more programs than it's currently used for,
> and could be made to work with those --enable/--with options: you'd be
> able to configure them entirely on the command line, or interactively
> with "./configure --menu" (runs menuconfig), or with a config file.

That's exactly what ptxdist does: add a Kconfig frontend to the
configurable switches. It does it for the user's convenience, although
the currently implemented method is really developer-unfriendly (but we
care about our users first).

But it's of absolutely no use to whine about the fact that the world is
such a curel place. *If* Kconfig had been there 20 years ago ... *if*
90% of the packages out there would have been Kconfig instead of
autotools... We have to live with *reality*, and reality is that
autotools solve real world problems, and they offer *one* standard user
interface to package building. I can cross build almost all autotoolized
packages in exactly the same way and people are used to it. All other
build systems I've seen invented their very special way of doing things,
leading to wheel-reinvention all over the place.

> The "make" / "make install" part is easy to retain even with other
> build systems, using a trivial Makefile which calls the other tools.

I still don't understand why all the scons, cmakes and jams out there
don't even try to provide the *standard* user interface everyone is used
to on a unix system.

> Lots of packages need special tools or other software installed to
> build them; this is no different.

autotools need only a shell and make

> Perhaps it might even be possible to write a very small, portable,
> specialised alternative to Make which is small enough to ship with
> packages that use it?

Why on earth would one want to reinvent make?

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Jamie Lokier]

Robert Schwebel wrote:
> On Fri, Jun 13, 2008 at 11:25:23PM +0100, Jamie Lokier wrote:
> > A trouble with that is some packages have hundreds of user-selectable
> > options - or even thousands.
> 
> I've not seen a package with thousands of user selectable options. It's
> not even desirable, because the more options you have, the more
> difficult it becomes to test all the combinations.

Kernels, uclibc, busybox.  All combinations can't be tested.  But it's
still _very_ useful to compile in only those parts wanted.

Most packages have far fewer.  But some have enough that the command
line is unwieldy and Kconfig would fit.  I'm thinking libcurl - one
configure option for each protocol supported.  iptables (userspace)
would be a candidate, when you have to link statically and save space.

Media players with lots of optional formats and drivers are another.
(They also have considerable problems with their Autoconf in my
experience).

Generally, anything with lots of parts that different applications
might not use, and space or library dependencies are an issue.

> > Some other packages _should_ have more options, but don't because it's
> > too unwieldy to make them highly configurable with Autoconf.  Imho,
> > Kconfig would be good for more programs than it's currently used for,
> > and could be made to work with those --enable/--with options: you'd be
> > able to configure them entirely on the command line, or interactively
> > with "./configure --menu" (runs menuconfig), or with a config file.
> 
> That's exactly what ptxdist does: add a Kconfig frontend to the
> configurable switches. It does it for the user's convenience, although
> the currently implemented method is really developer-unfriendly (but we
> care about our users first).

I agree.  (And it proved about not being able to test more
combinations: last time I tried to build ptxdist, an up to date
version (at the time), it failed in several places.)

> But it's of absolutely no use to whine about the fact that the world is
> such a curel place. *If* Kconfig had been there 20 years ago ... *if*
> 90% of the packages out there would have been Kconfig instead of
> autotools... We have to live with *reality*, and reality is that
> autotools solve real world problems, and they offer *one* standard user
> interface to package building. I can cross build almost all autotoolized
> packages in exactly the same way and people are used to it. All other
> build systems I've seen invented their very special way of doing things,
> leading to wheel-reinvention all over the place.

I agree with you.  There's no need to disagree, and who's whining?
We're trying to think of realistic improvments.

Reality is that Kconfig front end to autotools does work - as you've
proved.  It's a good idea. :-)

> > The "make" / "make install" part is easy to retain even with other
> > build systems, using a trivial Makefile which calls the other tools.
> 
> I still don't understand why all the scons, cmakes and jams out there
> don't even try to provide the *standard* user interface everyone is used
> to on a unix system.

Me neither.

> > Lots of packages need special tools or other software installed to
> > build them; this is no different.
> 
> autotools need only a shell and make

No, that's true only for very simple packages.

Most packages need lots of additional libraries installed - and the
development versions of those libraries, for that matter.  Too often
the right development version - not too recent, not too old.  With the wrong versions, there are surprises.

You said about too many user-selectable options.  Many large packages
_check_ for many installed libraries.  Get them wrong, and you have
the same problems of untested combinations.

Quite a lot of packages require extra tools to build, beyond shell,
make, GCC and Binutils.  Perl, Python are common.

Sure, autotools by themselves don't need much.  But that's not
interesting: Autotools are not used only by themselves.

> > Perhaps it might even be possible to write a very small, portable,
> > specialised alternative to Make which is small enough to ship with
> > packages that use it?
> 
> Why on earth would one want to reinvent make?

Have you felt uncomfortable shipping a package that does use Autoconf,
Automake and Libtool, knowing that the scripts generated by those
tools are huge compared with the entire source of your package?

Have you _written_ Autoconf tests recently?  Made any shell /
shellutils non-portability mistakes in the tests?

Have you _read_ a portable Makefile lately?  Have you tried writing
one for a complex package, confident that it's portable to different
quirky makes, quirky shells and quirky tools, outside the parts which
you might use Automake for?

That's a rationale for the project which is rewriting Autotools in GNU
Make, assuming that to be ubiquitous now, imho.  (Not all interesting
systems have a shell either.)

If you're going to rewrite Autotools using GNU Make, why not ask if
another tool would be better, perhaps a tool specially designed for
the job?

-- Jamie

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Jamie Lokier]

Enrico Weigelt wrote:
> * Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:
> 
> > E.g. in python there are tests which call functions and check 
> > their result to see if we are currently on a platform where 
> > that function is broken (I think there was such a test for 
> > poll() and some other functions).
> 
> IMHO, that's broken sw engineering from ground up.

Oh?  The alternative I see is to do the test at run-time.  But that
adds to executable size and run-time slowdown on most platforms.

Doing it at build time is an improvement, for those people who don't
care about cross-compilation.  (Not me, you understand.)

-- Jamie

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Robert Schwebel]

On Sat, Jun 14, 2008 at 01:07:49AM +0100, Jamie Lokier wrote:
> Kernels, uclibc, busybox.  All combinations can't be tested.  But it's
> still _very_ useful to compile in only those parts wanted.

Kernel (and thus kconifg) is a critical mass project of it's own;
however, kconfig does only solve the "configuration" problem. Autotools
is much more. However, I was talking about userspace software. Core
components have usually much different needs.

> Media players with lots of optional formats and drivers are another.
> (They also have considerable problems with their Autoconf in my
> experience).

Send patches :-)

> Generally, anything with lots of parts that different applications
> might not use, and space or library dependencies are an issue.

According to my experience, the configure part of autotools is not it's
problem. Its more

- configure scripts are slow
- libtool isn't sysroot/destdir aware
- complex cross scenarios are not well supported (i.e. ace/tao, where
  the IDL compiler needs most of libace and has to be compiled for both,
  the "build" and "host" system)

> > That's exactly what ptxdist does: add a Kconfig frontend to the
> > configurable switches. It does it for the user's convenience, although
> > the currently implemented method is really developer-unfriendly (but we
> > care about our users first).
>
> I agree. (And it proved about not being able to test more
> combinations: last time I tried to build ptxdist, an up to date
> version (at the time), it failed in several places.)

Did you tell the development team? We are known to fix 95% of the
problems in this universe in less than 10 minutes :-)

However, we have a stable policy in the meantime (1.0.x is in
maintenance only mode, regularly tested and quickly fixed), and the team
is really responsive. So it may be time to try it out again :-)

> > autotools need only a shell and make
> 
> No, that's true only for very simple packages.

Ok, plus normal unix tools like sed and awk ...

> Most packages need lots of additional libraries installed - and the
> development versions of those libraries, for that matter.  Too often
> the right development version - not too recent, not too old.  With the
> wrong versions, there are surprises.

Right. But this is the downside of being able to deal with all this
complexity: every other build system would have the same problem.

> You said about too many user-selectable options. Many large packages
> _check_ for many installed libraries. Get them wrong, and you have the
> same problems of untested combinations.

Yup, auto-testing is usually a problem if you build cross stuff.

> Sure, autotools by themselves don't need much.  But that's not
> interesting: Autotools are not used only by themselves.

They don't need a special runtime environment other than shell. Other
systems like Perl, Python, Java or whatever has the problem that
anything other than the very core is not very well defined and ends up
in the version hell.

We have once tried to write our automated test system for embedded
boards with python and xml; the idea was to have something fancy, new
and with good code-reuse. In the end it failed because the pexpect
package we used to do the pattern matching bitrotted so quickly that
four months later all the fancy tests didn't work any more, because it
is not part of the python core.

In the meantime we have migrated our automatic testing stuff to use
shell, ssh/rsh and kermit. It is rock solid, and the code reuse factor
is at least as good as with anything else.

> Have you felt uncomfortable shipping a package that does use Autoconf,
> Automake and Libtool, knowing that the scripts generated by those
> tools are huge compared with the entire source of your package?

No :-)

> Have you _written_ Autoconf tests recently?

Yea, all our packages are autotoolized.

> Made any shell / shellutils non-portability mistakes in the tests?

Yea, it happens in ptxdist all the time. People report about problems,
we add new tests and the next revision works even on Ubuntu :)

> Have you _read_ a portable Makefile lately? Have you tried writing
> one for a complex package, confident that it's portable to different
> quirky makes, quirky shells and quirky tools, outside the parts which
> you might use Automake for?
>
> That's a rationale for the project which is rewriting Autotools in GNU
> Make, assuming that to be ubiquitous now, imho.  (Not all interesting
> systems have a shell either.)

I have no problems with people writing fancy new things. It's just that
most people who try to do something better than autotools have only a
fraction of the features.

Open Source is darwinism: if there is something better, let's use it.
But compare apples with apples.

> If you're going to rewrite Autotools using GNU Make, why not ask if
> another tool would be better, perhaps a tool specially designed for
> the job?

Go ahead.

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Leon Woestenberg]

Hello all,

On Thu, Jun 12, 2008 at 2:41 AM, Rob Landley <rob@landley.net> wrote:
>
> Most packages don't cross compile at all.  Debian has somewhere north of
> 30,000 packages.  Every project that does large scale cross compiling
> (buildroot, gentoo embedded, timesys making fedora cross compile, etc) tends
> to have about 200 packages that cross compile more or less easily, another
> 400 or so that can be made to cross compile with _lot_ of effort and a large
> enough rock, and then the project stalls at about that size.
>

Agreed, OpenEmbedded has a few thousands, but your point is valid.
However, fleeing to target-native compilation is not the way to
improve the situation IMHO.

Moore's law on hardware also goes for the host, I think the progress
is even bigger on big iron.

Also, how much of the 30000 packages are useful for something like
your own firmware Linux?

> Distcc can take advantage of smp, but that won't help the ./configure stage
> and I need to do some work on distcc to teach it to understand more gcc
>
If you want to build 1000+ packages, you don't need to run configure
itself multithreaded. There are enough jobs available to keep 16/32
processors busy (beyond that, you probably end up in
inter-package-dependencies stalling the build). This is just a guess
from what I see during a multi-threaded bake and multi-threaded make
on OpenEmbedded.

> However, having one or more full-time engineers devoted to debugging
> cross-compile issues is quite a high price to pay too.  Moore's law really
> doesn't help that one.
>
How about 30+ volunteers.

> I'm not saying either solution is perfect, I'm just saying the "build under
> emulation" approach is a viable alternative that gets more attractive as time
> passes, both because of ongoing development on emulators and because of
> Moore's law on the hardware.
>
I cannot follow your reasoning - Moore's law will help you more on the
big iron side of things.

That said, I welcome any effort (such as yours) to help improve the
embedded Linux domain, I rather try to fix the cross-compile stuff of
the few thousand packages I am interested in.

Yes it hurts my brain.

Regards,
-- 
Leon

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Rob Landley]

On Sunday 15 June 2008 10:39:43 Leon Woestenberg wrote:
> Hello all,
>
> On Thu, Jun 12, 2008 at 2:41 AM, Rob Landley <rob@landley.net> wrote:
> > Most packages don't cross compile at all.  Debian has somewhere north of
> > 30,000 packages.  Every project that does large scale cross compiling
> > (buildroot, gentoo embedded, timesys making fedora cross compile, etc)
> > tends to have about 200 packages that cross compile more or less easily,
> > another 400 or so that can be made to cross compile with _lot_ of effort
> > and a large enough rock, and then the project stalls at about that size.
>
> Agreed, OpenEmbedded has a few thousands, but your point is valid.
> However, fleeing to target-native compilation is not the way to
> improve the situation IMHO.

You say it like fleeing is a bad thing. :)

I believe building natively under emulation is the Right Thing.  Cross 
compiling has always historically been a transitional step until native 
compiling became available on the target.

When Ken Thompson and Dennis Ritchie were originally creating Unix for the 
PDP-7, they cross compiled their code from a honking big GE mainframe because 
that was their only option.  One of the first things they wrote was a PDP-7 
assembler that ran on the PDP-7.  The reason they created the B programming 
language in the first place was to have a tiny compiler that could run 
natively on the PDP-7, and when they moved up to a PDP-11 Dennis had more 
space to work with and expanded B into C.

When they severed the mainframe umbilical cord as soon as they were able to 
get the system self-hosting, it wasn't because the PDP-7 had suddenly become 
faster than the GE mainframe.

Compiling natively where possible has been the normal way to build Unix 
software ever since.  Linux became a real project when Linus stopped needing 
Minix to cross-compile it.  Linus didn't "flee" Minix, he assures us he 
erased his minix partition purely by accident. :)

> Moore's law on hardware also goes for the host, 

Which is why people no longer regularly write application software in assembly 
language, because we don't need to do that anymore.  The result would be 
faster, but not better.

The rise of scripting languages like Python and javascript that run the source 
code directly is also related (and if you don't think people don't write 
complete applications in those you haven't seen any of the google apps).  The 
big push for Java in 1998 could happen because the hardware was now fast 
enough to run _everything_ under an emulator for a processor that didn't 
actually exist (until Rockwell built one, anyway).

Build environments are now literally thousands of times faster than when I 
started programming.  The first machine I compiled code on was a commodore 64 
(1mhz, 8 bits, the compiler was called "blitz" and the best accelerator for 
it was a book).  The slowest machine I ever ran Linux on was a 16 mhz 386sx.

According to my blog, I moved from a 166mhz laptop to a 266mhz one on April 
13, 2002.  I started building entire Linux From Scratch systems on the 166mhz 
machine, including a ton of optional packages (apache, postgresql, openssh, 
samba, plus it was based on glibc and coreutils and stuff back then so the 
build was _slow_), hence the necessity of scripting it and leaving the build 
to its own devices for a few hours.

Even without distcc calling out to the cross compiler, the emulated system 
running on my laptop is several times faster than the build environment I had 
7 years ago (2001), somewhat faster than the one I had 5 years ago (2003), 
and somewhat slower than the one I had 3 years ago (2005).  (That's emulating 
an x86 build environment on my x86_64 laptop.  I didn't _have_ a non-x86 
build enviornment 5 years ago for comparison purposes.)

> I think the progress is even bigger on big iron.

Not that I've noticed, unless by "big iron", you mean "PC clusters".  (You can 
expand laterally if you've got the money for it and your problem distributes 
well...)

> Also, how much of the 30000 packages are useful for something like
> your own firmware Linux?

None of them, because Firmware Linux has a strictly limited agenda: provide a 
native build environment on every system emulation supported by qemu.  That's 
the 1.0 release criteria.  (Some day I may add other emulators like hercules 
for s390, but the principle's the same.)

Once you have the native build environment, you can bootstrap Gentoo, or 
Debian, or Linux From Scratch, or whatever you like.  I've got instructions 
for some of 'em.

The buildroot project fell into the trap of becoming a distro and having to 
care about the interaction between hundreds of packages.  I'm not interested 
in repeating that mistake.

Figuring out what packages will other people might need is something I stopped 
trying to predict a long time ago.  If it exists, somebody wanted it.  People 
want/need the weirdest stuff: the accelerometer in laptops is used for 
rolling marble games, and the iPhone is a cell phone with 3D support for 
(among other things) video playback.

> > Distcc can take advantage of smp, but that won't help the ./configure
> > stage and I need to do some work on distcc to teach it to understand more
> > gcc
>
> If you want to build 1000+ packages, you don't need to run configure
> itself multithreaded. There are enough jobs available to keep 16/32
> processors busy (beyond that, you probably end up in
> inter-package-dependencies stalling the build). This is just a guess
> from what I see during a multi-threaded bake and multi-threaded make
> on OpenEmbedded.

If your package management system can work out the dependencies and launch the 
package builds in parallel, have at.  Another reason I'm not trying to 
address package management: lots of people have already done it, and put a 
lot more work into it than I ever would.  I'm just trying to help them do 
their thing while staying out of their way.

> > However, having one or more full-time engineers devoted to debugging
> > cross-compile issues is quite a high price to pay too.  Moore's law
> > really doesn't help that one.
>
> How about 30+ volunteers.

I presume you're referring to OpenEmbedded, rather than Gentoo Embedded, 
Debian Embedded, buildroot, openmoko, or any of the others?

Making cross compiling work is tied to your choice of package management 
systems.  Making native compling work, less so.  The point of my project is 
to _unbundle_ stuff.  Separate out that layer, and make it as thin as 
possible.

> > I'm not saying either solution is perfect, I'm just saying the "build
> > under emulation" approach is a viable alternative that gets more
> > attractive as time passes, both because of ongoing development on
> > emulators and because of Moore's law on the hardware.
>
> I cannot follow your reasoning - Moore's law will help you more on the
> big iron side of things.

Actually it helps exactly as much, just on about a 5 year delay.

Moore's Law does not reduce the need for your above mentioned 30+ volunteers 
to endlessly debug cross compiling bugs.

My point is that building under emulation didn't used to be even _feasible_.  
Now it's an option.  Building under an emulator in 2008 gives us about the 
speed of building natively around 2003, and that seemed tolerable at the 
time.  Both the emulators and the hardware should improve going forward.

> That said, I welcome any effort (such as yours) to help improve the
> embedded Linux domain, I rather try to fix the cross-compile stuff of
> the few thousand packages I am interested in.

I've gotten involved in too many big complicated projects that got sidelined 
by somebody figuring out how to substitute a simpler problem and solve that 
one instead.  (Remember how much work maintaining kernel headers was 
before "make headers_install"?)

These days when I see big problem that can most obviously be solved by lots of 
people doing lots of work for a long time, my first instinct is try to figure 
out how to avoid needing to do that.

> Yes it hurts my brain.

It hurts everybody's brain.

There are two ways to build natively: you can do so under an emulator, or you 
can do so on real hardware.

The Fedora for Arm project builds natively, and started by using real ARM 
hardware to do it.  (They've added qemu since; I note that the guy who runs 
that project used to be my boss back before we all left TimeSys.) 
http://fedoraproject.org/wiki/Architectures/ARM#Technical_Approach

The Ubuntu for Arm project did the same, bought high end arm machines and made 
a compile farm out of them: 
http://mojo.handhelds.org/files/HandheldsMojo_ELC2008.pdf

If you're going to throw money at the problem, you can get a lot of 
specialized native hardware to build on, and that's an option a number of 
large projects have been taking.

The advantage of using emulators is that it's cheaper and more flexible.  
Anybody can do it, it's just a build environment you can take with you on 
your laptop (no contention for a limited number of devices and your 
developers aren't tethered to them to get work/testing done).  PC hardware is 
the cheapest and most flexible way to buy raw computing power (including 
memory/disk/network), and pretty much everybody already has it or they 
wouldn't be doing Linux development in the first place.

You don't snag hobbyists by saying "buy this hardware to play".  You snag 
hobbyists by giving them something they can download and poke at with the 
lowest threshold possible, which is why I provide precompiled images ala
http://landley.net/code/firmware/downloads/system-image

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Rob Landley]

On Thursday 12 June 2008 13:18:07 Enrico Weigelt wrote:
> * Rob Landley <rob@landley.net> schrieb:
>
> Hi,
>
> > There's also qemu.  You can native build under emulation.
>
> did you ever consider that crosscompiling is not only good for
> some other arch, but a few more things ?

Sure, such as building a uClibc system on a glibc host, which my _previous_ 
firmware linux project (http://landley.net/code/firmware/old) was aimed at.

That used User Mode Linux instead of qemu, because "fakeroot" wasn't good 
enough and chroot A) requires the build to run as root, B) sometimes gets a 
little segfaulty if you build uClibc with newer kernel headers than the 
kernel in the system you're running on.

You can't get away from cross compiling whenever you want to bootstrap a new 
platform.  But cross compiling can be minimized and encapsulated.  It can be 
a stage you pass through to get it over with and no longer have to deal with 
it on the other side, which is the approach I take.

> > In addition, if you have a cross compiler but don't want to spend all
> > your time lying to ./configure, preventing gcc from linking against the
> > host's zlib or grabbing stuff out of /usr/include that your target hasn't
> > got, or
>
> #1: use a proper (sysroot'ed) toolchain

I break everything.  (I've broken native toolchains.  I just break them 
_less_.)

By my count sysroot is the fifth layer of path logic the gcc guys have added 
in an attempt to paint over the dry rot.

Personally I use a derivative of the old uClibc wrapper script that rewrites 
the command line to start with "--nostdinc --nostdlib" and then builds it 
back up again without having any paths in there it shouldn't.

> #2: fix broken configure.in's (and feed back to upstream or OSS-QM)

Whack-a-mole.  Fun for the whole family.  Only problem is, it never stops.

> #3: replace libtool by unitool

Uninstall libtool and don't replace it with anything, it's a NOP on Linux.

> > libraries are linked inside the emulator, anything that wants to look
> > at /proc or sysinfo does it natively inside the emulator...)
>
> Only crap sw looks at /proc at build time.
> Yes, there's *much* crap sw out there :(

99% of all the developers out there don't really care about portability, and 
never will.  Even if you eliminate the windows guys and the people who don't 
do C, 90% of the people who are _left_ get to work on the PC first, get it to 
work natively on other Linux platforms afterwards.

Cross compiling is a step beyond "portability".  They'll _never_ care about 
cross compiling.  If they get inspired to make it work on MacOS X, then 
you'll have to extract the source and _build_ it on MacOS X to make that 
work.  And 99% of all developers will nod their heads and go "quite right, as 
it should be".

This isn't going to change any time soon.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Rob Landley]

On Thursday 12 June 2008 13:34:21 Enrico Weigelt wrote:
> * Bill Gatliff <bgat@billgatliff.com> schrieb:
> > If the build system derives from autoconf, then a hacked-up
> > config.cache (or equivalent command-line args) often solves
> > problems for me.
>
> Only if you're working on *one specific* target for a long time.
> I, for example, have to support lots of different targets, so your
> approach does not work for me. Ah, and it's not *solving* any problem,
> just deferring it to some other day.

It's not deferring it, it's ripping out the failed automation and configuring 
it manually, answering each question by hand.  (That said, it's the approach 
I took to get bash to cross-compile in FWL.  Sometimes it's all you can 
do...)

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Robert Schwebel <r.schwebel@pengutronix.de> schrieb:

Hi,

> Instead of hacking around and inventing new things, you should have
> spent your time for improving libtool ...

No, not with libtool. 
I do not want to support that insane approach of tweaking command
lines in the middle - it's an Pandorra's Box. I've already spent 
too much time on it and decided to completely drop it.

Instead I prefer *clean* lines.

Unitool provides commands on an higher level than, gcc+co do.
These commands are on an higher functional level, hiding the 
individual platform's details. Also including things like importing
.la-libs, sysroot, etc.

Lt-unitool is a wrapper which parses the libtool commands and calls 
Unitool to do the actual work.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives

[Enrico Weigelt]

* Bernd Petrovitsch <bernd@firmix.at> schrieb:

> > Recent pkg-config supports sysroot.
> 
> FC-6 has "only" 0.21.

Not sure when the sysroot stuff got into upstream releaes, but
maybe you just should update ;-P

> > So you simply build your .pc files as usual (w/o sysroot prefix) and
> > set the sysroot prefix via env on the pkg-config call.
> 
> From a quick glance over the man page of 0.23, yes.

Yep, it doesn't take $SYSROOT (as my original patch did), but with 
some prefix ...

> The problem is that the build-time (cross-)linker needs to find the
> (cross-compiled) lib under /my/build/host/target/environment/opt/foo/lib
> at link time and the shared linker under /opt/foo/lib at run-time.
> Hmm, after digging into that old project, it seems that libtool and
> the .la files were the problem.

Yes, libtool doesn't understand anything like sysroot. It generates 
broken pathnames in the .la files. You could use unitool and it's 
libtool-replacement.

> > > Yes. And even worse the compiled lib "foo" had explicit dependencies (on
> > > lib "bar") on
> > > "/my/build/host/target/environment/opt/bar/lib/libbar.so.1.2.3.4". 
> > 
> > And that's even more broken.
> 
> Yup. Maybe it was a result of my trial to make libtool work somehow ..

Heh, I've given up trying to repair libtool for a long time ;-P

> > > And BTW pkg-config didn't support the concept of a "DESTDIR" variable
> > > (and I don't care about the name of that variable).
> > 
> > No, why should it ?! It does not install anything.
> 
> But it may "use" installed files.

Yes, but they have to be strictly from sysroot. As said, recent 
pkg-config can handle this properly.
 
> > Probably you're looking for sysroot ?
> 
> Yes, very probably.

:)


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Jamie Lokier <jamie@shareable.org> schrieb:

> > > E.g. in python there are tests which call functions and check 
> > > their result to see if we are currently on a platform where 
> > > that function is broken (I think there was such a test for 
> > > poll() and some other functions).
> > 
> > IMHO, that's broken sw engineering from ground up.
> 
> Oh?  The alternative I see is to do the test at run-time.  But that
> adds to executable size and run-time slowdown on most platforms.

There's no generic answer to this, we have to look at the details
carefully ;-P

Most times I've seen those checks, they silently enable some 
features, eg. if it looks for certain kernel devices. Definitively
the wrong way! It really should be in users/packagers control to
explicitly enable features. Nevertheless, the existence of some
file or device says nothing about whether it will be usable
(or *should* be used) at runtime. I've seen packages silently
enabling some feature and then failing at runtime since the 
previously detected device is missing later. What a nightamare 
for packagers.

Another point are broken syscalls. Well, you *have* check at runtime 
to be sure, or perhaps choose to ignore it and expect a sane system.

*If* you really want to do constraint checks, you should do it 
as an separate, optional step. Maybe issue a big-fat hint that 
one really should run that test (and follow certain instructions)
if he don't exactly know what he's doing.

> Doing it at build time is an improvement, for those people who don't
> care about cross-compilation.  (Not me, you understand.)

IMHO, it's just lazyness, at least for about 99% of the cases ;-P


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Robert Schwebel <r.schwebel@pengutronix.de> schrieb:
> On Fri, Jun 13, 2008 at 08:30:52AM +0200, Alexander Neundorf wrote:
> > Battle of Wesnoth is currently converted to both Scons and CMake, and
> > in the end they will decide about the winner. (since Eric is good at
> > arguing I guess it will be scons).
> 
> The thing is that 'configure && make && make install' plus the usuall
> --enable-foo / --disable-foo / --with-bla=blub semantics is simply *the*
> standard way of configuring stuff for unix systems. You don't need fancy
> tools, you get cross compiling almost for free and unix people simply
> know how to use it.

ACK. The ./configure script's syntax is quite convenient and semi-standard.
So, IMHO, all packages which have such an configure stage should support
that syntax, even those who code those scripts manually.
Strange, but some projects insist in having their config scripts incompatible
with autoconf's, just to show that it wasn't produced by it ;-o

> Been there, seen that. I maintain > 500 packets in PTXdist and guess
> which ones make 90% of the problems. Hint: they are not related to
> autotools ...

Well, if you count in the improber uses of autoconf (AC_TRY_RUN, etc),
then: yes.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Jamie Lokier <jamie@shareable.org> schrieb:

> A trouble with that is some packages have hundreds of user-selectable
> options - or even thousands.  It is unfeasible to use --enable-foo
> options for all of those when configuring then.

Well, not that much ;-o
But taking care of such feature switches is the job of an automated
distro builder tool, including things like dependency tracking.
Actually, I'm really too lazy for doing those stuff by hand ;-P

But you're right, some packages have too many optional features, 
which better should be their own packages, and there's sometimes
much code out there which should be reused ...

> Some other packages _should_ have more options, but don't because it's
> too unwieldy to make them highly configurable with Autoconf.  

Adding new feature switches w/ autoconf is almost trivial
(well, not completely ;-o)

> Imho, Kconfig would be good for more programs than it's currently used for,
> and could be made to work with those --enable/--with options: you'd be
> able to configure them entirely on the command line, or interactively
> with "./configure --menu" (runs menuconfig), or with a config file.

Yes, that would be fine. But for me the primary constraint is that
all switches/options can be specified by command line - otherwise
I'd need extra complexity for each package in my distbuilder tool.

> Perhaps it might even be possible to write a very small, portable,
> specialised alternative to Make which is small enough to ship with
> packages that use it?

No, I really wouldn't advise this. Make tools are, IMHO, part of 
the toolchain (in a wider sense). Once point is avoiding code 
duplication, but the really important one is: a central point of
adaption/configuration. That's eg. why I like pkg-config so much:
if I need some tweaking, I just pass my own command (or a wrapper).
If each package does it's library lookup completely by itself, I
also need to touch each single package in case I need some tweaks.
I had exactly that trouble w/ lots of packages, before I ported
them to pkg-config.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Jamie Lokier <jamie@shareable.org> schrieb:

> Media players with lots of optional formats and drivers are another.
> (They also have considerable problems with their Autoconf in my
> experience).

You probably mean their hand-written ./configure script, which is
intentionally incompatible w/ autoconf ("this is not autoconf" 
as primary directive" ;-P) ... I guess we've got the same one
in mind ;-)

> Reality is that Kconfig front end to autotools does work - as you've
> proved.  It's a good idea. :-)

Now, we just need an autoconf-alike frontend for Kconfig ;-)

> Most packages need lots of additional libraries installed - and the
> development versions of those libraries, for that matter.  Too often
> the right development version - not too recent, not too old.  
> With the wrong versions, there are surprises.

But that's not the problem of autoconf or any other buildsystem,
just bad engineering (often on both sides).

> You said about too many user-selectable options.  Many large packages
> _check_ for many installed libraries.  Get them wrong, and you have
> the same problems of untested combinations.

It even gets worse when they silently enable certain features on
presence/absence of some lib. That's btw one of the reasons why
sysroot is an primary constraint for me, even when building for the
platform+arch.

> Have you felt uncomfortable shipping a package that does use Autoconf,
> Automake and Libtool, knowing that the scripts generated by those
> tools are huge compared with the entire source of your package?

Yeah, that's one of those things in autotools I never understood:
why isn't there just one function for each type of check/action, 
which is just called with the right params ?


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives

[Peter Korsgaard]

>>>>> "Enrico" == Enrico Weigelt <weigelt@metux.de> writes:

Hi,

 Enrico> * Bernd Petrovitsch <bernd@firmix.at> schrieb:
 >> > Recent pkg-config supports sysroot.
 >> 
 >> FC-6 has "only" 0.21.

Notice that sysroot is broken in 0.23 (it removes everything else than
-I / -L words). Patches have been posted to the pkgconfig list several
times.

 >> > So you simply build your .pc files as usual (w/o sysroot prefix) and
 >> > set the sysroot prefix via env on the pkg-config call.
 >> 
 >> From a quick glance over the man page of 0.23, yes.

 Enrico> Yep, it doesn't take $SYSROOT (as my original patch did), but with 
 Enrico> some prefix ...

PKG_CONFIG_SYSROOT_DIR

-- 
Bye, Peter Korsgaard

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Saturday 14 June 2008 01:19:32 you wrote:
...
> I still don't understand why all the scons, cmakes and jams out there
> don't even try to provide the *standard* user interface everyone is used
> to on a unix system.

For cmake:
CMake has a cache, where the values of variables are stored, e.g. if an option 
is enabled or not, or where a library has been found (e.g. 
JPEG_LIBRARY=/usr/local/lib/libjpeg.so).
The way to influence the behaviour of cmake is to change the value of these 
variables, this can be done either via a GUI (curses based or with cmake 2.6 
also a graphical one), or via the command line:
$ cmake -D<name_of_variable>=<new_value> ...more options

There is no simple automatic mapping from the variables to enabling features.

Anyway, it is on the TODO for cmake to add some kind of support for 
project-configurable command line options, so that configure-compatible 
options could be provided.

> > Lots of packages need special tools or other software installed to
> > build them; this is no different.
>
> autotools need only a shell and make

Yes, and requiring a shell is a lot, in the case you want to port also to 
Windows.


Alex

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Friday 13 June 2008 16:51:19 Enrico Weigelt wrote:
...
> That's why I prefer an purely descriptive paragidm (= subset of
> delcarative, but practically no logic): a buildfile should only
> describe the package's structure (eg.: "i have some executable foo
> which consists of source [...] and imports libs [...]), so the
> buildtool (and user's config) can cope with it all.

This was also the plan for cmake when it was started. But it turned out that 
this is not enough, and for more complex projects some programming logic is 
required. So cmake turned from quite declarative to quite imperative over 
time.
So while declarative sounds nice, in practice it just isn't powerful enough.

Alex

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Monday 16 June 2008 01:12:41 Enrico Weigelt wrote:
> * Robert Schwebel <r.schwebel@pengutronix.de> schrieb:
>
> Hi,
>
> > Instead of hacking around and inventing new things, you should have
> > spent your time for improving libtool ...
>
> No, not with libtool.
> I do not want to support that insane approach of tweaking command
> lines in the middle - it's an Pandorra's Box. I've already spent
> too much time on it and decided to completely drop it.
>
> Instead I prefer *clean* lines.
>
> Unitool provides commands on an higher level than, gcc+co do.
> These commands are on an higher functional level, hiding the
> individual platform's details. Also including things like importing
> .la-libs, sysroot, etc.
>
> Lt-unitool is a wrapper which parses the libtool commands and calls
> Unitool to do the actual work.

Seriously, why is a wrapper for the compiler/linker required AT ALL if the 
calls to these tools are made from _generated_ files ?
The generated files should just contain the appropriate calls for the 
respective commands.
This layer of abstraction is unnecessary and IMO just adds confusion. 
(modifying libtool so that it calls unitool even seems to add yet another 
layer which can potentially break or bitrot etc.)

Alex

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Friday 13 June 2008 20:51:57 you wrote:
...
> Been there, seen that. I maintain > 500 packets in PTXdist and guess
> which ones make 90% of the problems. Hint: they are not related to
> autotools ...

If there are specific problems common to many cmake based packages please 
report that to cmake@cmake.org. If there is something many people get wrong 
something should be done about this in cmake (and if it just means improving 
the documentation).

Alex

Re: cross-compiling alternatives

[Bernd Petrovitsch]

On Fre, 2008-06-13 at 20:51 +0200, Robert Schwebel wrote:
> On Fri, Jun 13, 2008 at 08:30:52AM +0200, Alexander Neundorf wrote:
> > Battle of Wesnoth is currently converted to both Scons and CMake, and
> > in the end they will decide about the winner. (since Eric is good at
> > arguing I guess it will be scons).
> 
> The thing is that 'configure && make && make install' plus the usuall
> --enable-foo / --disable-foo / --with-bla=blub semantics is simply *the*
> standard way of configuring stuff for unix systems. You don't need fancy
> tools, you get cross compiling almost for free and unix people simply
> know how to use it.

As long as people avoid AC_TRY_RUN() and similar and allow the
"configurator" to tell `configure.sh` facts for the unavoidable cases
about the target (and there were some apps - and I forgot the names -
out there where this wasn't easily possible with editing the generated
configure.sh. Yes, that's not fault of autotools as such but autotools
make it IMHO far too easy to write that sort without generating lots of
warnings[0]).

> All the cool kids out there who think they know everything better
> usually start with "I hate autotools", then invent something which

That has IMHO 2 main reasons:
- For lost of apps, a Makefile and some coding discipline is more than
  enough to support Linux/*BSD/MacOS-X even on different hardware.
  And there are always cases where you need OS-specific code anyways
  (e.g. manipulating routes).
  Yes, that may need much more coding discipline than the average
  programmer is used to.
- Converting $PROJECT to autotools is not easy. One has to learn and
  understand how the tools work[1] and what should be done in which way.
  And AFAIU (which is not much for autotools) one has to adapt the
  source anyways here and there (so it is not really a drop-in
  replacement).
  And if people consider using autotools, it is probably quite large and
  complex ....

Add that to negative experiences with other autotools packages and I can
understand the above sentence.

> solves 0.1% of the problems (including their very special problem) and
> tell the rest of the world that their shiny new tools is *soooo cooool*.

	Bernd

[0]: Yes, this is free software and I could send patches etc. to fix
     that. But that price is IMHO higher than just not using autotools
     for new stuff.
[1]: And I didn't find a site yet with an easy understandable
     description for seasoned programmers with cross-compile
     and multi-hardware experience.
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: cross-compiling alternatives

[Bernd Petrovitsch]

On Mon, 2008-06-16 at 10:02 +0200, Alexander Neundorf wrote:
[...]
> Seriously, why is a wrapper for the compiler/linker required AT ALL if the 
> calls to these tools are made from _generated_ files ?

AFAIU the motivation of libtool to provide OS-independent (and toolchain
independent?) means to compile and link (etc.).

> The generated files should just contain the appropriate calls for the 
> respective commands.

But these calls are - e.g. for shared libraries - not identical. The Gnu
toolchain is not everything.

> This layer of abstraction is unnecessary and IMO just adds confusion. 
> (modifying libtool so that it calls unitool even seems to add yet another 
> layer which can potentially break or bitrot etc.)

I know of more than one "occasion" where "gcc" et.al. actually where
wrappers around the actually binaries to check and guarantee commandline
parameters and options (some must be there, some mustn't be there, ...).
Usually sane Makefiles (and sane autotools usages) are enough for that.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: cross-compiling alternatives

[Bernd Petrovitsch]

On Sam, 2008-06-14 at 01:07 +0100, Jamie Lokier wrote:
[...]
> You said about too many user-selectable options.  Many large packages

These are IME not a problem if they have somewhat sensible defaults.

> _check_ for many installed libraries.  Get them wrong, and you have
> the same problems of untested combinations.

As long as I can specify that libfoo support must be compiled in (and
thus libfoo must be present) and the tools throw an error if it doesn't
find it, I have no problem.
Otherwise all package builders have a serious problem.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: cross-compiling alternatives

[Alexander Neundorf]

On Monday 16 June 2008 10:28:30 Bernd Petrovitsch wrote:
> On Mon, 2008-06-16 at 10:02 +0200, Alexander Neundorf wrote:
> [...]
>
> > Seriously, why is a wrapper for the compiler/linker required AT ALL if
> > the calls to these tools are made from _generated_ files ?
>
> AFAIU the motivation of libtool to provide OS-independent (and toolchain
> independent?) means to compile and link (etc.).
>
> > The generated files should just contain the appropriate calls for the
> > respective commands.
>
> But these calls are - e.g. for shared libraries - not identical. The Gnu
> toolchain is not everything.

Yes, that's what I mean. The commands for calling the tools are generated. 
They should be generated appropriately for the respective toolchains (i.e. 
the generator must know how to call the different toolchains). No 
libtool/unitool/other wrappers required.

Alex

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Jamie Lokier]

Enrico Weigelt wrote:
> Most times I've seen those checks, they silently enable some 
> features, eg. if it looks for certain kernel devices. Definitively
> the wrong way!

Agreed.  Though, you do often have to check for headers etc.,
otherwise you won't have the definitions needed to work with those
devices.  (Think: inotify).

> It really should be in users/packagers control to
> explicitly enable features.

But then there's a problem with 100 or more --enable-foo command line
options.  The user's script will _calls_ ./configure will
bitrot... there's no way you'll be typing it on the command line.

A big package can easily depend on that many local features.

> Nevertheless, the existence of some file or device says nothing
> about whether it will be usable (or *should* be used) at
> runtime. I've seen packages silently enabling some feature and then
> failing at runtime since the previously detected device is missing
> later. What a nightamare for packagers.

I agree, assume they're all there is not sane.

> Another point are broken syscalls. Well, you *have* check at runtime 
> to be sure, or perhaps choose to ignore it and expect a sane system.

That said, to use rsync as an example.  It uses the utimes() system
call when available, and utime() otherwise, because the former is
better.

On current GNU/Linux it will use utimes().  Recently someone tried to
run it on an old kernel, and it failed.  If it tried utimes() first
then fell back to utime(), it was fine, and rsync has a patch to do
that now.

But here's the thing: do you really want every package have code
calling every different variation on a system call, at run time, until
it finds one that works?

-- Jamie

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[David Woodhouse]

On Mon, 2008-06-16 at 11:49 +0100, Jamie Lokier wrote:
> But here's the thing: do you really want every package have code
> calling every different variation on a system call, at run time, until
> it finds one that works?

No. That functionality lives in libc, if you want it at all.

-- 
dwmw2

Re: cross-compiling alternatives

[Jamie Lokier]

Bernd Petrovitsch wrote:
> > _check_ for many installed libraries.  Get them wrong, and you have
> > the same problems of untested combinations.
> 
> As long as I can specify that libfoo support must be compiled in (and
> thus libfoo must be present) and the tools throw an error if it doesn't
> find it, I have no problem.
> Otherwise all package builders have a serious problem.

They do have problems, when you want to repeatably build and deploy,
if the build environment isn't very similar each time.

Typically the way you specify that libfoo support must be compiled in
is --with-libfoo=/path/to/libfoo.

That way lies bitrot between your build script which calls ./configure
(since you won't by typing it manually with 20+ options like that each
time you rebuild), and the changing version of an upstream package you
configure.

To prevent it trying to compile in libs you don't want, you also need
--without-libfoo.  Using Kerberos as an example, which I remember when
building CVS ages ago: If you don't _prevent_ it using libraries you
don't want, you get different binariesn depending on whether a
Kerberos library was installed on the build system at build time.  You
might then send a built program to another system, and find it won't
run at all, or has unwanted behaviour.

Do you really see package building scripts with 20 --with-libfoo= and
--without-libfoo= options in them for every library?  Sometimes.  But
more often, not: instead, they more often have build-time installed
prerequisites.

-- Jamie

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Jamie Lokier]

Enrico Weigelt wrote:
> > Reality is that Kconfig front end to autotools does work - as you've
> > proved.  It's a good idea. :-)
> 
> Now, we just need an autoconf-alike frontend for Kconfig ;-)

I agree and would support:

    ./configure --menu
         Invokes a configuration menu / wizard for user-selectable options.
         Only works if you have the "autoconf-menu" tools already installed.

    ./configure --ask
         Invokes command line configuration questions / wizard for
         user-selectable options.

    ./configure --ask --from-config=OLDCONFIG
         Uses existing settings from a previous configuration.

    ./configure
         Without --menu or --ask, invokes traditional Autoconf
         automatic detection of capabilities.

> > You said about too many user-selectable options.  Many large packages
> > _check_ for many installed libraries.  Get them wrong, and you have
> > the same problems of untested combinations.
> 
> It even gets worse when they silently enable certain features on
> presence/absence of some lib. That's btw one of the reasons why
> sysroot is an primary constraint for me, even when building for the
> platform+arch.

That's a basic design feature of Autoconf.  And it really is very good
most of the time - much better than config scripts which ask a lot of
questions (Perl).

> > Have you felt uncomfortable shipping a package that does use Autoconf,
> > Automake and Libtool, knowing that the scripts generated by those
> > tools are huge compared with the entire source of your package?
> 
> Yeah, that's one of those things in autotools I never understood:
> why isn't there just one function for each type of check/action, 
> which is just called with the right params ?

It's because it's written in very portable Bourne shell, which does
not support shell functions.

(This is why I toy with the idea of writing a very portable C program,
similar to a simplified shell, specially designed for Autoconf, with
functions...)

It could simulate functions portably by creating lots of little
scripts at the start, and then calling them later on.  That would
shrink ./configure a lot.  But it's a big change; somebody's got to do
it, test it, look after it, and so on.

Libtool similarly does not have to be huge, but it would be a lot of
work to shrink it without breaking anything.  It's hard enough to
change it at all without breaking anything on some system, somewhere.

-- Jamie

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Jamie Lokier]

Robert Schwebel wrote:
> We have once tried to write our automated test system for embedded
> boards with python and xml; the idea was to have something fancy, new
> and with good code-reuse. In the end it failed because the pexpect
> package we used to do the pattern matching bitrotted so quickly that
> four months later all the fancy tests didn't work any more, because it
> is not part of the python core.
> 
> In the meantime we have migrated our automatic testing stuff to use
> shell, ssh/rsh and kermit. It is rock solid, and the code reuse factor
> is at least as good as with anything else.

It sounds like the basic problem there was using non-core Python.  You
can count on shell, ssh/rsh, kernel - they're stable.  You can count
on Pythong's core language.  But not on pexpect: if you depended on
that, you should probably have copied the code into your own test
system, or written an equivalent.  Easier said than done, I know.

> > Have you felt uncomfortable shipping a package that does use Autoconf,
> > Automake and Libtool, knowing that the scripts generated by those
> > tools are huge compared with the entire source of your package?
> 
> No :-)

It's really disturbed me, sometimes, shipping a package of 50kB source
and 2MB configure+libtool.

> > Have you _written_ Autoconf tests recently?
> 
> Yea, all our packages are autotoolized.
> 
> > Made any shell / shellutils non-portability mistakes in the tests?
> 
> Yea, it happens in ptxdist all the time. People report about problems,
> we add new tests and the next revision works even on Ubuntu :)

Right, so you have no idea about portability of your tests on, say, SCO
Unixserver or IRIX? :-)

> I have no problems with people writing fancy new things. It's just that
> most people who try to do something better than autotools have only a
> fraction of the features.
> 
> Open Source is darwinism: if there is something better, let's use it.
> But compare apples with apples.
> 
> > If you're going to rewrite Autotools using GNU Make, why not ask if
> > another tool would be better, perhaps a tool specially designed for
> > the job?
> 
> Go ahead.

The first part of going ahead is to brainstorm and find ideas and
likely interest...  It's too big (as you rightly note) to simply sit
down and do it by oneself in isolation.

-- Jamie

Re: cross-compiling alternatives

[Bernd Petrovitsch]

On Mon, 2008-06-16 at 12:17 +0100, Jamie Lokier wrote:
> Bernd Petrovitsch wrote:
> > > _check_ for many installed libraries.  Get them wrong, and you have
> > > the same problems of untested combinations.
> > 
> > As long as I can specify that libfoo support must be compiled in (and
> > thus libfoo must be present) and the tools throw an error if it doesn't
> > find it, I have no problem.
> > Otherwise all package builders have a serious problem.
> 
> They do have problems, when you want to repeatably build and deploy,
> if the build environment isn't very similar each time.

Sometimes you have a different build environments - if only that you
want to rebuild e.g. your .src.rpm on several versions of CentOS and
Fedora.

> Typically the way you specify that libfoo support must be compiled in
> is --with-libfoo=/path/to/libfoo.
> 
> That way lies bitrot between your build script which calls ./configure

Cannot be really avoided IMHO. 

> (since you won't by typing it manually with 20+ options like that each
> time you rebuild), and the changing version of an upstream package you
> configure.

So be it. At least one sees errors/bugs immediately.

> To prevent it trying to compile in libs you don't want, you also need
> --without-libfoo.  Using Kerberos as an example, which I remember when
> building CVS ages ago: If you don't _prevent_ it using libraries you
> don't want, you get different binariesn depending on whether a
> Kerberos library was installed on the build system at build time.  You
> might then send a built program to another system, and find it won't
> run at all, or has unwanted behaviour.
> 
> Do you really see package building scripts with 20 --with-libfoo= and
> --without-libfoo= options in them for every library?  Sometimes.  But

For (in an ideal world 100%) repeatable builds - for a .rpm. .deb, some
cross-compiled embedded device - one usually ends up with that explicit
list (and IMHO it's the smaller PITA than to cope with strange bug
reports because something was broken in the build weeks ago).
Mainly because the dependency information is also present elsewehre
(e.g. in the package). Or you really want control over the installed
software.

> more often, not: instead, they more often have build-time installed
> prerequisites.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Jamie Lokier]

Enrico Weigelt wrote:
> > Imho, Kconfig would be good for more programs than it's currently used for,
> > and could be made to work with those --enable/--with options: you'd be
> > able to configure them entirely on the command line, or interactively
> > with "./configure --menu" (runs menuconfig), or with a config file.
> 
> Yes, that would be fine. But for me the primary constraint is that
> all switches/options can be specified by command line - otherwise
> I'd need extra complexity for each package in my distbuilder tool.

Yes, I agree with that.  It's one of the nice things about Kconfig
that you can use it with pre-specific config files, even when the
config is from a different version and doesn't specify everything.
Doing so as ./configure --enable/--disable arguments seems like a
comfortable fit.

> 
> > Perhaps it might even be possible to write a very small, portable,
> > specialised alternative to Make which is small enough to ship with
> > packages that use it?
> 
> No, I really wouldn't advise this. Make tools are, IMHO, part of 
> the toolchain (in a wider sense). Once point is avoiding code 
> duplication, but the really important one is: a central point of
> adaption/configuration. That's eg. why I like pkg-config so much:
> if I need some tweaking, I just pass my own command (or a wrapper).
> If each package does it's library lookup completely by itself, I
> also need to touch each single package in case I need some tweaks.
> I had exactly that trouble w/ lots of packages, before I ported
> them to pkg-config.

That's interesting, thanks.  I have seen problems with pkg-config, but
adding your own wrapper is a nice way to fix any of them :-)

That said, if you can specify library lookup on the ./configure
command line, that fixes a lot of problems of repeatability too.

Only problem then is you don't always know _which_ options to pass on
the ./configure command line, if you have a package build script, and
a different version of the package.  There's a tendancy to drift apart
which manifests as silent mistakes, until some user reports a bug.

-- Jamie

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Jamie Lokier]

David Woodhouse wrote:
> On Mon, 2008-06-16 at 11:49 +0100, Jamie Lokier wrote:
> > But here's the thing: do you really want every package have code
> > calling every different variation on a system call, at run time, until
> > it finds one that works?
> 
> No. That functionality lives in libc, if you want it at all.

Sometimes that's ok.

But sometimes libc can't do that, because the point of a newer
function is it addresses some inadequacy of an old function.  If libc
does a "compatibility" thing and uses old syscalls on old kernels,
sometimes that introduces bugs.

E.g. Calls to pread should _not_ be implemented as lseek+read+lseek on
old kernels which don't have pread.  That leads to race conditions and
corruption in some applications.  (I think this has really occurred,
but I'm unable to find it now).

The same applications are fine if pread returns ENOSYS and they know
what they need to do with lseek and read.

The same is true of the rsync example, with utimes and utime.  It's
wrong for libc to "emulate" utimes using utime on old kernels: they
aren't the same.  So the application does it instead: the application
has a policy that it's fine with the different functionality when
utimes returns ENOSYS.

-- Jamie

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[David Woodhouse]

On Mon, 2008-06-16 at 12:52 +0100, Jamie Lokier wrote:
> E.g. Calls to pread should _not_ be implemented as lseek+read+lseek on
> old kernels which don't have pread.  That leads to race conditions and
> corruption in some applications.  (I think this has really occurred,
> but I'm unable to find it now).

Likewise pselect(). You're right -- it can't always be emulated.

-- 
dwmw2

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Monday 16 June 2008 13:39:46 you wrote:
...
> > > If you're going to rewrite Autotools using GNU Make, why not ask if
> > > another tool would be better, perhaps a tool specially designed for
> > > the job?
> >
> > Go ahead.
>
> The first part of going ahead is to brainstorm and find ideas and
> likely interest...  It's too big (as you rightly note) to simply sit
> down and do it by oneself in isolation.

You are not seriously talking about writing yet another buildsystem ?
With autotools, cmake, scons, ant, (b)jam, bitbake, makeng, etc. there should 
be enough available.

Alex

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Jamie Lokier]

Alexander Neundorf wrote:
> On Monday 16 June 2008 13:39:46 you wrote:
> ...
> > > > If you're going to rewrite Autotools using GNU Make, why not ask if
> > > > another tool would be better, perhaps a tool specially designed for
> > > > the job?
> > >
> > > Go ahead.
> >
> > The first part of going ahead is to brainstorm and find ideas and
> > likely interest...  It's too big (as you rightly note) to simply sit
> > down and do it by oneself in isolation.
> 
> You are not seriously talking about writing yet another buildsystem?
> With autotools, cmake, scons, ant, (b)jam, bitbake, makeng,
> etc. there should be enough available.

No, I'm talking about improving Autotools to handle some things better
than they do now.  Passing the high hurdles required to become part of
Autotools - especially compatibility - is part of the goal.

-- Jamie

Re: cross-compiling alternatives

[Enrico Weigelt]

* Peter Korsgaard <jacmet@sunsite.dk> schrieb:

Hi,

> Notice that sysroot is broken in 0.23 (it removes everything else than
> -I / -L words). Patches have been posted to the pkgconfig list several
> times.

ugh, why didn't they just apply my patch which only appends 
$sysroot to the pathes ?  .... grmpf ...

well, i'll stick w/ my 5-lines wrapper script ;-o

>  Enrico> Yep, it doesn't take $SYSROOT (as my original patch did), but with 
>  Enrico> some prefix ...
> 
> PKG_CONFIG_SYSROOT_DIR

ack.

cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:

> CMake has a cache, where the values of variables are stored, e.g. if an option 
> is enabled or not, or where a library has been found (e.g. 
> JPEG_LIBRARY=/usr/local/lib/libjpeg.so).
> The way to influence the behaviour of cmake is to change the value of these 
> variables, this can be done either via a GUI (curses based or with cmake 2.6 
> also a graphical one), or via the command line:
> $ cmake -D<name_of_variable>=<new_value> ...more options

Are these variables strictly specified or is all left to individual 
author's decision ?

> > autotools need only a shell and make
> 
> Yes, and requiring a shell is a lot, in the case you want to port 
> also to Windows.

hmm, why not just expecting an sane shell on the building system ?
(as you already have to expect a sane compiler)


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:

> > That's why I prefer an purely descriptive paragidm (= subset of
> > delcarative, but practically no logic): a buildfile should only
> > describe the package's structure (eg.: "i have some executable foo
> > which consists of source [...] and imports libs [...]), so the
> > buildtool (and user's config) can cope with it all.
> 
> This was also the plan for cmake when it was started. But it turned 
> out that this is not enough, and for more complex projects some 
> programming logic is required. So cmake turned from quite declarative 
> to quite imperative over time.

And so you just open the Pandorra's Box again ;-o
As soon as you try to build some one-fits-all-solution, you'll
sooner or later run into similar problems as autoconf has.

I won't to that w/ my TreeBuild. It is intentionally limited on
easily structured packages. People should either structure their
packages properly use something else ;-P


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bernhard Fischer]

On Mon, Jun 16, 2008 at 02:32:01PM +0100, Jamie Lokier wrote:

>No, I'm talking about improving Autotools to handle some things better
>than they do now.  Passing the high hurdles required to become part of
>Autotools - especially compatibility - is part of the goal.

If you look at the sh scripts generated by autotools one is tempted to
just ship a small, clean sh, written in C89 with autotools, use that
and drop the workarounds.

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Bernhard Fischer]

On Mon, Jun 16, 2008 at 12:52:18PM +0100, Jamie Lokier wrote:

>The same applications are fine if pread returns ENOSYS and they know
>what they need to do with lseek and read.

Note that they (pread, etc.) would better not exist at all instead of
being stubbed out (this requirement was dropped, AFAICS). If there is no
prototype then you don't need AC_TRY_RUN(other_arch_pread_test) in the
first place. Of course this assumes that you don't want to make that
decision at runtime but compile time.
>
>The same is true of the rsync example, with utimes and utime.  It's
>wrong for libc to "emulate" utimes using utime on old kernels: they
>aren't the same.  So the application does it instead: the application
>has a policy that it's fine with the different functionality when
>utimes returns ENOSYS.

You usually do the opposite, emulate utimes with utime ;)
link_warning(utimes, "the use of LEGACY `utimes' is discouraged, use `utime'")

Re: cross-compiling alternatives

[Bernhard Fischer]

On Mon, Jun 16, 2008 at 04:33:15PM +0200, Enrico Weigelt wrote:
>* Peter Korsgaard <jacmet@sunsite.dk> schrieb:
>
>Hi,
>
>> Notice that sysroot is broken in 0.23 (it removes everything else than
>> -I / -L words). Patches have been posted to the pkgconfig list several
>> times.

heh, so it keeps e.g. -I=/usr/include ? Fun ;)

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Adrian Bunk]

On Mon, Jun 16, 2008 at 06:00:37PM +0200, Enrico Weigelt wrote:
> * Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:
> 
> > > That's why I prefer an purely descriptive paragidm (= subset of
> > > delcarative, but practically no logic): a buildfile should only
> > > describe the package's structure (eg.: "i have some executable foo
> > > which consists of source [...] and imports libs [...]), so the
> > > buildtool (and user's config) can cope with it all.
> > 
> > This was also the plan for cmake when it was started. But it turned 
> > out that this is not enough, and for more complex projects some 
> > programming logic is required. So cmake turned from quite declarative 
> > to quite imperative over time.
> 
> And so you just open the Pandorra's Box again ;-o
> As soon as you try to build some one-fits-all-solution, you'll
> sooner or later run into similar problems as autoconf has.
> 
> I won't to that w/ my TreeBuild. It is intentionally limited on
> easily structured packages. People should either structure their
> packages properly use something else ;-P

For simple packages autoconf+automake+libtool is already near at your
descriptive paragidm.

And despite all nasty details of autoconf/automake/libtool they also 
have advantages:
- they are quite powerful when you know how to handle them
- they allow to build your software on non-Linux systems
- they are the de-facto standard in the open source world, and everyone
  building open source software knows them

And the last point is a very important one:

For me as someone who is often compiling software your plan of creating 
yet another build tool I have to handle does not sound like a good idea.

> cu

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Jamie Lokier]

Bernhard Fischer wrote:
> On Mon, Jun 16, 2008 at 02:32:01PM +0100, Jamie Lokier wrote:
> 
> >No, I'm talking about improving Autotools to handle some things better
> >than they do now.  Passing the high hurdles required to become part of
> >Autotools - especially compatibility - is part of the goal.
> 
> If you look at the sh scripts generated by autotools one is tempted to
> just ship a small, clean sh, written in C89 with autotools, use that
> and drop the workarounds.

Exactly!

Use an early, small Autoconf script written in very portable shell (as
now) to detect the small set of features needed for this "clean sh",
and a host compiler.  Then build it, then continue in clean sh.

But if you're doing that, why use sh?  It could be a subset of sh with
extensions especially for this job - or something else entirely - to
make Autoconf scripts cleaner, smaller, faster than they are now.  Or
even a tokenised sh, so Autoconf tests run directly on it but are
compact.  Possibly even _more_ portable (no need to depend on quirks
of some utilities; configure scripts more checkable for common
errors).  And easier to configure things on environments with no shell
(i.e. Windows).

It adds an extra requirement for a host C compiler, but that's
probably reasonable these days.

This conversation ought to be heading over to the autotools
lists... if anyone wants to take it up there.

-- Jamie

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Adrian Bunk]

On Mon, Jun 16, 2008 at 11:28:36PM +0100, Jamie Lokier wrote:
> Bernhard Fischer wrote:
> > On Mon, Jun 16, 2008 at 02:32:01PM +0100, Jamie Lokier wrote:
> > 
> > >No, I'm talking about improving Autotools to handle some things better
> > >than they do now.  Passing the high hurdles required to become part of
> > >Autotools - especially compatibility - is part of the goal.
> > 
> > If you look at the sh scripts generated by autotools one is tempted to
> > just ship a small, clean sh, written in C89 with autotools, use that
> > and drop the workarounds.
> 
> Exactly!
> 
> Use an early, small Autoconf script written in very portable shell (as
> now) to detect the small set of features needed for this "clean sh",
> and a host compiler.  Then build it, then continue in clean sh.
> 
> But if you're doing that, why use sh?  It could be a subset of sh with
> extensions especially for this job - or something else entirely - to
> make Autoconf scripts cleaner, smaller, faster than they are now.  Or
> even a tokenised sh, so Autoconf tests run directly on it but are
> compact.  Possibly even _more_ portable (no need to depend on quirks
> of some utilities; configure scripts more checkable for common
> errors).  And easier to configure things on environments with no shell
> (i.e. Windows).
> 
> It adds an extra requirement for a host C compiler, but that's
> probably reasonable these days.
> 
> This conversation ought to be heading over to the autotools
> lists... if anyone wants to take it up there.

All public ramblings about how autoconf might be improved don't bring 
anything if you do not implement your ideas yourself and show that your 
implementation is better than the status quo.

After all, open source development is not driven by people producing hot 
air but by people writing code...

> -- Jamie

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Monday 16 June 2008 17:15:37 Enrico Weigelt wrote:
> * Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:
> > CMake has a cache, where the values of variables are stored, e.g. if an
> > option is enabled or not, or where a library has been found (e.g.
> > JPEG_LIBRARY=/usr/local/lib/libjpeg.so).
> > The way to influence the behaviour of cmake is to change the value of
> > these variables, this can be done either via a GUI (curses based or with
> > cmake 2.6 also a graphical one), or via the command line:
> > $ cmake -D<name_of_variable>=<new_value> ...more options
>
> Are these variables strictly specified or is all left to individual
> author's decision ?

Authors decision.

> > > autotools need only a shell and make
> >
> > Yes, and requiring a shell is a lot, in the case you want to port
> > also to Windows.
>
> hmm, why not just expecting an sane shell on the building system ?
> (as you already have to expect a sane compiler)

Well, we could go so far to expect a "sane" operating system, but you can't 
change it, there are "insane" people in the world ;-)

Alex

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:
> On Monday 16 June 2008 17:15:37 Enrico Weigelt wrote:
> > * Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:
> > > CMake has a cache, where the values of variables are stored, e.g. if an
> > > option is enabled or not, or where a library has been found (e.g.
> > > JPEG_LIBRARY=/usr/local/lib/libjpeg.so).
> > > The way to influence the behaviour of cmake is to change the value of
> > > these variables, this can be done either via a GUI (curses based or with
> > > cmake 2.6 also a graphical one), or via the command line:
> > > $ cmake -D<name_of_variable>=<new_value> ...more options
> >
> > Are these variables strictly specified or is all left to individual
> > author's decision ?
> 
> Authors decision.

Then you've got the same problem as w/ autoconf's config.status:
You have to tweak it for each individual package separately :(

My destiny is to have strictly standardized variables for all the
common things, so you can use an global per-target configuration 
for *all* packages ever coming.
That's what Unitool's system properties db is for.

> > hmm, why not just expecting an sane shell on the building system ?
> > (as you already have to expect a sane compiler)
> 
> Well, we could go so far to expect a "sane" operating system, but you can't 
> change it, there are "insane" people in the world ;-)

Many, many things can be done within in the toolchain, eg. fixes 
for broken libc's. For example, I've seen packages adding several 
missing functions for certain platforms - this should be the job
of the toolchain.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Enrico Weigelt]

* Adrian Bunk <bunk@kernel.org> schrieb:

Hi,

> > I won't to that w/ my TreeBuild. It is intentionally limited on
> > easily structured packages. People should either structure their
> > packages properly use something else ;-P
> 
> For simple packages autoconf+automake+libtool is already near at your
> descriptive paragidm.

Yes, but mine is much simpler and syntactically stable. With autotools
there are uncountable things you can do wrong, without even noticing - 
it needs *very* much care. In TreeBuild there's almost no room for
those mistakes. A simple typo can lead autoconf to unpredictable 
behaviour - TreeBuild spits out an parse error.

So for these cases (which already make up a very large number of
total packages in the world), maintaining buildfiles becomes a lot
easier ;-P

> And despite all nasty details of autoconf/automake/libtool they also 
> have advantages:
> - they are quite powerful when you know how to handle them

The point is: you *must* know autotools very well, otherwise you
soon run into pitfalls.

> - they allow to build your software on non-Linux systems

TreeBuild does that too, as soon as someone wrote a proper config 
for another platform. The design is platform agnostic, just lacking
ports to other platforms, due lack of (my) time.

> - they are the de-facto standard in the open source world, and everyone
>   building open source software knows them

I don't think that this is a good argument. M$-Word is also an
"de facto" standard ;-P
 
> And the last point is a very important one:
> 
> For me as someone who is often compiling software your plan of creating 
> yet another build tool I have to handle does not sound like a good idea.

Yes, but that's not a problem of TreeBuild, but the total count of 
different build systems around the world. That's why a new buildsystem
should be an really good solution for an reasonable class of problems
and should be easy to learn. IMHO, TreeBuild does this - for simple
structed packages. (which make up about 99% of my own ones).

Of course, TreeBuild still in an early development stage. So I 
don't claim it's really usable for everyone - for now it's more
an academic issue.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Enrico Weigelt]

* Rob Landley <rob@landley.net> schrieb:

Hi,

> You can't get away from cross compiling whenever you want to bootstrap a new 
> platform.  But cross compiling can be minimized and encapsulated.  It can be 
> a stage you pass through to get it over with and no longer have to deal with 
> it on the other side, which is the approach I take.

That's not enought for me. I want more encapsulation (yes, but buildfarm 
is also running in a VZ, but still sysroot'ed). For example, I *want* certain
tests (like running target code) to fail, since I *do not* trust them.
(btw: I also let compiler warnings fail, just to be sure).

I, personally, prefer high build-time constraints, because runtime tests
are quite limited.

> By my count sysroot is the fifth layer of path logic the gcc guys have added 
> in an attempt to paint over the dry rot.

I see this as a quite good separation between running and target system,
just one more fence for clear borders. As long as you don't pass absolute
search pathes to it, you can be sure that it doesn't mix up target and host.

Actually, I never want to crosscompile w/o it. 

> Personally I use a derivative of the old uClibc wrapper script that rewrites 
> the command line to start with "--nostdinc --nostdlib" and then builds it 
> back up again without having any paths in there it shouldn't.

Might work, but then you're limited to some specific cases. 
If you *only* intent bootstrapping of an minimal system, fine, but for
my projects too complex to handle.

> > #2: fix broken configure.in's (and feed back to upstream or OSS-QM)
> 
> Whack-a-mole.  Fun for the whole family.  Only problem is, it never stops.

Most times, it as only to be done one per package. And it's an clean solution.

> > #3: replace libtool by unitool
> 
> Uninstall libtool and don't replace it with anything, it's a NOP on Linux.

The problem is: many packages are entirely built upon this. So you'll have
to de-libtoolize them. Very much work. As I already had Unitool, I preferred
investing just a few hours for creating an generic drop-in replacement for 
libtool instead of touching each single package.

> > Only crap sw looks at /proc at build time.
> > Yes, there's *much* crap sw out there :(
> 
> 99% of all the developers out there don't really care about portability, and 
> never will.  Even if you eliminate the windows guys and the people who don't 
> do C, 90% of the people who are _left_ get to work on the PC first, get it to 
> work natively on other Linux platforms afterwards.

True :(
Some packages out there even don't have an clean native build path, eg. 
requiring parts of the package already built and installed (-> firebird-db)
 
> Cross compiling is a step beyond "portability".  They'll _never_ care about 
> cross compiling.  If they get inspired to make it work on MacOS X, then 
> you'll have to extract the source and _build_ it on MacOS X to make that 
> work.  And 99% of all developers will nod their heads and go "quite right, as 
> it should be".
> 
> This isn't going to change any time soon.

Actually, I don't care much about that. I concentrate on getting those 
packages I need cleaned-up and crosscompile'able - even if this means 
going alone and maintaining an own branch.

If I sum up all the invested working hours of all the last years
on that and substract the total saved time from other projects 
where I'm reusing this work, I get a positive balance ;-P


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)

[Alexander Neundorf]

On Tuesday 17 June 2008 15:46:36 Enrico Weigelt wrote:
> * Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:
> > On Monday 16 June 2008 17:15:37 Enrico Weigelt wrote:
> > > * Alexander Neundorf <neundorf@eit.uni-kl.de> schrieb:
> > > > CMake has a cache, where the values of variables are stored, e.g. if
> > > > an option is enabled or not, or where a library has been found (e.g.
> > > > JPEG_LIBRARY=/usr/local/lib/libjpeg.so).
> > > > The way to influence the behaviour of cmake is to change the value of
> > > > these variables, this can be done either via a GUI (curses based or
> > > > with cmake 2.6 also a graphical one), or via the command line:
> > > > $ cmake -D<name_of_variable>=<new_value> ...more options
> > >
> > > Are these variables strictly specified or is all left to individual
> > > author's decision ?
> >
> > Authors decision.
>
> Then you've got the same problem as w/ autoconf's config.status:
> You have to tweak it for each individual package separately :(

Well, these are all more or less scripting languages, so people can create 
whichever variables they like, no way to enforce something.
Still one can suggest standards which should be followed.

Alex

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Denys Vlasenko]

On Wednesday 30 April 2008 21:11, David Woodhouse wrote:
> On Wed, 2008-04-30 at 20:22 +0200, Andi Kleen wrote:
> > David Woodhouse <dwmw2@infradead.org> writes:
> > 
> > > Andrew Morton has been saying recently that we need an 'embedded
> > > maintainer', to take responsibility for 'embedded issues' in the core
> > > kernel, as well as trying to improve our relationship with those using
> > > the Linux kernel for 'embedded' devices -- who have a reputation of
> > > not working with us very closely; to their detriment as well as our
> > > own.
> > 
> > I hope your job description doesn't include adding more and more
> > CONFIGs though.
> > 
> > I am sure there are lots of low hanging fruit where memory can be
> > saved and it's a good thing someone cares about that, but please don't
> > focus on the code size only. Or if you work on that don't do it 
> > using CONFIG or when you really add a new one find some other 
> > that is pointless and remove it first. 
> > 
> > There are simply already far too many of them and they make the 
> > kernel harder and harder to change.
> 
> I agree. And if we do want to pay attention to pure code size, there are
> other approaches -- like --gc-sections

I have some patches in this area too. Were submitted to Sam
but he was too busy it seems.
--
vda

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Denys Vlasenko]

On Thursday 01 May 2008 12:41, Andi Kleen wrote:
> > To a large extent, I agree. I certainly don't want to focus solely on
> > code size; there's a lot more to embedded Linux than that. But it _is_
> 
> Not only code size, far more important is dynamic memory consumption.
> [admittedly we right now lack a good instrumentation framework for this]
> 
> > There are some cases where we really _do_ want to have CONFIG options,
> > but I agree that we should keep them to a minimum. And when we _do_ have
> > CONFIG options, they don't have to litter the actual code with ifdefs.
> 
> The problem I see is more that really nobody can even compile not 
> alone test all these combinations anymore. Hidding the problem in inlines
> does not solve that. And no randconfig is not the solution either.

Because we allowed kernel to be developed without the requirement that
random config should be buildable for release kernels.

Had it been a requirement, keeping it in shape wouldn't be
too difficult.

Sure enough, _now_ fixing kernel to pass such a test on i386
would take several weeks of work at least. But it is doable.

I would even volunteer to do it if there are some
reasonable chances resulting patches would be viewed
as worthwhile for inclusion. I am somewhat tired
of killing weeks of my time only to find that my work
is deemed "not important enough for inclusion".
--
vda

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Adrian Bunk]

On Mon, Jun 23, 2008 at 07:28:09PM +0200, Denys Vlasenko wrote:
> On Thursday 01 May 2008 12:41, Andi Kleen wrote:
> > > To a large extent, I agree. I certainly don't want to focus solely on
> > > code size; there's a lot more to embedded Linux than that. But it _is_
> > 
> > Not only code size, far more important is dynamic memory consumption.
> > [admittedly we right now lack a good instrumentation framework for this]
> > 
> > > There are some cases where we really _do_ want to have CONFIG options,
> > > but I agree that we should keep them to a minimum. And when we _do_ have
> > > CONFIG options, they don't have to litter the actual code with ifdefs.
> > 
> > The problem I see is more that really nobody can even compile not 
> > alone test all these combinations anymore. Hidding the problem in inlines
> > does not solve that. And no randconfig is not the solution either.
> 
> Because we allowed kernel to be developed without the requirement that
> random config should be buildable for release kernels.
> 
> Had it been a requirement, keeping it in shape wouldn't be
> too difficult.
> 
> Sure enough, _now_ fixing kernel to pass such a test on i386
> would take several weeks of work at least. But it is doable.
>...

On i386 it might even already work today.

But guess how much time it costs to get at least all defconfigs 
compiling on the other 22 architectures.

Even getting allmodconfig/allyesconfig compiling isn't trivial for all 
architectures, and random configurations are _far_ from compiling.

And we are not talking about something to be done once, as soon as you 
leave x86 there are tons of regular breakages.

Plus the fact that you often get into situations where more options
mean complex and fragile stuff. Read the Kconfig files under 
drivers/media/ and check in git all commits to them since 2.6.25 alone, 
and you'll understand why "add an option for every bit" can result in
very high ongoing maintainance work required.

Not everything that is technically possible is also maintainable, and 
maintainability is a very important point in a project with several 
million lines changing each year.

> vda

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Denys Vlasenko]

On Monday 23 June 2008 19:45, Adrian Bunk wrote:
> Plus the fact that you often get into situations where more options
> mean complex and fragile stuff. Read the Kconfig files under 
> drivers/media/ and check in git all commits to them since 2.6.25 alone, 
> and you'll understand why "add an option for every bit" can result in
> very high ongoing maintainance work required.
> 
> Not everything that is technically possible is also maintainable, and 
> maintainability is a very important point in a project with several 
> million lines changing each year.

Well, I am not (and was not) disputing this. I agree with it.
CONFIGs should not be multiplying like rabbits.
--
vda

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Sam Ravnborg]

On Mon, Jun 23, 2008 at 07:22:10PM +0200, Denys Vlasenko wrote:
> On Wednesday 30 April 2008 21:11, David Woodhouse wrote:
> > On Wed, 2008-04-30 at 20:22 +0200, Andi Kleen wrote:
> > > David Woodhouse <dwmw2@infradead.org> writes:
> > > 
> > > > Andrew Morton has been saying recently that we need an 'embedded
> > > > maintainer', to take responsibility for 'embedded issues' in the core
> > > > kernel, as well as trying to improve our relationship with those using
> > > > the Linux kernel for 'embedded' devices -- who have a reputation of
> > > > not working with us very closely; to their detriment as well as our
> > > > own.
> > > 
> > > I hope your job description doesn't include adding more and more
> > > CONFIGs though.
> > > 
> > > I am sure there are lots of low hanging fruit where memory can be
> > > saved and it's a good thing someone cares about that, but please don't
> > > focus on the code size only. Or if you work on that don't do it 
> > > using CONFIG or when you really add a new one find some other 
> > > that is pointless and remove it first. 
> > > 
> > > There are simply already far too many of them and they make the 
> > > kernel harder and harder to change.
> > 
> > I agree. And if we do want to pay attention to pure code size, there are
> > other approaches -- like --gc-sections
> 
> I have some patches in this area too. Were submitted to Sam
> but he was too busy it seems.
They were not trivial to apply and so went down on the TODO list.
We could try to push the generic and x86 specific .lds stuff via
the arch maintainers?

	Sam

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Tim Bird]

Adrian Bunk wrote:
> On Mon, Jun 23, 2008 at 07:28:09PM +0200, Denys Vlasenko wrote:
>> Had it been a requirement, keeping it in shape wouldn't be
>> too difficult.
>>
>> Sure enough, _now_ fixing kernel to pass such a test on i386
>> would take several weeks of work at least. But it is doable.
>> ...
> 
> On i386 it might even already work today.
> 
> But guess how much time it costs to get at least all defconfigs 
> compiling on the other 22 architectures.
> 
> Even getting allmodconfig/allyesconfig compiling isn't trivial for all 
> architectures, and random configurations are _far_ from compiling.
> 
> And we are not talking about something to be done once, as soon as you 
> leave x86 there are tons of regular breakages.
> 
> Plus the fact that you often get into situations where more options
> mean complex and fragile stuff. Read the Kconfig files under 
> drivers/media/ and check in git all commits to them since 2.6.25 alone, 
> and you'll understand why "add an option for every bit" can result in
> very high ongoing maintainance work required.
> 
> Not everything that is technically possible is also maintainable, and 
> maintainability is a very important point in a project with several 
> million lines changing each year.

OK sure.  Nobody's going to disagree with that.  I would, however,
disagree with a characterization of Linux-tiny as "adding an option
for every bit".  Linux-tiny has been around about 5 years now, and
if you added the whole thing right now you'd add about 30 config
options.

If you're worried about this multiplying out of control, let me
just say that having to curtail the rate of patch submission by
embedded developers has not been our biggest problem.  :-)
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Denys Vlasenko]

On Monday 23 June 2008 20:57, Sam Ravnborg wrote:
> > > I agree. And if we do want to pay attention to pure code size, there are
> > > other approaches -- like --gc-sections
> > 
> > I have some patches in this area too. Were submitted to Sam
> > but he was too busy it seems.
>
> They were not trivial to apply and so went down on the TODO list.

I realize that they were not trivial to review, but that
was unavoidable. They were even more not trivial to create.

> We could try to push the generic and x86 specific .lds stuff via
> the arch maintainers?

IIRC I splitted the entire GC collection patch in a way
that first patches were doing exactly this easier first part
and I hoped that at least these first patches
will be taken. They were big, but somewhat trivial,
from "it's obviously safe" department.

Had they been applied, now making --gc-sections to work
would be easier.
--
vda

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Sam Ravnborg]

On Mon, Jun 23, 2008 at 09:12:30PM +0200, Denys Vlasenko wrote:
> On Monday 23 June 2008 20:57, Sam Ravnborg wrote:
> > > > I agree. And if we do want to pay attention to pure code size, there are
> > > > other approaches -- like --gc-sections
> > > 
> > > I have some patches in this area too. Were submitted to Sam
> > > but he was too busy it seems.
> >
> > They were not trivial to apply and so went down on the TODO list.
> 
> I realize that they were not trivial to review, but that
> was unavoidable. They were even more not trivial to create.
> 
> > We could try to push the generic and x86 specific .lds stuff via
> > the arch maintainers?
> 
> IIRC I splitted the entire GC collection patch in a way
> that first patches were doing exactly this easier first part
> and I hoped that at least these first patches
> will be taken. They were big, but somewhat trivial,
> from "it's obviously safe" department.

I do not recall anything wrong with the patch-set.

> 
> Had they been applied, now making --gc-sections to work
> would be easier.
Agreed. I should have asked you to push this via arch maintainers back then.

	Sam

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[James Chapman]

Adrian Bunk wrote:
> On Mon, Jun 23, 2008 at 07:28:09PM +0200, Denys Vlasenko wrote:
>> On Thursday 01 May 2008 12:41, Andi Kleen wrote:
>>>> To a large extent, I agree. I certainly don't want to focus solely on
>>>> code size; there's a lot more to embedded Linux than that. But it _is_
>>> Not only code size, far more important is dynamic memory consumption.
>>> [admittedly we right now lack a good instrumentation framework for this]
>>>
>>>> There are some cases where we really _do_ want to have CONFIG options,
>>>> but I agree that we should keep them to a minimum. And when we _do_ have
>>>> CONFIG options, they don't have to litter the actual code with ifdefs.
>>> The problem I see is more that really nobody can even compile not 
>>> alone test all these combinations anymore. Hidding the problem in inlines
>>> does not solve that. And no randconfig is not the solution either.
>> Because we allowed kernel to be developed without the requirement that
>> random config should be buildable for release kernels.
>>
>> Had it been a requirement, keeping it in shape wouldn't be
>> too difficult.
>>
>> Sure enough, _now_ fixing kernel to pass such a test on i386
>> would take several weeks of work at least. But it is doable.
>> ...
> 
> On i386 it might even already work today.
> 
> But guess how much time it costs to get at least all defconfigs 
> compiling on the other 22 architectures.
> 
> Even getting allmodconfig/allyesconfig compiling isn't trivial for all 
> architectures, and random configurations are _far_ from compiling.
 >
 > And we are not talking about something to be done once, as soon as you
 > leave x86 there are tons of regular breakages.

Could automated builds and build error reporting be used to help resolve 
these problems?

The good people at Simtec have an automated build report available as an 
e-mail digest. I use it to watch for architecture build breakages in 
subsystems or drivers that I use or touch. It covers defconfigs of ARM 
and MIPS architectures and reports compile errors/warnings, module size, 
kernel size etc. If this approach were extended/distributed to cover 
more architectures and random config builds, developers could with 
little effort spot problems and fix them. Hell, it might also encourage 
new developers to get involved and contribute.

Here's a link to a recent report for ARM, fyi:-

http://lists.simtec.co.uk/pipermail/kautobuild/2008-June/001299.html

> Plus the fact that you often get into situations where more options
> mean complex and fragile stuff. Read the Kconfig files under 
> drivers/media/ and check in git all commits to them since 2.6.25 alone, 
> and you'll understand why "add an option for every bit" can result in
> very high ongoing maintainance work required.
> 
> Not everything that is technically possible is also maintainable, and 
> maintainability is a very important point in a project with several 
> million lines changing each year.
> 
>> vda
> 
> cu
> Adrian

-- 
James Chapman
Katalix Systems Ltd
http://www.katalix.com
Catalysts for your Embedded Linux software development

Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list

[Adrian Bunk]

On Wed, Jun 25, 2008 at 10:50:43AM +0100, James Chapman wrote:
> Adrian Bunk wrote:
>> On Mon, Jun 23, 2008 at 07:28:09PM +0200, Denys Vlasenko wrote:
>>> On Thursday 01 May 2008 12:41, Andi Kleen wrote:
>>>>> To a large extent, I agree. I certainly don't want to focus solely on
>>>>> code size; there's a lot more to embedded Linux than that. But it _is_
>>>> Not only code size, far more important is dynamic memory consumption.
>>>> [admittedly we right now lack a good instrumentation framework for this]
>>>>
>>>>> There are some cases where we really _do_ want to have CONFIG options,
>>>>> but I agree that we should keep them to a minimum. And when we _do_ have
>>>>> CONFIG options, they don't have to litter the actual code with ifdefs.
>>>> The problem I see is more that really nobody can even compile not  
>>>> alone test all these combinations anymore. Hidding the problem in 
>>>> inlines
>>>> does not solve that. And no randconfig is not the solution either.
>>> Because we allowed kernel to be developed without the requirement that
>>> random config should be buildable for release kernels.
>>>
>>> Had it been a requirement, keeping it in shape wouldn't be
>>> too difficult.
>>>
>>> Sure enough, _now_ fixing kernel to pass such a test on i386
>>> would take several weeks of work at least. But it is doable.
>>> ...
>>
>> On i386 it might even already work today.
>>
>> But guess how much time it costs to get at least all defconfigs  
>> compiling on the other 22 architectures.
>>
>> Even getting allmodconfig/allyesconfig compiling isn't trivial for all  
>> architectures, and random configurations are _far_ from compiling.
> >
> > And we are not talking about something to be done once, as soon as you
> > leave x86 there are tons of regular breakages.
>
> Could automated builds and build error reporting be used to help resolve  
> these problems?
> 
> The good people at Simtec have an automated build report available as an  
> e-mail digest. I use it to watch for architecture build breakages in  
> subsystems or drivers that I use or touch. It covers defconfigs of ARM  
> and MIPS architectures and reports compile errors/warnings, module size,  
> kernel size etc. If this approach were extended/distributed to cover  
> more architectures

Jan Dittmer has a great page showing the build status and kernel size of 
the defconfigs of all architectures that is running since 2004 or 2005:
  http://l4x.org/k/

> and random config builds, developers could with  
> little effort spot problems and fix them. Hell, it might also encourage  
> new developers to get involved and contribute.

Perhaps in an ideal world...

In reality, I'd claim I'm one out of only two people who regularly fix 
architecture-specific build problems for all architectures.

> James Chapman

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed