* Re: Not as much ccache win as I expected
From: Jörn Engel @ 2008-06-15 17:58 UTC (permalink / raw)
To: Tim Bird; +Cc: Oleg Verych, linux-embedded, linux-kbuild
In-Reply-To: <4852E245.4020502@am.sony.com>
On Fri, 13 June 2008 14:10:29 -0700, Tim Bird wrote:
>
> Maybe I should just be grateful for any ccache hits I get.
ccache's usefulness depends on your workload. If you make a change to
include/linux/fs.h, close to 100% of the kernel is rebuilt, with or
without ccache. But when you revert that change, the build time differs
dramatically. Without ccache, fs.h was simply changed again and
everything is rebuild. With ccache, there are hits for the old version
and all is pulled from the cache - provided you have allotted enough
disk for it.
If you never revert to an old version or do some equivalent operation,
ccache can even be a net loss. On a fast machine, the additional disk
accesses are easily more expensive than the minimal cpu gains.
Jörn
--
Public Domain - Free as in Beer
General Public - Free as in Speech
BSD License - Free as in Enterprise
Shared Source - Free as in "Work will make you..."
--
To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Kernel boot problem on IXP422 Rev. A
From: Rob Landley @ 2008-06-15 19:17 UTC (permalink / raw)
To: Tim Bird; +Cc: Glenn Henshaw, linux-embedded
In-Reply-To: <4852D322.5060400@am.sony.com>
On Friday 13 June 2008 15:05:54 Tim Bird wrote:
> Rob,
>
> This is an excellent and concise description of the open
> source perspective on the problem. I'll add just one note below.
>
> Rob Landley wrote:
> > 1) Try to reproduce the bug under a current kernel. (Set up a _test_
> > system.)
>
> This sounds easy, but can be quite difficult.
It's not a question of difficult or easy: it's the procedure that works.
You don't get support from a commercial vendor unless you pay them money, and
you don't get support from open source developers unless you help us make the
next release just a little bit better. (We never said our help was free, we
just said it didn't cost _money_. Ok, the FSF did but they don't speak for
all of us...)
> Very often, product developers are several versions behind, with
> no easy way to use the current kernel version.
I'm aware of that. But if you can't set up a test system to reproduce the bug
on a current system, the rest of us haven't got a _chance_.
> For example, a
> common scenario is starting with a kernel that comes with a board
> (with source mind you), where the kernel came from the semi-conductor
> vendor, who paid a Linux vendor to do a port, and it was
> released in a time-frame relative to the Linux vendor's
> product schedule.
Then poke your vendor to fix the problem.
If you've decided to use a divergent fork from a vendor rather than the
mainstream version, then the vendor has to support that fork for you because
we're not going to be familiar with it. (You can _hire_ one of us to support
it for you, but we're not going to do so on a volunteer basis.)
We're happy to debug _our_ code. But "our code" is the current vanilla
release tarball. If you can't reproduce the problem in the current vanilla
tarball, then it's not our bug. If you can only reproduce it in an older
version: congratulations, we must have fixed it since. If you can only
reproduce it in some other fork, obviously their changes introduced the bug.
If it's "your code plus this patch", we need to see the patch.
If _you_ can't reproduce it in our code, how do you expect _us_ to?
> This is how you end up having people STARTING projects today
> using a 2.6.11 kernel. (I know of many).
Oldest I've seen a new project launch with this year is 2.6.15, but I agree
with your point.
Whoever decided backporting bug fixes to a 2.6.16 kernel forever was a good
idea seems to have muddied the waters a bit. Ironically I don't know anybody
actually _using_ that version, but I've seen several people point to it to
show that "the community" supports arbitrarily older versions forever, and
thus they don't have to upgrade to get support, and 2.6.18 is actually
_newer_ than that...
> The real difficulty, when a developer finds themselves in
> this position, is how to forward-port the BSP code necessary to
> reproduce the bug in the current kernel. Often, the code
> is not isolated well enough (this is a vendor problem that
> really needs attention. If you have the BSP in patches, it
> is usually not too bad to forward port even across several
> kernel versions. But many vendors don't ship stuff this way.)
Yup. Sucks, doesn't it? This is not a problem that improves with the passage
of time.
Might be a good idea to make it clear up front that even if your changes never
get mainlined, failure to break up and break out your patches is still likely
to cause maintenance problems down the road.
> The fact is, that by a series of small steps and delays by
> the linux vendor, chip vendor, board vendor,
> and product developer the code is out-of step.
Hence the importance of breaking out and breaking up the changes.
> It's easy to say "don't get in this position", but
> this even happens when everyone is playing nice and actively
> trying to mainline stuff. BSP support in arch trees often
> lag mainline by a version or two.
Getting out of sync is inevitable. Happens to full-time kernel developers,
that's why they have their own trees. That's a separate issue from asking
for patches and getting a source tarball that compiles instead. "Here's a
haystack, find the needle."
Mainlining changes and breaking them up into clean patches on top of some
vanilla version (_any_ vanilla version) are two separate things. You have to
win one battle before you can even start the other.
> The number of parties involved here is why, IMHO, it has
> taken so long to make improvements in this area.
The lack of a clear consistent message from us to the vendors hasn't helped.
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
^ permalink raw reply
* Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list
From: Rob Landley @ 2008-06-15 21:43 UTC (permalink / raw)
To: Leon Woestenberg
Cc: Greg Ungerer, Sam Ravnborg, David Woodhouse, linux-embedded
In-Reply-To: <c384c5ea0806150839l4b18b988pf69990bc87e52bcf@mail.gmail.com>
On Sunday 15 June 2008 10:39:43 Leon Woestenberg wrote:
> Hello all,
>
> On Thu, Jun 12, 2008 at 2:41 AM, Rob Landley <rob@landley.net> wrote:
> > Most packages don't cross compile at all. Debian has somewhere north of
> > 30,000 packages. Every project that does large scale cross compiling
> > (buildroot, gentoo embedded, timesys making fedora cross compile, etc)
> > tends to have about 200 packages that cross compile more or less easily,
> > another 400 or so that can be made to cross compile with _lot_ of effort
> > and a large enough rock, and then the project stalls at about that size.
>
> Agreed, OpenEmbedded has a few thousands, but your point is valid.
> However, fleeing to target-native compilation is not the way to
> improve the situation IMHO.
You say it like fleeing is a bad thing. :)
I believe building natively under emulation is the Right Thing. Cross
compiling has always historically been a transitional step until native
compiling became available on the target.
When Ken Thompson and Dennis Ritchie were originally creating Unix for the
PDP-7, they cross compiled their code from a honking big GE mainframe because
that was their only option. One of the first things they wrote was a PDP-7
assembler that ran on the PDP-7. The reason they created the B programming
language in the first place was to have a tiny compiler that could run
natively on the PDP-7, and when they moved up to a PDP-11 Dennis had more
space to work with and expanded B into C.
When they severed the mainframe umbilical cord as soon as they were able to
get the system self-hosting, it wasn't because the PDP-7 had suddenly become
faster than the GE mainframe.
Compiling natively where possible has been the normal way to build Unix
software ever since. Linux became a real project when Linus stopped needing
Minix to cross-compile it. Linus didn't "flee" Minix, he assures us he
erased his minix partition purely by accident. :)
> Moore's law on hardware also goes for the host,
Which is why people no longer regularly write application software in assembly
language, because we don't need to do that anymore. The result would be
faster, but not better.
The rise of scripting languages like Python and javascript that run the source
code directly is also related (and if you don't think people don't write
complete applications in those you haven't seen any of the google apps). The
big push for Java in 1998 could happen because the hardware was now fast
enough to run _everything_ under an emulator for a processor that didn't
actually exist (until Rockwell built one, anyway).
Build environments are now literally thousands of times faster than when I
started programming. The first machine I compiled code on was a commodore 64
(1mhz, 8 bits, the compiler was called "blitz" and the best accelerator for
it was a book). The slowest machine I ever ran Linux on was a 16 mhz 386sx.
According to my blog, I moved from a 166mhz laptop to a 266mhz one on April
13, 2002. I started building entire Linux From Scratch systems on the 166mhz
machine, including a ton of optional packages (apache, postgresql, openssh,
samba, plus it was based on glibc and coreutils and stuff back then so the
build was _slow_), hence the necessity of scripting it and leaving the build
to its own devices for a few hours.
Even without distcc calling out to the cross compiler, the emulated system
running on my laptop is several times faster than the build environment I had
7 years ago (2001), somewhat faster than the one I had 5 years ago (2003),
and somewhat slower than the one I had 3 years ago (2005). (That's emulating
an x86 build environment on my x86_64 laptop. I didn't _have_ a non-x86
build enviornment 5 years ago for comparison purposes.)
> I think the progress is even bigger on big iron.
Not that I've noticed, unless by "big iron", you mean "PC clusters". (You can
expand laterally if you've got the money for it and your problem distributes
well...)
> Also, how much of the 30000 packages are useful for something like
> your own firmware Linux?
None of them, because Firmware Linux has a strictly limited agenda: provide a
native build environment on every system emulation supported by qemu. That's
the 1.0 release criteria. (Some day I may add other emulators like hercules
for s390, but the principle's the same.)
Once you have the native build environment, you can bootstrap Gentoo, or
Debian, or Linux From Scratch, or whatever you like. I've got instructions
for some of 'em.
The buildroot project fell into the trap of becoming a distro and having to
care about the interaction between hundreds of packages. I'm not interested
in repeating that mistake.
Figuring out what packages will other people might need is something I stopped
trying to predict a long time ago. If it exists, somebody wanted it. People
want/need the weirdest stuff: the accelerometer in laptops is used for
rolling marble games, and the iPhone is a cell phone with 3D support for
(among other things) video playback.
> > Distcc can take advantage of smp, but that won't help the ./configure
> > stage and I need to do some work on distcc to teach it to understand more
> > gcc
>
> If you want to build 1000+ packages, you don't need to run configure
> itself multithreaded. There are enough jobs available to keep 16/32
> processors busy (beyond that, you probably end up in
> inter-package-dependencies stalling the build). This is just a guess
> from what I see during a multi-threaded bake and multi-threaded make
> on OpenEmbedded.
If your package management system can work out the dependencies and launch the
package builds in parallel, have at. Another reason I'm not trying to
address package management: lots of people have already done it, and put a
lot more work into it than I ever would. I'm just trying to help them do
their thing while staying out of their way.
> > However, having one or more full-time engineers devoted to debugging
> > cross-compile issues is quite a high price to pay too. Moore's law
> > really doesn't help that one.
>
> How about 30+ volunteers.
I presume you're referring to OpenEmbedded, rather than Gentoo Embedded,
Debian Embedded, buildroot, openmoko, or any of the others?
Making cross compiling work is tied to your choice of package management
systems. Making native compling work, less so. The point of my project is
to _unbundle_ stuff. Separate out that layer, and make it as thin as
possible.
> > I'm not saying either solution is perfect, I'm just saying the "build
> > under emulation" approach is a viable alternative that gets more
> > attractive as time passes, both because of ongoing development on
> > emulators and because of Moore's law on the hardware.
>
> I cannot follow your reasoning - Moore's law will help you more on the
> big iron side of things.
Actually it helps exactly as much, just on about a 5 year delay.
Moore's Law does not reduce the need for your above mentioned 30+ volunteers
to endlessly debug cross compiling bugs.
My point is that building under emulation didn't used to be even _feasible_.
Now it's an option. Building under an emulator in 2008 gives us about the
speed of building natively around 2003, and that seemed tolerable at the
time. Both the emulators and the hardware should improve going forward.
> That said, I welcome any effort (such as yours) to help improve the
> embedded Linux domain, I rather try to fix the cross-compile stuff of
> the few thousand packages I am interested in.
I've gotten involved in too many big complicated projects that got sidelined
by somebody figuring out how to substitute a simpler problem and solve that
one instead. (Remember how much work maintaining kernel headers was
before "make headers_install"?)
These days when I see big problem that can most obviously be solved by lots of
people doing lots of work for a long time, my first instinct is try to figure
out how to avoid needing to do that.
> Yes it hurts my brain.
It hurts everybody's brain.
There are two ways to build natively: you can do so under an emulator, or you
can do so on real hardware.
The Fedora for Arm project builds natively, and started by using real ARM
hardware to do it. (They've added qemu since; I note that the guy who runs
that project used to be my boss back before we all left TimeSys.)
http://fedoraproject.org/wiki/Architectures/ARM#Technical_Approach
The Ubuntu for Arm project did the same, bought high end arm machines and made
a compile farm out of them:
http://mojo.handhelds.org/files/HandheldsMojo_ELC2008.pdf
If you're going to throw money at the problem, you can get a lot of
specialized native hardware to build on, and that's an option a number of
large projects have been taking.
The advantage of using emulators is that it's cheaper and more flexible.
Anybody can do it, it's just a build environment you can take with you on
your laptop (no contention for a limited number of devices and your
developers aren't tethered to them to get work/testing done). PC hardware is
the cheapest and most flexible way to buy raw computing power (including
memory/disk/network), and pretty much everybody already has it or they
wouldn't be doing Linux development in the first place.
You don't snag hobbyists by saying "buy this hardware to play". You snag
hobbyists by giving them something they can download and poke at with the
lowest threshold possible, which is why I provide precompiled images ala
http://landley.net/code/firmware/downloads/system-image
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
^ permalink raw reply
* Re: [PATCH 0/1] Embedded Maintainer(s), linux-embedded@vger list
From: Rob Landley @ 2008-06-15 21:48 UTC (permalink / raw)
To: weigelt; +Cc: Linux Embedded Maillist
In-Reply-To: <20080612181806.GA7423@nibiru.local>
On Thursday 12 June 2008 13:18:07 Enrico Weigelt wrote:
> * Rob Landley <rob@landley.net> schrieb:
>
> Hi,
>
> > There's also qemu. You can native build under emulation.
>
> did you ever consider that crosscompiling is not only good for
> some other arch, but a few more things ?
Sure, such as building a uClibc system on a glibc host, which my _previous_
firmware linux project (http://landley.net/code/firmware/old) was aimed at.
That used User Mode Linux instead of qemu, because "fakeroot" wasn't good
enough and chroot A) requires the build to run as root, B) sometimes gets a
little segfaulty if you build uClibc with newer kernel headers than the
kernel in the system you're running on.
You can't get away from cross compiling whenever you want to bootstrap a new
platform. But cross compiling can be minimized and encapsulated. It can be
a stage you pass through to get it over with and no longer have to deal with
it on the other side, which is the approach I take.
> > In addition, if you have a cross compiler but don't want to spend all
> > your time lying to ./configure, preventing gcc from linking against the
> > host's zlib or grabbing stuff out of /usr/include that your target hasn't
> > got, or
>
> #1: use a proper (sysroot'ed) toolchain
I break everything. (I've broken native toolchains. I just break them
_less_.)
By my count sysroot is the fifth layer of path logic the gcc guys have added
in an attempt to paint over the dry rot.
Personally I use a derivative of the old uClibc wrapper script that rewrites
the command line to start with "--nostdinc --nostdlib" and then builds it
back up again without having any paths in there it shouldn't.
> #2: fix broken configure.in's (and feed back to upstream or OSS-QM)
Whack-a-mole. Fun for the whole family. Only problem is, it never stops.
> #3: replace libtool by unitool
Uninstall libtool and don't replace it with anything, it's a NOP on Linux.
> > libraries are linked inside the emulator, anything that wants to look
> > at /proc or sysinfo does it natively inside the emulator...)
>
> Only crap sw looks at /proc at build time.
> Yes, there's *much* crap sw out there :(
99% of all the developers out there don't really care about portability, and
never will. Even if you eliminate the windows guys and the people who don't
do C, 90% of the people who are _left_ get to work on the PC first, get it to
work natively on other Linux platforms afterwards.
Cross compiling is a step beyond "portability". They'll _never_ care about
cross compiling. If they get inspired to make it work on MacOS X, then
you'll have to extract the source and _build_ it on MacOS X to make that
work. And 99% of all developers will nod their heads and go "quite right, as
it should be".
This isn't going to change any time soon.
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Rob Landley @ 2008-06-15 21:51 UTC (permalink / raw)
To: weigelt; +Cc: Linux Embedded Maillist
In-Reply-To: <20080612183421.GD7423@nibiru.local>
On Thursday 12 June 2008 13:34:21 Enrico Weigelt wrote:
> * Bill Gatliff <bgat@billgatliff.com> schrieb:
> > If the build system derives from autoconf, then a hacked-up
> > config.cache (or equivalent command-line args) often solves
> > problems for me.
>
> Only if you're working on *one specific* target for a long time.
> I, for example, have to support lots of different targets, so your
> approach does not work for me. Ah, and it's not *solving* any problem,
> just deferring it to some other day.
It's not deferring it, it's ripping out the failed automation and configuring
it manually, answering each question by hand. (That said, it's the approach
I took to get bash to cross-compile in FWL. Sometimes it's all you can
do...)
Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Enrico Weigelt @ 2008-06-15 23:12 UTC (permalink / raw)
To: linux-embedded
In-Reply-To: <20080613184555.GR13599@pengutronix.de>
* Robert Schwebel <r.schwebel@pengutronix.de> schrieb:
Hi,
> Instead of hacking around and inventing new things, you should have
> spent your time for improving libtool ...
No, not with libtool.
I do not want to support that insane approach of tweaking command
lines in the middle - it's an Pandorra's Box. I've already spent
too much time on it and decided to completely drop it.
Instead I prefer *clean* lines.
Unitool provides commands on an higher level than, gcc+co do.
These commands are on an higher functional level, hiding the
individual platform's details. Also including things like importing
.la-libs, sysroot, etc.
Lt-unitool is a wrapper which parses the libtool commands and calls
Unitool to do the actual work.
cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT service - http://www.metux.de/
---------------------------------------------------------------------
Please visit the OpenSource QM Taskforce:
http://wiki.metux.de/public/OpenSource_QM_Taskforce
Patches / Fixes for a lot dozens of packages in dozens of versions:
http://patches.metux.de/
---------------------------------------------------------------------
^ permalink raw reply
* Re: cross-compiling alternatives
From: Enrico Weigelt @ 2008-06-16 4:08 UTC (permalink / raw)
To: Linux Embedded Maillist
In-Reply-To: <1213382709.17853.79.camel@tara.firmix.at>
* Bernd Petrovitsch <bernd@firmix.at> schrieb:
> > Recent pkg-config supports sysroot.
>
> FC-6 has "only" 0.21.
Not sure when the sysroot stuff got into upstream releaes, but
maybe you just should update ;-P
> > So you simply build your .pc files as usual (w/o sysroot prefix) and
> > set the sysroot prefix via env on the pkg-config call.
>
> From a quick glance over the man page of 0.23, yes.
Yep, it doesn't take $SYSROOT (as my original patch did), but with
some prefix ...
> The problem is that the build-time (cross-)linker needs to find the
> (cross-compiled) lib under /my/build/host/target/environment/opt/foo/lib
> at link time and the shared linker under /opt/foo/lib at run-time.
> Hmm, after digging into that old project, it seems that libtool and
> the .la files were the problem.
Yes, libtool doesn't understand anything like sysroot. It generates
broken pathnames in the .la files. You could use unitool and it's
libtool-replacement.
> > > Yes. And even worse the compiled lib "foo" had explicit dependencies (on
> > > lib "bar") on
> > > "/my/build/host/target/environment/opt/bar/lib/libbar.so.1.2.3.4".
> >
> > And that's even more broken.
>
> Yup. Maybe it was a result of my trial to make libtool work somehow ..
Heh, I've given up trying to repair libtool for a long time ;-P
> > > And BTW pkg-config didn't support the concept of a "DESTDIR" variable
> > > (and I don't care about the name of that variable).
> >
> > No, why should it ?! It does not install anything.
>
> But it may "use" installed files.
Yes, but they have to be strictly from sysroot. As said, recent
pkg-config can handle this properly.
> > Probably you're looking for sysroot ?
>
> Yes, very probably.
:)
cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT service - http://www.metux.de/
---------------------------------------------------------------------
Please visit the OpenSource QM Taskforce:
http://wiki.metux.de/public/OpenSource_QM_Taskforce
Patches / Fixes for a lot dozens of packages in dozens of versions:
http://patches.metux.de/
---------------------------------------------------------------------
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Enrico Weigelt @ 2008-06-16 4:23 UTC (permalink / raw)
To: linux-embedded
In-Reply-To: <20080614003102.GB30652@shareable.org>
* Jamie Lokier <jamie@shareable.org> schrieb:
> > > E.g. in python there are tests which call functions and check
> > > their result to see if we are currently on a platform where
> > > that function is broken (I think there was such a test for
> > > poll() and some other functions).
> >
> > IMHO, that's broken sw engineering from ground up.
>
> Oh? The alternative I see is to do the test at run-time. But that
> adds to executable size and run-time slowdown on most platforms.
There's no generic answer to this, we have to look at the details
carefully ;-P
Most times I've seen those checks, they silently enable some
features, eg. if it looks for certain kernel devices. Definitively
the wrong way! It really should be in users/packagers control to
explicitly enable features. Nevertheless, the existence of some
file or device says nothing about whether it will be usable
(or *should* be used) at runtime. I've seen packages silently
enabling some feature and then failing at runtime since the
previously detected device is missing later. What a nightamare
for packagers.
Another point are broken syscalls. Well, you *have* check at runtime
to be sure, or perhaps choose to ignore it and expect a sane system.
*If* you really want to do constraint checks, you should do it
as an separate, optional step. Maybe issue a big-fat hint that
one really should run that test (and follow certain instructions)
if he don't exactly know what he's doing.
> Doing it at build time is an improvement, for those people who don't
> care about cross-compilation. (Not me, you understand.)
IMHO, it's just lazyness, at least for about 99% of the cases ;-P
cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT service - http://www.metux.de/
---------------------------------------------------------------------
Please visit the OpenSource QM Taskforce:
http://wiki.metux.de/public/OpenSource_QM_Taskforce
Patches / Fixes for a lot dozens of packages in dozens of versions:
http://patches.metux.de/
---------------------------------------------------------------------
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Enrico Weigelt @ 2008-06-16 4:31 UTC (permalink / raw)
To: linux-embedded
In-Reply-To: <20080613185157.GT13599@pengutronix.de>
* Robert Schwebel <r.schwebel@pengutronix.de> schrieb:
> On Fri, Jun 13, 2008 at 08:30:52AM +0200, Alexander Neundorf wrote:
> > Battle of Wesnoth is currently converted to both Scons and CMake, and
> > in the end they will decide about the winner. (since Eric is good at
> > arguing I guess it will be scons).
>
> The thing is that 'configure && make && make install' plus the usuall
> --enable-foo / --disable-foo / --with-bla=blub semantics is simply *the*
> standard way of configuring stuff for unix systems. You don't need fancy
> tools, you get cross compiling almost for free and unix people simply
> know how to use it.
ACK. The ./configure script's syntax is quite convenient and semi-standard.
So, IMHO, all packages which have such an configure stage should support
that syntax, even those who code those scripts manually.
Strange, but some projects insist in having their config scripts incompatible
with autoconf's, just to show that it wasn't produced by it ;-o
> Been there, seen that. I maintain > 500 packets in PTXdist and guess
> which ones make 90% of the problems. Hint: they are not related to
> autotools ...
Well, if you count in the improber uses of autoconf (AC_TRY_RUN, etc),
then: yes.
cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT service - http://www.metux.de/
---------------------------------------------------------------------
Please visit the OpenSource QM Taskforce:
http://wiki.metux.de/public/OpenSource_QM_Taskforce
Patches / Fixes for a lot dozens of packages in dozens of versions:
http://patches.metux.de/
---------------------------------------------------------------------
^ permalink raw reply
* Re: Firmware Linux (was Re: Cross Compiler and loads of issues)
From: Enrico Weigelt @ 2008-06-16 4:38 UTC (permalink / raw)
To: linux-embedded
In-Reply-To: <200806131224.58568.rob@landley.net>
* Rob Landley <rob@landley.net> schrieb:
> Did you try my FWL project? :)
>
> http://landley.net/code/firmware
hmm, doesnt look like supporting sysroot ...
cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT service - http://www.metux.de/
---------------------------------------------------------------------
Please visit the OpenSource QM Taskforce:
http://wiki.metux.de/public/OpenSource_QM_Taskforce
Patches / Fixes for a lot dozens of packages in dozens of versions:
http://patches.metux.de/
---------------------------------------------------------------------
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Enrico Weigelt @ 2008-06-16 4:57 UTC (permalink / raw)
To: linux-embedded
In-Reply-To: <20080613222523.GA28805@shareable.org>
* Jamie Lokier <jamie@shareable.org> schrieb:
> A trouble with that is some packages have hundreds of user-selectable
> options - or even thousands. It is unfeasible to use --enable-foo
> options for all of those when configuring then.
Well, not that much ;-o
But taking care of such feature switches is the job of an automated
distro builder tool, including things like dependency tracking.
Actually, I'm really too lazy for doing those stuff by hand ;-P
But you're right, some packages have too many optional features,
which better should be their own packages, and there's sometimes
much code out there which should be reused ...
> Some other packages _should_ have more options, but don't because it's
> too unwieldy to make them highly configurable with Autoconf.
Adding new feature switches w/ autoconf is almost trivial
(well, not completely ;-o)
> Imho, Kconfig would be good for more programs than it's currently used for,
> and could be made to work with those --enable/--with options: you'd be
> able to configure them entirely on the command line, or interactively
> with "./configure --menu" (runs menuconfig), or with a config file.
Yes, that would be fine. But for me the primary constraint is that
all switches/options can be specified by command line - otherwise
I'd need extra complexity for each package in my distbuilder tool.
> Perhaps it might even be possible to write a very small, portable,
> specialised alternative to Make which is small enough to ship with
> packages that use it?
No, I really wouldn't advise this. Make tools are, IMHO, part of
the toolchain (in a wider sense). Once point is avoiding code
duplication, but the really important one is: a central point of
adaption/configuration. That's eg. why I like pkg-config so much:
if I need some tweaking, I just pass my own command (or a wrapper).
If each package does it's library lookup completely by itself, I
also need to touch each single package in case I need some tweaks.
I had exactly that trouble w/ lots of packages, before I ported
them to pkg-config.
cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT service - http://www.metux.de/
---------------------------------------------------------------------
Please visit the OpenSource QM Taskforce:
http://wiki.metux.de/public/OpenSource_QM_Taskforce
Patches / Fixes for a lot dozens of packages in dozens of versions:
http://patches.metux.de/
---------------------------------------------------------------------
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Enrico Weigelt @ 2008-06-16 5:11 UTC (permalink / raw)
To: linux-embedded
In-Reply-To: <20080614000749.GA30652@shareable.org>
* Jamie Lokier <jamie@shareable.org> schrieb:
> Media players with lots of optional formats and drivers are another.
> (They also have considerable problems with their Autoconf in my
> experience).
You probably mean their hand-written ./configure script, which is
intentionally incompatible w/ autoconf ("this is not autoconf"
as primary directive" ;-P) ... I guess we've got the same one
in mind ;-)
> Reality is that Kconfig front end to autotools does work - as you've
> proved. It's a good idea. :-)
Now, we just need an autoconf-alike frontend for Kconfig ;-)
> Most packages need lots of additional libraries installed - and the
> development versions of those libraries, for that matter. Too often
> the right development version - not too recent, not too old.
> With the wrong versions, there are surprises.
But that's not the problem of autoconf or any other buildsystem,
just bad engineering (often on both sides).
> You said about too many user-selectable options. Many large packages
> _check_ for many installed libraries. Get them wrong, and you have
> the same problems of untested combinations.
It even gets worse when they silently enable certain features on
presence/absence of some lib. That's btw one of the reasons why
sysroot is an primary constraint for me, even when building for the
platform+arch.
> Have you felt uncomfortable shipping a package that does use Autoconf,
> Automake and Libtool, knowing that the scripts generated by those
> tools are huge compared with the entire source of your package?
Yeah, that's one of those things in autotools I never understood:
why isn't there just one function for each type of check/action,
which is just called with the right params ?
cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT service - http://www.metux.de/
---------------------------------------------------------------------
Please visit the OpenSource QM Taskforce:
http://wiki.metux.de/public/OpenSource_QM_Taskforce
Patches / Fixes for a lot dozens of packages in dozens of versions:
http://patches.metux.de/
---------------------------------------------------------------------
^ permalink raw reply
* Re: cross-compiling alternatives
From: Peter Korsgaard @ 2008-06-16 7:31 UTC (permalink / raw)
To: weigelt; +Cc: Linux Embedded Maillist
In-Reply-To: <20080616040854.GG12962@nibiru.local>
>>>>> "Enrico" == Enrico Weigelt <weigelt@metux.de> writes:
Hi,
Enrico> * Bernd Petrovitsch <bernd@firmix.at> schrieb:
>> > Recent pkg-config supports sysroot.
>>
>> FC-6 has "only" 0.21.
Notice that sysroot is broken in 0.23 (it removes everything else than
-I / -L words). Patches have been posted to the pkgconfig list several
times.
>> > So you simply build your .pc files as usual (w/o sysroot prefix) and
>> > set the sysroot prefix via env on the pkg-config call.
>>
>> From a quick glance over the man page of 0.23, yes.
Enrico> Yep, it doesn't take $SYSROOT (as my original patch did), but with
Enrico> some prefix ...
PKG_CONFIG_SYSROOT_DIR
--
Bye, Peter Korsgaard
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Alexander Neundorf @ 2008-06-16 7:55 UTC (permalink / raw)
To: Linux Embedded Maillist
In-Reply-To: <20080613231932.GW13599@pengutronix.de>
On Saturday 14 June 2008 01:19:32 you wrote:
...
> I still don't understand why all the scons, cmakes and jams out there
> don't even try to provide the *standard* user interface everyone is used
> to on a unix system.
For cmake:
CMake has a cache, where the values of variables are stored, e.g. if an option
is enabled or not, or where a library has been found (e.g.
JPEG_LIBRARY=/usr/local/lib/libjpeg.so).
The way to influence the behaviour of cmake is to change the value of these
variables, this can be done either via a GUI (curses based or with cmake 2.6
also a graphical one), or via the command line:
$ cmake -D<name_of_variable>=<new_value> ...more options
There is no simple automatic mapping from the variables to enabling features.
Anyway, it is on the TODO for cmake to add some kind of support for
project-configurable command line options, so that configure-compatible
options could be provided.
> > Lots of packages need special tools or other software installed to
> > build them; this is no different.
>
> autotools need only a shell and make
Yes, and requiring a shell is a lot, in the case you want to port also to
Windows.
Alex
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Alexander Neundorf @ 2008-06-16 7:58 UTC (permalink / raw)
To: Linux Embedded Maillist
In-Reply-To: <20080613145119.GF11760@nibiru.local>
On Friday 13 June 2008 16:51:19 Enrico Weigelt wrote:
...
> That's why I prefer an purely descriptive paragidm (= subset of
> delcarative, but practically no logic): a buildfile should only
> describe the package's structure (eg.: "i have some executable foo
> which consists of source [...] and imports libs [...]), so the
> buildtool (and user's config) can cope with it all.
This was also the plan for cmake when it was started. But it turned out that
this is not enough, and for more complex projects some programming logic is
required. So cmake turned from quite declarative to quite imperative over
time.
So while declarative sounds nice, in practice it just isn't powerful enough.
Alex
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Alexander Neundorf @ 2008-06-16 8:02 UTC (permalink / raw)
To: Linux Embedded Maillist
In-Reply-To: <20080615231241.GA8899@nibiru.local>
On Monday 16 June 2008 01:12:41 Enrico Weigelt wrote:
> * Robert Schwebel <r.schwebel@pengutronix.de> schrieb:
>
> Hi,
>
> > Instead of hacking around and inventing new things, you should have
> > spent your time for improving libtool ...
>
> No, not with libtool.
> I do not want to support that insane approach of tweaking command
> lines in the middle - it's an Pandorra's Box. I've already spent
> too much time on it and decided to completely drop it.
>
> Instead I prefer *clean* lines.
>
> Unitool provides commands on an higher level than, gcc+co do.
> These commands are on an higher functional level, hiding the
> individual platform's details. Also including things like importing
> .la-libs, sysroot, etc.
>
> Lt-unitool is a wrapper which parses the libtool commands and calls
> Unitool to do the actual work.
Seriously, why is a wrapper for the compiler/linker required AT ALL if the
calls to these tools are made from _generated_ files ?
The generated files should just contain the appropriate calls for the
respective commands.
This layer of abstraction is unnecessary and IMO just adds confusion.
(modifying libtool so that it calls unitool even seems to add yet another
layer which can potentially break or bitrot etc.)
Alex
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Alexander Neundorf @ 2008-06-16 8:13 UTC (permalink / raw)
To: Linux Embedded Maillist
In-Reply-To: <20080613185157.GT13599@pengutronix.de>
On Friday 13 June 2008 20:51:57 you wrote:
...
> Been there, seen that. I maintain > 500 packets in PTXdist and guess
> which ones make 90% of the problems. Hint: they are not related to
> autotools ...
If there are specific problems common to many cmake based packages please
report that to cmake@cmake.org. If there is something many people get wrong
something should be done about this in cmake (and if it just means improving
the documentation).
Alex
^ permalink raw reply
* Re: cross-compiling alternatives
From: Bernd Petrovitsch @ 2008-06-16 8:21 UTC (permalink / raw)
To: Robert Schwebel; +Cc: Alexander Neundorf, linux-embedded
In-Reply-To: <20080613185157.GT13599@pengutronix.de>
On Fre, 2008-06-13 at 20:51 +0200, Robert Schwebel wrote:
> On Fri, Jun 13, 2008 at 08:30:52AM +0200, Alexander Neundorf wrote:
> > Battle of Wesnoth is currently converted to both Scons and CMake, and
> > in the end they will decide about the winner. (since Eric is good at
> > arguing I guess it will be scons).
>
> The thing is that 'configure && make && make install' plus the usuall
> --enable-foo / --disable-foo / --with-bla=blub semantics is simply *the*
> standard way of configuring stuff for unix systems. You don't need fancy
> tools, you get cross compiling almost for free and unix people simply
> know how to use it.
As long as people avoid AC_TRY_RUN() and similar and allow the
"configurator" to tell `configure.sh` facts for the unavoidable cases
about the target (and there were some apps - and I forgot the names -
out there where this wasn't easily possible with editing the generated
configure.sh. Yes, that's not fault of autotools as such but autotools
make it IMHO far too easy to write that sort without generating lots of
warnings[0]).
> All the cool kids out there who think they know everything better
> usually start with "I hate autotools", then invent something which
That has IMHO 2 main reasons:
- For lost of apps, a Makefile and some coding discipline is more than
enough to support Linux/*BSD/MacOS-X even on different hardware.
And there are always cases where you need OS-specific code anyways
(e.g. manipulating routes).
Yes, that may need much more coding discipline than the average
programmer is used to.
- Converting $PROJECT to autotools is not easy. One has to learn and
understand how the tools work[1] and what should be done in which way.
And AFAIU (which is not much for autotools) one has to adapt the
source anyways here and there (so it is not really a drop-in
replacement).
And if people consider using autotools, it is probably quite large and
complex ....
Add that to negative experiences with other autotools packages and I can
understand the above sentence.
> solves 0.1% of the problems (including their very special problem) and
> tell the rest of the world that their shiny new tools is *soooo cooool*.
Bernd
[0]: Yes, this is free software and I could send patches etc. to fix
that. But that price is IMHO higher than just not using autotools
for new stuff.
[1]: And I didn't find a site yet with an easy understandable
description for seasoned programmers with cross-compile
and multi-hardware experience.
--
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and Services
^ permalink raw reply
* Re: cross-compiling alternatives
From: Bernd Petrovitsch @ 2008-06-16 8:28 UTC (permalink / raw)
To: Alexander Neundorf; +Cc: Linux Embedded Maillist
In-Reply-To: <200806161002.25385.neundorf@eit.uni-kl.de>
On Mon, 2008-06-16 at 10:02 +0200, Alexander Neundorf wrote:
[...]
> Seriously, why is a wrapper for the compiler/linker required AT ALL if the
> calls to these tools are made from _generated_ files ?
AFAIU the motivation of libtool to provide OS-independent (and toolchain
independent?) means to compile and link (etc.).
> The generated files should just contain the appropriate calls for the
> respective commands.
But these calls are - e.g. for shared libraries - not identical. The Gnu
toolchain is not everything.
> This layer of abstraction is unnecessary and IMO just adds confusion.
> (modifying libtool so that it calls unitool even seems to add yet another
> layer which can potentially break or bitrot etc.)
I know of more than one "occasion" where "gcc" et.al. actually where
wrappers around the actually binaries to check and guarantee commandline
parameters and options (some must be there, some mustn't be there, ...).
Usually sane Makefiles (and sane autotools usages) are enough for that.
Bernd
--
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and Services
^ permalink raw reply
* Re: cross-compiling alternatives
From: Bernd Petrovitsch @ 2008-06-16 8:33 UTC (permalink / raw)
To: Jamie Lokier; +Cc: Robert Schwebel, Alexander Neundorf, linux-embedded
In-Reply-To: <20080614000749.GA30652@shareable.org>
On Sam, 2008-06-14 at 01:07 +0100, Jamie Lokier wrote:
[...]
> You said about too many user-selectable options. Many large packages
These are IME not a problem if they have somewhat sensible defaults.
> _check_ for many installed libraries. Get them wrong, and you have
> the same problems of untested combinations.
As long as I can specify that libfoo support must be compiled in (and
thus libfoo must be present) and the tools throw an error if it doesn't
find it, I have no problem.
Otherwise all package builders have a serious problem.
Bernd
--
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and Services
^ permalink raw reply
* Re: cross-compiling alternatives
From: Alexander Neundorf @ 2008-06-16 9:25 UTC (permalink / raw)
To: Linux Embedded Maillist
In-Reply-To: <1213604910.1710.32.camel@tara.firmix.at>
On Monday 16 June 2008 10:28:30 Bernd Petrovitsch wrote:
> On Mon, 2008-06-16 at 10:02 +0200, Alexander Neundorf wrote:
> [...]
>
> > Seriously, why is a wrapper for the compiler/linker required AT ALL if
> > the calls to these tools are made from _generated_ files ?
>
> AFAIU the motivation of libtool to provide OS-independent (and toolchain
> independent?) means to compile and link (etc.).
>
> > The generated files should just contain the appropriate calls for the
> > respective commands.
>
> But these calls are - e.g. for shared libraries - not identical. The Gnu
> toolchain is not everything.
Yes, that's what I mean. The commands for calling the tools are generated.
They should be generated appropriately for the respective toolchains (i.e.
the generator must know how to call the different toolchains). No
libtool/unitool/other wrappers required.
Alex
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Jamie Lokier @ 2008-06-16 10:49 UTC (permalink / raw)
To: Enrico Weigelt; +Cc: linux-embedded
In-Reply-To: <20080616042310.GH12962@nibiru.local>
Enrico Weigelt wrote:
> Most times I've seen those checks, they silently enable some
> features, eg. if it looks for certain kernel devices. Definitively
> the wrong way!
Agreed. Though, you do often have to check for headers etc.,
otherwise you won't have the definitions needed to work with those
devices. (Think: inotify).
> It really should be in users/packagers control to
> explicitly enable features.
But then there's a problem with 100 or more --enable-foo command line
options. The user's script will _calls_ ./configure will
bitrot... there's no way you'll be typing it on the command line.
A big package can easily depend on that many local features.
> Nevertheless, the existence of some file or device says nothing
> about whether it will be usable (or *should* be used) at
> runtime. I've seen packages silently enabling some feature and then
> failing at runtime since the previously detected device is missing
> later. What a nightamare for packagers.
I agree, assume they're all there is not sane.
> Another point are broken syscalls. Well, you *have* check at runtime
> to be sure, or perhaps choose to ignore it and expect a sane system.
That said, to use rsync as an example. It uses the utimes() system
call when available, and utime() otherwise, because the former is
better.
On current GNU/Linux it will use utimes(). Recently someone tried to
run it on an old kernel, and it failed. If it tried utimes() first
then fell back to utime(), it was fine, and rsync has a patch to do
that now.
But here's the thing: do you really want every package have code
calling every different variation on a system call, at run time, until
it finds one that works?
-- Jamie
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: David Woodhouse @ 2008-06-16 11:09 UTC (permalink / raw)
To: Jamie Lokier; +Cc: Enrico Weigelt, linux-embedded
In-Reply-To: <20080616104917.GA18857@shareable.org>
On Mon, 2008-06-16 at 11:49 +0100, Jamie Lokier wrote:
> But here's the thing: do you really want every package have code
> calling every different variation on a system call, at run time, until
> it finds one that works?
No. That functionality lives in libc, if you want it at all.
--
dwmw2
^ permalink raw reply
* Re: cross-compiling alternatives
From: Jamie Lokier @ 2008-06-16 11:17 UTC (permalink / raw)
To: Bernd Petrovitsch; +Cc: Robert Schwebel, Alexander Neundorf, linux-embedded
In-Reply-To: <1213605214.1710.38.camel@tara.firmix.at>
Bernd Petrovitsch wrote:
> > _check_ for many installed libraries. Get them wrong, and you have
> > the same problems of untested combinations.
>
> As long as I can specify that libfoo support must be compiled in (and
> thus libfoo must be present) and the tools throw an error if it doesn't
> find it, I have no problem.
> Otherwise all package builders have a serious problem.
They do have problems, when you want to repeatably build and deploy,
if the build environment isn't very similar each time.
Typically the way you specify that libfoo support must be compiled in
is --with-libfoo=/path/to/libfoo.
That way lies bitrot between your build script which calls ./configure
(since you won't by typing it manually with 20+ options like that each
time you rebuild), and the changing version of an upstream package you
configure.
To prevent it trying to compile in libs you don't want, you also need
--without-libfoo. Using Kerberos as an example, which I remember when
building CVS ages ago: If you don't _prevent_ it using libraries you
don't want, you get different binariesn depending on whether a
Kerberos library was installed on the build system at build time. You
might then send a built program to another system, and find it won't
run at all, or has unwanted behaviour.
Do you really see package building scripts with 20 --with-libfoo= and
--without-libfoo= options in them for every library? Sometimes. But
more often, not: instead, they more often have build-time installed
prerequisites.
-- Jamie
^ permalink raw reply
* Re: cross-compiling alternatives (was Re: [PATCH 0/1] Embedded Maintainer(s)...)
From: Jamie Lokier @ 2008-06-16 11:33 UTC (permalink / raw)
To: linux-embedded
In-Reply-To: <20080616051147.GD32139@nibiru.local>
Enrico Weigelt wrote:
> > Reality is that Kconfig front end to autotools does work - as you've
> > proved. It's a good idea. :-)
>
> Now, we just need an autoconf-alike frontend for Kconfig ;-)
I agree and would support:
./configure --menu
Invokes a configuration menu / wizard for user-selectable options.
Only works if you have the "autoconf-menu" tools already installed.
./configure --ask
Invokes command line configuration questions / wizard for
user-selectable options.
./configure --ask --from-config=OLDCONFIG
Uses existing settings from a previous configuration.
./configure
Without --menu or --ask, invokes traditional Autoconf
automatic detection of capabilities.
> > You said about too many user-selectable options. Many large packages
> > _check_ for many installed libraries. Get them wrong, and you have
> > the same problems of untested combinations.
>
> It even gets worse when they silently enable certain features on
> presence/absence of some lib. That's btw one of the reasons why
> sysroot is an primary constraint for me, even when building for the
> platform+arch.
That's a basic design feature of Autoconf. And it really is very good
most of the time - much better than config scripts which ask a lot of
questions (Perl).
> > Have you felt uncomfortable shipping a package that does use Autoconf,
> > Automake and Libtool, knowing that the scripts generated by those
> > tools are huge compared with the entire source of your package?
>
> Yeah, that's one of those things in autotools I never understood:
> why isn't there just one function for each type of check/action,
> which is just called with the right params ?
It's because it's written in very portable Bourne shell, which does
not support shell functions.
(This is why I toy with the idea of writing a very portable C program,
similar to a simplified shell, specially designed for Autoconf, with
functions...)
It could simulate functions portably by creating lots of little
scripts at the start, and then calling them later on. That would
shrink ./configure a lot. But it's a big change; somebody's got to do
it, test it, look after it, and so on.
Libtool similarly does not have to be huge, but it would be a lot of
work to shrink it without breaking anything. It's hard enough to
change it at all without breaking anything on some system, somewhere.
-- Jamie
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox