* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: H. Peter Anvin @ 2009-01-02 19:27 UTC (permalink / raw)
To: Sam Ravnborg
Cc: Embedded Linux mailing list, Paul Mundt, Rob Landley,
linux-kernel, Andrew Morton
In-Reply-To: <20090102180134.GB5818@uranus.ravnborg.org>
Sam Ravnborg wrote:
> Hi Wookey.
>
>> Given the
>> simplicitly of these patches I can't see any reason not to put them
>> in
>
> Please do NOT do the mistake and think this the same thing.
>
> Rob's patch simplyfy the timecost stuff - and will be applied on
> this merit alone assuming comments will be addressed.
>
> But the serie rased anohter topic: shall we ban use of perl
> for generating a kernel.
> And this is what primary is discussed and the outcome of
> that discussion will not prevent patches that stands on their
> own to be applied.
>
My personal opinion on this is that this is ridiculous. Given that you
need gcc, binutils, make etc. to build the kernel, and this is more than
inherent, you have to have a pretty bloody strangely constrained system
to disallow Perl, which is as close to a standard Unix utility you can
get without making it into SuS.
The only *real* motivation I have seen for this is a system that as far
I can tell is nothing other than a toy, specifically designed to show
off how little you need to build the kernel. In other words, it's not a
practical application, it's a show-off art piece.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
From: H. Peter Anvin @ 2009-01-02 19:33 UTC (permalink / raw)
To: Rob Landley
Cc: Sam Ravnborg, Embedded Linux mailing list, linux-kernel,
Andrew Morton
In-Reply-To: <200901020600.47847.rob@landley.net>
Rob Landley wrote:
>
> You mean "The new shell script is much simpler, about 1/4 the size, runs on
> Red Hat 9 from 2003, and isn't perl?" :)
>
And introduces unclear environment dependencies depending on how
external utilities are implemented.
The whole point of why that script was written in Perl was to have
access to arbitrary-precision arithmetic -- after it was shown that bc
would simply lock up on some systems.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Jamie Lokier @ 2009-01-03 3:22 UTC (permalink / raw)
To: Theodore Tso, Rob Landley, Christoph Hellwig,
Arkadiusz Miskiewicz, linux-kernel, Embed
In-Reply-To: <20090102140409.GA4758@mit.edu>
Theodore Tso wrote:
> perl is actually quite portable.
Portability aside, Perl has another fun issue. The number of times
I've had a Perl script break when copied to a newer system which had a
newer version of Perl is... noticable.
> I'd also suggest explicitly add a reminder to the shell scripts'
> comments to avoid bashisms for maximum portability, to remind
> developers in the future who might try to change the shell scripts
> to watch out for portability issues.
You can force Bash into POSIX mode if that's helpful.
-- Jamie
^ permalink raw reply
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
From: Harvey Harrison @ 2009-01-03 6:28 UTC (permalink / raw)
To: Rob Landley
Cc: Sam Ravnborg, Embedded Linux mailing list, linux-kernel,
Andrew Morton, H. Peter Anvin
In-Reply-To: <200901020600.47847.rob@landley.net>
On Fri, 2009-01-02 at 06:00 -0600, Rob Landley wrote:
> On Friday 02 January 2009 03:04:39 Sam Ravnborg wrote:
> > > +# Output start of header file
> > > +
> > > +cat << EOF
> > > +/* Automatically generated by kernel/timeconst.sh */
> > > +/* Conversion constants for HZ == $HZ */
> > > +
> > > +#ifndef KERNEL_TIMECONST_H
> > > +#define KERNEL_TIMECONST_H
> >
> > Please use __KERNEL_TIMECONST_H. The two underscores are standard.
>
> Sure thing. (I was just copying the perl there, I'll post an updated patch
> after I get some sleep.)
In the x86 discussions recently regarding header guards, I though a single
underscore was eventually decided on as 'standard'.
Harvey
^ permalink raw reply
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
From: Ingo Oeser @ 2009-01-03 12:28 UTC (permalink / raw)
To: Rob Landley
Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton,
H. Peter Anvin, Sam Ravnborg
In-Reply-To: <200901020213.30658.rob@landley.net>
Hi Robert,
I fully support your argument here: Requiring lots of hacks in various languages
to build a system core sucks. But now sth. more productive:
On Friday 02 January 2009, Rob Landley wrote:
> diff -r d0c7611dcfd6 kernel/Makefile
> --- a/kernel/Makefile Tue Dec 30 17:48:25 2008 -0800
> +++ b/kernel/Makefile Fri Jan 02 00:10:44 2009 -0600
> @@ -115,7 +115,7 @@
> $(obj)/time.o: $(obj)/timeconst.h
>
> quiet_cmd_timeconst = TIMEC $@
> - cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@
> + cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) > $@
> targets += timeconst.h
> -$(obj)/timeconst.h: $(src)/timeconst.pl FORCE
> +$(obj)/timeconst.h: $(src)/timeconst.sh FORCE
> $(call if_changed,timeconst)
> --- /dev/null 1969-12-31 00:00:00.000000000 -0600
> +++ hg/kernel/timeconst.sh 2009-01-01 23:53:17.000000000 -0600
> @@ -0,0 +1,91 @@
> +#!/bin/bash
> +
> +if [ $# -ne 1 ]
> +then
> + echo "Usage: timeconst.sh HZ"
> + exit 1
> +fi
> +
> +HZ=$1
> +
> +# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math.
> +
> +[ $((1<<32)) -lt 0 ] && exit 1
> +
> +# Output start of header file
> +
> +cat << EOF
> +/* Automatically generated by kernel/timeconst.sh */
> +/* Conversion constants for HZ == $HZ */
> +
> +#ifndef KERNEL_TIMECONST_H
> +#define KERNEL_TIMECONST_H
> +
> +#include <linux/param.h>
> +#include <linux/types.h>
> +
> +#if HZ != $HZ
> +#error "kernel/timeconst.h has the wrong HZ value!"
> +#endif
> +
> +EOF
> +
> +# For both Miliseconds and Microseconds
> +
> +for i in "MSEC 1000" "USEC 1000000"
> +do
> + NAME=$(echo $i | awk '{print $1}')
cut -d' ' -f1 does the same
> + PERIOD=$(echo $i | awk '{print $2}')
cut -d' ' -f2 does the same
Best Regards
Ingo Oeser
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Wolfgang Denk @ 2009-01-03 14:59 UTC (permalink / raw)
To: Paul Mundt
Cc: Rob Landley, Embedded Linux mailing list, linux-kernel,
Andrew Morton, H. Peter Anvin, Sam Ravnborg
In-Reply-To: <20090102095023.GA28078@linux-sh.org>
Dear Paul Mundt,
In message <20090102095023.GA28078@linux-sh.org> you wrote:
>
> Your main reasons against inclusion of perl seem to be that there is no
> realistic expectation for target systems that will be self-hosting will
> have perl included, or the inherent complexity in maintaining a coherent
> cross compiling environment. Both of these things are issues with your
> own environment, and in no way are these representative of the embedded
> development community at large.
I'm not sure how representative for the "embedded development
community at large" your statement is.
Just to add a data point to the statistice, I do agree with Rob's
opinion about needing Per to build the Linux kernel.
All other arguments aside, natively compiling the Linux kernel on a
target board, especially with root file system mounted over NFS, has
always been an excellent stress test (you may even call it regression
test) for Linux running on some target hardware.
Losing this ability just because some scripts are implemented in
language FOO when plain shell will do as well is something we should
try to avoid.
Of course, YMMV.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Include the success of others in your dreams for your own success.
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Rob Landley @ 2009-01-03 19:46 UTC (permalink / raw)
To: Matthieu CASTET
Cc: Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list,
Andrew Morton, H. Peter Anvin, Sam Ravnborg
In-Reply-To: <495E3AF8.6000006@parrot.com>
On Friday 02 January 2009 10:04:08 Matthieu CASTET wrote:
> Rob Landley a écrit :
> > On Friday 02 January 2009 03:26:37 Arkadiusz Miskiewicz wrote:
> >> On Friday 02 of January 2009, Rob Landley wrote:
> >>
> >> Heh,
> >
> > I believe all three scripts run under dash and busybox ash. (The
> > timeconst.sh one needs 64 bit math which dash only provides on 64 bit
> > hosts, which is a regression from Red Hat 9 in 2003 by the way.
>
> With dash 0.5.4-12 (from debian sid), I seems I got the 64 bit math for
> 32 bit hosts :
> $ uname -m
> i686
> $ dash -c 'echo $((1<<32))'
> 4294967296
Cool.
The "relatively recent" 32 bit image I have lying around for testing purposes
is xubuntu 7.10, and when dash was first introduced into ubuntu it had
_buckets_ of bugs. (If you backgrounded a task with & and then hit ctrl-z on
the command line, it would suspend the backgrounded task. It was Not Ready
for Prime Time in a big way.) Lack of 64 bit math could easily be one more.
(It _is_ a regression vs Red Hat 9.)
The dash in ubuntu 8.10 seems to have a lot of the more obvious problems
worked out. Good to know. :)
That said, while testing the new round of patches against various shells and
making it reproduce the full set of time constants that the old perl script
kept cached values for (24, 32, 48, 64, 100, 122, 128, 200, 250, 256, 300,
512, 1000, 1024, and 1200 hz), I found a bug.
The first patch is miscalculating USEC_TO_HZ_ADJ32 for 24 HZ and 122 HZ. All
the other values are fine.) It's an integer overflow. The GCD of 24 and
1000000 is 8, so that's 17 significant bits with a shift of 47... which is
exactly 64 bits, but the math is _signed_, so it goes boing.
For the record, the reason I can't just pregenerate all these suckers on a
system that's got an arbitrary precision calculator (ala dc) and then just
ship the resulting header files (more or less the what the first version of
that first patch did) is that some architectures (arm omap and and arm at91)
allow you to enter arbitrary HZ values in kconfig. (Their help text says that
in many cases values that aren't powers of two won't work, but nothing
enforces this.) So if we didn't have the capability to dynamically generate
these, you could enter a .config value that would break the build.
I'd be willing to use dc in the script if A) it was mentioned in SUSv3 (it's
not, although bc is), B) the version of dc in busybox wasn't crap (it's
floating point rather than arbitrary precision, and doesn't implement the left
shift operator). The reason I'm not looking more closely at what SUSv3 has to
say about bc is that A) it claims to be an entire programming language, which
is definitely overkill for this B) busybox hasn't bothered to implement it so
it can't be all that widely used anymore.
I'll fix this and resubmit, it just wasn't ready last night. (If the merge
window is closing soon I could resubmit the other two with Sam's suggestions
and resubmit this one next time 'round, but it was only a couple days to write
in the first place, once I finally figured out what the perl version was
trying to _do_...)
I believe ADJ32 is the only operation with any potential to overflow. The
pathological case for SHIFT is HZ 1, which for USEC conversions would give a
shift around 52 (32 significant bits plus 20 bits to divide by 1000000), but
MUL32 still wouldn't overflow because the shift loop stops when it finds 32
significant bits, and any larger HZ value would produce a smaller shift. The
problem with ADJ32 is it uses the MUL32 shift value, so a small $TO (24 HZ)
with a big $FROM (1000000 USEC, 19 signficant bits) and a small Greatest
Common Denominator (in this case 8) can overflow 64 bits. Pathological case
is still HZ 1. (Or any other smallish prime number.) If I make that work,
everything else has to.
So anyway, it's not _arbitrary_ precision math. It's more like 32+20+20=72
bits, and I can probably fake that pretty easily by breaking a couple of
operations into two stages...
Fallen a bit behind on the thread since I noticed this and went off to code,
I'll try to catch up later today.
Rob
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Rob Landley @ 2009-01-03 19:48 UTC (permalink / raw)
To: Sam Ravnborg
Cc: Embedded Linux mailing list, Paul Mundt, linux-kernel,
Andrew Morton, H. Peter Anvin
In-Reply-To: <20090102180134.GB5818@uranus.ravnborg.org>
On Friday 02 January 2009 12:01:34 Sam Ravnborg wrote:
> But the serie rased anohter topic: shall we ban use of perl
> for generating a kernel.
I dunno about "ban", but every time somebody adds perl to the "hot path" of
the kernel build it breaks my build system, and I write a removal patch
anyway. I have to maintain them anyway, so I might as well try to push 'em
upstream. (I posted the first patch in this series twice before, once for 25
and then an updated version to the linux-embedded list for .26.)
I didn't discover this topic independently. Somebody pinged me about it on
freenode back in February, and several other people sent me private email
about it, and it's been previously raised on several other mailing lists (such
as busybox and uclibc ones).
Unfortunately, most of the embedded developers I know aren't subscribed to
linux-kernel. (You either do kernel development, or you do everything else.
It's not really feasible to keep up with the guts of the kernel and uClibc and
busybox and gcc and qemu and the current offerings of three different hardware
vendors and whatever userspace application the board's supposed to run and
your build system and what INSANE things your EEdiot electrical engineer
decided to miswire this time and fighting off marketing's desire to switch
everything over to WinCE because they can get their entire advertising budget
subsidized and there's a trade show next week we're not READY for... Not only
can none of 'em read a meaningful subset of linux-kernel anymore, but if you
disappear into your own little niche for nine months, when you pop back up the
kernel's all different and sometimes even the patch submission policy's
migrated a bit. Heck, I'm three months behind reading the LWN kernel page
myself, and that's no substitute for kernel-traffic, RIP...)
Hopefully the cc: to linux-embedded is helping get more embedded guys involved
in the discussion than just me. :)
> And this is what primary is discussed and the outcome of
> that discussion will not prevent patches that stands on their
> own to be applied.
The best way to get a patch applied is always for that patch to stand on its
own merits. Larger agendas are secondary.
Whether or not the kernel decides on a policy of keeping perl out of the
kernel build's "hot path", I still need these patches for my own use, and plan
to keep coming up with them and submitting them. I haven't removed ones that
haven't broken my build yet, but just because I'm not using md today doesn't
mean I won't _start_. (And if enough other people keep poking me about the
kernel build I can tackle 'em to please them. I actually _do_ know some
embedded developers using raid for network attached storage and video servers
and such...)
> Sam
Rob
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Sam Ravnborg @ 2009-01-03 20:10 UTC (permalink / raw)
To: Rob Landley
Cc: Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel,
Embedded Linux mailing list, Andrew Morton, H. Peter Anvin
In-Reply-To: <200901031346.01325.rob@landley.net>
>
> I'll fix this and resubmit, it just wasn't ready last night. (If the merge
> window is closing soon I could resubmit the other two with Sam's suggestions
> and resubmit this one next time 'round, but it was only a couple days to write
> in the first place, once I finally figured out what the perl version was
> trying to _do_...)
For kbuild only fixes and trivial stuff will be merged until next merge window.
Neither of the three patches fall into that category.
With respect to your three patches the plan is to:
- add the updated timeconst patch to kbuild-next
- add the updated cpu-feature patch to kbuild-next
- the patch touching headers_install will not be merged.
The way forward for headers_install is to combine the
unifdef feature and the header modifications.
And this must be in a single program that can process
all headers in one go so the install process becomes so fast
that we do not worry about if it was done before or not.
Then we can avoid all the .* files in the directory
where we isntall the headers.
The program is a prime candidate for a small C program
and I hope someone can take the challenge to write it.
Migrating from perl to shell does not help us here
and the shell version was less readable than the perl version.
Sam
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: H. Peter Anvin @ 2009-01-03 20:50 UTC (permalink / raw)
To: Sam Ravnborg
Cc: Rob Landley, Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel,
Embedded Linux mailing list, Andrew Morton
In-Reply-To: <20090103201059.GA4875@uranus.ravnborg.org>
Sam Ravnborg wrote:
> With respect to your three patches the plan is to:
> - add the updated timeconst patch to kbuild-next
If you add this, you take the responsibility for the breakages that will
occur. The reason his patch is "simpler" is because he removes the
arbitrary-precision arithmetic, and simply hopes that the system
utilities that he uses uses an integer size which happens to be big enough.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Leon Woestenberg @ 2009-01-03 22:54 UTC (permalink / raw)
To: Rob Landley
Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton,
H. Peter Anvin, Sam Ravnborg
In-Reply-To: <200901020207.30359.rob@landley.net>
Hello all,
On Fri, Jan 2, 2009 at 9:07 AM, Rob Landley <rob@landley.net> wrote:
> Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 )
> building a Linux kernel never required perl to be installed on the build
> system. (Various development and debugging scripts were written in perl and
> python and such, but they weren't involved in actually building a kernel.)
> Building a kernel before 2.6.25 could be done with a minimal system built from
> gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, and nothing
> else. (Embedded developers creating clean cross compile environments that
>
I agree with Rob that the amount of required dependencies should be
kept to a minimum.
If we only use 0.5% of a certain language (or: dependent package),
then rather implement
that 0.5% in the existing language.
Dependencies very quickly become dependency hell. If A requires B,
then A also inherits all
(future) requirements of B, etc. etc.
In my daily software development work with Linux and GNU software in
general, 10% of it is spent fighting/removing these extremely "thin"
or false depencies, so that it is usuable in embedded devices.
Regards,
--
Leon
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: H. Peter Anvin @ 2009-01-03 23:03 UTC (permalink / raw)
To: Leon Woestenberg
Cc: Rob Landley, Embedded Linux mailing list, linux-kernel,
Andrew Morton, Sam Ravnborg
In-Reply-To: <c384c5ea0901031454n451bc899w166bb146c96f6164@mail.gmail.com>
Leon Woestenberg wrote:
>
> I agree with Rob that the amount of required dependencies should be
> kept to a minimum.
>
> If we only use 0.5% of a certain language (or: dependent package),
> then rather implement that 0.5% in the existing language.
>
> Dependencies very quickly become dependency hell. If A requires B,
> then A also inherits all (future) requirements of B, etc. etc.
>
> In my daily software development work with Linux and GNU software in
> general, 10% of it is spent fighting/removing these extremely "thin"
> or false depencies, so that it is usuable in embedded devices.
>
First of all, I largely consider this a joke. All real-life embedded
kernel builds take place on hosted platforms; anything else seems to be
done "just because it can be done", as a kind of show-off art project.
Cute, but hardly worth impeding the rest of the kernel community for.
We're not talking about general platform dependencies here, but build
dependencies for the kernel. A platform that can build the kernel is
not a small platform.
Second of all, these patches are not fullworthy replacements. The
original patch using bc had less dependencies, but bc failed on some
platforms, mysteriously. The new patches have *more* environmental
dependencies than that ever did.
Third, if someone actually cares to do it right, I have a smallish
bignum library at http://git.zytor.com/?p=lib/pbn.git;a=summary that
might be a starting point.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Leon Woestenberg @ 2009-01-04 0:37 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Rob Landley, Embedded Linux mailing list, linux-kernel,
Andrew Morton, Sam Ravnborg
In-Reply-To: <495FEEAF.5020005@zytor.com>
Hello,
On Sun, Jan 4, 2009 at 12:03 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>> Dependencies very quickly become dependency hell. If A requires B,
>> then A also inherits all (future) requirements of B, etc. etc.
>>
>> In my daily software development work with Linux and GNU software in
>> general, 10% of it is spent fighting/removing these extremely "thin"
>> or false depencies, so that it is usuable in embedded devices.
>>
>
> First of all, I largely consider this a joke. All real-life embedded
> kernel builds take place on hosted platforms; anything else seems to be
> done "just because it can be done", as a kind of show-off art project.
> Cute, but hardly worth impeding the rest of the kernel community for.
>
Let me explain why it is not a joke for me, although yes I agree it's
a funny way of how software engineering works.
My argument on thin dependencies indeed mostly holds for run-time
dependencies (to reduce size) but also for build dependency (to reduce
complexity)*.
In general the right version of the right tool is not available on the
build host.
If I cross-build 30 packages all of which need a build-host-native
perl during their build, consider the chance of these packages
building with the one version of perl that lives on the system. It's
near 0% for the average mix of packages.
Yes, the host is fat enough so that we can build (say three) different
versions of perl to accomodate the build.
The truth often is, of the 30 build dependencies 25 of them are thin
(build) dependencies.
So yes, the say extra 10% of build tool dependency needed to make 0.5%
of the build system work, can be solved on the fat build platform, but
I do not always consider it a well engineered solution.
For a (dangerous) metaphor: In mechanical engineering, if there are 30
joints using nuts and bolts, and a 31st is designed in, it is strongly
prefered to make it a nut and bolt as well, even if using a screw
would work equally well or even better. The overall costs (over
complexity, tool dependency, maintainability) would be lower.
Regards,
--
Leon
* Since Intel, since recently, pays another team of excellent people
maintaining a system dealing with the (cross) build tool dependency
hell, I rest my case. :-)
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Robert Hancock @ 2009-01-04 0:44 UTC (permalink / raw)
To: Rob Landley
Cc: Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel,
Embedded Linux mailing list, Andrew Morton, H. Peter Anvin,
Sam Ravnborg
In-Reply-To: <200901031346.01325.rob@landley.net>
Rob Landley wrote:
> For the record, the reason I can't just pregenerate all these suckers on a
> system that's got an arbitrary precision calculator (ala dc) and then just
> ship the resulting header files (more or less the what the first version of
> that first patch did) is that some architectures (arm omap and and arm at91)
> allow you to enter arbitrary HZ values in kconfig. (Their help text says that
> in many cases values that aren't powers of two won't work, but nothing
> enforces this.) So if we didn't have the capability to dynamically generate
> these, you could enter a .config value that would break the build.
Is there a good reason that these archs allow you enter arbitrary HZ
values? The use case for using custom HZ values at all nowadays seems
fairly low now that dynticks is around (if that arch supports it
anyway), let alone being able to specify wierd obscure values for it.
Especially if nothing can ensure that all values it allows will actually
result in a functional kernel..
^ permalink raw reply
* PATCH [0/3]: Simplify the kernel build by removing perl v2
From: Rob Landley @ 2009-01-04 1:24 UTC (permalink / raw)
To: Embedded Linux mailing list
Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg
In-Reply-To: <200901020207.30359.rob@landley.net>
Here's an updated set of patches to remove use of perl from the kernel build's
"hot path" (roughly defined as "make allnoconfig; make; make
headers_install").
This update incorporates feedback from Sam Ravnborg, Ted Tso, Joe Perches,
Ingo Oeser, and others. It also fixes an integer overflow error in the first
patch, and all three scripts have been retested under dash.
^ permalink raw reply
* PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2)
From: Rob Landley @ 2009-01-04 1:27 UTC (permalink / raw)
To: Embedded Linux mailing list
Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg
In-Reply-To: <200901031924.15869.rob@landley.net>
From: Rob Landley <rob@landley.net>
Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell script
is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003.
It requires a shell which can do 64 bit math, such as bash, busybox ash,
or dash running on a 64 bit host.
Changes from previous version:
Redo ADJ32 math to avoid integer overflow for small HZ sizes (such as 24 or
122). In the pathological case (HZ=1) both versions now produce
USEC_TO_HZ_ADJ32 of 0x7ffff79c842fa. (See source comments for details.)
Also expand usage message, add error message when no 64 bit math available in
shell (and suggest some shells that support it), add whitespace around
operators in equations, added underscores before __KERNEL_TIMECONST_H, change
makefile so script is responsible for creating output file, make script delete
output file on error, change shebang to #!/bin/sh and test with dash and bash.
Signed-off-by: Rob Landley <rob@landley.net>
---
kernel/Makefile | 4
kernel/timeconst.pl | 378 ------------------------------------------
kernel/timeconst.sh | 149 ++++++++++++++++
3 files changed, 151 insertions(+), 380 deletions(-)
--- linux-2.6.28/kernel/timeconst.pl 2008-12-24 17:26:37.000000000 -0600
+++ /dev/null 2008-11-21 04:46:41.000000000 -0600
@@ -1,378 +0,0 @@
-#!/usr/bin/perl
-# -----------------------------------------------------------------------
-#
-# Copyright 2007-2008 rPath, Inc. - All Rights Reserved
-#
-# This file is part of the Linux kernel, and is made available under
-# the terms of the GNU General Public License version 2 or (at your
-# option) any later version; incorporated herein by reference.
-#
-# -----------------------------------------------------------------------
-#
-
-#
-# Usage: timeconst.pl HZ > timeconst.h
-#
-
-# Precomputed values for systems without Math::BigInt
-# Generated by:
-# timeconst.pl --can 24 32 48 64 100 122 128 200 250 256 300 512 1000 1024 1200
-%canned_values = (
- 24 => [
- '0xa6aaaaab','0x2aaaaaa',26,
- 125,3,
- '0xc49ba5e4','0x1fbe76c8b4',37,
- 3,125,
- '0xa2c2aaab','0xaaaa',16,
- 125000,3,
- '0xc9539b89','0x7fffbce4217d',47,
- 3,125000,
- ], 32 => [
- '0xfa000000','0x6000000',27,
- 125,4,
- '0x83126e98','0xfdf3b645a',36,
- 4,125,
- '0xf4240000','0x0',17,
- 31250,1,
- '0x8637bd06','0x3fff79c842fa',46,
- 1,31250,
- ], 48 => [
- '0xa6aaaaab','0x6aaaaaa',27,
- 125,6,
- '0xc49ba5e4','0xfdf3b645a',36,
- 6,125,
- '0xa2c2aaab','0x15555',17,
- 62500,3,
- '0xc9539b89','0x3fffbce4217d',46,
- 3,62500,
- ], 64 => [
- '0xfa000000','0xe000000',28,
- 125,8,
- '0x83126e98','0x7ef9db22d',35,
- 8,125,
- '0xf4240000','0x0',18,
- 15625,1,
- '0x8637bd06','0x1fff79c842fa',45,
- 1,15625,
- ], 100 => [
- '0xa0000000','0x0',28,
- 10,1,
- '0xcccccccd','0x733333333',35,
- 1,10,
- '0x9c400000','0x0',18,
- 10000,1,
- '0xd1b71759','0x1fff2e48e8a7',45,
- 1,10000,
- ], 122 => [
- '0x8325c53f','0xfbcda3a',28,
- 500,61,
- '0xf9db22d1','0x7fbe76c8b',35,
- 61,500,
- '0x8012e2a0','0x3ef36',18,
- 500000,61,
- '0xffda4053','0x1ffffbce4217',45,
- 61,500000,
- ], 128 => [
- '0xfa000000','0x1e000000',29,
- 125,16,
- '0x83126e98','0x3f7ced916',34,
- 16,125,
- '0xf4240000','0x40000',19,
- 15625,2,
- '0x8637bd06','0xfffbce4217d',44,
- 2,15625,
- ], 200 => [
- '0xa0000000','0x0',29,
- 5,1,
- '0xcccccccd','0x333333333',34,
- 1,5,
- '0x9c400000','0x0',19,
- 5000,1,
- '0xd1b71759','0xfff2e48e8a7',44,
- 1,5000,
- ], 250 => [
- '0x80000000','0x0',29,
- 4,1,
- '0x80000000','0x180000000',33,
- 1,4,
- '0xfa000000','0x0',20,
- 4000,1,
- '0x83126e98','0x7ff7ced9168',43,
- 1,4000,
- ], 256 => [
- '0xfa000000','0x3e000000',30,
- 125,32,
- '0x83126e98','0x1fbe76c8b',33,
- 32,125,
- '0xf4240000','0xc0000',20,
- 15625,4,
- '0x8637bd06','0x7ffde7210be',43,
- 4,15625,
- ], 300 => [
- '0xd5555556','0x2aaaaaaa',30,
- 10,3,
- '0x9999999a','0x1cccccccc',33,
- 3,10,
- '0xd0555556','0xaaaaa',20,
- 10000,3,
- '0x9d495183','0x7ffcb923a29',43,
- 3,10000,
- ], 512 => [
- '0xfa000000','0x7e000000',31,
- 125,64,
- '0x83126e98','0xfdf3b645',32,
- 64,125,
- '0xf4240000','0x1c0000',21,
- 15625,8,
- '0x8637bd06','0x3ffef39085f',42,
- 8,15625,
- ], 1000 => [
- '0x80000000','0x0',31,
- 1,1,
- '0x80000000','0x0',31,
- 1,1,
- '0xfa000000','0x0',22,
- 1000,1,
- '0x83126e98','0x1ff7ced9168',41,
- 1,1000,
- ], 1024 => [
- '0xfa000000','0xfe000000',32,
- 125,128,
- '0x83126e98','0x7ef9db22',31,
- 128,125,
- '0xf4240000','0x3c0000',22,
- 15625,16,
- '0x8637bd06','0x1fff79c842f',41,
- 16,15625,
- ], 1200 => [
- '0xd5555556','0xd5555555',32,
- 5,6,
- '0x9999999a','0x66666666',31,
- 6,5,
- '0xd0555556','0x2aaaaa',22,
- 2500,3,
- '0x9d495183','0x1ffcb923a29',41,
- 3,2500,
- ]
-);
-
-$has_bigint = eval 'use Math::BigInt qw(bgcd); 1;';
-
-sub bint($)
-{
- my($x) = @_;
- return Math::BigInt->new($x);
-}
-
-#
-# Constants for division by reciprocal multiplication.
-# (bits, numerator, denominator)
-#
-sub fmul($$$)
-{
- my ($b,$n,$d) = @_;
-
- $n = bint($n);
- $d = bint($d);
-
- return scalar (($n << $b)+$d-bint(1))/$d;
-}
-
-sub fadj($$$)
-{
- my($b,$n,$d) = @_;
-
- $n = bint($n);
- $d = bint($d);
-
- $d = $d/bgcd($n, $d);
- return scalar (($d-bint(1)) << $b)/$d;
-}
-
-sub fmuls($$$) {
- my($b,$n,$d) = @_;
- my($s,$m);
- my($thres) = bint(1) << ($b-1);
-
- $n = bint($n);
- $d = bint($d);
-
- for ($s = 0; 1; $s++) {
- $m = fmul($s,$n,$d);
- return $s if ($m >= $thres);
- }
- return 0;
-}
-
-# Generate a hex value if the result fits in 64 bits;
-# otherwise skip.
-sub bignum_hex($) {
- my($x) = @_;
- my $s = $x->as_hex();
-
- return (length($s) > 18) ? undef : $s;
-}
-
-# Provides mul, adj, and shr factors for a specific
-# (bit, time, hz) combination
-sub muladj($$$) {
- my($b, $t, $hz) = @_;
- my $s = fmuls($b, $t, $hz);
- my $m = fmul($s, $t, $hz);
- my $a = fadj($s, $t, $hz);
- return (bignum_hex($m), bignum_hex($a), $s);
-}
-
-# Provides numerator, denominator values
-sub numden($$) {
- my($n, $d) = @_;
- my $g = bgcd($n, $d);
- return ($n/$g, $d/$g);
-}
-
-# All values for a specific (time, hz) combo
-sub conversions($$) {
- my ($t, $hz) = @_;
- my @val = ();
-
- # HZ_TO_xx
- push(@val, muladj(32, $t, $hz));
- push(@val, numden($t, $hz));
-
- # xx_TO_HZ
- push(@val, muladj(32, $hz, $t));
- push(@val, numden($hz, $t));
-
- return @val;
-}
-
-sub compute_values($) {
- my($hz) = @_;
- my @val = ();
- my $s, $m, $a, $g;
-
- if (!$has_bigint) {
- die "$0: HZ == $hz not canned and ".
- "Math::BigInt not available\n";
- }
-
- # MSEC conversions
- push(@val, conversions(1000, $hz));
-
- # USEC conversions
- push(@val, conversions(1000000, $hz));
-
- return @val;
-}
-
-sub outputval($$)
-{
- my($name, $val) = @_;
- my $csuf;
-
- if (defined($val)) {
- if ($name !~ /SHR/) {
- $val = "U64_C($val)";
- }
- printf "#define %-23s %s\n", $name.$csuf, $val.$csuf;
- }
-}
-
-sub output($@)
-{
- my($hz, @val) = @_;
- my $pfx, $bit, $suf, $s, $m, $a;
-
- print "/* Automatically generated by kernel/timeconst.pl */\n";
- print "/* Conversion constants for HZ == $hz */\n";
- print "\n";
- print "#ifndef KERNEL_TIMECONST_H\n";
- print "#define KERNEL_TIMECONST_H\n";
- print "\n";
-
- print "#include <linux/param.h>\n";
- print "#include <linux/types.h>\n";
-
- print "\n";
- print "#if HZ != $hz\n";
- print "#error \"kernel/timeconst.h has the wrong HZ value!\"\n";
- print "#endif\n";
- print "\n";
-
- foreach $pfx ('HZ_TO_MSEC','MSEC_TO_HZ',
- 'HZ_TO_USEC','USEC_TO_HZ') {
- foreach $bit (32) {
- foreach $suf ('MUL', 'ADJ', 'SHR') {
- outputval("${pfx}_$suf$bit", shift(@val));
- }
- }
- foreach $suf ('NUM', 'DEN') {
- outputval("${pfx}_$suf", shift(@val));
- }
- }
-
- print "\n";
- print "#endif /* KERNEL_TIMECONST_H */\n";
-}
-
-# Pretty-print Perl values
-sub perlvals(@) {
- my $v;
- my @l = ();
-
- foreach $v (@_) {
- if (!defined($v)) {
- push(@l, 'undef');
- } elsif ($v =~ /^0x/) {
- push(@l, "\'".$v."\'");
- } else {
- push(@l, $v.'');
- }
- }
- return join(',', @l);
-}
-
-($hz) = @ARGV;
-
-# Use this to generate the %canned_values structure
-if ($hz eq '--can') {
- shift(@ARGV);
- @hzlist = sort {$a <=> $b} (@ARGV);
-
- print "# Precomputed values for systems without Math::BigInt\n";
- print "# Generated by:\n";
- print "# timeconst.pl --can ", join(' ', @hzlist), "\n";
- print "\%canned_values = (\n";
- my $pf = "\t";
- foreach $hz (@hzlist) {
- my @values = compute_values($hz);
- print "$pf$hz => [\n";
- while (scalar(@values)) {
- my $bit;
- foreach $bit (32) {
- my $m = shift(@values);
- my $a = shift(@values);
- my $s = shift(@values);
- print "\t\t", perlvals($m,$a,$s), ",\n";
- }
- my $n = shift(@values);
- my $d = shift(@values);
- print "\t\t", perlvals($n,$d), ",\n";
- }
- print "\t]";
- $pf = ', ';
- }
- print "\n);\n";
-} else {
- $hz += 0; # Force to number
- if ($hz < 1) {
- die "Usage: $0 HZ\n";
- }
-
- @val = @{$canned_values{$hz}};
- if (!defined(@val)) {
- @val = compute_values($hz);
- }
- output($hz, @val);
-}
-exit 0;
--- /dev/null 2008-11-21 04:46:41.000000000 -0600
+++ linux-2.6.28-new/kernel/timeconst.sh 2009-01-03 15:24:01.000000000 -0600
@@ -0,0 +1,147 @@
+#!/bin/sh
+
+if [ $# -ne 2 ]
+then
+ echo "Usage: timeconst.sh HZ FILENAME"
+ echo
+ echo "Generate a header file with constants for coverting between"
+ echo "decimal HZ timer ticks and milisecond or microsecond delays."
+ echo
+ exit 1
+fi
+
+HZ=$1
+shift
+FILENAME=$1
+
+# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math.
+
+if [ $((1 << 32)) -lt 0 ]
+then
+ echo "timeconst.sh needs a shell with 64 bit math, such as bash,"
+ echo "busybox ash, or dash running on a 64 bit host."
+ exit 1
+fi
+
+# If this script exits for any reason before this trap is removed,
+# delete the output file so a partial file won't confuse the build.
+
+trap "rm $FILENAME" EXIT
+
+# Output start of header file
+
+cat > $FILENAME << EOF || exit 1
+/* Automatically generated by kernel/timeconst.sh */
+/* Conversion constants for HZ == $HZ */
+
+#ifndef __KERNEL_TIMECONST_H
+#define __KERNEL_TIMECONST_H
+
+#include <linux/param.h>
+#include <linux/types.h>
+
+#if HZ != $HZ
+#error "kernel/timeconst.h has the wrong HZ value!"
+#endif
+
+EOF
+
+# For both Milliseconds and Microseconds
+
+for i in "MSEC 1000" "USEC 1000000"
+do
+ NAME=$(echo $i | cut -d ' ' -f 1)
+ PERIOD=$(echo $i | cut -d ' ' -f 2)
+
+ # Find greatest common denominator (using Euclid's algorithm)
+
+ A=$HZ
+ B=$PERIOD
+
+ while [ $B -ne 0 ]
+ do
+ C=$(( $A % $B ))
+ A=$B
+ B=$C
+ done
+
+ GCD=$A
+
+ # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ)
+
+ for DIRECTION in 0 1
+ do
+ if [ $DIRECTION -eq 0 ]
+ then
+ CONVERT="HZ_TO_${NAME}"
+ FROM=$HZ
+ TO=$PERIOD
+ else
+ CONVERT="${NAME}_TO_HZ"
+ FROM=$PERIOD
+ TO=$HZ
+ fi
+
+ # Calculate 32 significant bits of MUL32 data.
+
+ SHIFT=0
+ while true
+ do
+ # This can't overflow 64 bit math. Pathological case
+ # (TO=1, FROM=1000000) uses around 32+20=52 bits.
+
+ MUL32=$(( ( ( $TO << $SHIFT ) + $FROM - 1 ) / $FROM ))
+
+ # Keep increasing $SHIFT until we've got 32 bits.
+
+ [ $MUL32 -gt $(( 1 << 31 )) ] && break
+ SHIFT=$(( $SHIFT + 1 ))
+ done
+ MUL32=$( printf %x $MUL32 )
+
+ # ADJ32 is just (((FROM/GCD)-1)<<SHIFT)/(FROM/GCD) but this
+ # can overflow 64 bit math (examples, HZ=24 or HZ=122).
+ # Pathological case could use 32+20+20=72 bits. (And this is
+ # the pathological case because a larger $HZ results in a
+ # smaller $SHIFT, so even insane HZ>USEC cases should be ok.)
+
+ # To get around this, we chop the bottom 32 bits off the
+ # calculation and then reassemble it to avoid overflow:
+ # 32+64=96, which is > 72.
+
+ ADJ32=$(( $FROM / $GCD ))
+ if [ $SHIFT -gt 32 ]
+ then
+ UPPER=$(( ( $ADJ32 - 1 ) << ( $SHIFT - 32 ) ))
+ LOWER=$(( ( $UPPER % $ADJ32 ) << 32 ))
+ ADJ32=$(( ( ( $UPPER / $ADJ32 ) << 32 ) + ( $LOWER / $ADJ32 )))
+ else
+ ADJ32=$(( ( ( $ADJ32 - 1 ) << $SHIFT) / $ADJ32 ))
+ fi
+ ADJ32=$( printf %x $ADJ32 )
+
+ NUM=$(( $TO / $GCD ))
+ DEN=$(( $FROM / $GCD ))
+
+ # Output next chunk of header data to file
+
+ (
+ echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)" &&
+ echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)" &&
+ echo "#define ${CONVERT}_SHR32 $SHIFT" &&
+ echo "#define ${CONVERT}_NUM U64_C($NUM)" &&
+ echo "#define ${CONVERT}_DEN U64_C($DEN)"
+ ) >> $FILENAME || exit 1
+ done
+done
+
+(
+ echo
+ echo "#endif /* __KERNEL_TIMECHONST_H */"
+) >> $FILENAME || exit 1
+
+# Don't rm $FILENAME on exit anymore.
+
+trap "" EXIT
+
+exit 0
--- linux-2.6.28/kernel/Makefile 2008-12-24 17:26:37.000000000 -0600
+++ linux-2.6.28-new/kernel/Makefile 2009-01-03 00:40:21.000000000 -0600
@@ -116,7 +116,7 @@
$(obj)/time.o: $(obj)/timeconst.h
quiet_cmd_timeconst = TIMEC $@
- cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@
+ cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) $@
targets += timeconst.h
-$(obj)/timeconst.h: $(src)/timeconst.pl FORCE
+$(obj)/timeconst.h: $(src)/timeconst.sh FORCE
$(call if_changed,timeconst)
\0
^ permalink raw reply
* PATCH [2/3]: Remove perl from make headers_install.
From: Rob Landley @ 2009-01-04 1:28 UTC (permalink / raw)
To: Embedded Linux mailing list
Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg
In-Reply-To: <200901031924.15869.rob@landley.net>
From: Rob Landley <rob@landley.net>
Remove perl from make headers_install by replacing a perl script (doing
a simple regex search and replace) with a smaller and faster shell script
implementation. The new shell script is a single for loop calling sed and
piping its output through unifdef to produce the target file.
Changes from previous version: Added help text and a check for the right
number of arguments. Removed unused ARCH input from script and makefile
(the makefile incorporates ARCH into INDIR, so the script doesn't care),
fixed a whitespace mistake in the makefile pointed out by Sam Ravnborg,
changed the shebang to #!/bin/sh and tested under bash and dash.
put_changelog_here
Signed-off-by: Rob Landley <rob@landley.net>
---
scripts/Makefile.headersinst | 6 ++--
scripts/headers_install.pl | 46 ---------------------------------
scripts/headers_install.sh | 36 +++++++++++++++++++++++++
3 files changed, 39 insertions(+), 49 deletions(-)
diff -ruN linux-2.6.28/scripts/headers_install.sh linux-2.6.28-new/scripts/headers_install.sh
--- linux-2.6.28/scripts/headers_install.sh 1969-12-31 18:00:00.000000000 -0600
+++ linux-2.6.28-new/scripts/headers_install.sh 2009-01-02 22:35:17.000000000 -0600
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+if [ $# -lt 2 ]
+then
+ echo "Usage: headers_install.sh INDIR OUTDIR [FILES...]
+ echo
+ echo "Prepares kernel header files for use by user space, by removing"
+ echo "all compiler.h definitions and #includes, and removing any"
+ echo "#ifdef __KERNEL__ sections."
+ echo
+ echo "INDIR: directory to read each kernel header FILE from."
+ echo "OUTDIR: directory to write each userspace header FILE to."
+ echo "FILES: list of header files to operate on."
+
+ exit 1
+fi
+
+# Grab arguments
+
+INDIR="$1"
+shift
+OUTDIR="$1"
+shift
+
+# Iterate through files listed on command line
+
+for i in "$@"
+do
+ sed -r \
+ -e 's/([ \t(])(__user|__force|__iomem)[ \t]/\1/g' \
+ -e 's/__attribute_const__([ \t]|$)/\1/g' \
+ -e 's@^#include <linux/compiler.h>@@' "$INDIR/$i" |
+ scripts/unifdef -U__KERNEL__ - > "$OUTDIR/$i"
+done
+
+exit 0
diff -ruN linux-2.6.28/scripts/Makefile.headersinst linux-2.6.28-new/scripts/Makefile.headersinst
--- linux-2.6.28/scripts/Makefile.headersinst 2008-12-24 17:26:37.000000000 -0600
+++ linux-2.6.28-new/scripts/Makefile.headersinst 2009-01-02 22:36:42.000000000 -0600
@@ -44,8 +44,8 @@
quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
file$(if $(word 2, $(all-files)),s))
cmd_install = \
- $(PERL) $< $(srctree)/$(obj) $(install) $(SRCARCH) $(header-y); \
- $(PERL) $< $(objtree)/$(obj) $(install) $(SRCARCH) $(objhdr-y); \
+ $(CONFIG_SHELL) $< $(srctree)/$(obj) $(install) $(header-y); \
+ $(CONFIG_SHELL) $< $(objtree)/$(obj) $(install) $(objhdr-y); \
touch $@
quiet_cmd_remove = REMOVE $(unwanted)
@@ -64,7 +64,7 @@
@:
targets += $(install-file)
-$(install-file): scripts/headers_install.pl $(input-files) FORCE
+$(install-file): scripts/headers_install.sh $(input-files) FORCE
$(if $(unwanted),$(call cmd,remove),)
$(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
$(call if_changed,install)
diff -ruN linux-2.6.28/scripts/headers_install.pl linux-2.6.28-new/scripts/headers_install.pl
--- linux-2.6.28/scripts/headers_install.pl 2008-12-24 17:26:37.000000000 -0600
+++ linux-2.6.28-new/scripts/headers_install.pl 1969-12-31 18:00:00.000000000 -0600
@@ -1,46 +0,0 @@
-#!/usr/bin/perl -w
-#
-# headers_install prepare the listed header files for use in
-# user space and copy the files to their destination.
-#
-# Usage: headers_install.pl readdir installdir arch [files...]
-# readdir: dir to open files
-# installdir: dir to install the files
-# arch: current architecture
-# arch is used to force a reinstallation when the arch
-# changes because kbuild then detect a command line change.
-# files: list of files to check
-#
-# Step in preparation for users space:
-# 1) Drop all use of compiler.h definitions
-# 2) Drop include of compiler.h
-# 3) Drop all sections defined out by __KERNEL__ (using unifdef)
-
-use strict;
-
-my ($readdir, $installdir, $arch, @files) = @ARGV;
-
-my $unifdef = "scripts/unifdef -U__KERNEL__";
-
-foreach my $file (@files) {
- local *INFILE;
- local *OUTFILE;
- my $tmpfile = "$installdir/$file.tmp";
- open(INFILE, "<$readdir/$file")
- or die "$readdir/$file: $!\n";
- open(OUTFILE, ">$tmpfile") or die "$tmpfile: $!\n";
- while (my $line = <INFILE>) {
- $line =~ s/([\s(])__user\s/$1/g;
- $line =~ s/([\s(])__force\s/$1/g;
- $line =~ s/([\s(])__iomem\s/$1/g;
- $line =~ s/\s__attribute_const__\s/ /g;
- $line =~ s/\s__attribute_const__$//g;
- $line =~ s/^#include <linux\/compiler.h>//;
- printf OUTFILE "%s", $line;
- }
- close OUTFILE;
- close INFILE;
- system $unifdef . " $tmpfile > $installdir/$file";
- unlink $tmpfile;
-}
-exit 0;
\0
^ permalink raw reply
* PATCH [3/3]: Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh
From: Rob Landley @ 2009-01-04 1:29 UTC (permalink / raw)
To: Embedded Linux mailing list
Cc: linux-kernel, Andrew Morton, H. Peter Anvin, Sam Ravnborg
In-Reply-To: <200901031924.15869.rob@landley.net>
From: Rob Landley <rob@landley.net>
Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh.
This script generates kernel/cpu/capflags.c from include/asm/cpufeature.h.
Changes from last time: changed shebang to #!/bin/sh and tested under bash
and dash.
Signed-off-by: Rob Landley <rob@landley.net>
---
arch/x86/kernel/cpu/Makefile | 4 +--
arch/x86/kernel/cpu/mkcapflags.pl | 32 ----------------------------
arch/x86/kernel/cpu/mkcapflags.sh | 28 ++++++++++++++++++++++++
3 files changed, 30 insertions(+), 34 deletions(-)
diff -ruN linux-2.6.28/arch/x86/kernel/cpu/Makefile linux-2.6.28-new/arch/x86/kernel/cpu/Makefile
--- linux-2.6.28/arch/x86/kernel/cpu/Makefile 2008-12-24 17:26:37.000000000 -0600
+++ linux-2.6.28-new/arch/x86/kernel/cpu/Makefile 2009-01-02 01:10:00.000000000 -0600
@@ -23,10 +23,10 @@
obj-$(CONFIG_X86_LOCAL_APIC) += perfctr-watchdog.o
quiet_cmd_mkcapflags = MKCAP $@
- cmd_mkcapflags = $(PERL) $(srctree)/$(src)/mkcapflags.pl $< $@
+ cmd_mkcapflags = $(CONFIG_SHELL) $(srctree)/$(src)/mkcapflags.sh $< $@
cpufeature = $(src)/../../include/asm/cpufeature.h
targets += capflags.c
-$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.pl FORCE
+$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.sh FORCE
$(call if_changed,mkcapflags)
diff -ruN linux-2.6.28/arch/x86/kernel/cpu/mkcapflags.pl linux-2.6.28-new/arch/x86/kernel/cpu/mkcapflags.pl
--- linux-2.6.28/arch/x86/kernel/cpu/mkcapflags.pl 2008-12-24 17:26:37.000000000 -0600
+++ linux-2.6.28-new/arch/x86/kernel/cpu/mkcapflags.pl 1969-12-31 18:00:00.000000000 -0600
@@ -1,32 +0,0 @@
-#!/usr/bin/perl
-#
-# Generate the x86_cap_flags[] array from include/asm-x86/cpufeature.h
-#
-
-($in, $out) = @ARGV;
-
-open(IN, "< $in\0") or die "$0: cannot open: $in: $!\n";
-open(OUT, "> $out\0") or die "$0: cannot create: $out: $!\n";
-
-print OUT "#include <asm/cpufeature.h>\n\n";
-print OUT "const char * const x86_cap_flags[NCAPINTS*32] = {\n";
-
-while (defined($line = <IN>)) {
- if ($line =~ /^\s*\#\s*define\s+(X86_FEATURE_(\S+))\s+(.*)$/) {
- $macro = $1;
- $feature = $2;
- $tail = $3;
- if ($tail =~ /\/\*\s*\"([^"]*)\".*\*\//) {
- $feature = $1;
- }
-
- if ($feature ne '') {
- printf OUT "\t%-32s = \"%s\",\n",
- "[$macro]", "\L$feature";
- }
- }
-}
-print OUT "};\n";
-
-close(IN);
-close(OUT);
diff -ruN linux-2.6.28/arch/x86/kernel/cpu/mkcapflags.sh linux-2.6.28-
new/arch/x86/kernel/cpu/mkcapflags.sh
--- linux-2.6.28/arch/x86/kernel/cpu/mkcapflags.sh 1969-12-31 18:00:00.000000000 -0600
+++ linux-2.6.28-new/arch/x86/kernel/cpu/mkcapflags.sh 2009-01-02 01:10:00.000000000 -0600
@@ -0,0 +1,28 @@
+#!/bin/sh
+#
+# Generate the x86_cap_flags[] array from include/asm/cpufeature.h
+#
+
+IN=$1
+OUT=$2
+
+(
+ echo "#include <asm/cpufeature.h>"
+ echo ""
+ echo "const char * const x86_cap_flags[NCAPINTS*32] = {"
+
+ # Iterate through any input lines starting with #define X86_FEATURE_
+ sed -n -e 's/\t/ /g' -e 's/^ *# *define *X86_FEATURE_//p' $IN |
+ while read i
+ do
+ # Name is everything up to the first whitespace
+ NAME="$(echo "$i" | sed 's/ .*//')"
+
+ # If the /* comment */ starts with a quote string, grab that.
+ VALUE="$(echo "$i" | sed -n 's@.*/\* *\("[^"]*"\).*\*/@\1@p')"
+ [ -z "$VALUE" ] && VALUE="\"$(echo "$NAME" | tr A-Z a-z)\""
+
+ [ "$VALUE" != '""' ] && echo " [X86_FEATURE_$NAME] = $VALUE,"
+ done
+ echo "};"
+) > $OUT
\0
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Rob Landley @ 2009-01-04 1:32 UTC (permalink / raw)
To: Matthieu CASTET
Cc: Arkadiusz Miskiewicz, linux-kernel, Embedded Linux mailing list,
Andrew Morton, H. Peter Anvin, Sam Ravnborg
In-Reply-To: <495E3AF8.6000006@parrot.com>
On Friday 02 January 2009 10:04:08 Matthieu CASTET wrote:
> Rob Landley a écrit :
> > On Friday 02 January 2009 03:26:37 Arkadiusz Miskiewicz wrote:
> >> On Friday 02 of January 2009, Rob Landley wrote:
> >>
> >> Heh,
> >
> > I believe all three scripts run under dash and busybox ash. (The
> > timeconst.sh one needs 64 bit math which dash only provides on 64 bit
> > hosts, which is a regression from Red Hat 9 in 2003 by the way.
>
> With dash 0.5.4-12 (from debian sid), I seems I got the 64 bit math for
> 32 bit hosts :
> $ uname -m
> i686
> $ dash -c 'echo $((1<<32))'
> 4294967296
>
>
> Matthieu
Alas, my attempt to install a 32 bit version of xubuntu 8.10 under qemu hung
at "Scanning files: 15%", and has been there for an hour now. I'll have to
take your word for it. (All three scripts work fine under 64 bit dash.)
I encountered one bug in busybox, which I pinged that list about, but
otherwise busybox ash works on 'em all too.
Rob
^ permalink raw reply
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
From: Rob Landley @ 2009-01-04 1:32 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Sam Ravnborg, Embedded Linux mailing list, linux-kernel,
Andrew Morton
In-Reply-To: <495E6BEE.1000805@zytor.com>
On Friday 02 January 2009 13:33:02 H. Peter Anvin wrote:
> Rob Landley wrote:
> > You mean "The new shell script is much simpler, about 1/4 the size, runs
> > on Red Hat 9 from 2003, and isn't perl?" :)
>
> And introduces unclear environment dependencies depending on how
> external utilities are implemented.
I note that sed and printf and such are all susv3. I have an explicit test
for 32 bit math in the script that cares, and this worked in Red Hat 9 circa
2003.
I consider this a step up from code with an implicit dependency on a CPAN
library.
> The whole point of why that script was written in Perl was to have
> access to arbitrary-precision arithmetic -- after it was shown that bc
> would simply lock up on some systems.
A) I'm not using bc.
B) You don't need arbitrary precision arithmetic, you need around 72 bits
worth of arithmetic for the pathological case.
C) Your definition of "access to arbitrary-precision arithmetic" includes the
following, cut and paste verbatim from your old script:
# Precomputed values for systems without Math::BigInt
# Generated by:
# timeconst.pl --can 24 32 48 64 100 122 128 200 250 256 300 512 1000 1024
1200
%canned_values = (
24 => [
'0xa6aaaaab','0x2aaaaaa',26,
125,3,
'0xc49ba5e4','0x1fbe76c8b4',37,
3,125,
'0xa2c2aaab','0xaaaa',16,
125000,3,
'0xc9539b89','0x7fffbce4217d',47,
3,125000,
], 32 => [
'0xfa000000','0x6000000',27,
125,4,
'0x83126e98','0xfdf3b645a',36,
4,125,
'0xf4240000','0x0',17,
31250,1,
'0x8637bd06','0x3fff79c842fa',46,
1,31250,
], 48 => [
'0xa6aaaaab','0x6aaaaaa',27,
125,6,
'0xc49ba5e4','0xfdf3b645a',36,
6,125,
'0xa2c2aaab','0x15555',17,
62500,3,
'0xc9539b89','0x3fffbce4217d',46,
3,62500,
], 64 => [
'0xfa000000','0xe000000',28,
125,8,
'0x83126e98','0x7ef9db22d',35,
8,125,
'0xf4240000','0x0',18,
15625,1,
'0x8637bd06','0x1fff79c842fa',45,
1,15625,
], 100 => [
'0xa0000000','0x0',28,
10,1,
'0xcccccccd','0x733333333',35,
1,10,
'0x9c400000','0x0',18,
10000,1,
'0xd1b71759','0x1fff2e48e8a7',45,
1,10000,
], 122 => [
'0x8325c53f','0xfbcda3a',28,
500,61,
'0xf9db22d1','0x7fbe76c8b',35,
61,500,
'0x8012e2a0','0x3ef36',18,
500000,61,
'0xffda4053','0x1ffffbce4217',45,
61,500000,
], 128 => [
'0xfa000000','0x1e000000',29,
125,16,
'0x83126e98','0x3f7ced916',34,
16,125,
'0xf4240000','0x40000',19,
15625,2,
'0x8637bd06','0xfffbce4217d',44,
2,15625,
], 200 => [
'0xa0000000','0x0',29,
5,1,
'0xcccccccd','0x333333333',34,
1,5,
'0x9c400000','0x0',19,
5000,1,
'0xd1b71759','0xfff2e48e8a7',44,
1,5000,
], 250 => [
'0x80000000','0x0',29,
4,1,
'0x80000000','0x180000000',33,
1,4,
'0xfa000000','0x0',20,
4000,1,
'0x83126e98','0x7ff7ced9168',43,
1,4000,
], 256 => [
'0xfa000000','0x3e000000',30,
125,32,
'0x83126e98','0x1fbe76c8b',33,
32,125,
'0xf4240000','0xc0000',20,
15625,4,
'0x8637bd06','0x7ffde7210be',43,
4,15625,
], 300 => [
'0xd5555556','0x2aaaaaaa',30,
10,3,
'0x9999999a','0x1cccccccc',33,
3,10,
'0xd0555556','0xaaaaa',20,
10000,3,
'0x9d495183','0x7ffcb923a29',43,
3,10000,
], 512 => [
'0xfa000000','0x7e000000',31,
125,64,
'0x83126e98','0xfdf3b645',32,
64,125,
'0xf4240000','0x1c0000',21,
15625,8,
'0x8637bd06','0x3ffef39085f',42,
8,15625,
], 1000 => [
'0x80000000','0x0',31,
1,1,
'0x80000000','0x0',31,
1,1,
'0xfa000000','0x0',22,
1000,1,
'0x83126e98','0x1ff7ced9168',41,
1,1000,
], 1024 => [
'0xfa000000','0xfe000000',32,
125,128,
'0x83126e98','0x7ef9db22',31,
128,125,
'0xf4240000','0x3c0000',22,
15625,16,
'0x8637bd06','0x1fff79c842f',41,
16,15625,
], 1200 => [
'0xd5555556','0xd5555555',32,
5,6,
'0x9999999a','0x66666666',31,
6,5,
'0xd0555556','0x2aaaaa',22,
2500,3,
'0x9d495183','0x1ffcb923a29',41,
3,2500,
]
);
Plus a decent chunk of the remaining logic was code to regenerate that table,
and to figure out when to use the table and when to compute new values. (And
erroring out if the system wasn't capable of doing so.) I don't understand
why you didn't just precompute the actual header file output instead or
precomputing perl source, but that's a side issue.
Rob
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Rob Landley @ 2009-01-04 1:35 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Sam Ravnborg, Embedded Linux mailing list, Paul Mundt,
linux-kernel, Andrew Morton
In-Reply-To: <495E6AB1.2060707@zytor.com>
On Friday 02 January 2009 13:27:45 H. Peter Anvin wrote:
> Sam Ravnborg wrote:
> > Hi Wookey.
> >
> >> Given the
> >> simplicitly of these patches I can't see any reason not to put them
> >> in
> >
> > Please do NOT do the mistake and think this the same thing.
> >
> > Rob's patch simplyfy the timecost stuff - and will be applied on
> > this merit alone assuming comments will be addressed.
> >
> > But the serie rased anohter topic: shall we ban use of perl
> > for generating a kernel.
> > And this is what primary is discussed and the outcome of
> > that discussion will not prevent patches that stands on their
> > own to be applied.
>
> My personal opinion on this is that this is ridiculous. Given that you
> need gcc, binutils, make etc. to build the kernel,
I believe Intel's icc builds the kernel, and tinycc previously built a subset
of the kernel. The pcc and llvm/clang projects are getting close to being
able to build the kernel. Ever since c99 came out, lots of gcc-isms with c99
equivalents have been swapped over, most of the rest is testing.
> and this is more than
> inherent, you have to have a pretty bloody strangely constrained system
> to disallow Perl, which is as close to a standard Unix utility you can
> get without making it into SuS.
Please show me A) the standard perl implements, B) the second implementation
of that standard ala IETF guidelines.
> The only *real* motivation I have seen for this is a system that as far
> I can tell is nothing other than a toy, specifically designed to show
> off how little you need to build the kernel. In other words, it's not a
> practical application, it's a show-off art piece.
I'm glad you think my Firmware Linux project is a work of art, but if you'd
like to hear directly from my users I can ask them to complain at you in
person, if you like. I'm not sure what that would prove, though.
When cross compiling, it's good to have as constrained an environment as
possible, because otherwise bits of the host system leak into the target
system. If you don't tightly control your cross compiling environment, it
won't work. That's just about an axiom in embedded development.
I know every single dependency my system has. I can list them, explicitly. I
did this because it's very _USEFUL_ in this context.
Add perl scripts that call cpan, and this is no longer true.
> -hpa
Rob
^ permalink raw reply
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
From: H. Peter Anvin @ 2009-01-04 1:35 UTC (permalink / raw)
To: Rob Landley
Cc: Sam Ravnborg, Embedded Linux mailing list, linux-kernel,
Andrew Morton
In-Reply-To: <200901031932.51873.rob@landley.net>
Rob Landley wrote:
>
> I consider this a step up from code with an implicit dependency on a CPAN
> library.
>
There is no CPAN library in use. Math::BigInt is a standard part of
Perl, and the canned values is there only to support extremely old
versions of Perl, or weird system configurations, as a
belt-and-suspenders measure.
-hpa
^ permalink raw reply
* Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh
From: Rob Landley @ 2009-01-04 1:36 UTC (permalink / raw)
To: Ingo Oeser
Cc: Embedded Linux mailing list, linux-kernel, Andrew Morton,
H. Peter Anvin, Sam Ravnborg
In-Reply-To: <200901031328.23079.ioe-lkml@rameria.de>
On Saturday 03 January 2009 06:28:22 Ingo Oeser wrote:
> > +for i in "MSEC 1000" "USEC 1000000"
> > +do
> > + NAME=$(echo $i | awk '{print $1}')
>
> cut -d' ' -f1 does the same
>
> > + PERIOD=$(echo $i | awk '{print $2}')
>
> cut -d' ' -f2 does the same
From a standards perspective
http://www.opengroup.org/onlinepubs/9699919799/utilities/cut.html vs
http://www.opengroup.org/onlinepubs/9699919799/utilities/awk.html is probably
a wash, but from a simplicity perspective using the tool that _isn't_ its own
programming language is probably a win. :)
I made the change in the second round of patches I just posted.
Thanks,
Rob
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: David Brownell @ 2009-01-04 1:39 UTC (permalink / raw)
To: Robert Hancock
Cc: Rob Landley, Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel,
Embedded Linux mailing list, Andrew Morton, H. Peter Anvin,
Sam Ravnborg
In-Reply-To: <4960068A.3040109@shaw.ca>
On Saturday 03 January 2009, Robert Hancock wrote:
> Rob Landley wrote:
> > ... some architectures (arm omap and and arm at91)
> > allow you to enter arbitrary HZ values in kconfig. (Their help text says that
> > in many cases values that aren't powers of two won't work, but nothing
> > enforces this.)
>
> Is there a good reason that these archs allow you enter arbitrary HZ
> values?
Power-of-two can be desirable when using a 32 KiHz oscillator, because
other values accumulate rounding errors ... you can't make 100 Hz, or
250 Hz, or 300 Hz, or 1000 Hz, by a binary division of 32 KiHz.
Other values were supported to help work around stupid software making
bad assumptions about HZ. IMO, enforcing power-of-two would be better;
that software breaks with dyntick anyway, and needs fixing.
> The use case for using custom HZ values at all nowadays seems
> fairly low now that dynticks is around (if that arch supports it
> anyway),
A better argument would be that GENERIC_TIME exists (and works
on OMAP and AT91), which avoids some flavors of rounding error.
ISTR those CONFIG_HZ options predate GENERIC_TIME support.
However, the issue remains that most kernel times are measured in
jiffies not ktime_t -- they're easier and more efficient, all
those 64-bit multiplies can hurt on ARM (32-bit, non-GHz) -- so
it's still good to be able to ensure that jiffies-centric logic
won't always be inserting easily avoidable errors.
- Dave
^ permalink raw reply
* Re: PATCH [0/3]: Simplify the kernel build by removing perl.
From: Rob Landley @ 2009-01-04 1:45 UTC (permalink / raw)
To: Sam Ravnborg
Cc: Matthieu CASTET, Arkadiusz Miskiewicz, linux-kernel,
Embedded Linux mailing list, Andrew Morton, H. Peter Anvin
In-Reply-To: <20090103201059.GA4875@uranus.ravnborg.org>
On Saturday 03 January 2009 14:10:59 Sam Ravnborg wrote:
> > I'll fix this and resubmit, it just wasn't ready last night. (If the
> > merge window is closing soon I could resubmit the other two with Sam's
> > suggestions and resubmit this one next time 'round, but it was only a
> > couple days to write in the first place, once I finally figured out what
> > the perl version was trying to _do_...)
>
> For kbuild only fixes and trivial stuff will be merged until next merge
> window. Neither of the three patches fall into that category.
*shrug* I poke my head into kernel development every few months, and have
just enough familiarity with it to remember that "changes go in during the
merge window". Seemed a good time to post 'em.
> With respect to your three patches the plan is to:
> - add the updated timeconst patch to kbuild-next
> - add the updated cpu-feature patch to kbuild-next
>
> - the patch touching headers_install will not be merged.
> The way forward for headers_install is to combine the
> unifdef feature and the header modifications.
Since you're turning down an existing patch in favor of a theoretical patch, I
assume you have plans to do this yourself?
> And this must be in a single program that can process
> all headers in one go so the install process becomes so fast
> that we do not worry about if it was done before or not.
> Then we can avoid all the .* files in the directory
> where we isntall the headers.
What if they run out of disk space halfway through writing a file and thus it
creates a short file (or a 0 length file where the dentry was created but no
blocks could be allocated for the write)?
I expected headers_install to overwrite destination files and create
directories with -p, so if you interrupt it you can just re-run it again with
the same arguments and it could install everything again cleanly over existing
partial output. I take it this isn't what's happening, or that isn't
sufficient somehow?
I didn't look too closely at what the makefile was doing (it makes my eyes
bleed), I just rewrote the perl script and changed the call. I could try to
upgrade the script to not need the makefile to tell it what files to work on,
and just take the appropriate top level include directory and the output
directory and figure out which files it needs to operate on by itself so it
_does_ work that way. Figuring out where the make file is getting this info
from now shouldn't be too much harder than reading perl scripts and figuring
out what they're doing.
Not during this merge window, though.
> The program is a prime candidate for a small C program
> and I hope someone can take the challenge to write it.
Good luck with that. Having written most of the busybox sed implementation,
and before that having written my own regex implementation back under OS/2,
I've pretty much gotten my fill of doing regexes in C, at least without a good
reason.
I suppose if I was feeling really bored I could try to implement unifdef as a
sed script, but the only way to get a single invocation of sed to work on a
bunch of individual files coherently is to use -i mode, which A) ain't in
susv3 (although busybox sed supports it), B) would involve a cp of all the
files to the destination first, which is kind of ugly.
> Migrating from perl to shell does not help us here
> and the shell version was less readable than the perl version.
The shell version is less readable but you never noticed that the perl version
isn't using its $ARCH argument? *shrug* Ok.
I can try to make the shell version more readable, and more powerful. It's
already noticeably faster than the perl version. I have no objections to
making unifdef do all of this, I just haven't got any interest either.
> Sam
Rob
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox