is it desirable to improve the build system?

is it desirable to improve the build system?

[Mark Galeck]

Dear Linux-Kernel Community,

I am a consultant specializing in builds, and I recently worked for a large client company, a world-wide leader in its field, where I overhauled their build system: sped it up by more of an order of magnitude, and improved maintainability, for example making comment-to-code ratio approach 1:1. 

A part of their build is a modified Linux kernel. I rebuilt it countless times in various configurations, but made only a few further changes, because those improvements would have a small effect on the whole system, and because they want to stay close to your current release for ease of porting.

>From that limited experience, it nevertheless seemed to me, that the Linux kernel build, while correct, is somewhat slow, and the sources could be more readable.

Does the Linux-Kernel Community perceive that is the case?

If so, do you think it is possible to improve?

If so, would such an attempt be welcome, including and especially by, the current maintainer(s) of the build?  Of course it would have to be completely backwards-compatible, including to the text output interface and requirements for modules makefiles.

I do apologize if my impressions are simply the result of unfamilliarity and naivete, and that I don't understand the deep reasons why "it has to be this way", and that I am unaware that such attempts were already made by some very skilled people.  

Thank you,

Mark

Re: is it desirable to improve the build system?

[Greg KH]

On Mon, Jul 01, 2013 at 05:12:01PM -0700, Mark Galeck wrote:
> Dear Linux-Kernel Community,
> 
> I am a consultant specializing in builds, and I recently worked for a
> large client company, a world-wide leader in its field, where I
> overhauled their build system: sped it up by more of an order of
> magnitude, and improved maintainability, for example making
> comment-to-code ratio approach 1:1. 
> 
> A part of their build is a modified Linux kernel. I rebuilt it
> countless times in various configurations, but made only a few further
> changes, because those improvements would have a small effect on the
> whole system, and because they want to stay close to your current
> release for ease of porting.
> 
> From that limited experience, it nevertheless seemed to me, that the
> Linux kernel build, while correct, is somewhat slow, and the sources
> could be more readable.

How is it "slow"?  And it has to be correct, fast and non-correct
doesn't work well, does it :)

What "sources" are you referring to as being not readable?

> Does the Linux-Kernel Community perceive that is the case?
> 
> If so, do you think it is possible to improve?
> 
> If so, would such an attempt be welcome, including and especially by,
> the current maintainer(s) of the build?  Of course it would have to be
> completely backwards-compatible, including to the text output
> interface and requirements for modules makefiles.

Patches are always gladly accepted, if they work well, please feel free
to submit them.

> I do apologize if my impressions are simply the result of
> unfamilliarity and naivete, and that I don't understand the deep
> reasons why "it has to be this way", and that I am unaware that such
> attempts were already made by some very skilled people.  

What do you not understand that you think could be changed?

Have you looked at the history of the build code to help understand why
things were changed to be they way they are?  git should help you out
here.

thanks,

greg k-h

Re: is it desirable to improve the build system?

[Mark Galeck]

>> Linux kernel build, while correct, is somewhat slow, and the sources

>> could be more readable.

Greg wrote:
>How is it "slow"?  

Well, the proportion of time spent by the CPU cores on activities other than compiling seemed high.


>What "sources" are you referring to as being not readable?

Not "not readable at all", just "could be made more readable" - the main Makefile and other main Make files; it seemed that one reason, IMHO, was not enough comments. 


>What do you not understand that you think could be changed?

I did not make careful notes in that area (details would come back to me if I look at this carefully), but right now I remember two things.

As every child in kindergarten knows, recursive make is bad (except when it is good, which you learn in primary school). One reason is that all that re-parsing costs time. Linux kernel build is very heavy recursive.

Frequent use of FORCE phony prerequisites to circumvent the normal GNU Make recipe avoidance mechanism, and then using a custom recipe mechanism to decide what to execute, seems to go against the philosophy of the tool being used (GNU Make) and as such seems, IMHO, to also waste time.

Of course if one were to attempt a change, the first thing would be to do look carefully at the amount of time spent on such activities, rather than using words such as "seems". 


What I don't understand of course is the reasons behind these choices have been made.


>Have you looked at the history of the build code to help understand why
things were changed to be they way they are?  git should help you out
here.

No. IMHO, looking at source, and especially past history thereof, to understand it, is a very inefficient use of one's time, only to be undertaken if all else fails. Much better is looking at comments and documentation, if available, and also, asking well-informed persons such as yourself.  


Mark

Re: is it desirable to improve the build system?

[Greg KH]

On Tue, Jul 02, 2013 at 01:46:45AM -0700, Mark Galeck wrote:
> >> Linux kernel build, while correct, is somewhat slow, and the sources
> 
> >> could be more readable.
> 
> Greg wrote:
> >How is it "slow"?  
> 
> Well, the proportion of time spent by the CPU cores on activities
> other than compiling seemed high.

Really?  What is that proportion?

> >What "sources" are you referring to as being not readable?
> 
> Not "not readable at all", just "could be made more readable" - the
> main Makefile and other main Make files; it seemed that one reason,
> IMHO, was not enough comments. 

Everyone always wants more comments :)

Have you read the documentation about the build system that we have in
the kernel Documentation/ directory?

There is also a mailing list just for the build system as well, see the
MAINTAINERS file for details.

greg k-h

Re: is it desirable to improve the build system?

[Pavel Machek]

Hi!
> 
> Does the Linux-Kernel Community perceive that is the case?
> 
> If so, do you think it is possible to improve?
> 
> If so, would such an attempt be welcome, including and especially by, the current maintainer(s) of the build?  Of course it would have to be completely backwards-compatible, including to the text output interface and requirements for modules makefiles.
> 

Yes, speeding up build would be nice. Especially nice would be
speeding up "empty" build, when only one or two files changed...

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: is it desirable to improve the build system?

[Bjorn Helgaas]

On Thu, Jul 11, 2013 at 5:38 AM, Pavel Machek <pavel@ucw.cz> wrote:
> Hi!
>>
>> Does the Linux-Kernel Community perceive that is the case?
>>
>> If so, do you think it is possible to improve?
>>
>> If so, would such an attempt be welcome, including and especially by, the current maintainer(s) of the build?  Of course it would have to be completely backwards-compatible, including to the text output interface and requirements for modules makefiles.
>>
>
> Yes, speeding up build would be nice. Especially nice would be
> speeding up "empty" build, when only one or two files changed...

The answer to "is it desirable to improve X?" is always "yes."  But
the only way to make progress in Linux is to actually post patches
that "improve X."  This is unlike many corporate environments, where
you might need to get somebody's approval before you even start
trying.  In Linux, there's no such thing as prior approval.  Even if
you could get somebody to say "this looks like a good thing to try,"
that would be no guarantee that subsequent patches would be accepted.

It's hard enough to get people to comment on actual Linux code
changes, i.e., patches.  It's almost impossible to get useful feedback
on proposals without code.

If you posted patches that made significant improvements in build
time, I think people would be very interested.

Bjorn

Re: is it desirable to improve the build system?

[Mark Galeck]

>The answer to "is it desirable to improve X?" is always "yes."  But

the only way to make progress in Linux is to actually post patches
that "improve X."  This is unlike many corporate environments, where
you might need to get somebody's approval


Precisely. Please excuse me coming from a corporate background. 
In corporate, even if you "get somebody's approval", there may be "other stakeholders" who may "feel ownership" etc etc


Thank you

Mark

Re: is it desirable to improve the build system?

[Richard Cochran]

On Thu, Jul 11, 2013 at 02:22:14PM -0700, Mark Galeck wrote:
> >The answer to "is it desirable to improve X?" is always "yes."  But
> 
> the only way to make progress in Linux is to actually post patches
> that "improve X."  This is unlike many corporate environments, where
> you might need to get somebody's approval
> 
> 
> Precisely. Please excuse me coming from a corporate background. 
> In corporate, even if you "get somebody's approval", there may be "other stakeholders" who may "feel ownership" etc etc

Yes, your approved project may get canned later. But the point was,
that you aren't allowed to even get started without someone's
approval.

Thanks,
Richard