From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stephen Warren Date: Mon, 11 Apr 2016 11:07:29 -0600 Subject: [U-Boot] [PATCH] buildman: allow more incremental building In-Reply-To: References: <1459204775-23510-1-git-send-email-swarren@wwwdotorg.org> <20160401133534.GW23166@bill-the-cat> <570BCDBA.2020508@wwwdotorg.org> Message-ID: <570BD9D1.3050806@wwwdotorg.org> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: u-boot@lists.denx.de On 04/11/2016 10:58 AM, Simon Glass wrote: > Hi Stephen, > > On 11 April 2016 at 10:15, Stephen Warren wrote: >> On 04/11/2016 06:04 AM, Simon Glass wrote: >>> >>> Hi, >>> >>> On 1 April 2016 at 07:35, Tom Rini wrote: >>>> >>>> On Mon, Mar 28, 2016 at 04:39:35PM -0600, Stephen Warren wrote: >>>> >>>>> From: Stephen Warren >>>>> >>>>> One use-case for buildman is to continually run it interactively after >>>>> each small step in a large refactoring operation. This gives more >>>>> immediate feedback than making a number of commits and then going back >>>>> and >>>>> testing them. For this to work well, buildman needs to be extremely >>>>> fast. >>>>> At present, a couple issues prevent it being as fast as it could be: >>>>> >>>>> 1) Each time buildman runs "make %_defconfig", it runs "make mrproper" >>>>> first. This throws away all previous build results, requiring a >>>>> from-scratch build. Optionally avoiding this would speed up the build, >>>>> at >>>>> the cost of potentially causing or missing some build issues. >>> >>> >>> Does it actually cause problems? I think I have seen things go wrong >>> before, which is why I added it, but I can't remember the detail. >> >> >> I have not noticed any issues using -I and hence skipping the mrproper. It >> is possible you added the mrproper before the conversion to Kbuild; Kbuild >> seems to handle rebuilding after code changes much better than the old >> system, which definitely had some flaws that prevented incremental builds in >> many cases. > > OK I see. For next release we should consider making -I the default. >> >>> Threads work by configuring and building the first commit for a board, >>> then incrementally building each commit for that same board. So it >>> seems like change should only be useful when all the boards you are >>> building have a similar config. >> >> >> No, because of point 2 below. >> >>>>> 2) A build tree is created per thread rather than per board. When a >>>>> thread >>>>> switches between building different boards, this often causes many files >>>>> to be rebuilt due to changing config options. Using a separate build >>>>> tree >>>>> for each board would avoid this. This does put more strain on the >>>>> system's >>>>> disk cache, but it is worth it on my system at least. >>> >>> >>> I'm not sure why this is a win, given what I said above. I'm starting >>> to feel that I don't understand what is going on. >> >> >> Since each board gets built in a separate directory, there's no build churn >> due to the configuration being changed when threads switch between building >> different boards. Right now what happens on a thread (even without mrproper) >> is: >> >> Thread 1 builds board X >> - Some stuff gets rebuilt due to source changes >> Thread 1 builds board Y >> - Some stuff gets rebuilt due to source changes >> - Lots more gets rebuilt because the config changed between the two boards, >> and bother were built in the same directory. >> >> After this patch, the following happens: >> >> Thread 1 builds board X >> - Some stuff gets rebuilt due to source changes >> Thread 1 builds board Y >> - Some stuff gets rebuilt due to source changes >> - Nothing else gets rebuilt >> >> If the source changes happen not to affect some particular board, then this >> new scheme means that nothing gets rebuilt for it, which significantly >> speeds up the build. > > I understand what you are saying, but the intention of > builderthread.RunJob is to avoid this churn. It runs in a loop through > all commits, building the same board for each. What am I missing? Each thread can build multiple boards in turn. This is not a problem building a series of commits for a board, but rather a problem when building multiple boards in turn. Also, see V2 of the patch which adds a longer explanation to the README.