From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.lixom.net ([70.86.134.90]:48770 "EHLO mail.lixom.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752001Ab0GMSKy (ORCPT ); Tue, 13 Jul 2010 14:10:54 -0400 Date: Tue, 13 Jul 2010 13:04:02 -0500 From: Olof Johansson Subject: Re: optimized script [Was: ARM defconfig files] Message-ID: <20100713180402.GA1422@lixom.net> References: <20100712173228.GC9897@n2100.arm.linux.org.uk> <20100712185029.GB14425@pengutronix.de> <20100713070741.GB26442@pengutronix.de> <20100713080705.GA20978@pengutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20100713080705.GA20978@pengutronix.de> Sender: linux-arm-msm-owner@vger.kernel.org List-ID: To: Uwe =?iso-8859-1?Q?Kleine-K=F6nig?= Cc: Linus Torvalds , Grant Likely , Nicolas Pitre , Daniel Walker , Russell King - ARM Linux , Kevin Hilman , linux-arm-msm@vger.kernel.org, Linux Kernel Mailing List , linux-omap@vger.kernel.org, Eric Miao , linux-arm-kernel@lists.infradead.org, Stephen Rothwell , Benjamin Herrenschmidt , linuxppc-dev@lists.ozlabs.org On Tue, Jul 13, 2010 at 10:07:05AM +0200, Uwe Kleine-König wrote: > Hello, > > On Tue, Jul 13, 2010 at 09:07:41AM +0200, Uwe Kleine-König wrote: > > Hi > > > > On Mon, Jul 12, 2010 at 01:50:47PM -0600, Grant Likely wrote: > > > On Mon, Jul 12, 2010 at 1:34 PM, Linus Torvalds > > > wrote: > > > > On Mon, Jul 12, 2010 at 12:17 PM, Nicolas Pitre wrote: > > > >> I think Uwe could provide his script and add it to the kernel tree. > > > >> Then all architectures could benefit from it.  Having the defconfig > > > >> files contain only those options which are different from the defaults > > > >> is certainly more readable, even on x86. > > > > > > > > Quite possible. But maintainers would need to be on the lookout of > > > > people actually using the script, and refusing to apply patches that > > > > re-introduce the whole big thing. > > > > > > I can (partially) speak for powerpc. If ARM uses this approach, then > > > I think we can do the same. After the defconfigs are trimmed, I > > > certainly won't pick up any more full defconfigs. > > I just restarted my script on the powerpc defconfigs basing on rc5, I > > assume they complete in a few days time. > So Stephen was faster than me. I don't know yet how he optimised my > script, meanwhile I put some efforts into it, too by just checking lines > that match "^(# )?CONFIG_". > > Find it attached. > > I will start to reduce the remaining configs (i.e. all but arm and > powerpc). I added just a simple heuristic: If I could remove a line, I attempted to remove twice the amount next time around (and fall back to 1 if it failed). I.e. main loop: i = 0 lines = 1 while i < len(config): print 'test for %r + %d' % (config[i], lines) defconfig = open(defconfig_src, 'w') defconfig.writelines(config[:i]) defconfig.writelines(config[i + lines:]) defconfig.close() subprocess.check_call(['make', '-s', 'ARCH=%s' % arch, target]) if os.stat('.config').st_size == config_size and list(open('.config')) == origconfig: del config[i:i+lines] lines *= 2 else: if lines > 1: lines = 1 else: i += 1 I didn't measure what the actual improvement was, but I saw a fair amount of 2/4/8-attempts passing, so I let it run. Stephen beat me to posting the resulting patch though. :P While this script is great, it is somewhat painful to run given that it attempts one config per line. Even on a fast machine that tends to take a while. -Olof From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olof Johansson Subject: Re: optimized script [Was: ARM defconfig files] Date: Tue, 13 Jul 2010 13:04:02 -0500 Message-ID: <20100713180402.GA1422@lixom.net> References: <20100712173228.GC9897@n2100.arm.linux.org.uk> <20100712185029.GB14425@pengutronix.de> <20100713070741.GB26442@pengutronix.de> <20100713080705.GA20978@pengutronix.de> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: <20100713080705.GA20978@pengutronix.de> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-arm-kernel-bounces@lists.infradead.org Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=m.gmane.org@lists.infradead.org To: Uwe =?iso-8859-1?Q?Kleine-K=F6nig?= Cc: Stephen Rothwell , Daniel Walker , Russell King - ARM Linux , Nicolas Pitre , Kevin Hilman , linux-arm-msm@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Linux Kernel Mailing List , Grant Likely , Eric Miao , Benjamin Herrenschmidt , linux-omap@vger.kernel.org, Linus Torvalds , linux-arm-kernel@lists.infradead.org List-Id: linux-omap@vger.kernel.org On Tue, Jul 13, 2010 at 10:07:05AM +0200, Uwe Kleine-K=F6nig wrote: > Hello, > = > On Tue, Jul 13, 2010 at 09:07:41AM +0200, Uwe Kleine-K=F6nig wrote: > > Hi > > = > > On Mon, Jul 12, 2010 at 01:50:47PM -0600, Grant Likely wrote: > > > On Mon, Jul 12, 2010 at 1:34 PM, Linus Torvalds > > > wrote: > > > > On Mon, Jul 12, 2010 at 12:17 PM, Nicolas Pitre = wrote: > > > >> I think Uwe could provide his script and add it to the kernel tree. > > > >> Then all architectures could benefit from it. =A0Having the defcon= fig > > > >> files contain only those options which are different from the defa= ults > > > >> is certainly more readable, even on x86. > > > > > > > > Quite possible. But maintainers would need to be on the lookout of > > > > people actually using the script, and refusing to apply patches that > > > > re-introduce the whole big thing. > > > = > > > I can (partially) speak for powerpc. If ARM uses this approach, then > > > I think we can do the same. After the defconfigs are trimmed, I > > > certainly won't pick up any more full defconfigs. > > I just restarted my script on the powerpc defconfigs basing on rc5, I > > assume they complete in a few days time. > So Stephen was faster than me. I don't know yet how he optimised my > script, meanwhile I put some efforts into it, too by just checking lines > that match "^(# )?CONFIG_". > = > Find it attached. > = > I will start to reduce the remaining configs (i.e. all but arm and > powerpc). I added just a simple heuristic: If I could remove a line, I attempted to remove twice the amount next time around (and fall back to 1 if it faile= d). I.e. main loop: i =3D 0 lines =3D 1 while i < len(config): print 'test for %r + %d' % (config[i], lines) defconfig =3D open(defconfig_src, 'w') defconfig.writelines(config[:i]) defconfig.writelines(config[i + lines:]) defconfig.close() subprocess.check_call(['make', '-s', 'ARCH=3D%s' % arch, target]) if os.stat('.config').st_size =3D=3D config_size and list(open('.config= ')) =3D=3D origconfig: del config[i:i+lines] lines *=3D 2 else: if lines > 1: lines =3D 1 else: i +=3D 1 I didn't measure what the actual improvement was, but I saw a fair amount of 2/4/8-attempts passing, so I let it run. Stephen beat me to posting the resulting patch though. :P While this script is great, it is somewhat painful to run given that it attempts one config per line. Even on a fast machine that tends to take a while. -Olof From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.lixom.net (mail.lixom.net [70.86.134.90]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 01384B6F19 for ; Wed, 14 Jul 2010 04:10:56 +1000 (EST) Date: Tue, 13 Jul 2010 13:04:02 -0500 From: Olof Johansson To: Uwe =?iso-8859-1?Q?Kleine-K=F6nig?= Subject: Re: optimized script [Was: ARM defconfig files] Message-ID: <20100713180402.GA1422@lixom.net> References: <20100712173228.GC9897@n2100.arm.linux.org.uk> <20100712185029.GB14425@pengutronix.de> <20100713070741.GB26442@pengutronix.de> <20100713080705.GA20978@pengutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 In-Reply-To: <20100713080705.GA20978@pengutronix.de> Cc: Stephen Rothwell , Daniel Walker , Russell King - ARM Linux , Nicolas Pitre , Kevin Hilman , linux-arm-msm@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Linux Kernel Mailing List , Eric Miao , linux-omap@vger.kernel.org, Linus Torvalds , linux-arm-kernel@lists.infradead.org List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Tue, Jul 13, 2010 at 10:07:05AM +0200, Uwe Kleine-König wrote: > Hello, > > On Tue, Jul 13, 2010 at 09:07:41AM +0200, Uwe Kleine-König wrote: > > Hi > > > > On Mon, Jul 12, 2010 at 01:50:47PM -0600, Grant Likely wrote: > > > On Mon, Jul 12, 2010 at 1:34 PM, Linus Torvalds > > > wrote: > > > > On Mon, Jul 12, 2010 at 12:17 PM, Nicolas Pitre wrote: > > > >> I think Uwe could provide his script and add it to the kernel tree. > > > >> Then all architectures could benefit from it.  Having the defconfig > > > >> files contain only those options which are different from the defaults > > > >> is certainly more readable, even on x86. > > > > > > > > Quite possible. But maintainers would need to be on the lookout of > > > > people actually using the script, and refusing to apply patches that > > > > re-introduce the whole big thing. > > > > > > I can (partially) speak for powerpc. If ARM uses this approach, then > > > I think we can do the same. After the defconfigs are trimmed, I > > > certainly won't pick up any more full defconfigs. > > I just restarted my script on the powerpc defconfigs basing on rc5, I > > assume they complete in a few days time. > So Stephen was faster than me. I don't know yet how he optimised my > script, meanwhile I put some efforts into it, too by just checking lines > that match "^(# )?CONFIG_". > > Find it attached. > > I will start to reduce the remaining configs (i.e. all but arm and > powerpc). I added just a simple heuristic: If I could remove a line, I attempted to remove twice the amount next time around (and fall back to 1 if it failed). I.e. main loop: i = 0 lines = 1 while i < len(config): print 'test for %r + %d' % (config[i], lines) defconfig = open(defconfig_src, 'w') defconfig.writelines(config[:i]) defconfig.writelines(config[i + lines:]) defconfig.close() subprocess.check_call(['make', '-s', 'ARCH=%s' % arch, target]) if os.stat('.config').st_size == config_size and list(open('.config')) == origconfig: del config[i:i+lines] lines *= 2 else: if lines > 1: lines = 1 else: i += 1 I didn't measure what the actual improvement was, but I saw a fair amount of 2/4/8-attempts passing, so I let it run. Stephen beat me to posting the resulting patch though. :P While this script is great, it is somewhat painful to run given that it attempts one config per line. Even on a fast machine that tends to take a while. -Olof From mboxrd@z Thu Jan 1 00:00:00 1970 From: olof@lixom.net (Olof Johansson) Date: Tue, 13 Jul 2010 13:04:02 -0500 Subject: optimized script [Was: ARM defconfig files] In-Reply-To: <20100713080705.GA20978@pengutronix.de> References: <20100712173228.GC9897@n2100.arm.linux.org.uk> <20100712185029.GB14425@pengutronix.de> <20100713070741.GB26442@pengutronix.de> <20100713080705.GA20978@pengutronix.de> Message-ID: <20100713180402.GA1422@lixom.net> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On Tue, Jul 13, 2010 at 10:07:05AM +0200, Uwe Kleine-K?nig wrote: > Hello, > > On Tue, Jul 13, 2010 at 09:07:41AM +0200, Uwe Kleine-K?nig wrote: > > Hi > > > > On Mon, Jul 12, 2010 at 01:50:47PM -0600, Grant Likely wrote: > > > On Mon, Jul 12, 2010 at 1:34 PM, Linus Torvalds > > > wrote: > > > > On Mon, Jul 12, 2010 at 12:17 PM, Nicolas Pitre wrote: > > > >> I think Uwe could provide his script and add it to the kernel tree. > > > >> Then all architectures could benefit from it. ?Having the defconfig > > > >> files contain only those options which are different from the defaults > > > >> is certainly more readable, even on x86. > > > > > > > > Quite possible. But maintainers would need to be on the lookout of > > > > people actually using the script, and refusing to apply patches that > > > > re-introduce the whole big thing. > > > > > > I can (partially) speak for powerpc. If ARM uses this approach, then > > > I think we can do the same. After the defconfigs are trimmed, I > > > certainly won't pick up any more full defconfigs. > > I just restarted my script on the powerpc defconfigs basing on rc5, I > > assume they complete in a few days time. > So Stephen was faster than me. I don't know yet how he optimised my > script, meanwhile I put some efforts into it, too by just checking lines > that match "^(# )?CONFIG_". > > Find it attached. > > I will start to reduce the remaining configs (i.e. all but arm and > powerpc). I added just a simple heuristic: If I could remove a line, I attempted to remove twice the amount next time around (and fall back to 1 if it failed). I.e. main loop: i = 0 lines = 1 while i < len(config): print 'test for %r + %d' % (config[i], lines) defconfig = open(defconfig_src, 'w') defconfig.writelines(config[:i]) defconfig.writelines(config[i + lines:]) defconfig.close() subprocess.check_call(['make', '-s', 'ARCH=%s' % arch, target]) if os.stat('.config').st_size == config_size and list(open('.config')) == origconfig: del config[i:i+lines] lines *= 2 else: if lines > 1: lines = 1 else: i += 1 I didn't measure what the actual improvement was, but I saw a fair amount of 2/4/8-attempts passing, so I let it run. Stephen beat me to posting the resulting patch though. :P While this script is great, it is somewhat painful to run given that it attempts one config per line. Even on a fast machine that tends to take a while. -Olof