From mboxrd@z Thu Jan 1 00:00:00 1970 From: Neil Horman Subject: Re: [PATCH] validate_abi: build faster by augmenting make with job count Date: Thu, 21 Jul 2016 14:34:01 -0400 Message-ID: <20160721183401.GE10032@hmsreliant.think-freely.org> References: <1469034588-1847-1-git-send-email-nhorman@tuxdriver.com> <6330653.K6A9CLrOZy@xps13> <20160720174849.GE28844@hmsreliant.think-freely.org> <20160720201617.GA23515@hmsreliant.think-freely.org> <9309C1B3-D86C-4265-97A4-58E3D7958944@intel.com> <20160721135433.GA9628@hmsreliant.think-freely.org> <9B6D664A-6C02-4982-BBA1-628C94514164@intel.com> <20160721150604.GC10032@hmsreliant.think-freely.org> <64F83F10-008B-4DB0-9662-989A8F57BE80@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Neil Horman , Thomas Monjalon , "dev@dpdk.org" , "Mcnamara, John" To: "Wiles, Keith" Return-path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id 76C3E5908 for ; Thu, 21 Jul 2016 20:34:05 +0200 (CEST) Content-Disposition: inline In-Reply-To: <64F83F10-008B-4DB0-9662-989A8F57BE80@intel.com> List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Thu, Jul 21, 2016 at 03:22:45PM +0000, Wiles, Keith wrote: >=20 > > On Jul 21, 2016, at 10:06 AM, Neil Horman wrote: > >=20 > > On Thu, Jul 21, 2016 at 02:09:19PM +0000, Wiles, Keith wrote: > >>=20 > >>> On Jul 21, 2016, at 8:54 AM, Neil Horman wr= ote: > >>>=20 > >>> On Wed, Jul 20, 2016 at 10:32:28PM +0000, Wiles, Keith wrote: > >>>>=20 > >>>>> On Jul 20, 2016, at 3:16 PM, Neil Horman = wrote: > >>>>>=20 > >>>>> On Wed, Jul 20, 2016 at 07:47:32PM +0000, Wiles, Keith wrote: > >>>>>>=20 > >>>>>>> On Jul 20, 2016, at 12:48 PM, Neil Horman = wrote: > >>>>>>>=20 > >>>>>>> On Wed, Jul 20, 2016 at 07:40:49PM +0200, Thomas Monjalon wrote= : > >>>>>>>> 2016-07-20 13:09, Neil Horman: > >>>>>>>>> From: Neil Horman > >>>>>>>>>=20 > >>>>>>>>> John Mcnamara and I were discussing enhacing the validate_abi= script to build > >>>>>>>>> the dpdk tree faster with multiple jobs. Theres no reason no= t to do it, so this > >>>>>>>>> implements that requirement. It uses a MAKE_JOBS variable th= at can be set by > >>>>>>>>> the user to limit the job count. By default the job count is= set to the number > >>>>>>>>> of online cpus. > >>>>>>>>=20 > >>>>>>>> Please could you use the variable name DPDK_MAKE_JOBS? > >>>>>>>> This name is already used in scripts/test-build.sh. > >>>>>>>>=20 > >>>>>>> Sure > >>>>>>>=20 > >>>>>>>>> +if [ -z "$MAKE_JOBS" ] > >>>>>>>>> +then > >>>>>>>>> + # This counts the number of cpus on the system > >>>>>>>>> + MAKE_JOBS=3D`lscpu -p=3Dcpu | grep -v "#" | wc -l` > >>>>>>>>> +fi > >>>>>>>>=20 > >>>>>>>> Is lscpu common enough? > >>>>>>>>=20 > >>>>>>> I'm not sure how to answer that. lscpu is part of the util-lin= ux package, which > >>>>>>> is part of any base install. Theres a variant for BSD, but I'm= not sure how > >>>>>>> common it is there. > >>>>>>> Neil > >>>>>>>=20 > >>>>>>>> Another acceptable default would be just "-j" without any numb= er. > >>>>>>>> It would make the number of jobs unlimited. > >>>>>>=20 > >>>>>> I think the best is just use -j as it tries to use the correct n= umber of jobs based on the number of cores, right? > >>>>>>=20 > >>>>> -j with no argument (or -j 0), is sort of, maybe what you want. = With either of > >>>>> those options, make will just issue jobs as fast as it processes = dependencies. > >>>>> Dependent on how parallel the build is, that can lead to tons of = waiting process > >>>>> (i.e. more than your number of online cpus), which can actually h= urt your build > >>>>> time. > >>>>=20 > >>>> I read the manual and looked at the code, which supports your stat= ement. (I think I had some statement on stack overflow and the last time = I believe anything on the internet :-) I have not seen a lot of differenc= es in compile times with -j on my system. Mostly I suspect it is the numb= er of paths in the dependency, cores and memory on the system. > >>>>=20 > >>>> I have 72 lcores or 2 sockets, 18 cores per socket. Xeon 2.3Ghz co= res. > >>>>=20 > >>>> $ export RTE_TARGET=3Dx86_64-native-linuxapp-gcc=20 > >>>>=20 > >>>> $ time make install T=3D${RTE_TARGET} > >>>> real 0m59.445s user 0m27.344s sys 0m7.040s > >>>>=20 > >>>> $ time make install T=3D${RTE_TARGET} -j > >>>> real 0m26.584s user 0m14.380s sys 0m5.120s > >>>>=20 > >>>> # Remove the x86_64-native-linuxapp-gcc > >>>>=20 > >>>> $ time make install T=3D${RTE_TARGET} -j 72 > >>>> real 0m23.454s user 0m10.832s sys 0m4.664s > >>>>=20 > >>>> $ time make install T=3D${RTE_TARGET} -j 8 > >>>> real 0m23.812s user 0m10.672s sys 0m4.276s > >>>>=20 > >>>> cd x86_64-native-linuxapp-gcc > >>>> $ make clean > >>>> $ time make > >>>> real 0m28.539s user 0m9.820s sys 0m3.620s > >>>>=20 > >>>> # Do a make clean between each build. > >>>>=20 > >>>> $ time make -j > >>>> real 0m7.217s user 0m6.532s sys 0m2.332s > >>>>=20 > >>>> $ time make -j 8 > >>>> real 0m8.256s user 0m6.472s sys 0m2.456s > >>>>=20 > >>>> $ time make -j 72 > >>>> real 0m6.866s user 0m6.184s sys 0m2.216s > >>>>=20 > >>>> Just the real time numbers in the following table. > >>>>=20 > >>>> processes real Time depdirs > >>>> no -j 59.4s Yes > >>>> -j 8 23.8s Yes > >>>> -j 72 23.5s Yes > >>>> -j 26.5s Yes > >>>>=20 > >>>> no -j 28.5s No > >>>> -j 8 8.2s No > >>>> -j 72 6.8s No > >>>> -j 7.2s No > >>>>=20 > >>>> Looks like the depdirs build time on my system: > >>>> $ make clean -j > >>>> $ rm .depdirs > >>>> $ time make -j > >>>> real 0m23.734s user 0m11.228s sys 0m4.844s > >>>>=20 > >>>> About 16 seconds, which is not a lot of savings. Now the differenc= e from no -j to -j is a lot, but the difference between -j and -j is not a huge saving. This leads me back to over engineering the pro= blem when =E2=80=98-j=E2=80=99 would work just as well here. > >>>>=20 > >>>> Even on my MacBook Pro i7 system the difference is not that much 1= m8s without depdirs build for -j in a VirtualBox with all 4 cores 8G RAM.= Compared to 1m13s with -j 4 option. > >>>>=20 > >>>> I just wonder if it makes a lot of sense to use cpuinfo in this gi= ven case if it turns out to be -j works with the 80% rule? > >>>>=20 > >>> It may, but that seems to be reason to me to just set DPDK_MAKE_JOB= S=3D0, and > >>> you'll get that behavior > >>=20 > >> Just to be sure, =E2=80=98make -j 0=E2=80=99 is not a valid argument= to the -j option. It looks like you have to do =E2=80=98-j=E2=80=99 or =E2= =80=98-j N=E2=80=99 or no option where N !=3D 0 > >>=20 > >> I think we just use -j which gets us the 80% rule and the best perfo= rmance without counting cores. > >>=20 > > Thats odd, specifying 0 works for me. If it doesn't for you, specify= $MAX_INT > > or some other huge number would be comparable >=20 > rkwiles@supermicro (master):~/.../dpdk/x86_64-native-linuxapp-gcc$ make= --version > GNU Make 4.1 > Built for x86_64-pc-linux-gnu > Copyright (C) 1988-2014 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. >=20 > rkwiles@supermicro (master):~/.../dpdk/x86_64-native-linuxapp-gcc$ make= -j 0 > make: the '-j' option requires a positive integer argument >=20 > rkwiles@supermicro (master):~/.../dpdk/x86_64-native-linuxapp-gcc$ lsb_= release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 16.04.1 LTS > Release: 16.04 > Codename: xenial >=20 I'm not saying your variant doesn't work, only that my copy of make does,= but its possible that I have some alternately patched version (I used to fix = make bugs way back when, so I may have an impure copy). Regardless, my commen= t is still valid, if you want to have unlimited jobs, you can just export DPDK_MAKE_JOBS=3D Neil > >=20 > > Neil > >=20 > >>>=20 > >>> Neil > >>>=20 > >>>> On some other project with a lot more files like the FreeBSD or Li= nux distro, yes it would make a fair amount of real time difference. > >>>>=20 > >>>> Keith > >>>>=20 > >>>>>=20 > >>>>> While its fine in los of cases, its not always fine, and with thi= s > >>>>> implementation you can still opt in to that behavior by setting D= PDK_MAKE_JOBS=3D0 > >>>>>=20 > >>>>> Neil >=20