* [Buildroot] [PATCH] package-infra: limit the number of // jobs @ 2013-05-09 22:03 Yann E. MORIN 2013-05-09 22:13 ` Yann E. MORIN 2013-05-10 13:09 ` Nathan Lynch 0 siblings, 2 replies; 4+ messages in thread From: Yann E. MORIN @ 2013-05-09 22:03 UTC (permalink / raw) To: buildroot From: "Yann E. MORIN" <yann.morin.1998@free.fr> The current code spawns as many jobs as up to twice the number of CPUs. On small-class machines like laptops, with a limitted amount of memory, but still a few CPUs (real or hyperthreads), the HDD becomes a bottleneck, and it becomes almost impossible to do anythiong else while there is a build in progress. Limit the number of jobs to the number of CPUs plus one. Even on fast machines with fast HDDs, this settings keeps the machine fully busy (for those packages that can build in parallel, of course). For example, building qemu or the linux kernel kept my hyperthreaded hexa Core i7 with 18GiB of RAM, busy at 99% (I never ever managed to get 100% even with more jobs, not even 200); while on my hyperthreaded dual Core i5 with only 4GiB and a slow HDD, I still topped at 100% CPU, while still able to do some work involving the HDD. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> --- package/Makefile.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package/Makefile.in b/package/Makefile.in index a449089..a575f17 100644 --- a/package/Makefile.in +++ b/package/Makefile.in @@ -11,7 +11,7 @@ HOSTMAKE :=$(shell which $(HOSTMAKE) || type -p $(HOSTMAKE) || echo make) # while waiting on I/O. ifeq ($(BR2_JLEVEL),0) PARALLEL_JOBS:=$(shell echo \ - $$((2 * `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1`))) + $$((2 + `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 0`))) else PARALLEL_JOBS:=$(BR2_JLEVEL) endif -- 1.8.1.2 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH] package-infra: limit the number of // jobs 2013-05-09 22:03 [Buildroot] [PATCH] package-infra: limit the number of // jobs Yann E. MORIN @ 2013-05-09 22:13 ` Yann E. MORIN 2013-05-10 13:09 ` Nathan Lynch 1 sibling, 0 replies; 4+ messages in thread From: Yann E. MORIN @ 2013-05-09 22:13 UTC (permalink / raw) To: buildroot All, On Fri, May 10, 2013 at 12:03:20AM +0200, Yann E. MORIN wrote: [--SNIP--] > Limit the number of jobs to the number of CPUs plus one. [--SNIP--] > @@ -11,7 +11,7 @@ HOSTMAKE :=$(shell which $(HOSTMAKE) || type -p $(HOSTMAKE) || echo make) > # while waiting on I/O. > ifeq ($(BR2_JLEVEL),0) > PARALLEL_JOBS:=$(shell echo \ > - $$((2 * `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1`))) > + $$((2 + `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 0`))) I forgot to 'git add' before I 'git commit --amend'. So this should have been a '1 +' not a '2 +'. Sigh... :-/ Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH] package-infra: limit the number of // jobs 2013-05-09 22:03 [Buildroot] [PATCH] package-infra: limit the number of // jobs Yann E. MORIN 2013-05-09 22:13 ` Yann E. MORIN @ 2013-05-10 13:09 ` Nathan Lynch 2013-05-10 13:24 ` Yann E. MORIN 1 sibling, 1 reply; 4+ messages in thread From: Nathan Lynch @ 2013-05-10 13:09 UTC (permalink / raw) To: buildroot Hi Yann, On Fri, 2013-05-10 at 00:03 +0200, Yann E. MORIN wrote: > From: "Yann E. MORIN" <yann.morin.1998@free.fr> > > The current code spawns as many jobs as up to twice the number of CPUs. > > On small-class machines like laptops, with a limitted amount of memory, > but still a few CPUs (real or hyperthreads), the HDD becomes a bottleneck, > and it becomes almost impossible to do anythiong else while there is a > build in progress. > > Limit the number of jobs to the number of CPUs plus one. > > Even on fast machines with fast HDDs, this settings keeps the machine > fully busy (for those packages that can build in parallel, of course). > > For example, building qemu or the linux kernel kept my hyperthreaded > hexa Core i7 with 18GiB of RAM, busy at 99% (I never ever managed to > get 100% even with more jobs, not even 200); while on my hyperthreaded > dual Core i5 with only 4GiB and a slow HDD, I still topped at 100% CPU, > while still able to do some work involving the HDD. > > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> > --- > package/Makefile.in | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/package/Makefile.in b/package/Makefile.in > index a449089..a575f17 100644 > --- a/package/Makefile.in > +++ b/package/Makefile.in > @@ -11,7 +11,7 @@ HOSTMAKE :=$(shell which $(HOSTMAKE) || type -p $(HOSTMAKE) || echo make) > # while waiting on I/O. > ifeq ($(BR2_JLEVEL),0) > PARALLEL_JOBS:=$(shell echo \ > - $$((2 * `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1`))) > + $$((2 + `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 0`))) You'll need to update/remove the comment preceding this code too... When I submitted this feature there was some discussion of what the heuristic should be here. Given your experience I am fine with making it more conservative, especially since this is the default behavior. ^ permalink raw reply [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH] package-infra: limit the number of // jobs 2013-05-10 13:09 ` Nathan Lynch @ 2013-05-10 13:24 ` Yann E. MORIN 0 siblings, 0 replies; 4+ messages in thread From: Yann E. MORIN @ 2013-05-10 13:24 UTC (permalink / raw) To: buildroot Nathan, All, On Fri, May 10, 2013 at 08:09:11AM -0500, Nathan Lynch wrote: > On Fri, 2013-05-10 at 00:03 +0200, Yann E. MORIN wrote: > > From: "Yann E. MORIN" <yann.morin.1998@free.fr> > > > > The current code spawns as many jobs as up to twice the number of CPUs. > > > > On small-class machines like laptops, with a limitted amount of memory, > > but still a few CPUs (real or hyperthreads), the HDD becomes a bottleneck, > > and it becomes almost impossible to do anythiong else while there is a > > build in progress. > > > > Limit the number of jobs to the number of CPUs plus one. > > > > Even on fast machines with fast HDDs, this settings keeps the machine > > fully busy (for those packages that can build in parallel, of course). > > > > For example, building qemu or the linux kernel kept my hyperthreaded > > hexa Core i7 with 18GiB of RAM, busy at 99% (I never ever managed to > > get 100% even with more jobs, not even 200); while on my hyperthreaded > > dual Core i5 with only 4GiB and a slow HDD, I still topped at 100% CPU, > > while still able to do some work involving the HDD. > > > > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> > > --- > > package/Makefile.in | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/package/Makefile.in b/package/Makefile.in > > index a449089..a575f17 100644 > > --- a/package/Makefile.in > > +++ b/package/Makefile.in > > @@ -11,7 +11,7 @@ HOSTMAKE :=$(shell which $(HOSTMAKE) || type -p $(HOSTMAKE) || echo make) > > # while waiting on I/O. > > ifeq ($(BR2_JLEVEL),0) > > PARALLEL_JOBS:=$(shell echo \ > > - $$((2 * `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1`))) > > + $$((2 + `getconf _NPROCESSORS_ONLN 2>/dev/null || echo 0`))) > > You'll need to update/remove the comment preceding this code too... Right, good catch. I'll send an updated patch soonish. > When I submitted this feature there was some discussion of what the > heuristic should be here. Given your experience I am fine with making > it more conservative, especially since this is the default behavior. Yes. In the Good Old Times (TM), it was customary to set jobs=2*CPU, because disks (or network) was very slow, and IO did not catch up with CPU bandwidth; this is not much true these days. Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2013-05-10 13:24 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-05-09 22:03 [Buildroot] [PATCH] package-infra: limit the number of // jobs Yann E. MORIN 2013-05-09 22:13 ` Yann E. MORIN 2013-05-10 13:09 ` Nathan Lynch 2013-05-10 13:24 ` Yann E. MORIN
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox