* [PATCH] bitbake.conf: set the default pigz thread
@ 2020-08-11 7:22 Yu, Mingli
2020-08-11 7:57 ` [OE-core] " Richard Purdie
0 siblings, 1 reply; 5+ messages in thread
From: Yu, Mingli @ 2020-08-11 7:22 UTC (permalink / raw)
To: openembedded-core
From: Mingli Yu <mingli.yu@windriver.com>
pigz, which stands for parallel implementation of gzip, is a
fully functional replacement for gzip that exploits multiple
processors and multiple cores to the hilt when compressing data.
do_package uses pigz to compress data to speed up the time
if there is pigz available on the build server.
But for some big package such as clang, there comes below
error during do_package.
gzip: stdout: Cannot allocate memory
tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs-linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cfe8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192 of 10240 bytes
tar: Child returned status 1
tar: Error is not recoverable: exiting now
So set the default pigz thread and the user also can customize
the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p 3"
for the individual package in local.conf to avoid eating so much
memory.
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
---
meta/conf/bitbake.conf | 3 +++
1 file changed, 3 insertions(+)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 353caacef9..bc87e78d78 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
+# Default parallelism and resource usage for pigz
+export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"
+
##################################################################
# Magic Cookie for SANITY CHECK
##################################################################
--
2.26.2
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH] bitbake.conf: set the default pigz thread
2020-08-11 7:22 [PATCH] bitbake.conf: set the default pigz thread Yu, Mingli
@ 2020-08-11 7:57 ` Richard Purdie
2020-08-11 8:09 ` Yu, Mingli
0 siblings, 1 reply; 5+ messages in thread
From: Richard Purdie @ 2020-08-11 7:57 UTC (permalink / raw)
To: Yu, Mingli, openembedded-core
On Tue, 2020-08-11 at 15:22 +0800, Yu, Mingli wrote:
> From: Mingli Yu <mingli.yu@windriver.com>
>
> pigz, which stands for parallel implementation of gzip, is a
> fully functional replacement for gzip that exploits multiple
> processors and multiple cores to the hilt when compressing data.
>
> do_package uses pigz to compress data to speed up the time
> if there is pigz available on the build server.
>
> But for some big package such as clang, there comes below
> error during do_package.
> gzip: stdout: Cannot allocate memory
> tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs-linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cfe8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192 of 10240 bytes
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
>
> So set the default pigz thread and the user also can customize
> the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p 3"
> for the individual package in local.conf to avoid eating so much
> memory.
>
> Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
> ---
> meta/conf/bitbake.conf | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 353caacef9..bc87e78d78 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
> XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
>
> +# Default parallelism and resource usage for pigz
> +export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"
Why not just set this in the cases you need it in and leave the
defaults alone? You can just export it in the tasks which need it too.
This change actually makes the sstate number of cpus specific which we
definitely don't want.
Cheers,
Richard
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH] bitbake.conf: set the default pigz thread
2020-08-11 7:57 ` [OE-core] " Richard Purdie
@ 2020-08-11 8:09 ` Yu, Mingli
2020-08-11 8:19 ` Richard Purdie
0 siblings, 1 reply; 5+ messages in thread
From: Yu, Mingli @ 2020-08-11 8:09 UTC (permalink / raw)
To: Richard Purdie, openembedded-core
Hi RP,
On 8/11/20 3:57 PM, Richard Purdie wrote:
> On Tue, 2020-08-11 at 15:22 +0800, Yu, Mingli wrote:
>> From: Mingli Yu <mingli.yu@windriver.com>
>>
>> pigz, which stands for parallel implementation of gzip, is a
>> fully functional replacement for gzip that exploits multiple
>> processors and multiple cores to the hilt when compressing data.
>>
>> do_package uses pigz to compress data to speed up the time
>> if there is pigz available on the build server.
>>
>> But for some big package such as clang, there comes below
>> error during do_package.
>> gzip: stdout: Cannot allocate memory
>> tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs-linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cfe8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192 of 10240 bytes
>> tar: Child returned status 1
>> tar: Error is not recoverable: exiting now
>>
>> So set the default pigz thread and the user also can customize
>> the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p 3"
>> for the individual package in local.conf to avoid eating so much
>> memory.
>>
>> Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
>> ---
>> meta/conf/bitbake.conf | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index 353caacef9..bc87e78d78 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
>> XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
>> XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
>>
>> +# Default parallelism and resource usage for pigz
>> +export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"
>
> Why not just set this in the cases you need it in and leave the
> defaults alone? You can just export it in the tasks which need it too.
Actually I just want to use this logic in this patch altogether
PIGZ_pn-clang = "-p 3" in local.conf to limit the pigz thread number for
clang.
Without this patch, I think PIGZ_pn-clang = "-p 3" doesn't take effect.
>
> This change actually makes the sstate number of cpus specific which we
> definitely don't want.
From my understanding, the default behavior for pigz is to take all
online cores, it should be the same with export PIGZ = "-p
${@oe.utils.cpu_count(at_least=2)}".
Thanks,
>
> Cheers,
>
> Richard
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH] bitbake.conf: set the default pigz thread
2020-08-11 8:19 ` Richard Purdie
@ 2020-08-11 8:19 ` Yu, Mingli
0 siblings, 0 replies; 5+ messages in thread
From: Yu, Mingli @ 2020-08-11 8:19 UTC (permalink / raw)
To: Richard Purdie, openembedded-core
On 8/11/20 4:19 PM, Richard Purdie wrote:
> On Tue, 2020-08-11 at 16:09 +0800, Yu, Mingli wrote:
>> Hi RP,
>>
>> On 8/11/20 3:57 PM, Richard Purdie wrote:
>>> On Tue, 2020-08-11 at 15:22 +0800, Yu, Mingli wrote:
>>>> From: Mingli Yu <mingli.yu@windriver.com>
>>>>
>>>> pigz, which stands for parallel implementation of gzip, is a
>>>> fully functional replacement for gzip that exploits multiple
>>>> processors and multiple cores to the hilt when compressing data.
>>>>
>>>> do_package uses pigz to compress data to speed up the time
>>>> if there is pigz available on the build server.
>>>>
>>>> But for some big package such as clang, there comes below
>>>> error during do_package.
>>>> gzip: stdout: Cannot allocate memory
>>>> tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs-
>>>> linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cf
>>>> e8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192
>>>> of 10240 bytes
>>>> tar: Child returned status 1
>>>> tar: Error is not recoverable: exiting now
>>>>
>>>> So set the default pigz thread and the user also can customize
>>>> the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p
>>>> 3"
>>>> for the individual package in local.conf to avoid eating so much
>>>> memory.
>>>>
>>>> Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
>>>> ---
>>>> meta/conf/bitbake.conf | 3 +++
>>>> 1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>>>> index 353caacef9..bc87e78d78 100644
>>>> --- a/meta/conf/bitbake.conf
>>>> +++ b/meta/conf/bitbake.conf
>>>> @@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least
>>>> =2)}"
>>>> XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --
>>>> threads=${XZ_THREADS}"
>>>> XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
>>>>
>>>> +# Default parallelism and resource usage for pigz
>>>> +export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"
>>>
>>> Why not just set this in the cases you need it in and leave the
>>> defaults alone? You can just export it in the tasks which need it
>>> too.
>>
>> Actually I just want to use this logic in this patch altogether
>> PIGZ_pn-clang = "-p 3" in local.conf to limit the pigz thread number
>> for clang.
>>
>> Without this patch, I think PIGZ_pn-clang = "-p 3" doesn't take
>> effect.
>
> You'll have to use something like:
>
> PIGZEXPORT = "0"
> PIGZEXPORT_pn-clang = "1"
> PIGZ[export] = "${PIGZEXPORT}"
>
>>> This change actually makes the sstate number of cpus specific which
>>> we
>>> definitely don't want.
>>
>> From my understanding, the default behavior for pigz is to take all
>> online cores, it should be the same with export PIGZ = "-p
>> ${@oe.utils.cpu_count(at_least=2)}".
>
> I understand why you want to do it and what it does. It breaks sstate
> signatures for everyone else though so it is not acceptable.
Got it!
Thanks,
>
> Cheers,
>
> Richard
>
>
>
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH] bitbake.conf: set the default pigz thread
2020-08-11 8:09 ` Yu, Mingli
@ 2020-08-11 8:19 ` Richard Purdie
2020-08-11 8:19 ` Yu, Mingli
0 siblings, 1 reply; 5+ messages in thread
From: Richard Purdie @ 2020-08-11 8:19 UTC (permalink / raw)
To: Yu, Mingli, openembedded-core
On Tue, 2020-08-11 at 16:09 +0800, Yu, Mingli wrote:
> Hi RP,
>
> On 8/11/20 3:57 PM, Richard Purdie wrote:
> > On Tue, 2020-08-11 at 15:22 +0800, Yu, Mingli wrote:
> > > From: Mingli Yu <mingli.yu@windriver.com>
> > >
> > > pigz, which stands for parallel implementation of gzip, is a
> > > fully functional replacement for gzip that exploits multiple
> > > processors and multiple cores to the hilt when compressing data.
> > >
> > > do_package uses pigz to compress data to speed up the time
> > > if there is pigz available on the build server.
> > >
> > > But for some big package such as clang, there comes below
> > > error during do_package.
> > > gzip: stdout: Cannot allocate memory
> > > tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs-
> > > linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cf
> > > e8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192
> > > of 10240 bytes
> > > tar: Child returned status 1
> > > tar: Error is not recoverable: exiting now
> > >
> > > So set the default pigz thread and the user also can customize
> > > the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p
> > > 3"
> > > for the individual package in local.conf to avoid eating so much
> > > memory.
> > >
> > > Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
> > > ---
> > > meta/conf/bitbake.conf | 3 +++
> > > 1 file changed, 3 insertions(+)
> > >
> > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > > index 353caacef9..bc87e78d78 100644
> > > --- a/meta/conf/bitbake.conf
> > > +++ b/meta/conf/bitbake.conf
> > > @@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least
> > > =2)}"
> > > XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --
> > > threads=${XZ_THREADS}"
> > > XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> > >
> > > +# Default parallelism and resource usage for pigz
> > > +export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"
> >
> > Why not just set this in the cases you need it in and leave the
> > defaults alone? You can just export it in the tasks which need it
> > too.
>
> Actually I just want to use this logic in this patch altogether
> PIGZ_pn-clang = "-p 3" in local.conf to limit the pigz thread number
> for clang.
>
> Without this patch, I think PIGZ_pn-clang = "-p 3" doesn't take
> effect.
You'll have to use something like:
PIGZEXPORT = "0"
PIGZEXPORT_pn-clang = "1"
PIGZ[export] = "${PIGZEXPORT}"
> > This change actually makes the sstate number of cpus specific which
> > we
> > definitely don't want.
>
> From my understanding, the default behavior for pigz is to take all
> online cores, it should be the same with export PIGZ = "-p
> ${@oe.utils.cpu_count(at_least=2)}".
I understand why you want to do it and what it does. It breaks sstate
signatures for everyone else though so it is not acceptable.
Cheers,
Richard
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-08-11 8:21 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-08-11 7:22 [PATCH] bitbake.conf: set the default pigz thread Yu, Mingli
2020-08-11 7:57 ` [OE-core] " Richard Purdie
2020-08-11 8:09 ` Yu, Mingli
2020-08-11 8:19 ` Richard Purdie
2020-08-11 8:19 ` Yu, Mingli
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox