[PATCH] gcc: enable zstd compression of LTO bytecode

[PATCH] gcc: enable zstd compression of LTO bytecode

[Enrico Scholz]

gcc-10+ supports zstd compression of LTO bytecode.  Install the
corresponding package to enable this feature in a deterministic way.

NOTE: previously built LTO object files (without this compression)
must be regenerated; gcc will fail else with

  | lto1: internal compiler error: original not compressed with zstd

in this case.

NOTE: it seems to be possible that zstd support is enabled non
deterministically (perhaps by host system pollution).

I had the problem that the SDK gcc was built with zstd while the
cross gcc was built without it.  Libraries (built by cross gcc and
uncompressed hence) could not be used with the SDK gcc.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-devtools/gcc/gcc-11.2.inc
index f07a5be507b3..62db035760aa 100644
--- a/meta/recipes-devtools/gcc/gcc-11.2.inc
+++ b/meta/recipes-devtools/gcc/gcc-11.2.inc
@@ -11,7 +11,7 @@ BINV = "11.2.0"
 FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:"
 
 DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
-NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native"
+NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native"
 
 LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
 
-- 
2.34.1

Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode

[Khem Raj]

On Thu, Feb 3, 2022 at 9:26 AM Enrico Scholz via
lists.openembedded.org
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> wrote:
>
> gcc-10+ supports zstd compression of LTO bytecode.  Install the
> corresponding package to enable this feature in a deterministic way.
>
> NOTE: previously built LTO object files (without this compression)
> must be regenerated; gcc will fail else with
>
>   | lto1: internal compiler error: original not compressed with zstd
>
> in this case.
>
> NOTE: it seems to be possible that zstd support is enabled non
> deterministically (perhaps by host system pollution).
>
> I had the problem that the SDK gcc was built with zstd while the
> cross gcc was built without it.  Libraries (built by cross gcc and
> uncompressed hence) could not be used with the SDK gcc.
>

this means regenerating entire shared state isnt it ?

> Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> ---
>  meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-devtools/gcc/gcc-11.2.inc
> index f07a5be507b3..62db035760aa 100644
> --- a/meta/recipes-devtools/gcc/gcc-11.2.inc
> +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc
> @@ -11,7 +11,7 @@ BINV = "11.2.0"
>  FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:"
>
>  DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
> -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native"
> +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native"
>
>  LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
>
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161292): https://lists.openembedded.org/g/openembedded-core/message/161292
> Mute This Topic: https://lists.openembedded.org/mt/88888010/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode

[Enrico Scholz]

Khem Raj <raj.khem@gmail.com> writes:

>> gcc-10+ supports zstd compression of LTO bytecode.  Install the
>> corresponding package to enable this feature in a deterministic way.
>>
>> NOTE: previously built LTO object files (without this compression)
>> must be regenerated; gcc will fail else with
>>
>>   | lto1: internal compiler error: original not compressed with zstd
>>
>> in this case.
>>
>> NOTE: it seems to be possible that zstd support is enabled non
>> deterministically (perhaps by host system pollution).
>>
>> I had the problem that the SDK gcc was built with zstd while the
>> cross gcc was built without it.  Libraries (built by cross gcc and
>> uncompressed hence) could not be used with the SDK gcc.
>
> this means regenerating entire shared state isnt it ?

yes (at least when lto.inc is used (which is *not* by default)).  When a
bad sstate really matters for 'master', perhaps there can be wait for a
change in glibc or so before applying this patch.

Or do you think that adding a DISTRO_FEATURE like 'lto-zstd' would make
sense?  When this flag is missing, '--without-zstd' must be added to
EXTRA_OECONF.  Also some early [vardeps] must be set (where?) to avoid
the sstate problem.

Globally enabling 'zstd' appears easier because drawbacks (costs for
compression + decompression) should be negligible for zstd.



Enrico

Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode

[Khem Raj]

On Thu, Feb 3, 2022 at 1:40 PM Enrico Scholz
<enrico.scholz@sigma-chemnitz.de> wrote:
>
> Khem Raj <raj.khem@gmail.com> writes:
>
> >> gcc-10+ supports zstd compression of LTO bytecode.  Install the
> >> corresponding package to enable this feature in a deterministic way.
> >>
> >> NOTE: previously built LTO object files (without this compression)
> >> must be regenerated; gcc will fail else with
> >>
> >>   | lto1: internal compiler error: original not compressed with zstd
> >>
> >> in this case.
> >>
> >> NOTE: it seems to be possible that zstd support is enabled non
> >> deterministically (perhaps by host system pollution).
> >>
> >> I had the problem that the SDK gcc was built with zstd while the
> >> cross gcc was built without it.  Libraries (built by cross gcc and
> >> uncompressed hence) could not be used with the SDK gcc.
> >
> > this means regenerating entire shared state isnt it ?
>
> yes (at least when lto.inc is used (which is *not* by default)).  When a
> bad sstate really matters for 'master', perhaps there can be wait for a
> change in glibc or so before applying this patch.
>
> Or do you think that adding a DISTRO_FEATURE like 'lto-zstd' would make
> sense?  When this flag is missing, '--without-zstd' must be added to
> EXTRA_OECONF.  Also some early [vardeps] must be set (where?) to avoid
> the sstate problem.
>
> Globally enabling 'zstd' appears easier because drawbacks (costs for
> compression + decompression) should be negligible for zstd.

I think switching to zstd is fine.

>
>
>
> Enrico

Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode

[Richard Purdie]

On Thu, 2022-02-03 at 18:26 +0100, Enrico Scholz via lists.openembedded.org
wrote:
> gcc-10+ supports zstd compression of LTO bytecode.  Install the
> corresponding package to enable this feature in a deterministic way.
> 
> NOTE: previously built LTO object files (without this compression)
> must be regenerated; gcc will fail else with
> 
>   | lto1: internal compiler error: original not compressed with zstd
> 
> in this case.
> 
> NOTE: it seems to be possible that zstd support is enabled non
> deterministically (perhaps by host system pollution).
> 
> I had the problem that the SDK gcc was built with zstd while the
> cross gcc was built without it.  Libraries (built by cross gcc and
> uncompressed hence) could not be used with the SDK gcc.
> 
> Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> ---
>  meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-devtools/gcc/gcc-11.2.inc
> index f07a5be507b3..62db035760aa 100644
> --- a/meta/recipes-devtools/gcc/gcc-11.2.inc
> +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc
> @@ -11,7 +11,7 @@ BINV = "11.2.0"
>  FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:"
>  
>  DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
> -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native"
> +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native"
>  
>  LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
>  

Should we be passing some flag to gcc explicitly to configure this
deterministicly?

Cheers,

Richard

Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2452 bytes --]

On Thu, Feb 3, 2022 at 1:59 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Thu, 2022-02-03 at 18:26 +0100, Enrico Scholz via
> lists.openembedded.org
> wrote:
> > gcc-10+ supports zstd compression of LTO bytecode.  Install the
> > corresponding package to enable this feature in a deterministic way.
> >
> > NOTE: previously built LTO object files (without this compression)
> > must be regenerated; gcc will fail else with
> >
> >   | lto1: internal compiler error: original not compressed with zstd
> >
> > in this case.
> >
> > NOTE: it seems to be possible that zstd support is enabled non
> > deterministically (perhaps by host system pollution).
> >
> > I had the problem that the SDK gcc was built with zstd while the
> > cross gcc was built without it.  Libraries (built by cross gcc and
> > uncompressed hence) could not be used with the SDK gcc.
> >
> > Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> > ---
> >  meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc
> b/meta/recipes-devtools/gcc/gcc-11.2.inc
> > index f07a5be507b3..62db035760aa 100644
> > --- a/meta/recipes-devtools/gcc/gcc-11.2.inc
> > +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc
> > @@ -11,7 +11,7 @@ BINV = "11.2.0"
> >  FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:"
> >
> >  DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
> > -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native
> flex-native"
> > +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native
> flex-native zstd-native"
> >
> >  LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
> >
>
> Should we be passing some flag to gcc explicitly to configure this
> deterministicly?


Perhaps use —with-zstd configure switch maybe via package config but it’s
better to enable it always

>
>
> Cheers,
>
> Richard
>
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161323):
> https://lists.openembedded.org/g/openembedded-core/message/161323
> Mute This Topic: https://lists.openembedded.org/mt/88888010/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3795 bytes --]

Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode

[Richard Purdie]

On Thu, 2022-02-03 at 14:11 -0800, Khem Raj wrote:
> 
> 
> On Thu, Feb 3, 2022 at 1:59 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Thu, 2022-02-03 at 18:26 +0100, Enrico Scholz via lists.openembedded.org
> > wrote:
> > > gcc-10+ supports zstd compression of LTO bytecode.  Install the
> > > corresponding package to enable this feature in a deterministic way.
> > > 
> > > NOTE: previously built LTO object files (without this compression)
> > > must be regenerated; gcc will fail else with
> > > 
> > >    | lto1: internal compiler error: original not compressed with zstd
> > > 
> > > in this case.
> > > 
> > > NOTE: it seems to be possible that zstd support is enabled non
> > > deterministically (perhaps by host system pollution).
> > > 
> > > I had the problem that the SDK gcc was built with zstd while the
> > > cross gcc was built without it.  Libraries (built by cross gcc and
> > > uncompressed hence) could not be used with the SDK gcc.
> > > 
> > > Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> > > ---
> > >   meta/recipes-devtools/gcc/gcc-11.2.inc | 2 +-
> > >   1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/meta/recipes-devtools/gcc/gcc-11.2.inc b/meta/recipes-
> > devtools/gcc/gcc-11.2.inc
> > > index f07a5be507b3..62db035760aa 100644
> > > --- a/meta/recipes-devtools/gcc/gcc-11.2.inc
> > > +++ b/meta/recipes-devtools/gcc/gcc-11.2.inc
> > > @@ -11,7 +11,7 @@ BINV = "11.2.0"
> > >   FILESEXTRAPATHS =. "${FILE_DIRNAME}/gcc:${FILE_DIRNAME}/gcc/backport:"
> > >   
> > >   DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
> > > -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-
> > > native"
> > > +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-
> > > native
> > zstd-native"
> > >   
> > >   LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
> > >   
> > 
> > Should we be passing some flag to gcc explicitly to configure this
> > deterministicly?
> > 
> 
> 
> Perhaps use —with-zstd configure switch maybe via package config but it’s
> better to enable it always 
> 

Agreed, I'm fine with enabling it, I'd just like to be explicit about what we're
doing.

Cheers,

Richard

Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

>>   | lto1: internal compiler error: original not compressed with zstd
>>
>>  DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
>> -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native"
>> +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native"
>>  
>>  LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
>>  
>
> Should we be passing some flag to gcc explicitly to configure this
> deterministicly?

Both '--with-zstd' and '--without-zstd' make it deterministic (require
zstd.h and abort when non-existing resp. ignore it when existing).

Perhaps '--without-zstd' should be added in the hardknott and honister
branches to prevent pollution by host environment.



Enrico

Re: [OE-core] [PATCH] gcc: enable zstd compression of LTO bytecode

[Khem Raj]

On Fri, Feb 4, 2022 at 11:00 AM Enrico Scholz via
lists.openembedded.org
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> wrote:
>
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>
> >>   | lto1: internal compiler error: original not compressed with zstd
> >>
> >>  DEPENDS =+ "mpfr gmp libmpc zlib flex-native"
> >> -NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native"
> >> +NATIVEDEPS = "mpfr-native gmp-native libmpc-native zlib-native flex-native zstd-native"
> >>
> >>  LICENSE = "GPL-3.0-with-GCC-exception & GPLv3"
> >>
> >
> > Should we be passing some flag to gcc explicitly to configure this
> > deterministicly?
>
> Both '--with-zstd' and '--without-zstd' make it deterministic (require
> zstd.h and abort when non-existing resp. ignore it when existing).
>
> Perhaps '--without-zstd' should be added in the hardknott and honister
> branches to prevent pollution by host environment.
>

yes makes sense.

>
>
> Enrico
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161379): https://lists.openembedded.org/g/openembedded-core/message/161379
> Mute This Topic: https://lists.openembedded.org/mt/88888010/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>