[OE-core] [dunfell] [PATCH] cmake-native: Use cmake-provided zstd library; supported host distro zstd may be too old.

[OE-core] [dunfell] [PATCH] cmake-native: Use cmake-provided zstd library; supported host distro zstd may be too old.

[Leon Woestenberg]

cmake-native uses the system provided libraries due to:

CMAKE_EXTRACONF = "<...> -DCMAKE_USE_SYSTEM_LIBRARIES=1 <...>"

Now, iff the libzstd(-dev) is installed but too old, which can happen on Ubuntu 16.04:

dpkg -l | grep zstd
ii  libzstd-dev  0.5.1-1
ii  libzstd0 0.5.1-1

cmake configure will use the zstd system library:

cat ./tmp/work/x86_64-linux/cmake-native/3.15.3-r0/temp/log.do_configure | grep -i ZSTD
-- Using system-installed ZSTD
-- Found ZSTD: /usr/lib/x86_64-linux-gnu/libzstd.so

but will fail to compile due to:

<...>/tmp/work/x86_64-linux/cmake-native/3.15.3-r0/cmake-3.15.3/Utilities/cmlibarchive/libarchive/archive_read_support_filter_zstd.c:59:2: error: unknown type name ‘ZSTD_DStream’
|   ZSTD_DStream *dstream;
|   ^

The library does not contain that datatype:

grep -rne ZSTD_DStream /usr/include/zstd.h
<no output>

Apparently the cmake-native library check is not checking the library version or features.

This fix is to not depend on the system library and use the zstd library provided with cmake.

CMAKE_EXTRACONF = "\
<...>
    -DCMAKE_USE_SYSTEM_LIBRARY_ZSTD=0 \
<...>

Signed-off-by: Leon Woestenberg <leon@sidebranch.com>
---
 meta/recipes-devtools/cmake/cmake-native_3.16.5.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb b/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb
index b2952ee..e0ac3f8 100644
--- a/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb
+++ b/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb
@@ -21,6 +21,7 @@ CMAKE_EXTRACONF = "\
     -DCMAKE_USE_SYSTEM_LIBRARY_LIBARCHIVE=0 \
     -DCMAKE_USE_SYSTEM_LIBRARY_LIBUV=0 \
     -DCMAKE_USE_SYSTEM_LIBRARY_LIBRHASH=0 \
+    -DCMAKE_USE_SYSTEM_LIBRARY_ZSTD=0 \
     -DENABLE_ACL=0 -DHAVE_ACL_LIBACL_H=0 \
     -DHAVE_SYS_ACL_H=0 \
 "
-- 
2.7.4

Re: [OE-core] [dunfell] [PATCH] cmake-native: Use cmake-provided zstd library; supported host distro zstd may be too old.

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2130 bytes --]

This needs to go to master first probably?

Alex

On Fri, 14 Aug 2020 at 22:46, Leon Woestenberg <leon@sidebranch.com> wrote:

> cmake-native uses the system provided libraries due to:
>
> CMAKE_EXTRACONF = "<...> -DCMAKE_USE_SYSTEM_LIBRARIES=1 <...>"
>
> Now, iff the libzstd(-dev) is installed but too old, which can happen on
> Ubuntu 16.04:
>
> dpkg -l | grep zstd
> ii  libzstd-dev  0.5.1-1
> ii  libzstd0 0.5.1-1
>
> cmake configure will use the zstd system library:
>
> cat ./tmp/work/x86_64-linux/cmake-native/3.15.3-r0/temp/log.do_configure |
> grep -i ZSTD
> -- Using system-installed ZSTD
> -- Found ZSTD: /usr/lib/x86_64-linux-gnu/libzstd.so
>
> but will fail to compile due to:
>
> <...>/tmp/work/x86_64-linux/cmake-native/3.15.3-r0/cmake-3.15.3/Utilities/cmlibarchive/libarchive/archive_read_support_filter_zstd.c:59:2:
> error: unknown type name ‘ZSTD_DStream’
> |   ZSTD_DStream *dstream;
> |   ^
>
> The library does not contain that datatype:
>
> grep -rne ZSTD_DStream /usr/include/zstd.h
> <no output>
>
> Apparently the cmake-native library check is not checking the library
> version or features.
>
> This fix is to not depend on the system library and use the zstd library
> provided with cmake.
>
> CMAKE_EXTRACONF = "\
> <...>
>     -DCMAKE_USE_SYSTEM_LIBRARY_ZSTD=0 \
> <...>
>
> Signed-off-by: Leon Woestenberg <leon@sidebranch.com>
> ---
>  meta/recipes-devtools/cmake/cmake-native_3.16.5.bb | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb
> b/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb
> index b2952ee..e0ac3f8 100644
> --- a/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb
> +++ b/meta/recipes-devtools/cmake/cmake-native_3.16.5.bb
> @@ -21,6 +21,7 @@ CMAKE_EXTRACONF = "\
>      -DCMAKE_USE_SYSTEM_LIBRARY_LIBARCHIVE=0 \
>      -DCMAKE_USE_SYSTEM_LIBRARY_LIBUV=0 \
>      -DCMAKE_USE_SYSTEM_LIBRARY_LIBRHASH=0 \
> +    -DCMAKE_USE_SYSTEM_LIBRARY_ZSTD=0 \
>      -DENABLE_ACL=0 -DHAVE_ACL_LIBACL_H=0 \
>      -DHAVE_SYS_ACL_H=0 \
>  "
> --
> 2.7.4
>
> 
>

[-- Attachment #2: Type: text/html, Size: 3089 bytes --]

Re: [OE-core] [dunfell] [PATCH] cmake-native: Use cmake-provided zstd library; supported host distro zstd may be too old.

[Leon Woestenberg]

Hello Alexander,

On Fri, Aug 14, 2020 at 11:44 PM Alexander Kanavin
<alex.kanavin@gmail.com> wrote:
>
> This needs to go to master first probably?
> Alex
>
> On Fri, 14 Aug 2020 at 22:46, Leon Woestenberg <leon@sidebranch.com> wrote:
>>
>> cmake-native uses the system provided libraries due to:
>> <...>
>> This fix is to not depend on the system library and use the zstd library provided with cmake.
>>

Yes most probably, but master did not build for me so was untestable
for me within reasonable time span I had for this issue.

Regards,

Leon.

Re: [OE-core] [dunfell] [PATCH] cmake-native: Use cmake-provided zstd library; supported host distro zstd may be too old.

[Adrian Bunk]

On Fri, Aug 14, 2020 at 11:44:17PM +0200, Alexander Kanavin wrote:
> This needs to go to master first probably?

Better for master might be moving zstd from meta-openembedded,
and then DEPENDS on zstd-native?

IMHO moving target zstd to OE-core is already overdue,
so this would not be solely for cmake.

> Alex
>...

cu
Adrian

Re: [OE-core] [dunfell] [PATCH] cmake-native: Use cmake-provided zstd library; supported host distro zstd may be too old.

[Leon Woestenberg]

[-- Attachment #1: Type: text/plain, Size: 2080 bytes --]

Hello Adrian, all,

On Sat, Aug 15, 2020 at 9:55 AM Adrian Bunk <bunk@stusta.de> wrote:

> On Fri, Aug 14, 2020 at 11:44:17PM +0200, Alexander Kanavin wrote:
> > This needs to go to master first probably?
>
> Better for master might be moving zstd from meta-openembedded,
> and then DEPENDS on zstd-native?
>
> IMHO moving target zstd to OE-core is already overdue,
> so this would not be solely for cmake.
>
>
Good point, makes better sense then my proposed fix for cmake-native only.

More breakage of master against Ubuntu 16.04 (still supported by Yocto)
that supports Adrian's suggestion:


|
../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:
In function ‘int lto_normalized_zstd_level()’:
|
../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:120:36:
error: ‘ZSTD_maxCLevel’ was not declared in this scope
|    else if (level > ZSTD_maxCLevel ())
|                                     ^
|
../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:
In function ‘void lto_uncompression_zstd(lto_compression_stream*)’:
|
../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:160:74:
error: ‘ZSTD_getFrameContentSize’ was not declared in this scope
|    unsigned long long const rsize = ZSTD_getFrameContentSize (cursor,
size);
|
^
|
../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:161:16:
error: ‘ZSTD_CONTENTSIZE_ERROR’ was not declared in this scope
|    if (rsize == ZSTD_CONTENTSIZE_ERROR)
|                 ^
|
../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:163:21:
error: ‘ZSTD_CONTENTSIZE_UNKNOWN’ was not declared in this scope
|    else if (rsize == ZSTD_CONTENTSIZE_UNKNOWN)
|                      ^
| Makefile:1118: recipe for target 'lto-compress.o' failed
| make[1]: *** [lto-compress.o] Error 1
| make[1]: *** Waiting for unfinished jobs....

Did not deep-dive into this, nor test with Adrian's suggestion yet.

Regards,

Leon.

[-- Attachment #2: Type: text/html, Size: 2830 bytes --]

Re: [OE-core] [dunfell] [PATCH] cmake-native: Use cmake-provided zstd library; supported host distro zstd may be too old.

[Andre McCurdy]

On Wed, Aug 19, 2020 at 1:49 PM Leon Woestenberg <leon@sidebranch.com> wrote:
>
> Hello Adrian, all,
>
> On Sat, Aug 15, 2020 at 9:55 AM Adrian Bunk <bunk@stusta.de> wrote:
>>
>> On Fri, Aug 14, 2020 at 11:44:17PM +0200, Alexander Kanavin wrote:
>> > This needs to go to master first probably?
>>
>> Better for master might be moving zstd from meta-openembedded,
>> and then DEPENDS on zstd-native?
>>
>> IMHO moving target zstd to OE-core is already overdue,
>> so this would not be solely for cmake.
>>
>
> Good point, makes better sense then my proposed fix for cmake-native only.
>
> More breakage of master against Ubuntu 16.04 (still supported by Yocto) that supports Adrian's suggestion:

If cmake-native is auto detecting zstd support from the host (and
therefore building non-deterministically depending on whether or not
the host has zstd development libraries) then the build breakage is
just a symptom. Adding zstd to oe-core would be a good approach to
fixing the bug for master but not for older releases.

>
> | ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c: In function ‘int lto_normalized_zstd_level()’:
> | ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:120:36: error: ‘ZSTD_maxCLevel’ was not declared in this scope
> |    else if (level > ZSTD_maxCLevel ())
> |                                     ^
> | ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c: In function ‘void lto_uncompression_zstd(lto_compression_stream*)’:
> | ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:160:74: error: ‘ZSTD_getFrameContentSize’ was not declared in this scope
> |    unsigned long long const rsize = ZSTD_getFrameContentSize (cursor, size);
> |                                                                           ^
> | ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:161:16: error: ‘ZSTD_CONTENTSIZE_ERROR’ was not declared in this scope
> |    if (rsize == ZSTD_CONTENTSIZE_ERROR)
> |                 ^
> | ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:163:21: error: ‘ZSTD_CONTENTSIZE_UNKNOWN’ was not declared in this scope
> |    else if (rsize == ZSTD_CONTENTSIZE_UNKNOWN)
> |                      ^
> | Makefile:1118: recipe for target 'lto-compress.o' failed
> | make[1]: *** [lto-compress.o] Error 1
> | make[1]: *** Waiting for unfinished jobs....
>
> Did not deep-dive into this, nor test with Adrian's suggestion yet.
>
> Regards,
>
> Leon.
>
>