Re: [OE-core] [PATCH v2] linux-firmware: Upgrade to 20241017 and allow compressing firmware

[Quentin Schulz <quentin.schulz@cherry.de>]

Hi Zoltan,

On 10/18/24 11:26 AM, Zoltan Boszormenyi via lists.openembedded.org wrote:
> The new 20241017 version contains some breaking changes
> regarding firmware file deduplication.
> 
> Disable running check_whence.py in copy-firmware.sh to avoid
> build issues. Instead, assume that WHENCE as accepted into
> release tarballs is correct.
> 

Which build issues?

Which breaking changes?

Please extend the commit log to provide more info on those so we know 
what we're talking about and if this is indeed the best way to handle 
those issues (now, or in the future). Also, adding comments in the 
do_install where we disable check_whence.py would be nice.

Finally, is this something that was done on purpose by linux-firmware 
maintainers? If yes, are they aware they are breaking stuff, or should 
we file a bug report/send a patch?

> The kernel, dracut and systemd all support compressed firmware
> files. Allow compressing them either as xz or zst which allows
> significant disk space savings.
> 

Please send this as a separate patch so it's easier to review and revert 
if necessary.

> Ship non-compressed firmware as default.
> 
> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> ---
> v2: Fix non-compressed firmware symlinking in do_install:append
> 
>   ...20240909.bb => linux-firmware_20241017.bb} | 364 +++++++++---------
>   1 file changed, 190 insertions(+), 174 deletions(-)
>   rename meta/recipes-kernel/linux-firmware/{linux-firmware_20240909.bb => linux-firmware_20241017.bb} (89%)
> 
> diff --git a/meta/recipes-kernel/linux-firmware/linux-firmware_20240909.bb b/meta/recipes-kernel/linux-firmware/linux-firmware_20241017.bb
> similarity index 89%
> rename from meta/recipes-kernel/linux-firmware/linux-firmware_20240909.bb
> rename to meta/recipes-kernel/linux-firmware/linux-firmware_20241017.bb
> index 4bcda67c8f..f40723cbca 100644
> --- a/meta/recipes-kernel/linux-firmware/linux-firmware_20240909.bb
> +++ b/meta/recipes-kernel/linux-firmware/linux-firmware_20241017.bb
> @@ -158,7 +158,7 @@ LIC_FILES_CHKSUM = "file://LICENCE.Abilis;md5=b5ee3f410780e56711ad48eadc22b8bc \
>                       "
>   # WHENCE checksum is defined separately to ease overriding it if
>   # class-devupstream is selected.
> -WHENCE_CHKSUM  = "6ae5ffd807c84809977286ad0b37acdb"
> +WHENCE_CHKSUM  = "f82849fb6325a8a14e21a4feacc5ebb0"
>   
>   # These are not common licenses, set NO_GENERIC_LICENSE for them
>   # so that the license files will be copied from fetched source
> @@ -247,23 +247,35 @@ SRC_URI:class-devupstream = "git://git.kernel.org/pub/scm/linux/kernel/git/firmw
>   # Pin this to the 20220509 release, override this in local.conf
>   SRCREV:class-devupstream ?= "b19cbdca78ab2adfd210c91be15a22568e8b8cae"
>   
> -SRC_URI[sha256sum] = "943fbd19883cf8eadf89e0b22422549db056557b1ecd30a56400615971369671"
> +SRC_URI[sha256sum] = "a26c38ef5a83272f2b98ce8bf8ca1865a852a3deea49ce5a8dd804b914351273"
>   
>   inherit allarch
>   
>   CLEANBROKEN = "1"
>   
> -# Use PACKAGECONFIG_CONFARGS to set the Makefile target
>   PACKAGECONFIG ??= ""
> -# Enabling dedup will turn duplicate firmware files into links
> -PACKAGECONFIG[deduplicate] = "install,install-nodedup,rdfind-native"
> +PACKAGECONFIG[deduplicate] = ",,rdfind-native"
> +
> +# Possible values are "xz" and "zst".
> +FIRMWARE_COMPRESSION ?= ""
> +
> +def fw_compr_suffix(d):
> +    compr = d.getVar('FIRMWARE_COMPRESSION')
> +    if compr == '':
> +        return ''
> +    return '-' + compr
>   
>   do_compile() {
>   	:
>   }
>   
>   do_install() {
> -        oe_runmake 'DESTDIR=${D}' 'FIRMWAREDIR=${nonarch_base_libdir}/firmware' ${PACKAGECONFIG_CONFARGS}
> +        sed -i 's:^./check_whence.py:#./check_whence.py:' ${S}/copy-firmware.sh
> +
> +        oe_runmake 'DESTDIR=${D}' 'FIRMWAREDIR=${nonarch_base_libdir}/firmware' install${@fw_compr_suffix(d)}
> +        if [ "${@bb.utils.contains('PACKAGECONFIG', 'deduplicate', '1', '0', d)}" = "1" ]; then
> +                oe_runmake 'DESTDIR=${D}' 'FIRMWAREDIR=${nonarch_base_libdir}/firmware' dedup
> +        fi
>           cp LICEN[CS]E.* WHENCE ${D}${nonarch_base_libdir}/firmware/
>           cp wfx/LICEN[CS]E.* ${D}${nonarch_base_libdir}/firmware/wfx/
>   }
> @@ -437,7 +449,7 @@ RDEPENDS:${PN}-amphion-vpu += "${PN}-amphion-vpu-license"
>   LICENSE:${PN}-cw1200 = "Firmware-cw1200"
>   LICENSE:${PN}-cw1200-license = "Firmware-cw1200"
>   
> -FILES:${PN}-cw1200 = "${nonarch_base_libdir}/firmware/wsm_22.bin"
> +FILES:${PN}-cw1200 = "${nonarch_base_libdir}/firmware/wsm_22.bin*"

I assume this is due to the file now being compressed?

I think we should rather explicitly add the compression extension 
because this could be catching more than intended.

This is also probably a bit of an error-prone process for new firmware 
for which we shouldn't forget about adding the * or ${COMP_EXT}. Is 
there something we can do with some python/shell logic to update the 
paths to automatically add this compression extension/suffix? Not sure 
it's really wiser, but suggesting anyway.

Cheers,
Quentin