Re: [Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses

["Yann E. MORIN" <yann.morin.1998@free.fr>]

Fabrice, All,

+Hervé, +Thomas, +Arnout, +Romain

Thanks for this patch, that is a very good example of a very tricky
problem we've been stumbling upon since we started doing top-level
parallel builds...

On 2021-08-29 20:39 +0200, Fabrice Fontaine spake thusly:
> Fix per-package build with ncurses and toolchains providing ncurses.h by
> adding --update parameter to rsync call. Indeed, without this parameter,
> the curses.h file installed by ncurses (with wchar support) could be
> overriden by a curses.h file provided by the toolchain (without wchar
> support) resulting in the following build failure on bmon or rtorrent:
> 
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host
> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/rtorrent/host

So, if we had had file overwrite detection in place (which I have been
reviewing and testing this WE), the build would have failed earlier, at
the end of the install step of ncurses, with:

    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/ncurses.pc: FAILED
    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/menu.pc: FAILED
    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/form.pc: FAILED
    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/panel.pc: FAILED
    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/curses.h: FAILED
    /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/term.h: FAILED
    md5sum: WARNING: 6 computed checksums did NOT match
    ERROR: package ncurses has overwritten files installed by a previous package, aborting.
    make[1]: *** [package/pkg-generic.mk:467: /home/ymorin/dev/buildroot/O/build/ncurses-6.1/.stamp_installed] Error 1
    make: *** [Makefile:23: _all] Error 2

> [...]
> 
> checking for NcursesW wide-character library... yes
> checking for working ncursesw/curses.h... no
> checking for working ncursesw.h... no
> checking for working ncurses.h... no
> configure: WARNING: could not find a working ncursesw/curses.h, ncursesw.h or ncurses.h
> 
> Fixes:
>  - http://autobuild.buildroot.org/results/8c0d13e3cc3ddd2a21f99477cad4e39522e9deec
>  - http://autobuild.buildroot.org/results/d2f18ccad6a0330d78e97d70c810ac89b84d2261
> 
> Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
> ---
>  package/pkg-utils.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index ae3c7f9da9..5af34d4ee0 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -184,7 +184,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
>  define per-package-rsync
>  	mkdir -p $(3)
>  	$(foreach pkg,$(1),\
> -		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
> +		rsync -au --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \

Sorry, this is not the correct solution. Indeed, we can not accept that
a package overwrites a file provided by another package. See the
explanations Thomas and I discussed earlier today:

    https://lore.kernel.org/buildroot/20210829133955.74d921e9@windsurf/
    https://lore.kernel.org/buildroot/20210829164031.GB1053080@scaer/

This case is however very special, and I have no idea (yet) on what the
best solution is. I guess the curses headers in the toolchain are due to
the presence of gdb with TUI support...

I would be tempted to say "this toolchain is not pure, we can't use it",
and defer to the Bootlin toolchain builder project to fix their
toolchains, but this is probably a bit of shuffling the dust under the
rag...

So, any input and insight would be appreciated...

Regards,
Yann E. MORIN.

>  		$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
>  		$(3)$(sep))
>  endef
> -- 
> 2.32.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot