* [Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses @ 2021-08-29 18:39 Fabrice Fontaine 2021-08-29 19:37 ` Yann E. MORIN 0 siblings, 1 reply; 4+ messages in thread From: Fabrice Fontaine @ 2021-08-29 18:39 UTC (permalink / raw) To: buildroot; +Cc: Fabrice Fontaine Fix per-package build with ncurses and toolchains providing ncurses.h by adding --update parameter to rsync call. Indeed, without this parameter, the curses.h file installed by ncurses (with wchar support) could be overriden by a curses.h file provided by the toolchain (without wchar support) resulting in the following build failure on bmon or rtorrent: rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/rtorrent/host rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/rtorrent/host rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/rtorrent/host rsync -a --link-dest=/tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/rtorrent/host rsync -a --link-dest=/tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host rsync -a --link-dest=/tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/rtorrent/host [...] checking for NcursesW wide-character library... yes checking for working ncursesw/curses.h... no checking for working ncursesw.h... no checking for working ncurses.h... no configure: WARNING: could not find a working ncursesw/curses.h, ncursesw.h or ncurses.h Fixes: - http://autobuild.buildroot.org/results/8c0d13e3cc3ddd2a21f99477cad4e39522e9deec - http://autobuild.buildroot.org/results/d2f18ccad6a0330d78e97d70c810ac89b84d2261 Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com> --- package/pkg-utils.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk index ae3c7f9da9..5af34d4ee0 100644 --- a/package/pkg-utils.mk +++ b/package/pkg-utils.mk @@ -184,7 +184,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define per-package-rsync mkdir -p $(3) $(foreach pkg,$(1),\ - rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ + rsync -au --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ $(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ $(3)$(sep)) endef -- 2.32.0 _______________________________________________ buildroot mailing list buildroot@busybox.net http://lists.busybox.net/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses 2021-08-29 18:39 [Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses Fabrice Fontaine @ 2021-08-29 19:37 ` Yann E. MORIN 2021-08-30 9:20 ` Arnout Vandecappelle 2021-08-30 21:51 ` Thomas Petazzoni 0 siblings, 2 replies; 4+ messages in thread From: Yann E. MORIN @ 2021-08-29 19:37 UTC (permalink / raw) To: Fabrice Fontaine; +Cc: Romain Naour, Herve Codina, Thomas Petazzoni, buildroot Fabrice, All, +Hervé, +Thomas, +Arnout, +Romain Thanks for this patch, that is a very good example of a very tricky problem we've been stumbling upon since we started doing top-level parallel builds... On 2021-08-29 20:39 +0200, Fabrice Fontaine spake thusly: > Fix per-package build with ncurses and toolchains providing ncurses.h by > adding --update parameter to rsync call. Indeed, without this parameter, > the curses.h file installed by ncurses (with wchar support) could be > overriden by a curses.h file provided by the toolchain (without wchar > support) resulting in the following build failure on bmon or rtorrent: > > rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/rtorrent/host > rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host > rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/rtorrent/host > rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/rtorrent/host > rsync -a --link-dest=/tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/rtorrent/host > rsync -a --link-dest=/tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host > rsync -a --link-dest=/tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/rtorrent/host So, if we had had file overwrite detection in place (which I have been reviewing and testing this WE), the build would have failed earlier, at the end of the install step of ncurses, with: /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/ncurses.pc: FAILED /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/menu.pc: FAILED /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/form.pc: FAILED /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/panel.pc: FAILED /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/curses.h: FAILED /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/term.h: FAILED md5sum: WARNING: 6 computed checksums did NOT match ERROR: package ncurses has overwritten files installed by a previous package, aborting. make[1]: *** [package/pkg-generic.mk:467: /home/ymorin/dev/buildroot/O/build/ncurses-6.1/.stamp_installed] Error 1 make: *** [Makefile:23: _all] Error 2 > [...] > > checking for NcursesW wide-character library... yes > checking for working ncursesw/curses.h... no > checking for working ncursesw.h... no > checking for working ncurses.h... no > configure: WARNING: could not find a working ncursesw/curses.h, ncursesw.h or ncurses.h > > Fixes: > - http://autobuild.buildroot.org/results/8c0d13e3cc3ddd2a21f99477cad4e39522e9deec > - http://autobuild.buildroot.org/results/d2f18ccad6a0330d78e97d70c810ac89b84d2261 > > Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com> > --- > package/pkg-utils.mk | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk > index ae3c7f9da9..5af34d4ee0 100644 > --- a/package/pkg-utils.mk > +++ b/package/pkg-utils.mk > @@ -184,7 +184,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) > define per-package-rsync > mkdir -p $(3) > $(foreach pkg,$(1),\ > - rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ > + rsync -au --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ Sorry, this is not the correct solution. Indeed, we can not accept that a package overwrites a file provided by another package. See the explanations Thomas and I discussed earlier today: https://lore.kernel.org/buildroot/20210829133955.74d921e9@windsurf/ https://lore.kernel.org/buildroot/20210829164031.GB1053080@scaer/ This case is however very special, and I have no idea (yet) on what the best solution is. I guess the curses headers in the toolchain are due to the presence of gdb with TUI support... I would be tempted to say "this toolchain is not pure, we can't use it", and defer to the Bootlin toolchain builder project to fix their toolchains, but this is probably a bit of shuffling the dust under the rag... So, any input and insight would be appreciated... Regards, Yann E. MORIN. > $(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ > $(3)$(sep)) > endef > -- > 2.32.0 > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@busybox.net http://lists.busybox.net/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses 2021-08-29 19:37 ` Yann E. MORIN @ 2021-08-30 9:20 ` Arnout Vandecappelle 2021-08-30 21:51 ` Thomas Petazzoni 1 sibling, 0 replies; 4+ messages in thread From: Arnout Vandecappelle @ 2021-08-30 9:20 UTC (permalink / raw) To: Yann E. MORIN, Fabrice Fontaine Cc: Romain Naour, Herve Codina, Thomas Petazzoni, buildroot On 29/08/2021 21:37, Yann E. MORIN wrote: > Fabrice, All, > > +Hervé, +Thomas, +Arnout, +Romain > > Thanks for this patch, that is a very good example of a very tricky > problem we've been stumbling upon since we started doing top-level > parallel builds... > > On 2021-08-29 20:39 +0200, Fabrice Fontaine spake thusly: >> Fix per-package build with ncurses and toolchains providing ncurses.h by >> adding --update parameter to rsync call. Indeed, without this parameter, >> the curses.h file installed by ncurses (with wchar support) could be >> overriden by a curses.h file provided by the toolchain (without wchar >> support) resulting in the following build failure on bmon or rtorrent: >> >> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/host-pkgconf/host/ /tmp/instance-0/output-1/per-package/rtorrent/host >> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/host-skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host >> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/libcurl/host/ /tmp/instance-0/output-1/per-package/rtorrent/host >> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/libtorrent/host/ /tmp/instance-0/output-1/per-package/rtorrent/host >> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/ncurses/host/ /tmp/instance-0/output-1/per-package/rtorrent/host >> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/skeleton/host/ /tmp/instance-0/output-1/per-package/rtorrent/host >> rsync -a --link-dest=/tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/toolchain/host/ /tmp/instance-0/output-1/per-package/rtorrent/host > > So, if we had had file overwrite detection in place (which I have been > reviewing and testing this WE), the build would have failed earlier, at > the end of the install step of ncurses, with: > > /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/ncurses.pc: FAILED > /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/menu.pc: FAILED > /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/form.pc: FAILED > /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/pkgconfig/panel.pc: FAILED > /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/curses.h: FAILED > /home/ymorin/dev/buildroot/O/per-package/ncurses/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/term.h: FAILED > md5sum: WARNING: 6 computed checksums did NOT match > ERROR: package ncurses has overwritten files installed by a previous package, aborting. > make[1]: *** [package/pkg-generic.mk:467: /home/ymorin/dev/buildroot/O/build/ncurses-6.1/.stamp_installed] Error 1 > make: *** [Makefile:23: _all] Error 2 > >> [...] >> >> checking for NcursesW wide-character library... yes >> checking for working ncursesw/curses.h... no >> checking for working ncursesw.h... no >> checking for working ncurses.h... no >> configure: WARNING: could not find a working ncursesw/curses.h, ncursesw.h or ncurses.h >> >> Fixes: >> - http://autobuild.buildroot.org/results/8c0d13e3cc3ddd2a21f99477cad4e39522e9deec >> - http://autobuild.buildroot.org/results/d2f18ccad6a0330d78e97d70c810ac89b84d2261 >> >> Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com> >> --- >> package/pkg-utils.mk | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk >> index ae3c7f9da9..5af34d4ee0 100644 >> --- a/package/pkg-utils.mk >> +++ b/package/pkg-utils.mk >> @@ -184,7 +184,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) >> define per-package-rsync >> mkdir -p $(3) >> $(foreach pkg,$(1),\ >> - rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ >> + rsync -au --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ > > Sorry, this is not the correct solution. Indeed, we can not accept that > a package overwrites a file provided by another package. See the > explanations Thomas and I discussed earlier today: > > https://lore.kernel.org/buildroot/20210829133955.74d921e9@windsurf/ > https://lore.kernel.org/buildroot/20210829164031.GB1053080@scaer/ > > This case is however very special, and I have no idea (yet) on what the > best solution is. I guess the curses headers in the toolchain are due to > the presence of gdb with TUI support... > > I would be tempted to say "this toolchain is not pure, we can't use it", > and defer to the Bootlin toolchain builder project to fix their > toolchains, but this is probably a bit of shuffling the dust under the > rag... > > So, any input and insight would be appreciated... I think we need to start a wiki page for this discussion, on the mailing list the conclusions will get lost too easily... I don't have time for that now though, maybe tonight. Regards, Arnout > > Regards, > Yann E. MORIN. > >> $(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ >> $(3)$(sep)) >> endef >> -- >> 2.32.0 >> >> _______________________________________________ >> buildroot mailing list >> buildroot@busybox.net >> http://lists.busybox.net/mailman/listinfo/buildroot > _______________________________________________ buildroot mailing list buildroot@busybox.net http://lists.busybox.net/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses 2021-08-29 19:37 ` Yann E. MORIN 2021-08-30 9:20 ` Arnout Vandecappelle @ 2021-08-30 21:51 ` Thomas Petazzoni 1 sibling, 0 replies; 4+ messages in thread From: Thomas Petazzoni @ 2021-08-30 21:51 UTC (permalink / raw) To: Yann E. MORIN; +Cc: Romain Naour, Herve Codina, Fabrice Fontaine, buildroot On Sun, 29 Aug 2021 21:37:48 +0200 "Yann E. MORIN" <yann.morin.1998@free.fr> wrote: > Sorry, this is not the correct solution. Indeed, we can not accept that > a package overwrites a file provided by another package. See the > explanations Thomas and I discussed earlier today: > > https://lore.kernel.org/buildroot/20210829133955.74d921e9@windsurf/ > https://lore.kernel.org/buildroot/20210829164031.GB1053080@scaer/ > > This case is however very special, and I have no idea (yet) on what the > best solution is. I guess the curses headers in the toolchain are due to > the presence of gdb with TUI support... gdb with TUI support would mean to have ncurses (library and headers) installed for the host, but here it's in the sysroot of the toolchain, while for the target we only build gdbserver. The two build failures that Fabrice is pointing to are RISC-V 64-bit, and I believe until recently gdbserver was not available on RISC-V 64-bit. So it means that when we enable BR2_PACKAGE_GDB=y when building our RISC-V 64-bit toolchains, instead of getting just the default of gdbserver, we get the full gdb, which pulls in ncurses. This is clearly a bug in the toolchain: we don't want to provide a full gdb for the target. Confirmed with: $ tar tvf aarch64--glibc--bleeding-edge-2020.08-1.tar.bz2 | grep ncurses.h lrwxrwxrwx root/root 0 2020-09-30 21:49 aarch64--glibc--bleeding-edge-2020.08-1/include/ncurses/ncurses.h -> curses.h $ tar tvf riscv64--glibc--bleeding-edge-2020.08-1.tar.bz2 | grep ncurses.h lrwxrwxrwx root/root 0 2020-10-01 10:22 riscv64--glibc--bleeding-edge-2020.08-1/riscv64-buildroot-linux-gnu/sysroot/usr/include/ncurses.h -> curses.h lrwxrwxrwx root/root 0 2020-10-01 10:21 riscv64--glibc--bleeding-edge-2020.08-1/include/ncurses/ncurses.h -> curses.h But in fact, I see we don't even provide gdbserver anywhere in fact, since we're building the toolchain tarball with "make sdk", that doesn't preserve the target directory. So we're building the target gdb package for no reason... Meh. Stuff to fix. Thomas -- Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list buildroot@busybox.net http://lists.busybox.net/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-08-30 21:51 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-08-29 18:39 [Buildroot] [PATCH 1/1] package/pkg-utils.mk: fix per-package build with ncurses Fabrice Fontaine 2021-08-29 19:37 ` Yann E. MORIN 2021-08-30 9:20 ` Arnout Vandecappelle 2021-08-30 21:51 ` Thomas Petazzoni
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox