Re: [Buildroot] [PATCH] toolchain-external: Fix external toolchains when BR2_ROOTFS_MERGED_USR

[Charlie Jenkins <charlie@rivosinc.com>]

On Wed, Mar 26, 2025 at 09:01:01AM +0100, Arnout Vandecappelle wrote:
> 
> 
> On 26/03/2025 03:57, Charlie Jenkins wrote:
> > On Tue, Mar 25, 2025 at 10:47:12PM +0100, Arnout Vandecappelle wrote:
> > > 
> > > 
> > > On 11/03/2025 03:48, Charlie Jenkins wrote:
> > > > External toolchains like crosstool-ng place libraries at /lib.
> > > > When BR2_ROOTFS_MERGED_USR is enabled, these libraries were not being
> > > > copied into the target. This caused these toolchains to be unusable with
> > > > this option.
> > > > 
> > > > WHen BR2_ROOTFS_MERGED_USR is selected, copy the /lib and /lib64
> > > > directories into /usr/lib and /usr/lib64 respectively.
> > > 
> > >   This should already happen, because the lib -> usr/lib symlink should be
> > > created in staging dir before the toolchain sysroot is copied.
> > > 
> > >   But you reported in the other thread that the problem is actually with the
> > > symlink:
> > > 
> > > > $ ls -l ./output/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/libm.so
> > > > lrwxrwxrwx 1 charlie rvs 6 Mar  9 22:58 ./output/host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/libm.so -> ../../
> > > 
> > >   So this looks liek the symlink creation is going wrong. That is done by the
> > > following code:
> > > 
> > > 	for link in $$(find $(STAGING_DIR) -type l); do \
> > > 		target=$$(readlink $${link}) ; \
> > > 		if [ "$${target}" == "$${target$(SHARP_SIGN)/}" ] ; then \
> > > 			continue ; \
> > > 		fi ; \
> > > 		relpath="$(call relpath_prefix,$${target$(SHARP_SIGN)/})" ; \
> > > 		echo "Fixing symlink $${link} from $${target} to
> > > $${relpath}$${target$(SHARP_SIGN)/}" ; \
> > > 		ln -sf $${relpath}$${target$(SHARP_SIGN)/} $${link} ; \
> > > 	done ; \
> > > 	relpath="$(call relpath_prefix,$${ARCH_LIB_DIR})" ; \
> > > 	if [ "$${relpath}" != "" ]; then \
> > > 		for i in $$(find -H $(STAGING_DIR)/$${ARCH_LIB_DIR}
> > > $(STAGING_DIR)/usr/$${ARCH_LIB_DIR} -type l -xtype l); do \
> > > 			LINKTARGET=`readlink $$i` ; \
> > > 			NEWLINKTARGET=$${LINKTARGET\#$$relpath} ; \
> > > 			ln -sf $${NEWLINKTARGET} $$i ; \
> > > 			$(call simplify_symlink,$$i,$(STAGING_DIR)) ; \
> > > 		done ; \
> > > 	fi ; \
> > > 
> > >   So I guess there is a symlink from usr/lib/libm.so to ../../lib/libm.so in
> > > the ct-ng toolchain, and that this doesn't get handled correctly by the code
> > > above. But honestly that code is not xxactly readable...
> > 
> > Are you suggesting that this is a problem with crosstool-ng and not with
> > Buildroot?
> 
>  No not really. It's annoying that ct-ng doesn't use merged usr in its
> sysroot, but not wrong as such.
> 
>  What I'm suggesting is that Buildroot should already copy all the
> libraries, but it is just not creating symlinks correctly. And just rsyncing
> stuff willy-nilly is probably going to break other external toolchains.
> 
> > 
> > > 
> > > > 
> > > > Signed-off-by: Charlie Jenkins <charlie@rivosinc.com>
> > > > ---
> > > > I originally reported this issue here [1].
> > > > 
> > > > [1] https://lore.kernel.org/buildroot/Z86AGdrKHaUqgLVt@ghost/T/#t
> > > > ---
> > > >    toolchain/helpers.mk | 7 +++++++
> > > >    1 file changed, 7 insertions(+)
> > > > 
> > > > diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
> > > > index f3fdaaec07c5a01688da685c4d0fd4e2cb357b95..b565da8a3a8f12567920f14dd3b6a18e548da6c7 100644
> > > > --- a/toolchain/helpers.mk
> > > > +++ b/toolchain/helpers.mk
> > > > @@ -55,6 +55,9 @@ copy_toolchain_lib_root = \
> > > >    # corresponding architecture variants), and we don't want to import
> > > >    # them.
> > > >    #
> > > > +# Furthermore, when BR2_ROOTFS_MERGED_USR is enabled, the 'lib*' directories
> > > > +# need to be copied over to usr/'lib*'.
> > > > +#
> > > >    # If ARCH_LIB_DIR is not a singular directory component, e.g.
> > > >    # 'lib32/octeon2', then symbolic links in ARCH_LIB_DIR and
> > > >    # usr/ARCH_LIB_DIR may be broken because Buildroot will flatten the
> > > > @@ -109,6 +112,10 @@ copy_toolchain_sysroot = \
> > > >    	ARCH_SUBDIR="$(strip $3)"; \
> > > >    	ARCH_LIB_DIR="$(strip $4)" ; \
> > > >    	SUPPORT_LIB_DIR="$(strip $5)" ; \
> > > > +	if [ "$(BR2_ROOTFS_MERGED_USR)" == "y" ]; then \
> > > > +		rsync -au --chmod=u=rwX,go=rX $${ARCH_SYSROOT_DIR}/lib $(STAGING_DIR)/usr/lib 1>&2 ; \
> > > > +		rsync -au --chmod=u=rwX,go=rX $${ARCH_SYSROOT_DIR}/$${ARCH_LIB_DIR} $(STAGING_DIR)/usr/$${ARCH_LIB_DIR} 1>&2 ; \
> > > 
> > >   It looks suspicious to me that you copy both lib and ARCH_LIB_DIR - only
> > > one of them should be valid (or, usually, one is a symlink to the other).
> > > Also, ARCH_LIB_DIR is already handled in the loop below (in a slightly
> > > different way, because thanks to the trailing /, the lib -> usr/lib symlink
> > > is preserved).
> > 
> > Without both, the symlinks are not preserved.
> 
>  Well, obviously symlinks are not preserved correctly, because the Buildroot
> logic takes the libraries from several places and puts all of them in
> usr/lib. So cross-directory symlinks are not correctly preserved. That's why
> there is additional code after the rsync loop that repairs symlinks.
> 
>  I'm not entirely sure how it is possible that these additional rsyncs
> preserve it. I suspect that the second one destroys the lib64->lib symlink
> and this somehow makes the libm.so symlink valid again.
> 
> > Getting rid of the
> > ARCH_LIB_DIR statement here results in the symlink being garbage:
> > 
> > ./host/riscv64-buildroot-linux-gnu/sysroot/usr/lib/libm.so -> ../../
> 
>  I suspect that this garbage is created by the symlink-repairing code that I
> pointed out above. So the correct fix is to fix that symlink code, not to do
> additional rsyncs that duplicate the rsyncs that already exist.
> 

Ooh I see, that makes sense.

> 
>  If you put the toolchain in some location where I can downoad it, I could
> give it a try (but don't count on it too much, my Buildroot contributions
> are unfortunately very unreliable).
> 


I didn't know where to put it so I put it in git with LFS [1]. I also made a
config "external_toolchain_testing_defconfig" that can be used. The
commit that has the workaround is included. Reverting that commit will
cause compilation to fail and the broken symlinks can be found.

Much appreciated if you have any time to look!

[1] https://github.com/charlie-rivos/buildroot

- Charlie

>  Regards,
>  Arnout
> 
> > 
> > - Charlie
> > 
> > > 
> > >   Regards,
> > >   Arnout
> > > 
> > > > +	fi ; \
> > > >    	for i in etc $${ARCH_LIB_DIR} sbin usr usr/$${ARCH_LIB_DIR}; do \
> > > >    		if [ ! -d $${ARCH_SYSROOT_DIR}/$$i ] ; then \
> > > >    			continue ; \
> > > > 
> > > > ---
> > > > base-commit: 4809690d42f9b395ccb76ee7583c199fdd8d42c5
> > > > change-id: 20250310-fix_external_toolchains-d1b8c4da0be8
> > > 
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot