[Buildroot] Weird issue with uClibc RISC-V 32-bit in Qemu

[Buildroot] Weird issue with uClibc RISC-V 32-bit in Qemu

[Thomas Petazzoni via buildroot]

Hello,

I am working on updating the toolchains of toolchains.bootlin.com, and
as part of that rebuilding all toolchains with Buildroot 2024.05 + a
few patches:

  https://github.com/bootlin/buildroot-toolchains/tree/toolchains.bootlin.com-2024.05

On riscv32-ilp32d, I have the uClibc stable toolchain that generates a
system that doesn't boot under Qemu: the kernel boots fine, but when
switching to user-space, nothing happens:

  https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7378190992

The interesting thing is that the uClibc bleeding-edge toolchain works
fine. The only difference is that GCC is 14.x instead of 13.x, and
binutils is 2.42 instead of 2.41. Bleeding edge toolchain:

  https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7378190991

I took just the rootfs.ext2 of the "working" build, and put it together
with the firmware+kernel of the "non working" build, and it works fine:
so the problem is in the rootfs.

In the broken build, if I pass init=/bin/sh, then I'm dropped into a
shell, but some things are weird:

 - When I run ls, the first line of output starts right after the
   prompt, not in a newline

 - After ls has finished, it doesn't got back to the shell prompt,
   unless I press Enter again

Using init=/bin/sh with the fully working rootfs does not exhibit this
issue.

You can retrieve the non-working artefacts and defconfig at:

  https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7378190992/artifacts/browse/test/output/

And the working artefact and defconfig at:

  https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7378190991/artifacts/browse/test/output/

Do you have any clue at what could be happening here?

Thanks for your support!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Weird issue with uClibc RISC-V 32-bit in Qemu

[Waldemar Brodkorb]

Hi Thomas,
Thomas Petazzoni wrote,

> Hello,
> 
> I am working on updating the toolchains of toolchains.bootlin.com, and
> as part of that rebuilding all toolchains with Buildroot 2024.05 + a
> few patches:
> 
>   https://github.com/bootlin/buildroot-toolchains/tree/toolchains.bootlin.com-2024.05
> 
> On riscv32-ilp32d, I have the uClibc stable toolchain that generates a
> system that doesn't boot under Qemu: the kernel boots fine, but when
> switching to user-space, nothing happens:
> 
>   https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7378190992
> 
> The interesting thing is that the uClibc bleeding-edge toolchain works
> fine. The only difference is that GCC is 14.x instead of 13.x, and
> binutils is 2.42 instead of 2.41. Bleeding edge toolchain:
> 
>   https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7378190991
> 
> I took just the rootfs.ext2 of the "working" build, and put it together
> with the firmware+kernel of the "non working" build, and it works fine:
> so the problem is in the rootfs.
> 
> In the broken build, if I pass init=/bin/sh, then I'm dropped into a
> shell, but some things are weird:
> 
>  - When I run ls, the first line of output starts right after the
>    prompt, not in a newline
> 
>  - After ls has finished, it doesn't got back to the shell prompt,
>    unless I press Enter again
> 
> Using init=/bin/sh with the fully working rootfs does not exhibit this
> issue.
> 
> You can retrieve the non-working artefacts and defconfig at:
> 
>   https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7378190992/artifacts/browse/test/output/
> 
> And the working artefact and defconfig at:
> 
>   https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7378190991/artifacts/browse/test/output/
> 
> Do you have any clue at what could be happening here?
> 
> Thanks for your support!

That is indeed strange behaviour. I can reproduce the issue with the
broken artefacts. The bad news, I can't reproduce it with OpenADK
nor with a fresh compile of buildroot-toolchains (the buildroot
fork) on my laptop. I just used the riscv32 defconfig.

Is the gitlab infrastructure using anything uncommon, like maybe
ccache? 

When adding strace to the rootfs.ext2 I see in the broken image
with strace -f /bin/ash:
...
uname({sysname="Linux", nodename="(none)", ...}) = 0
statx(AT_FDCWD, "/", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=STATX_ATTR_MOUNT_ROOT, stx_mode=S_IFDIR|0755, stx_size=1024, ...}) = 0
statx(AT_FDCWD, ".", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=STATX_ATTR_MOUNT_ROOT, stx_mode=S_IFDIR|0755, stx_size=1024, ...}) = 0
ioctl(0, TCGETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
ioctl(1, TCGETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
rt_sigaction(SIGHUP, {sa_handler=SIG_DFL, sa_mask=[USR1 PIPE CHLD CONT STOP TSTP XFSZ VTALRM PROF IO RT_1], sa_flags=SA_RESTART}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x6860f1fc, sa_mask=[RT_1 RT_2 RT_3 RT_4 RT_5 RT_6 RT_7 RT_8 RT_9 RT_10 RT_11 RT_12 RT_13 RT_14 RT_15 RT_16 RT_17 RT_18 RT_19 RT_20 RT_21 RT_22 RT_23 RT_24 RT_25 RT_26 RT_27 RT_28 RT_29 RT_30 RT_31 RT_32], sa_flags=0}, NULL, 8) = 0
rt_sigaction(SIGQUIT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=SIG_IGN, sa_mask=[RT_1 RT_2 RT_3 RT_4 RT_5 RT_6 RT_7 RT_8 RT_9 RT_10 RT_11 RT_12 RT_13 RT_14 RT_15 RT_16 RT_17 RT_18 RT_19 RT_20 RT_21 RT_22 RT_23 RT_24 RT_25 RT_26 RT_27 RT_28 RT_29 RT_30 RT_31 RT_32], sa_flags=0}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGTERM, {sa_handler=SIG_IGN, sa_mask=[RT_1 RT_2 RT_3 RT_4 RT_5 RT_6 RT_7 RT_8 RT_9 RT_10 RT_11 RT_12 RT_13 RT_14 RT_15 RT_16 RT_17 RT_18 RT_19 RT_20 RT_21 RT_22 RT_23 RT_24 RT_25 RT_26 RT_27 RT_28 RT_29 RT_30 RT_31 RT_32], sa_flags=0}, NULL, 8) = 0
brk(0x686a6000)                         = 0x686a6000
openat(AT_FDCWD, "/dev/tty", O_RDWR|O_LARGEFILE) = -1 ENXIO (No such device or address)
ioctl(2, TCGETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
fcntl64(2, F_DUPFD_CLOEXEC, 10)         = 10
ioctl(10, TIOCGPGRP, 0x9c371c3c)        = -1 ENOTTY (Inappropriate ioctl for device)
write(2, "/bin/ash", 8/bin/ash)                 = 8
write(2, ": ", 2: )                       = 2
write(2, "can't access tty; job control tu"..., 40can't access tty; job control turned off) = 40
write(2, "\n", 1
)                       = 1
close(10)                               = 0
brk(0x686a7000)                         = 0x686a7000
ioctl(0, TCGETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
openat(AT_FDCWD, "/.ash_history", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(0, SNDCTL_TMR_START or TCSETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
ioctl(0, TCGETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
ioctl(0, TIOCGWINSZ, {ws_row=0, ws_col=0, ws_xpixel=0, ws_ypixel=0}) = 0
geteuid()                               = 0
rt_sigaction(SIGWINCH, {sa_handler=0x6866d618, sa_mask=[], sa_flags=SA_RESTART}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
write(1, "~ # ", 4~ # )                     = 4
prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=4*1024}) = 0
syscall_0x48(0x1, 0x9c3718b0, 0x9c371880, 0x9c371850, 0, 0) = -1 ENOSYS (Function not implemented)
read(0, 

And in my working image:
...
uname({sysname="Linux", nodename="(none)", ...}) = 0
statx(AT_FDCWD, "/", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=STATX_ATTR_MOUNT_ROOT, stx_mode=S_IFDIR|0755, stx_size=1024, ...}) = 0
statx(AT_FDCWD, ".", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=STATX_ATTR_MOUNT_ROOT, stx_mode=S_IFDIR|0755, stx_size=1024, ...}) = 0
ioctl(0, TCGETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
ioctl(1, TCGETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
rt_sigaction(SIGHUP, {sa_handler=SIG_DFL, sa_mask=[HUP], sa_flags=SA_RESTART}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x685ac040, sa_mask=~[RTMIN RT_1], sa_flags=0}, NULL, 8) = 0
rt_sigaction(SIGQUIT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=SIG_IGN, sa_mask=~[RTMIN RT_1], sa_flags=0}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGTERM, {sa_handler=SIG_IGN, sa_mask=~[RTMIN RT_1], sa_flags=0}, NULL, 8) = 0
openat(AT_FDCWD, "/dev/tty", O_RDWR|O_LARGEFILE) = -1 ENXIO (No such device or address)
ioctl(2, TCGETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
fcntl64(2, F_DUPFD_CLOEXEC, 10)         = 10
ioctl(10, TIOCGPGRP, 0x9c3fdb8c)        = -1 ENOTTY (Inappropriate ioctl for device)
write(2, "/bin/ash: ", 10/bin/ash: )              = 10
write(2, "can't access tty; job control tu"..., 40can't access tty; job control turned off) = 40
write(2, "\n", 1
)                       = 1
close(10)                               = 0
ioctl(0, TCGETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
openat(AT_FDCWD, "/.ash_history", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
ioctl(0, SNDCTL_TMR_START or TCSETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
ioctl(0, TIOCGWINSZ, {ws_row=0, ws_col=0, ws_xpixel=0, ws_ypixel=0}) = 0
geteuid()                               = 0
statx(1, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFCHR|0600, stx_size=0, ...}) = 0
ioctl(1, TCGETS, {c_iflag=ICRNL|IXON, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B9600|CS8|CREAD|HUPCL|CLOCAL, c_lflag=ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
rt_sigaction(SIGWINCH, {sa_handler=0x6860d9b8, sa_mask=[], sa_flags=SA_RESTART}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
write(1, "~ # ", 4~ # )                     = 4
ppoll_time64([{fd=0, events=POLLIN}], 1, NULL, NULL, 0

Can you explain the difference in behaviour? Seems like some time64 issue, but why it happens on gitlab, but not on my laptop.

You can get a copy of the static strace binary for playing around from here:
https://debug.openadk.org/strace.riscv32

best regards
 Waldemar

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Weird issue with uClibc RISC-V 32-bit in Qemu

[Waldemar Brodkorb]

Hi Thomas,
Thomas Petazzoni wrote,

> Hello,
> 
> I am working on updating the toolchains of toolchains.bootlin.com, and
> as part of that rebuilding all toolchains with Buildroot 2024.05 + a
> few patches:
> 
>   https://github.com/bootlin/buildroot-toolchains/tree/toolchains.bootlin.com-2024.05
> 
> On riscv32-ilp32d, I have the uClibc stable toolchain that generates a
> system that doesn't boot under Qemu: the kernel boots fine, but when
> switching to user-space, nothing happens:
> 
>   https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/7378190992
> 
> The interesting thing is that the uClibc bleeding-edge toolchain works
> fine. The only difference is that GCC is 14.x instead of 13.x, and
> binutils is 2.42 instead of 2.41. Bleeding edge toolchain:

Okay, that is not the only difference.
You are using pre-time64 headers in your gcc 13.x defconfig
(BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_19=y) and newer headers in your
gcc 14.x defconfig (BR2_TOOLCHAIN_EXTERNAL_HEADERS_5_15=y).

That seems the reason for the weird behaviour.
This might be a bug in uClibc-ng time64 implementation. I am unsure
about it.
@Dmitry: What do you think? 

best regards
 Waldemar
 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot