Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Christian Schrefl]

On 12/14/25 12:54 AM, Rudraksha Gupta wrote:
> Hello all,
> 
> 
> I have the following problem: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635
> 
> 
> In short, what is happening is the following:
> 
> 
> - The kernel boots and outputs via UART when I build the kernel with the following:
> 
> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
> 
> 
> - The kernel doesn't boot and there is no output via UART when I build the kernel with the following:
> 
> make LLVM=1 ARCH="$arm"
> 
> 
> The only difference being: CC="${CC:-gcc}". Is this expected? I think this was present in the Linux kernel ever since Rust was enabled for ARMv7, and I never encountered it because postmarketOS was originally building the first way.


I've managed to the get the build setup for qemu-armv7. For some reason
I could not get past the initrd even on kernels that are supposed to work,
but I think that is unrelated (and not a kernel issue).
On the linux-next kernel I didn't get any output on the console from qemu so I
think I've reproduced the issue. Changing CONFIG_RUST=n did not change the behavior.

So I this is almost certainly a LLVM/clang issue and not a Rust issue. I'll try to
do a bit more digging, but I'm not sure how much I'll get done.

Adding the LLVM and ARM lists to this conversation.

Cheers,
Christian

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Rudraksha Gupta]

On 12/16/25 06:41, Christian Schrefl wrote:
> On 12/14/25 12:54 AM, Rudraksha Gupta wrote:
>> Hello all,
>>
>>
>> I have the following problem: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635
>>
>>
>> In short, what is happening is the following:
>>
>>
>> - The kernel boots and outputs via UART when I build the kernel with the following:
>>
>> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
>>
>>
>> - The kernel doesn't boot and there is no output via UART when I build the kernel with the following:
>>
>> make LLVM=1 ARCH="$arm"
>>
>>
>> The only difference being: CC="${CC:-gcc}". Is this expected? I think this was present in the Linux kernel ever since Rust was enabled for ARMv7, and I never encountered it because postmarketOS was originally building the first way.
>
> I've managed to the get the build setup for qemu-armv7. For some reason
> I could not get past the initrd even on kernels that are supposed to work,
> but I think that is unrelated (and not a kernel issue).

Yep, I just got qemu-arm working to drop into a debug shell for now. I 
have to look into why other things aren't behaving nicely (but that's a 
problem for later me :P). For now, it seems to demonstrate the problem 
nicely:

https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635#note_521740


> On the linux-next kernel I didn't get any output on the console from qemu so I
> think I've reproduced the issue. Changing CONFIG_RUST=n did not change the behavior.
>
> So I this is almost certainly a LLVM/clang issue and not a Rust issue. I'll try to
> do a bit more digging, but I'm not sure how much I'll get done.

Did a little more testing in addition to the testing in the gitlab issue 
mentioned above:

- Removed Rust configs from linux-next/pmos.config -> didn't boot on 
qemu-arm and my phone

- Then I removed Rust dependencies from linux-next/APKBUILD -> didn't 
boot on qemu-arm and my phone

- used linux-stable instead of linux-next -> booted on qemu-arm to a 
debug shell

linux-stable is built via gcc:
https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/community/linux-stable/APKBUILD#L179

linux-next is built via clang:
https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/APKBUILD#L68



>
> Adding the LLVM and ARM lists to this conversation.
>
> Cheers,
> Christian

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Nathan Chancellor]

Hi Rudraksha,

On Wed, Dec 17, 2025 at 02:21:11AM -0800, Rudraksha Gupta wrote:
> On 12/16/25 06:41, Christian Schrefl wrote:
> > On 12/14/25 12:54 AM, Rudraksha Gupta wrote:
> > > Hello all,
> > > 
> > > 
> > > I have the following problem: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635
> > > 
> > > 
> > > In short, what is happening is the following:
> > > 
> > > 
> > > - The kernel boots and outputs via UART when I build the kernel with the following:
> > > 
> > > make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
> > > 
> > > 
> > > - The kernel doesn't boot and there is no output via UART when I build the kernel with the following:
> > > 
> > > make LLVM=1 ARCH="$arm"
> > > 
> > > 
> > > The only difference being: CC="${CC:-gcc}". Is this expected? I think this was present in the Linux kernel ever since Rust was enabled for ARMv7, and I never encountered it because postmarketOS was originally building the first way.
> > 
> > I've managed to the get the build setup for qemu-armv7. For some reason
> > I could not get past the initrd even on kernels that are supposed to work,
> > but I think that is unrelated (and not a kernel issue).
> 
> Yep, I just got qemu-arm working to drop into a debug shell for now. I have
> to look into why other things aren't behaving nicely (but that's a problem
> for later me :P). For now, it seems to demonstrate the problem nicely:
> 
> https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635#note_521740
> 
> 
> > On the linux-next kernel I didn't get any output on the console from qemu so I
> > think I've reproduced the issue. Changing CONFIG_RUST=n did not change the behavior.
> > 
> > So I this is almost certainly a LLVM/clang issue and not a Rust issue. I'll try to
> > do a bit more digging, but I'm not sure how much I'll get done.
> 
> Did a little more testing in addition to the testing in the gitlab issue
> mentioned above:
> 
> - Removed Rust configs from linux-next/pmos.config -> didn't boot on
> qemu-arm and my phone
> 
> - Then I removed Rust dependencies from linux-next/APKBUILD -> didn't boot
> on qemu-arm and my phone
> 
> - used linux-stable instead of linux-next -> booted on qemu-arm to a debug
> shell
> 
> linux-stable is built via gcc:
> https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/community/linux-stable/APKBUILD#L179
> 
> linux-next is built via clang:
> https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/APKBUILD#L68

It certainly seems like LLVM / clang is a factor here based on the fact
that LLVM binutils were being used with GCC based on the original report
using 'LLVM=1 CC=gcc'. A few additional ideas for narrowing this down:

  * Does this reproduce with GNU binutils + clang (i.e.,
    CROSS_COMPILE=... CC=clang)? This would further confirm that clang
    is the cuplrit since GNU binutils and GCC are confirmed working with
    linux-stable, correct?

  * Does this reproduce when linux-stable is built with clang / LLVM=1?
    This would rule out a -next specific regression as well as allow
    diffing the linux-stable GCC configuration with the clang
    configuration to see if there are any configurations that get
    enabled only with clang, which could be another reason no issue is
    seen with GCC.

  * Our continuous integration boot tests several ARM configurations in
    QEMU, including Alpine Linux's:

    https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/20379046102/job/58575229973

    So it is possible that a postmarketOS configuration option
    introduces this?

The results of that testing might give us a more obvious direction to go
for troubleshooting, especially since this appears to reproduce in QEMU,
which should make debugging in gdb possible.

Also, what version of clang is this?

Cheers,
Nathan

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Christian Schrefl]

On 12/19/25 10:11 PM, Nathan Chancellor wrote:
> Hi Rudraksha,
> 
> On Wed, Dec 17, 2025 at 02:21:11AM -0800, Rudraksha Gupta wrote:
>> On 12/16/25 06:41, Christian Schrefl wrote:
>>> On 12/14/25 12:54 AM, Rudraksha Gupta wrote:
>>>> Hello all,
>>>>
>>>>
>>>> I have the following problem: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635
>>>>
>>>>
>>>> In short, what is happening is the following:
>>>>
>>>>
>>>> - The kernel boots and outputs via UART when I build the kernel with the following:
>>>>
>>>> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
>>>>
>>>>
>>>> - The kernel doesn't boot and there is no output via UART when I build the kernel with the following:
>>>>
>>>> make LLVM=1 ARCH="$arm"
>>>>
>>>>
>>>> The only difference being: CC="${CC:-gcc}". Is this expected? I think this was present in the Linux kernel ever since Rust was enabled for ARMv7, and I never encountered it because postmarketOS was originally building the first way.
>>>
>>> I've managed to the get the build setup for qemu-armv7. For some reason
>>> I could not get past the initrd even on kernels that are supposed to work,
>>> but I think that is unrelated (and not a kernel issue).
>>
>> Yep, I just got qemu-arm working to drop into a debug shell for now. I have
>> to look into why other things aren't behaving nicely (but that's a problem
>> for later me :P). For now, it seems to demonstrate the problem nicely:
>>
>> https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635#note_521740
>>
>>
>>> On the linux-next kernel I didn't get any output on the console from qemu so I
>>> think I've reproduced the issue. Changing CONFIG_RUST=n did not change the behavior.
>>>
>>> So I this is almost certainly a LLVM/clang issue and not a Rust issue. I'll try to
>>> do a bit more digging, but I'm not sure how much I'll get done.
>>
>> Did a little more testing in addition to the testing in the gitlab issue
>> mentioned above:
>>
>> - Removed Rust configs from linux-next/pmos.config -> didn't boot on
>> qemu-arm and my phone
>>
>> - Then I removed Rust dependencies from linux-next/APKBUILD -> didn't boot
>> on qemu-arm and my phone
>>
>> - used linux-stable instead of linux-next -> booted on qemu-arm to a debug
>> shell
>>
>> linux-stable is built via gcc:
>> https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/community/linux-stable/APKBUILD#L179
>>
>> linux-next is built via clang:
>> https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/APKBUILD#L68
> 
> It certainly seems like LLVM / clang is a factor here based on the fact
> that LLVM binutils were being used with GCC based on the original report
> using 'LLVM=1 CC=gcc'. A few additional ideas for narrowing this down:
> 
>   * Does this reproduce with GNU binutils + clang (i.e.,
>     CROSS_COMPILE=... CC=clang)? This would further confirm that clang
>     is the cuplrit since GNU binutils and GCC are confirmed working with
>     linux-stable, correct?
> 
>   * Does this reproduce when linux-stable is built with clang / LLVM=1?
>     This would rule out a -next specific regression as well as allow
>     diffing the linux-stable GCC configuration with the clang
>     configuration to see if there are any configurations that get
>     enabled only with clang, which could be another reason no issue is
>     seen with GCC.
> 
>   * Our continuous integration boot tests several ARM configurations in
>     QEMU, including Alpine Linux's:
> 
>     https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/20379046102/job/58575229973
> 
>     So it is possible that a postmarketOS configuration option
>     introduces this?
> 
> The results of that testing might give us a more obvious direction to go
> for troubleshooting, especially since this appears to reproduce in QEMU,
> which should make debugging in gdb possible.

I've found the issue (at least on qemu), it was just that the driver for
the PL011 serial was not included in the config.

I don't know why Rudraksha would have observed any difference disabling
Rust or clang though.
Maybe it's an entirely different issue on Hardware?

> 
> Also, what version of clang is this?
>

Alpine clang version 21.1.8

Cheers,
Christian

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Rudraksha Gupta]

Hi all,


So it seems that the boot.img size was getting too large for lk2nd (the 
bootloader) to handle and was giving these errors as described. Seems 
like lk2nd can't handle ~21 MB boot.img.


After removing networking from the Linux kernel (as it was the simplest 
way to massively reduce the Linux kernel size) as well as building the 
rust modules into the kernel itself, it seems to boot fine now:

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 6.18.0 (root@9e6cbc09906e) (Alpine clang 
version 20.1.8, LLD 20.1.8) #4 SMP PREEMPT Sun Dec 21 23:44:57 UTC 2025
[    0.000000] CPU: ARMv7 Processor [511f04d4] revision 4 (ARMv7), 
cr=10c5787d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction 
cache
[    0.000000] OF: fdt: Machine model: Samsung Galaxy Express SGH-I437

[ ... ]


[    1.487035] rust_minimal: Rust minimal sample (init)
[    1.487297] rust_minimal: Am I built-in? true
[    1.492632] rust_print: Rust printing macros sample (init)
[    1.496873] rust_print: Emergency message (level 0) without args
[    1.502244] rust_print: Alert message (level 1) without args
[    1.508439] rust_print: Critical message (level 2) without args
[    1.514025] rust_print: Error message (level 3) without args
[    1.519702] rust_print: Warning message (level 4) without args
[    1.525592] rust_print: Notice message (level 5) without args
[    1.531239] rust_print: Info message (level 6) without args
[    1.537037] rust_print: A line that is continued without args
[    1.542440] rust_print: Emergency message (level 0) with args
[    1.548348] rust_print: Alert message (level 1) with args
[    1.554069] rust_print: Critical message (level 2) with args
[    1.559470] rust_print: Error message (level 3) with args
[    1.565177] rust_print: Warning message (level 4) with args
[    1.570458] rust_print: Notice message (level 5) with args
[    1.575829] rust_print: Info message (level 6) with args
[    1.581444] rust_print: A line that is continued with args
[    1.586893] rust_print: 1
[    1.586925] rust_print: "hello, world"
[    1.592188] rust_print: [../samples/rust/rust_print_main.rs:35:5] c = 
"hello, world"
[    1.598511] rust_print: Arc<dyn Display> says 42
[    1.598542] rust_print: Arc<dyn Display> says hello, world
[    1.606445] rust_print: "hello, world"


[ ... ]

Welcome to postmarketOS
Kernel 6.18.0 on an armv7l (/dev/ttyMSM0)
samsung-expressatt login: user
Password:
login: can't set groups: Function not implemented


Sorry for wasting everyone's time. I appreciate the helpful responses :)


Thanks,

Rudraksha