ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Rudraksha Gupta]

Hello all,


I have the following problem: 
https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635


In short, what is happening is the following:


- The kernel boots and outputs via UART when I build the kernel with the 
following:

make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"


- The kernel doesn't boot and there is no output via UART when I build 
the kernel with the following:

make LLVM=1 ARCH="$arm"


The only difference being: CC="${CC:-gcc}". Is this expected? I think 
this was present in the Linux kernel ever since Rust was enabled for 
ARMv7, and I never encountered it because postmarketOS was originally 
building the first way.


Thanks,

Rudraksha

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Miguel Ojeda]

On Sun, Dec 14, 2025 at 12:54 AM Rudraksha Gupta <guptarud@gmail.com> wrote:
>
> - The kernel boots and outputs via UART when I build the kernel with the
> following:
>
> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
>
> - The kernel doesn't boot and there is no output via UART when I build
> the kernel with the following:
>
> make LLVM=1 ARCH="$arm"
>
> The only difference being: CC="${CC:-gcc}". Is this expected?

It depends on what that resolves to, i.e. your environment etc., i.e.
that is resolved before Kbuild is called.

The normal way of calling would be the latter anyway -- with the
former you are setting a custom `CC` (either whatever you have there
or the `gcc` default). By default, `LLVM=1` means `CC=clang`.

So it could be that you are using a completely different compiler
(even Clang vs. GCC, or different versions, and so on). Hard to say.
And depending on that, you may end up with a very different kernel
image. Even substantial Kconfig options may get changed etc.

I would suggest comparing the kernel configuration of those two ways
(attaching them here could also be nice to see what compilers you are
using and so on).

Cc'ing Kbuild too so that they are in the loop.

I hope that helps.

Cheers,
Miguel

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Rudraksha Gupta]

On 12/13/25 22:06, Miguel Ojeda wrote:
> On Sun, Dec 14, 2025 at 12:54 AM Rudraksha Gupta <guptarud@gmail.com> wrote:
>> - The kernel boots and outputs via UART when I build the kernel with the
>> following:
>>
>> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
>>
>> - The kernel doesn't boot and there is no output via UART when I build
>> the kernel with the following:
>>
>> make LLVM=1 ARCH="$arm"
>>
>> The only difference being: CC="${CC:-gcc}". Is this expected?
> It depends on what that resolves to, i.e. your environment etc., i.e.
> that is resolved before Kbuild is called.

Sorry about that, I should've specified in the original email. The CC 
resolves to armv7-alpine-linux-musleabihf-gcc.

When both LLVM=1 and the CC=gcc are used, I can insmod the sample rust 
modules just fine. However, if I only use LLVM=1, my phone doesn't 
output anything over UART, and I assume that it fails to boot. 
Interestingly enough, if I just specify LLVM=1 (with no CC=gcc), and 
remove the rust related configs from the pmos.config fragment, then my 
phone boots and I can see logs over UART.


> The normal way of calling would be the latter anyway -- with the
> former you are setting a custom `CC` (either whatever you have there
> or the `gcc` default). By default, `LLVM=1` means `CC=clang`.
>
> So it could be that you are using a completely different compiler
> (even Clang vs. GCC, or different versions, and so on). Hard to say.
> And depending on that, you may end up with a very different kernel
> image. Even substantial Kconfig options may get changed etc.
>
> I would suggest comparing the kernel configuration of those two ways
> (attaching them here could also be nice to see what compilers you are
> using and so on).

postmarketOS uses kernel config fragments and tracks the latest linux-next:

- 
https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/devices.config

- 
https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/pmos.config

- build recipe: 
https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/APKBUILD


The only thing that changed was whether CC=gcc was specified or not:

https://gitlab.postmarketos.org/postmarketOS/pmaports/-/commit/b9102ac5718b8d18acb6801a62e1cd4cc7edc0cb

> Cc'ing Kbuild too so that they are in the loop.
>
> I hope that helps.

Thanks for your help! Always appreciate your presence. :)


> Cheers,
> Miguel

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Gary Guo]

On Sat, 13 Dec 2025 23:34:31 -0800
Rudraksha Gupta <guptarud@gmail.com> wrote:

> On 12/13/25 22:06, Miguel Ojeda wrote:
> > On Sun, Dec 14, 2025 at 12:54 AM Rudraksha Gupta <guptarud@gmail.com> wrote:  
> >> - The kernel boots and outputs via UART when I build the kernel with the
> >> following:
> >>
> >> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
> >>
> >> - The kernel doesn't boot and there is no output via UART when I build
> >> the kernel with the following:
> >>
> >> make LLVM=1 ARCH="$arm"
> >>
> >> The only difference being: CC="${CC:-gcc}". Is this expected?  
> > It depends on what that resolves to, i.e. your environment etc., i.e.
> > that is resolved before Kbuild is called.  
> 
> Sorry about that, I should've specified in the original email. The CC 
> resolves to armv7-alpine-linux-musleabihf-gcc.
> 
> When both LLVM=1 and the CC=gcc are used, I can insmod the sample rust 
> modules just fine. However, if I only use LLVM=1, my phone doesn't 
> output anything over UART, and I assume that it fails to boot. 
> Interestingly enough, if I just specify LLVM=1 (with no CC=gcc), and 
> remove the rust related configs from the pmos.config fragment, then my 
> phone boots and I can see logs over UART.

Which drivers have you enabled that use Rust? Just having core Rust
infrastructure enabled shouldn't cause issues on its own, apart from
slightly bigger kernel image.

If just enabling Rust but none of Rust drivers cause issue, I would start
looking at

1) if there're any symbols somehow being overwritten by the Rust object
files.

2) if the size of kernel is pushed above a certain threshold that your
bootloader/firmware is unhappy.

Best,
Gary

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Christian Schrefl]

On 12/14/25 8:34 AM, Rudraksha Gupta wrote:
> On 12/13/25 22:06, Miguel Ojeda wrote:
>> On Sun, Dec 14, 2025 at 12:54 AM Rudraksha Gupta <guptarud@gmail.com> wrote:
>>> - The kernel boots and outputs via UART when I build the kernel with the
>>> following:
>>>
>>> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
>>>
>>> - The kernel doesn't boot and there is no output via UART when I build
>>> the kernel with the following:
>>>
>>> make LLVM=1 ARCH="$arm"
>>>
>>> The only difference being: CC="${CC:-gcc}". Is this expected?
>> It depends on what that resolves to, i.e. your environment etc., i.e.
>> that is resolved before Kbuild is called.
> 
> Sorry about that, I should've specified in the original email. The CC resolves to armv7-alpine-linux-musleabihf-gcc.
> 
> When both LLVM=1 and the CC=gcc are used, I can insmod the sample rust modules just fine. However, if I only use LLVM=1, my phone doesn't output anything over UART, and I assume that it fails to boot. Interestingly enough, if I just specify LLVM=1 (with no CC=gcc), and remove the rust related configs from the pmos.config fragment, then my phone boots and I can see logs over UART.

Did you try other architectures / devices as well (especially can you reproduce it on qemu-arm)?

Did you try a LLVM=1 build without rust enabled?

> 
>> The normal way of calling would be the latter anyway -- with the
>> former you are setting a custom `CC` (either whatever you have there
>> or the `gcc` default). By default, `LLVM=1` means `CC=clang`.
>>
>> So it could be that you are using a completely different compiler
>> (even Clang vs. GCC, or different versions, and so on). Hard to say.
>> And depending on that, you may end up with a very different kernel
>> image. Even substantial Kconfig options may get changed etc.
>>
>> I would suggest comparing the kernel configuration of those two ways
>> (attaching them here could also be nice to see what compilers you are
>> using and so on).
> 
> postmarketOS uses kernel config fragments and tracks the latest linux-next:
> 
> - https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/devices.config
> 
> - https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/pmos.config
> 
> - build recipe: https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/APKBUILD
> 
> 
> The only thing that changed was whether CC=gcc was specified or not:
> 
> https://gitlab.postmarketos.org/postmarketOS/pmaports/-/commit/b9102ac5718b8d18acb6801a62e1cd4cc7edc0cb
> 

I'm not familiar with pmbootstrap, what is required to reproduce this?
Do I just need to use the edge channel with linux-next or is something special required?

I might habe time to look into trying to reproduce it this week, but I'm not sure.
I have a armv7 based asus-flo device to try it out. Its not very well supported, but
It should be sufficient for this.

Cheers,
Christian

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Christian Schrefl]

On 12/14/25 12:54 AM, Rudraksha Gupta wrote:
> Hello all,
> 
> 
> I have the following problem: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635
> 
> 
> In short, what is happening is the following:
> 
> 
> - The kernel boots and outputs via UART when I build the kernel with the following:
> 
> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
> 
> 
> - The kernel doesn't boot and there is no output via UART when I build the kernel with the following:
> 
> make LLVM=1 ARCH="$arm"
> 
> 
> The only difference being: CC="${CC:-gcc}". Is this expected? I think this was present in the Linux kernel ever since Rust was enabled for ARMv7, and I never encountered it because postmarketOS was originally building the first way.


I've managed to the get the build setup for qemu-armv7. For some reason
I could not get past the initrd even on kernels that are supposed to work,
but I think that is unrelated (and not a kernel issue).
On the linux-next kernel I didn't get any output on the console from qemu so I
think I've reproduced the issue. Changing CONFIG_RUST=n did not change the behavior.

So I this is almost certainly a LLVM/clang issue and not a Rust issue. I'll try to
do a bit more digging, but I'm not sure how much I'll get done.

Adding the LLVM and ARM lists to this conversation.

Cheers,
Christian

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Rudraksha Gupta]

On 12/15/25 03:19, Gary Guo wrote:
> On Sat, 13 Dec 2025 23:34:31 -0800
> Rudraksha Gupta <guptarud@gmail.com> wrote:
>
>> On 12/13/25 22:06, Miguel Ojeda wrote:
>>> On Sun, Dec 14, 2025 at 12:54 AM Rudraksha Gupta <guptarud@gmail.com> wrote:
>>>> - The kernel boots and outputs via UART when I build the kernel with the
>>>> following:
>>>>
>>>> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
>>>>
>>>> - The kernel doesn't boot and there is no output via UART when I build
>>>> the kernel with the following:
>>>>
>>>> make LLVM=1 ARCH="$arm"
>>>>
>>>> The only difference being: CC="${CC:-gcc}". Is this expected?
>>> It depends on what that resolves to, i.e. your environment etc., i.e.
>>> that is resolved before Kbuild is called.
>> Sorry about that, I should've specified in the original email. The CC
>> resolves to armv7-alpine-linux-musleabihf-gcc.
>>
>> When both LLVM=1 and the CC=gcc are used, I can insmod the sample rust
>> modules just fine. However, if I only use LLVM=1, my phone doesn't
>> output anything over UART, and I assume that it fails to boot.
>> Interestingly enough, if I just specify LLVM=1 (with no CC=gcc), and
>> remove the rust related configs from the pmos.config fragment, then my
>> phone boots and I can see logs over UART.
> Which drivers have you enabled that use Rust? Just having core Rust
> infrastructure enabled shouldn't cause issues on its own, apart from
> slightly bigger kernel image.

Just these ones:

https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/pmos.config#L264


I'm starting to think this might be an clang issue. Ran this on qemu-arm 
and encountering similar issues:

https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635#note_521740

> If just enabling Rust but none of Rust drivers cause issue, I would start
> looking at
>
> 1) if there're any symbols somehow being overwritten by the Rust object
> files.

If I suspect it to be a clang issue now, is this still the best way to 
go about looking into the problem? This is a little bit out of my 
domain, but happy to learn.


> 2) if the size of kernel is pushed above a certain threshold that your
> bootloader/firmware is unhappy.

I believe lk2nd (fastboot bootloader) would complain if this were the 
case. I'm currently flashing lk2nd into the boot partition (to override 
the proprietary bootloader's defaults) and then flashing posmarketOS 
kernel and rootfs in data and system partitions.


>
> Best,
> Gary

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Rudraksha Gupta]

On 12/15/25 04:17, Christian Schrefl wrote:
> On 12/14/25 8:34 AM, Rudraksha Gupta wrote:
>> On 12/13/25 22:06, Miguel Ojeda wrote:
>>> On Sun, Dec 14, 2025 at 12:54 AM Rudraksha Gupta <guptarud@gmail.com> wrote:
>>>> - The kernel boots and outputs via UART when I build the kernel with the
>>>> following:
>>>>
>>>> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
>>>>
>>>> - The kernel doesn't boot and there is no output via UART when I build
>>>> the kernel with the following:
>>>>
>>>> make LLVM=1 ARCH="$arm"
>>>>
>>>> The only difference being: CC="${CC:-gcc}". Is this expected?
>>> It depends on what that resolves to, i.e. your environment etc., i.e.
>>> that is resolved before Kbuild is called.
>> Sorry about that, I should've specified in the original email. The CC resolves to armv7-alpine-linux-musleabihf-gcc.
>>
>> When both LLVM=1 and the CC=gcc are used, I can insmod the sample rust modules just fine. However, if I only use LLVM=1, my phone doesn't output anything over UART, and I assume that it fails to boot. Interestingly enough, if I just specify LLVM=1 (with no CC=gcc), and remove the rust related configs from the pmos.config fragment, then my phone boots and I can see logs over UART.
> Did you try other architectures / devices as well (especially can you reproduce it on qemu-arm)?
>
> Did you try a LLVM=1 build without rust enabled?

Seems like you've found the preliminary qemu-arm patches I posted, but 
for completeness sake for others, this is reproducible on qemu-arm:

https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635#note_521740


As Christian mentioned, and I also now seem to conclude, it seems to be 
a clang issue.


>>> The normal way of calling would be the latter anyway -- with the
>>> former you are setting a custom `CC` (either whatever you have there
>>> or the `gcc` default). By default, `LLVM=1` means `CC=clang`.
>>>
>>> So it could be that you are using a completely different compiler
>>> (even Clang vs. GCC, or different versions, and so on). Hard to say.
>>> And depending on that, you may end up with a very different kernel
>>> image. Even substantial Kconfig options may get changed etc.
>>>
>>> I would suggest comparing the kernel configuration of those two ways
>>> (attaching them here could also be nice to see what compilers you are
>>> using and so on).
>> postmarketOS uses kernel config fragments and tracks the latest linux-next:
>>
>> - https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/devices.config
>>
>> - https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/pmos.config
>>
>> - build recipe: https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/APKBUILD
>>
>>
>> The only thing that changed was whether CC=gcc was specified or not:
>>
>> https://gitlab.postmarketos.org/postmarketOS/pmaports/-/commit/b9102ac5718b8d18acb6801a62e1cd4cc7edc0cb
>>
> I'm not familiar with pmbootstrap, what is required to reproduce this?
> Do I just need to use the edge channel with linux-next or is something special required?
>
> I might habe time to look into trying to reproduce it this week, but I'm not sure.
> I have a armv7 based asus-flo device to try it out. Its not very well supported, but
> It should be sufficient for this.

asus-flo would probably also encounter this. Currently it's a bit broken 
in pmbootstrap, but I've provided a general guide here on how to go 
about fixing it:

https://postmarketos.org/edge/2025/11/17/apq8064-kernel-removed/


Would love to help you setup the asus-flo, since it also seems that 
upstream's IOMMU is broken and likely other things as well. If you'd 
like to drop by in the Matrix chat, we'd love to help you there if needed.

Re: ARMv7 Linux + Rust doesn't boot when compiling with only LLVM=1

[Rudraksha Gupta]

On 12/16/25 06:41, Christian Schrefl wrote:
> On 12/14/25 12:54 AM, Rudraksha Gupta wrote:
>> Hello all,
>>
>>
>> I have the following problem: https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635
>>
>>
>> In short, what is happening is the following:
>>
>>
>> - The kernel boots and outputs via UART when I build the kernel with the following:
>>
>> make LLVM=1 ARCH="$arm" CC="${CC:-gcc}"
>>
>>
>> - The kernel doesn't boot and there is no output via UART when I build the kernel with the following:
>>
>> make LLVM=1 ARCH="$arm"
>>
>>
>> The only difference being: CC="${CC:-gcc}". Is this expected? I think this was present in the Linux kernel ever since Rust was enabled for ARMv7, and I never encountered it because postmarketOS was originally building the first way.
>
> I've managed to the get the build setup for qemu-armv7. For some reason
> I could not get past the initrd even on kernels that are supposed to work,
> but I think that is unrelated (and not a kernel issue).

Yep, I just got qemu-arm working to drop into a debug shell for now. I 
have to look into why other things aren't behaving nicely (but that's a 
problem for later me :P). For now, it seems to demonstrate the problem 
nicely:

https://gitlab.postmarketos.org/postmarketOS/pmbootstrap/-/issues/2635#note_521740


> On the linux-next kernel I didn't get any output on the console from qemu so I
> think I've reproduced the issue. Changing CONFIG_RUST=n did not change the behavior.
>
> So I this is almost certainly a LLVM/clang issue and not a Rust issue. I'll try to
> do a bit more digging, but I'm not sure how much I'll get done.

Did a little more testing in addition to the testing in the gitlab issue 
mentioned above:

- Removed Rust configs from linux-next/pmos.config -> didn't boot on 
qemu-arm and my phone

- Then I removed Rust dependencies from linux-next/APKBUILD -> didn't 
boot on qemu-arm and my phone

- used linux-stable instead of linux-next -> booted on qemu-arm to a 
debug shell

linux-stable is built via gcc:
https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/community/linux-stable/APKBUILD#L179

linux-next is built via clang:
https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/device/testing/linux-next/APKBUILD#L68



>
> Adding the LLVM and ARM lists to this conversation.
>
> Cheers,
> Christian