[PATCH V5 2/3] rust: Use llvm instead of rust-llvm

[Deepesh.Varatharajan@windriver.com]

From: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com>

Updated the Rust build to depend on llvm instead.

*Summary of discussion with the rust upstream about using latest LLVM instead of Rust maintained LLVM fork.
https://internals.rust-lang.org/t/can-we-use-proper-clang-instead-of-llvm-fork-what-rust-uses/23489

*Upstream LLVM is generally compatible:
- Rust does support building with upstream (vanilla) LLVM, especially the latest
major release and the one or two preceding ones.
https://rustc-dev-guide.rust-lang.org/backend/updating-llvm.html#updating-llvm

*Impact on Yocto Rust upgrades:
- Rust upgrades shall always check for updates on rust forked llvm and backport
the relevant patches to llvm.

*Regarding the rust forked llvm local patches:
- There are no local patches on rust forked llvm other than the backported fixes
from llvm master.

*We are copying the natively built `llvm-config` binary into the target sysroot and running
it. However, this `llvm-config` has compile time dependencies on various other arch's LLVM
libraries because native-llvm is built for all oe-core supported targets.

Attempting to work around this by symlinking the missing libraries from the native sysroot
into the target sysroot leads to mixed architectures in the final `.rlib`. Specifically,
the object files extracted from those symlinked libraries within `librustc_llvm-<hash>.rlib`
are built for the host, while others are correctly built for the target This results in linker
failures due to file format not recognized.

To resolve this, we now build llvm-target also for all oe-core supported architectures in
addition to the native-llvm build. This ensures that `llvm-config` and all associated
libraries are built for the correct target, eliminating cross-architecture contamination
and linker issues.

*We are enabling -DLLVM_INSTALL_UTILS=ON to ensure essential LLVM utilities like FileCheck
are available, as they are required by the Rust build.

Without this, the build fails with an error as below:
| thread 'main' panicked at src/bootstrap/src/core/sanity.rs:315:21:
| FileCheck executable "poky/build/tmp/work/x86_64-linux/rust-native/1.90.0/recipe-sysroot
-native/usr/bin/FileCheck" does not exist

*We now add these flags "-Clink-arg=-lz -Clink-arg=-lzstd" because of this following
diff otherwise we will get errors during link time.

Setup in rust-llvm
-DLLVM_ENABLE_ZLIB=OFF \
-DLLVM_ENABLE_ZSTD=OFF \
-DLLVM_ENABLE_FFI=OFF \

Setup in llvm
-DLLVM_ENABLE_FFI=ON \

*When multilibs enabled:

llvm-config expects static libraries to be located in the lib directory rather than
lib64. However, since we are copying the natively built llvm-config to target sysroot
and running it and llvm-config doesn't know anything about lib64 existence. To accommodate
this without breaking multilib behavior, we are creating a symlink from 'lib' to 'lib64'
directory.

Previously, when we depended on rust-llvm, this worked because we specified:
-DCMAKE_INSTALL_PREFIX:PATH=${libdir}/llvm-rust

With this setup, llvm-config was installed inside ${libdir}/llvm-rust, which included
its own bin and lib directories. Thus, llvm-config located in bin would correctly find
the libraries in the adjacent lib directory.

Even when multilib was enabled or not, llvm-config would still look for libraries under
lib in this structure, so everything functioned as expected.

Signed-off-by: Deepesh Varatharajan <Deepesh.Varatharajan@windriver.com>
---
 meta/recipes-devtools/clang/llvm_git.bb   |  2 +-
 meta/recipes-devtools/rust/rust_1.90.0.bb | 39 ++++++++++++++++++++---
 2 files changed, 36 insertions(+), 5 deletions(-)

diff --git a/meta/recipes-devtools/clang/llvm_git.bb b/meta/recipes-devtools/clang/llvm_git.bb
index d2b060ff88..2f47af8d7a 100644
--- a/meta/recipes-devtools/clang/llvm_git.bb
+++ b/meta/recipes-devtools/clang/llvm_git.bb
@@ -27,7 +27,6 @@ OECMAKE_SOURCEPATH = "${S}/llvm"
 # https://github.com/llvm/llvm-project/blob/main/llvm/CMakeLists.txt
 LLVM_TARGETS_GPU ?= "${@bb.utils.contains_any('DISTRO_FEATURES', 'opencl opengl vulkan', 'AMDGPU;NVPTX;SPIRV', '', d)}"
 LLVM_TARGETS_TO_BUILD ?= "AArch64;ARM;BPF;Mips;PowerPC;RISCV;X86;LoongArch;${LLVM_TARGETS_GPU}"
-LLVM_TARGETS_TO_BUILD:class-target ?= "${@get_clang_host_arch(bb, d)};BPF;${LLVM_TARGETS_GPU}"
 
 LLVM_EXPERIMENTAL_TARGETS_TO_BUILD ?= ""
 
@@ -37,6 +36,7 @@ HF[vardepvalue] = "${HF}"
 
 EXTRA_OECMAKE += "-DCMAKE_BUILD_TYPE=MinSizeRel \
                   -DLLVM_ENABLE_BINDINGS=OFF \
+                  -DLLVM_INSTALL_UTILS=ON \
                   -DLLVM_ENABLE_FFI=ON \
                   -DLLVM_ENABLE_RTTI=ON \
                   -DLLVM_TARGETS_TO_BUILD='${LLVM_TARGETS_TO_BUILD}' \
diff --git a/meta/recipes-devtools/rust/rust_1.90.0.bb b/meta/recipes-devtools/rust/rust_1.90.0.bb
index 5d804c7398..0319d73b93 100644
--- a/meta/recipes-devtools/rust/rust_1.90.0.bb
+++ b/meta/recipes-devtools/rust/rust_1.90.0.bb
@@ -7,7 +7,7 @@ LIC_FILES_CHKSUM = "file://COPYRIGHT;md5=11a3899825f4376896e438c8c753f8dc"
 inherit rust
 inherit cargo_common
 
-DEPENDS += "rust-llvm"
+DEPENDS += "llvm"
 # native rust uses cargo/rustc from binary snapshots to bootstrap
 # but everything else should use our native builds
 DEPENDS:append:class-target = " cargo-native rust-native"
@@ -28,8 +28,8 @@ PV .= "${@bb.utils.contains('RUST_CHANNEL', 'stable', '', '-${RUST_CHANNEL}', d)
 
 export FORCE_CRATE_HASH = "${BB_TASKHASH}"
 
-RUST_ALTERNATE_EXE_PATH ?= "${STAGING_LIBDIR}/llvm-rust/bin/llvm-config"
-RUST_ALTERNATE_EXE_PATH_NATIVE = "${STAGING_LIBDIR_NATIVE}/llvm-rust/bin/llvm-config"
+RUST_ALTERNATE_EXE_PATH ?= "${STAGING_BINDIR}/llvm-config"
+RUST_ALTERNATE_EXE_PATH_NATIVE = "${STAGING_BINDIR_NATIVE}/llvm-config"
 
 # We don't want to use bitbakes vendoring because the rust sources do their
 # own vendoring.
@@ -188,6 +188,37 @@ python do_configure() {
     bb.build.exec_func("setup_cargo_environment", d)
 }
 
+# llvm-config expects static libraries to be in the 'lib' directory rather than 'lib64' when
+# multilibs enabled. Since we are copying the natively built llvm-config into the target sysroot
+# and executing it there, it will default to searching in 'lib', as it is unaware of the 'lib64'
+# directory. To ensure llvm-config can locate the necessary libraries, create a symlink from 'lib'
+do_compile:append:class-target() {
+    # Ensure llvm-config can find static libraries in multilib setup
+    lib64_dir="${STAGING_DIR_TARGET}/usr/lib64"
+    lib_dir="${STAGING_DIR_TARGET}/usr/lib"
+
+    if [ -d "$lib64_dir" ]; then
+        # If lib does not exist, symlink it to lib64
+        if [ ! -e "$lib_dir" ]; then
+            ln -s lib64 "$lib_dir"
+        fi
+
+        # Only do per-file symlinking if lib is a real directory (not symlink)
+        if [ -d "$lib_dir" ] && [ ! -L "$lib_dir" ]; then
+            for lib64_file in "${lib64_dir}"/libLLVM*.a; do
+                if [ -e "$lib64_file" ]; then
+                    lib_name=$(basename "${lib64_file}")
+                    target_link="${lib_dir}/${lib_name}"
+
+                    if [ ! -e "${target_link}" ]; then
+                        ln -s "../lib64/${lib_name}" "${target_link}"
+                    fi
+                fi
+            done
+        fi
+    fi
+}
+
 rust_runx () {
     echo "COMPILE ${PN}" "$@"
 
@@ -199,7 +230,7 @@ rust_runx () {
     unset CXXFLAGS
     unset CPPFLAGS
 
-    export RUSTFLAGS="${RUST_DEBUG_REMAP}"
+    export RUSTFLAGS="${RUST_DEBUG_REMAP} -Clink-arg=-lz -Clink-arg=-lzstd"
 
     # Copy the natively built llvm-config into the target so we can run it. Horrible,
     # but works!
-- 
2.49.0