[PATCH v7] rust: use host dylib naming convention

[PATCH v7] rust: use host dylib naming convention

[Tamir Duberstein]

Because the `macros` crate exposes procedural macros, it must be
compiled as a dynamic library (so it can be loaded by the compiler at
compile-time).

Before this change the resulting artifact was always named
`libmacros.so`, which works on hosts where this matches the naming
convention for dynamic libraries. However the proper name on macOS would
be `libmacros.dylib`.

This turns out to matter even when the dependency is passed with a path
(`--extern macros=path/to/libmacros.so` rather than `--extern macros`)
because rustc uses the file name to infer the type of the library (see
link). This is because there's no way to specify both the path to and
the type of the external library via CLI flags. The compiler could
speculatively parse the file to determine its type, but it does not do
so today.

This means that libraries that match neither rustc's naming convention
for static libraries nor the platform's naming convention for dynamic
libraries are *rejected*.

The only solution I've found is to follow the host platform's naming
convention. This patch does that by querying the compiler to determine
the appropriate name for the artifact. This allows the kernel to build
with CONFIG_RUST=y on macOS.

Link: https://github.com/rust-lang/rust/blob/d829780/compiler/rustc_metadata/src/locator.rs#L728-L752
Tested-by: Daniel Gomez <da.gomez@samsung.com>
Co-developed-by: Fiona Behrens <me@kloenk.dev>
Signed-off-by: Fiona Behrens <me@kloenk.dev>
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
Changes in v7:
- Rebase on d072acda4862 ("rust: use custom FFI integer types") [0]
- Link: https://lore.kernel.org/all/20240913213041.395655-4-gary@garyguo.net/ [0]
- Link to v6: https://lore.kernel.org/r/20241016-b4-dylib-host-macos-v6-1-bbeab5029199@gmail.com

Changes in V6: Removed setting `no-clean-files`. Rewrote description.

Changes in V5: Added missing `shell` in rust/Makefile.

Changes in V4: Added motivation. Added missing Signed-off-by.

Changes in V3: Added .strip() to rustc output to remove errant newline.

Changes in V2:
- De-duplicated and sorted imports.
- Changed Signed-off-by to Co-developed-by.
---
 .gitignore                        |  1 +
 Makefile                          |  2 +-
 rust/Makefile                     | 20 +++++++++++---------
 scripts/generate_rust_analyzer.py | 15 +++++++++++----
 4 files changed, 24 insertions(+), 14 deletions(-)

diff --git a/.gitignore b/.gitignore
index 6839cf84acda0d2d3c236a2e42b0cb0fe1b14965..5937c74d3dc1e4ca56d93ea0b8d5ff9fc64ec839 100644
--- a/.gitignore
+++ b/.gitignore
@@ -22,6 +22,7 @@
 *.dtb.S
 *.dtbo.S
 *.dwo
+*.dylib
 *.elf
 *.gcno
 *.gcda
diff --git a/Makefile b/Makefile
index e5b8a8832c0caa16fa05159611dc2529066095c1..80d9943d9744f8288d3fc88bed8213988b0d15c2 100644
--- a/Makefile
+++ b/Makefile
@@ -1571,7 +1571,7 @@ MRPROPER_FILES += include/config include/generated          \
 		  certs/x509.genkey \
 		  vmlinux-gdb.py \
 		  rpmbuild \
-		  rust/libmacros.so
+		  rust/libmacros.so rust/libmacros.dylib
 
 # clean - Delete most, but leave enough to build external modules
 #
diff --git a/rust/Makefile b/rust/Makefile
index a40a3936126d603836e0ec9b42a1285916b60e45..c95d7294f4425a7875a177d005bf0b28d0a9cfb2 100644
--- a/rust/Makefile
+++ b/rust/Makefile
@@ -11,9 +11,6 @@ always-$(CONFIG_RUST) += exports_core_generated.h
 obj-$(CONFIG_RUST) += helpers/helpers.o
 CFLAGS_REMOVE_helpers/helpers.o = -Wmissing-prototypes -Wmissing-declarations
 
-always-$(CONFIG_RUST) += libmacros.so
-no-clean-files += libmacros.so
-
 always-$(CONFIG_RUST) += bindings/bindings_generated.rs bindings/bindings_helpers_generated.rs
 obj-$(CONFIG_RUST) += bindings.o kernel.o
 always-$(CONFIG_RUST) += exports_helpers_generated.h \
@@ -38,9 +35,14 @@ obj-$(CONFIG_RUST_KERNEL_DOCTESTS) += doctests_kernel_generated_kunit.o
 
 always-$(subst y,$(CONFIG_RUST),$(CONFIG_JUMP_LABEL)) += kernel/generated_arch_static_branch_asm.rs
 
-# Avoids running `$(RUSTC)` for the sysroot when it may not be available.
+# Avoids running `$(RUSTC)` when it may not be available.
 ifdef CONFIG_RUST
 
+libmacros_name := $(shell $(RUSTC) --print file-names --crate-name macros --crate-type proc-macro - < /dev/null)
+libmacros_extension := $(patsubst libmacros.%,%,$(libmacros_name))
+
+always-$(CONFIG_RUST) += $(libmacros_name)
+
 # `$(rust_flags)` is passed in case the user added `--sysroot`.
 rustc_sysroot := $(shell MAKEFLAGS= $(RUSTC) $(rust_flags) --print sysroot)
 rustc_host_target := $(shell $(RUSTC) --version --verbose | grep -F 'host: ' | cut -d' ' -f2)
@@ -109,10 +111,10 @@ rustdoc-ffi: $(src)/ffi.rs rustdoc-core FORCE
 	+$(call if_changed,rustdoc)
 
 rustdoc-kernel: private rustc_target_flags = --extern ffi \
-    --extern build_error --extern macros=$(objtree)/$(obj)/libmacros.so \
+    --extern build_error --extern macros \
     --extern bindings --extern uapi
 rustdoc-kernel: $(src)/kernel/lib.rs rustdoc-core rustdoc-ffi rustdoc-macros \
-    rustdoc-compiler_builtins $(obj)/libmacros.so \
+    rustdoc-compiler_builtins $(obj)/$(libmacros_name) \
     $(obj)/bindings.o FORCE
 	+$(call if_changed,rustdoc)
 
@@ -362,10 +364,10 @@ quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@
 		-Clink-args='$(call escsq,$(KBUILD_HOSTLDFLAGS))' \
 		--emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \
 		--crate-type proc-macro \
-		--crate-name $(patsubst lib%.so,%,$(notdir $@)) $<
+		--crate-name $(patsubst lib%.$(libmacros_extension),%,$(notdir $@)) $<
 
 # Procedural macros can only be used with the `rustc` that compiled it.
-$(obj)/libmacros.so: $(src)/macros/lib.rs FORCE
+$(obj)/$(libmacros_name): $(src)/macros/lib.rs FORCE
 	+$(call if_changed_dep,rustc_procmacro)
 
 quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L $@
@@ -443,7 +445,7 @@ $(obj)/uapi.o: $(src)/uapi/lib.rs \
 $(obj)/kernel.o: private rustc_target_flags = --extern ffi \
     --extern build_error --extern macros --extern bindings --extern uapi
 $(obj)/kernel.o: $(src)/kernel/lib.rs $(obj)/build_error.o \
-    $(obj)/libmacros.so $(obj)/bindings.o $(obj)/uapi.o FORCE
+    $(obj)/$(libmacros_name) $(obj)/bindings.o $(obj)/uapi.o FORCE
 	+$(call if_changed_rule,rustc_library)
 
 ifdef CONFIG_JUMP_LABEL
diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index 09e1d166d8d236fcf8c2b2375624b243ebf6e7f7..aa8ea1a4dbe5f9037c8c231d87ddc8d95c297f12 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -8,6 +8,7 @@ import json
 import logging
 import os
 import pathlib
+import subprocess
 import sys
 
 def args_crates_cfgs(cfgs):
@@ -35,8 +36,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     crates_cfgs = args_crates_cfgs(cfgs)
 
     def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False):
-        crates_indexes[display_name] = len(crates)
-        crates.append({
+        crate = {
             "display_name": display_name,
             "root_module": str(root_module),
             "is_workspace_member": is_workspace_member,
@@ -47,7 +47,15 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
             "env": {
                 "RUST_MODFILE": "This is only for rust-analyzer"
             }
-        })
+        }
+        if is_proc_macro:
+            proc_macro_dylib_name = subprocess.check_output(
+                [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
+                stdin=subprocess.DEVNULL,
+            ).decode('utf-8').strip()
+            crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
+        crates_indexes[display_name] = len(crates)
+        crates.append(crate)
 
     # First, the ones in `rust/` since they are a bit special.
     append_crate(
@@ -70,7 +78,6 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         [],
         is_proc_macro=True,
     )
-    crates[-1]["proc_macro_dylib_path"] = f"{objtree}/rust/libmacros.so"
 
     append_crate(
         "build_error",

---
base-commit: 92797e9898f1093641b5b28f6cf5f12da3448a45
change-id: 20241016-b4-dylib-host-macos-16757c350b49

Best regards,
-- 
Tamir Duberstein <tamird@gmail.com>

Re: [PATCH v7] rust: use host dylib naming convention

[Andreas Hindborg]

"Tamir Duberstein" <tamird@gmail.com> writes:

> Because the `macros` crate exposes procedural macros, it must be
> compiled as a dynamic library (so it can be loaded by the compiler at
> compile-time).
>
> Before this change the resulting artifact was always named
> `libmacros.so`, which works on hosts where this matches the naming
> convention for dynamic libraries. However the proper name on macOS would
> be `libmacros.dylib`.
>
> This turns out to matter even when the dependency is passed with a path
> (`--extern macros=path/to/libmacros.so` rather than `--extern macros`)
> because rustc uses the file name to infer the type of the library (see
> link). This is because there's no way to specify both the path to and
> the type of the external library via CLI flags. The compiler could
> speculatively parse the file to determine its type, but it does not do
> so today.
>
> This means that libraries that match neither rustc's naming convention
> for static libraries nor the platform's naming convention for dynamic
> libraries are *rejected*.
>
> The only solution I've found is to follow the host platform's naming
> convention. This patch does that by querying the compiler to determine
> the appropriate name for the artifact. This allows the kernel to build
> with CONFIG_RUST=y on macOS.
>
> Link: https://github.com/rust-lang/rust/blob/d829780/compiler/rustc_metadata/src/locator.rs#L728-L752
> Tested-by: Daniel Gomez <da.gomez@samsung.com>
> Co-developed-by: Fiona Behrens <me@kloenk.dev>
> Signed-off-by: Fiona Behrens <me@kloenk.dev>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

I don't have a mac to test on, but this does not break anything for me
when building with rust enabled on linux.


Tested-by: Andreas Hindborg <a.hindborg@kernel.org>


Best regards,
Andreas Hindborg

Re: [PATCH v7] rust: use host dylib naming convention

[Miguel Ojeda]

On Mon, Dec 16, 2024 at 4:54 PM Tamir Duberstein <tamird@gmail.com> wrote:
>
> the appropriate name for the artifact. This allows the kernel to build
> with CONFIG_RUST=y on macOS.

Applied to `rust-next` -- thanks everyone!

    [ Added `MAKEFLAGS=`s to avoid jobserver warnings. Removed space.
      Reworded title. - Miguel ]

Cheers,
Miguel