[PATCH v4 00/11] rust: generate_rust_analyzer.py: define host crates and scripts

[PATCH v4 00/11] rust: generate_rust_analyzer.py: define host crates and scripts

[Tamir Duberstein]

This series updates rust-project.json to differentiate between host and
target crates, where the former are used as dependencies of the `macros`
crate. Please see individual commit messages for details.

The first 3 commits contain mechanical formatting changes and are
optional. The series can be taken without them.

I avoided more significant formatting or changes where possible to
reduce the diff. Unfortunately `scripts/generate_rust_analyzer.py` is
not consistently formatted before nor after this series.

The 5th commit ("scripts: generate_rust_analyzer.py: use
str(pathlib.Path)") can also be considered optional. It removes an
inconsistency I noticed while working on this series and which occurs on
a line which churns in this series anyway.

The last 3 commits can also be considered optional, as they can be
submitted separately. I included them in this series because they depend
on it, but they can be split out if this is preferred.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
Changes in v4:
- Fix typo (s/generate/generated/).
- Pull Trevor's suggested change into a separate patch with a
  Suggested-by tag.
- Add patch to avoid file descriptor leak.
- Add patch to generate rust-project.json entries for scripts/*.rs.
- Add patch to use `cfg_groups` to reduce size of `rust-project.json` by
  90%.
- Link to v3: https://lore.kernel.org/r/20250319-rust-analyzer-host-v3-0-311644ee23d2@gmail.com

Changes in v3:
- Rebase on linux-next. This is needed to pick up all the conflicts from
  both rust-next and rust-fixes.
- Drop `uv` from `mypy` command. (Trevor Gross)
- Add `--python-version 3.8` to `mypy` command. (Trevor Gross)
- `from typings import ...` directly. (Trevor Gross)
- Extract `build_crate` and `register_crate` to avoid peeking into
  `crates[-1]`. (Trevor Gross)
- Link to v2: https://lore.kernel.org/r/20250311-rust-analyzer-host-v2-0-30220e116511@gmail.com

Changes in v2:
- Rebased on "rust: fix rust-analyzer configuration for generated files" [1]
  Link: https://lore.kernel.org/all/CANiq72nv7nQ+1BinCHe2qsvwdUb-y9t7x=RGSppi_n9TBXNHpw@mail.gmail.com/ [1]
- Link to v1: https://lore.kernel.org/r/20250209-rust-analyzer-host-v1-0-a2286a2a2fa3@gmail.com

---
Tamir Duberstein (11):
      scripts: generate_rust_analyzer.py: add missing whitespace
      scripts: generate_rust_analyzer.py: use double quotes
      scripts: generate_rust_analyzer.py: add trailing comma
      scripts: generate_rust_analyzer.py: extract `{build,register}_crate`
      scripts: generate_rust_analyzer.py: add type hints
      scripts: generate_rust_analyzer.py: use str(pathlib.Path)
      scripts: generate_rust_analyzer.py: identify crates explicitly
      scripts: generate_rust_analyzer.py: define host crates
      scripts: generate_rust_analyzer.py: avoid FD leak
      scripts: generate_rust_analyzer.py: define scripts
      scripts: generate_rust_analyzer.py: use `cfg_groups`

 scripts/generate_rust_analyzer.py | 257 ++++++++++++++++++++++++++------------
 1 file changed, 180 insertions(+), 77 deletions(-)
---
base-commit: 9388ec571cb1adba59d1cded2300eeb11827679c
change-id: 20250209-rust-analyzer-host-43b108655578

Best regards,
-- 
Tamir Duberstein <tamird@gmail.com>

[PATCH v4 01/11] scripts: generate_rust_analyzer.py: add missing whitespace

[Tamir Duberstein]

Add a space before the `/` operator for consistency with surrounding
code and code formatting tools. Add a second newline between top-level
items in accordance with PEP 8[1]:

> Surround top-level function and class definitions with two blank
lines.

This change was made by a code formatting tool.

Link: https://peps.python.org/pep-0008/ [1]
Reviewed-by: Fiona Behrens <me@kloenk.dev>
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index a0e5a0aef444..fc1788764b31 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -118,7 +118,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     ):
         append_crate(
             display_name,
-            srctree / "rust"/ display_name / "lib.rs",
+            srctree / "rust" / display_name / "lib.rs",
             deps,
             cfg=cfg,
         )
@@ -193,5 +193,6 @@ def main():
 
     json.dump(rust_project, sys.stdout, sort_keys=True, indent=4)
 
+
 if __name__ == "__main__":
     main()

-- 
2.48.1

[PATCH v4 02/11] scripts: generate_rust_analyzer.py: use double quotes

[Tamir Duberstein]

Replace inconsistent use of single quotes with double quotes.

This change was made by a code formatting tool.

Reviewed-by: Fiona Behrens <me@kloenk.dev>
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index fc1788764b31..e2bc4a717f87 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -169,8 +169,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument('--verbose', '-v', action='store_true')
-    parser.add_argument('--cfgs', action='append', default=[])
+    parser.add_argument("--verbose", "-v", action="store_true")
+    parser.add_argument("--cfgs", action="append", default=[])
     parser.add_argument("srctree", type=pathlib.Path)
     parser.add_argument("objtree", type=pathlib.Path)
     parser.add_argument("sysroot", type=pathlib.Path)

-- 
2.48.1

[PATCH v4 03/11] scripts: generate_rust_analyzer.py: add trailing comma

[Tamir Duberstein]

Add missing trailing comma on multi-line function call as suggested by
PEP-8:

> The pattern is to put each value (etc.) on a line by itself, always
> adding a trailing comma, and add the close parenthesis/bracket/brace
> on the next line.

This change was made by a code formatting tool.

Reviewed-by: Fiona Behrens <me@kloenk.dev>
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index e2bc4a717f87..e997d923268d 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -180,7 +180,7 @@ def main():
 
     logging.basicConfig(
         format="[%(asctime)s] [%(levelname)s] %(message)s",
-        level=logging.INFO if args.verbose else logging.WARNING
+        level=logging.INFO if args.verbose else logging.WARNING,
     )
 
     # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.

-- 
2.48.1

[PATCH v4 04/11] scripts: generate_rust_analyzer.py: extract `{build,register}_crate`

[Tamir Duberstein]

Extract helpers from `append_crate` to avoid the need to peek into
`crates[-1]`. This improves readability.

Suggested-by: Trevor Gross <tmgross@umich.edu>
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index e997d923268d..03f55cce673c 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -35,7 +35,14 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     crates_indexes = {}
     crates_cfgs = args_crates_cfgs(cfgs)
 
-    def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False):
+    def build_crate(
+        display_name,
+        root_module,
+        deps,
+        cfg=[],
+        is_workspace_member=True,
+        is_proc_macro=False,
+    ):
         crate = {
             "display_name": display_name,
             "root_module": str(root_module),
@@ -54,9 +61,26 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
                 stdin=subprocess.DEVNULL,
             ).decode('utf-8').strip()
             crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
-        crates_indexes[display_name] = len(crates)
+        return crate
+
+    def register_crate(crate):
+        crates_indexes[crate["display_name"]] = len(crates)
         crates.append(crate)
 
+    def append_crate(
+        display_name,
+        root_module,
+        deps,
+        cfg=[],
+        is_workspace_member=True,
+        is_proc_macro=False,
+    ):
+        register_crate(
+            build_crate(
+                display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro
+            )
+        )
+
     def append_sysroot_crate(
         display_name,
         deps,
@@ -116,20 +140,21 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         display_name,
         deps,
     ):
-        append_crate(
+        crate = build_crate(
             display_name,
             srctree / "rust" / display_name / "lib.rs",
             deps,
             cfg=cfg,
         )
-        crates[-1]["env"]["OBJTREE"] = str(objtree.resolve(True))
-        crates[-1]["source"] = {
+        crate["env"]["OBJTREE"] = str(objtree.resolve(True))
+        crate["source"] = {
             "include_dirs": [
                 str(srctree / "rust" / display_name),
                 str(objtree / "rust")
             ],
             "exclude_dirs": [],
         }
+        register_crate(crate)
 
     append_crate_with_generated("bindings", ["core"])
     append_crate_with_generated("uapi", ["core"])

-- 
2.48.1

[PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints

[Tamir Duberstein]

Python type hints allow static analysis tools like mypy to detect type
errors during development, improving the developer experience.

Python type hints have been present in the kernel since 2019 at the
latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for
running KUnit tests").

Run `mypy --strict scripts/generate_rust_analyzer.py --python-version
3.8` to verify. Note that `mypy` no longer supports python < 3.8.

This removes `"is_proc_macro": false` from `rust-project.json` in
exchange for stricter types. This field is interpreted as false if
absent[1] so this doesn't change the behavior of rust-analyzer.

Link: https://github.com/rust-lang/rust-analyzer/blob/8d01570b5e812a49daa1f08404269f6ea5dd73a1/crates/project-model/src/project_json.rs#L372-L373 [1]
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 172 +++++++++++++++++++++++++-------------
 1 file changed, 112 insertions(+), 60 deletions(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index 03f55cce673c..0772ea309f94 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -10,8 +10,10 @@ import os
 import pathlib
 import subprocess
 import sys
+from typing import Dict, Iterable, List, Literal, Optional, TypedDict
 
-def args_crates_cfgs(cfgs):
+
+def args_crates_cfgs(cfgs: Iterable[str]) -> Dict[str, List[str]]:
     crates_cfgs = {}
     for cfg in cfgs:
         crate, vals = cfg.split("=", 1)
@@ -19,7 +21,45 @@ def args_crates_cfgs(cfgs):
 
     return crates_cfgs
 
-def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
+
+class Dependency(TypedDict):
+    crate: int
+    name: str
+
+
+class Source(TypedDict):
+    include_dirs: List[str]
+    exclude_dirs: List[str]
+
+
+class Crate(TypedDict):
+    display_name: str
+    root_module: str
+    is_workspace_member: bool
+    deps: List[Dependency]
+    cfg: List[str]
+    edition: Literal["2021"]
+    env: Dict[str, str]
+
+
+# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11.
+class ProcMacroCrate(Crate):
+    is_proc_macro: Literal[True]
+    proc_macro_dylib_path: Optional[str]  # `pathlib.Path` is not JSON serializable.
+
+
+# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11.
+class CrateWithGenerated(Crate):
+    source: Optional[Source]
+
+
+def generate_crates(
+    srctree: pathlib.Path,
+    objtree: pathlib.Path,
+    sysroot_src: pathlib.Path,
+    external_src: pathlib.Path,
+    cfgs: List[str],
+) -> List[Crate]:
     # Generate the configuration list.
     cfg = []
     with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
@@ -31,67 +71,75 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     # Now fill the crates list -- dependencies need to come first.
     #
     # Avoid O(n^2) iterations by keeping a map of indexes.
-    crates = []
-    crates_indexes = {}
+    crates: List[Crate] = []
+    crates_indexes: Dict[str, int] = {}
     crates_cfgs = args_crates_cfgs(cfgs)
 
     def build_crate(
-        display_name,
-        root_module,
-        deps,
-        cfg=[],
-        is_workspace_member=True,
-        is_proc_macro=False,
-    ):
-        crate = {
+        display_name: str,
+        root_module: pathlib.Path,
+        deps: List[str],
+        cfg: List[str] = [],
+        is_workspace_member: bool = True,
+    ) -> Crate:
+        return {
             "display_name": display_name,
             "root_module": str(root_module),
             "is_workspace_member": is_workspace_member,
-            "is_proc_macro": is_proc_macro,
             "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps],
             "cfg": cfg,
             "edition": "2021",
             "env": {
                 "RUST_MODFILE": "This is only for rust-analyzer"
-            }
+            },
         }
-        if is_proc_macro:
-            proc_macro_dylib_name = subprocess.check_output(
-                [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
-                stdin=subprocess.DEVNULL,
-            ).decode('utf-8').strip()
-            crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
-        return crate
-
-    def register_crate(crate):
+
+    def register_crate(crate: Crate) -> None:
         crates_indexes[crate["display_name"]] = len(crates)
         crates.append(crate)
 
     def append_crate(
-        display_name,
-        root_module,
-        deps,
-        cfg=[],
-        is_workspace_member=True,
-        is_proc_macro=False,
-    ):
+        display_name: str,
+        root_module: pathlib.Path,
+        deps: List[str],
+        cfg: List[str] = [],
+        is_workspace_member: bool = True,
+    ) -> None:
         register_crate(
-            build_crate(
-                display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro
-            )
+            build_crate(display_name, root_module, deps, cfg, is_workspace_member)
         )
 
+    def append_proc_macro_crate(
+        display_name: str,
+        root_module: pathlib.Path,
+        deps: List[str],
+        cfg: List[str] = [],
+    ) -> None:
+        crate = build_crate(display_name, root_module, deps, cfg)
+        proc_macro_dylib_name = subprocess.check_output(
+            [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
+            stdin=subprocess.DEVNULL,
+        ).decode('utf-8').strip()
+        proc_macro_crate: ProcMacroCrate = {
+            **crate,
+            "is_proc_macro": True,
+            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
+        }
+        register_crate(proc_macro_crate)
+
     def append_sysroot_crate(
-        display_name,
-        deps,
-        cfg=[],
-    ):
-        append_crate(
-            display_name,
-            sysroot_src / display_name / "src" / "lib.rs",
-            deps,
-            cfg,
-            is_workspace_member=False,
+        display_name: str,
+        deps: List[str],
+        cfg: List[str] = [],
+    ) -> None:
+        register_crate(
+            build_crate(
+                display_name,
+                sysroot_src / display_name / "src" / "lib.rs",
+                deps,
+                cfg,
+                is_workspace_member=False,
+            )
         )
 
     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
@@ -108,11 +156,10 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         [],
     )
 
-    append_crate(
+    append_proc_macro_crate(
         "macros",
         srctree / "rust" / "macros" / "lib.rs",
         ["std", "proc_macro"],
-        is_proc_macro=True,
     )
 
     append_crate(
@@ -121,12 +168,11 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         ["core", "compiler_builtins"],
     )
 
-    append_crate(
+    append_proc_macro_crate(
         "pin_init_internal",
         srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs",
         [],
         cfg=["kernel"],
-        is_proc_macro=True,
     )
 
     append_crate(
@@ -137,9 +183,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     )
 
     def append_crate_with_generated(
-        display_name,
-        deps,
-    ):
+        display_name: str,
+        deps: List[str],
+    ) -> None:
         crate = build_crate(
             display_name,
             srctree / "rust" / display_name / "lib.rs",
@@ -147,20 +193,23 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
             cfg=cfg,
         )
         crate["env"]["OBJTREE"] = str(objtree.resolve(True))
-        crate["source"] = {
-            "include_dirs": [
-                str(srctree / "rust" / display_name),
-                str(objtree / "rust")
-            ],
-            "exclude_dirs": [],
+        crate_with_generated: CrateWithGenerated = {
+            **crate,
+            "source": {
+                "include_dirs": [
+                    str(srctree / "rust" / display_name),
+                    str(objtree / "rust")
+                ],
+                "exclude_dirs": [],
+            }
         }
-        register_crate(crate)
+        register_crate(crate_with_generated)
 
     append_crate_with_generated("bindings", ["core"])
     append_crate_with_generated("uapi", ["core"])
     append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "pin_init", "uapi"])
 
-    def is_root_crate(build_file, target):
+    def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
         try:
             return f"{target}.o" in open(build_file).read()
         except FileNotFoundError:
@@ -169,7 +218,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     # Then, the rest outside of `rust/`.
     #
     # We explicitly mention the top-level folders we want to cover.
-    extra_dirs = map(lambda dir: srctree / dir, ("samples", "drivers"))
+    extra_dirs: Iterable[pathlib.Path] = map(
+        lambda dir: srctree / dir, ("samples", "drivers")
+    )
     if external_src is not None:
         extra_dirs = [external_src]
     for folder in extra_dirs:
@@ -192,7 +243,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
 
     return crates
 
-def main():
+
+def main() -> None:
     parser = argparse.ArgumentParser()
     parser.add_argument("--verbose", "-v", action="store_true")
     parser.add_argument("--cfgs", action="append", default=[])

-- 
2.48.1

[PATCH v4 06/11] scripts: generate_rust_analyzer.py: use str(pathlib.Path)

[Tamir Duberstein]

Use the `/` operator on `pathlib.Path` rather than directly crafting a
string. This is consistent with all other path manipulation in this
script.

Reviewed-by: Fiona Behrens <me@kloenk.dev>
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index 0772ea309f94..1ee079c6d916 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -123,7 +123,7 @@ def generate_crates(
         proc_macro_crate: ProcMacroCrate = {
             **crate,
             "is_proc_macro": True,
-            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
+            "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name),
         }
         register_crate(proc_macro_crate)
 

-- 
2.48.1

[PATCH v4 07/11] scripts: generate_rust_analyzer.py: identify crates explicitly

[Tamir Duberstein]

Use the return of `append_crate` to declare dependency on that crate.
This allows multiple crates with the same display_name be defined, which
we'll use to define host crates separately from target crates.

Reviewed-by: Fiona Behrens <me@kloenk.dev>
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 70 +++++++++++++++++++--------------------
 1 file changed, 34 insertions(+), 36 deletions(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index 1ee079c6d916..de1193117161 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -68,17 +68,14 @@ def generate_crates(
             line = line.replace("\n", "")
             cfg.append(line)
 
-    # Now fill the crates list -- dependencies need to come first.
-    #
-    # Avoid O(n^2) iterations by keeping a map of indexes.
+    # Now fill the crates list.
     crates: List[Crate] = []
-    crates_indexes: Dict[str, int] = {}
     crates_cfgs = args_crates_cfgs(cfgs)
 
     def build_crate(
         display_name: str,
         root_module: pathlib.Path,
-        deps: List[str],
+        deps: List[Dependency],
         cfg: List[str] = [],
         is_workspace_member: bool = True,
     ) -> Crate:
@@ -86,7 +83,7 @@ def generate_crates(
             "display_name": display_name,
             "root_module": str(root_module),
             "is_workspace_member": is_workspace_member,
-            "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps],
+            "deps": deps,
             "cfg": cfg,
             "edition": "2021",
             "env": {
@@ -94,27 +91,28 @@ def generate_crates(
             },
         }
 
-    def register_crate(crate: Crate) -> None:
-        crates_indexes[crate["display_name"]] = len(crates)
+    def register_crate(crate: Crate) -> Dependency:
+        index = len(crates)
         crates.append(crate)
+        return {"crate": index, "name": crate["display_name"]}
 
     def append_crate(
         display_name: str,
         root_module: pathlib.Path,
-        deps: List[str],
+        deps: List[Dependency],
         cfg: List[str] = [],
         is_workspace_member: bool = True,
-    ) -> None:
-        register_crate(
+    ) -> Dependency:
+        return register_crate(
             build_crate(display_name, root_module, deps, cfg, is_workspace_member)
         )
 
     def append_proc_macro_crate(
         display_name: str,
         root_module: pathlib.Path,
-        deps: List[str],
+        deps: List[Dependency],
         cfg: List[str] = [],
-    ) -> None:
+    ) -> Dependency:
         crate = build_crate(display_name, root_module, deps, cfg)
         proc_macro_dylib_name = subprocess.check_output(
             [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
@@ -125,14 +123,14 @@ def generate_crates(
             "is_proc_macro": True,
             "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name),
         }
-        register_crate(proc_macro_crate)
+        return register_crate(proc_macro_crate)
 
     def append_sysroot_crate(
         display_name: str,
-        deps: List[str],
+        deps: List[Dependency],
         cfg: List[str] = [],
-    ) -> None:
-        register_crate(
+    ) -> Dependency:
+        return register_crate(
             build_crate(
                 display_name,
                 sysroot_src / display_name / "src" / "lib.rs",
@@ -145,47 +143,47 @@ def generate_crates(
     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
     # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
     # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
-    append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
-    append_sysroot_crate("alloc", ["core"])
-    append_sysroot_crate("std", ["alloc", "core"])
-    append_sysroot_crate("proc_macro", ["core", "std"])
+    core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
+    alloc = append_sysroot_crate("alloc", [core])
+    std = append_sysroot_crate("std", [alloc, core])
+    proc_macro = append_sysroot_crate("proc_macro", [core, std])
 
-    append_crate(
+    compiler_builtins = append_crate(
         "compiler_builtins",
         srctree / "rust" / "compiler_builtins.rs",
         [],
     )
 
-    append_proc_macro_crate(
+    macros = append_proc_macro_crate(
         "macros",
         srctree / "rust" / "macros" / "lib.rs",
-        ["std", "proc_macro"],
+        [std, proc_macro],
     )
 
-    append_crate(
+    build_error = append_crate(
         "build_error",
         srctree / "rust" / "build_error.rs",
-        ["core", "compiler_builtins"],
+        [core, compiler_builtins],
     )
 
-    append_proc_macro_crate(
+    pin_init_internal = append_proc_macro_crate(
         "pin_init_internal",
         srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs",
         [],
         cfg=["kernel"],
     )
 
-    append_crate(
+    pin_init = append_crate(
         "pin_init",
         srctree / "rust" / "pin-init" / "src" / "lib.rs",
-        ["core", "pin_init_internal", "macros"],
+        [core, pin_init_internal, macros],
         cfg=["kernel"],
     )
 
     def append_crate_with_generated(
         display_name: str,
-        deps: List[str],
-    ) -> None:
+        deps: List[Dependency],
+    ) -> Dependency:
         crate = build_crate(
             display_name,
             srctree / "rust" / display_name / "lib.rs",
@@ -203,11 +201,11 @@ def generate_crates(
                 "exclude_dirs": [],
             }
         }
-        register_crate(crate_with_generated)
+        return register_crate(crate_with_generated)
 
-    append_crate_with_generated("bindings", ["core"])
-    append_crate_with_generated("uapi", ["core"])
-    append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "pin_init", "uapi"])
+    bindings = append_crate_with_generated("bindings", [core])
+    uapi = append_crate_with_generated("uapi", [core])
+    kernel = append_crate_with_generated("kernel", [core, macros, build_error, bindings, pin_init, uapi])
 
     def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
         try:
@@ -237,7 +235,7 @@ def generate_crates(
             append_crate(
                 name,
                 path,
-                ["core", "kernel"],
+                [core, kernel],
                 cfg=cfg,
             )
 

-- 
2.48.1

[PATCH v4 08/11] scripts: generate_rust_analyzer.py: define host crates

[Tamir Duberstein]

Define host crates used by the `macros` crate separately from target
crates, now that we can uniquely identify crates with the same name.

This avoids rust-analyzer thinking the host `core` crate has our target
configs applied to it.

Suggested-by: Miguel Ojeda <ojeda@kernel.org>
Link: https://lore.kernel.org/all/CANiq72mw83RmLYeFFoJW6mUUygoyiA_f1ievSC2pmBESsQew+w@mail.gmail.com/
Reviewed-by: Fiona Behrens <me@kloenk.dev>
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index de1193117161..bd6e321a6aa5 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -143,10 +143,12 @@ def generate_crates(
     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
     # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
     # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
+    host_core = append_sysroot_crate("core", [])
+    host_alloc = append_sysroot_crate("alloc", [host_core])
+    host_std = append_sysroot_crate("std", [host_alloc, host_core])
+    host_proc_macro = append_sysroot_crate("proc_macro", [host_core, host_std])
+
     core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
-    alloc = append_sysroot_crate("alloc", [core])
-    std = append_sysroot_crate("std", [alloc, core])
-    proc_macro = append_sysroot_crate("proc_macro", [core, std])
 
     compiler_builtins = append_crate(
         "compiler_builtins",
@@ -157,7 +159,7 @@ def generate_crates(
     macros = append_proc_macro_crate(
         "macros",
         srctree / "rust" / "macros" / "lib.rs",
-        [std, proc_macro],
+        [host_std, host_proc_macro],
     )
 
     build_error = append_crate(

-- 
2.48.1

[PATCH v4 09/11] scripts: generate_rust_analyzer.py: avoid FD leak

[Tamir Duberstein]

Use a context manager to avoid leaking file descriptors.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index bd6e321a6aa5..ccb15aa66929 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -211,7 +211,8 @@ def generate_crates(
 
     def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
         try:
-            return f"{target}.o" in open(build_file).read()
+            with open(build_file) as f:
+                return f"{target}.o" in f.read()
         except FileNotFoundError:
             return False
 

-- 
2.48.1

[PATCH v4 10/11] scripts: generate_rust_analyzer.py: define scripts

[Tamir Duberstein]

Generate rust-project.json entries for scripts written in Rust. This is
possible now that we have a definition for `std` built for the host.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index ccb15aa66929..957b413fe0b6 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -209,6 +209,19 @@ def generate_crates(
     uapi = append_crate_with_generated("uapi", [core])
     kernel = append_crate_with_generated("kernel", [core, macros, build_error, bindings, pin_init, uapi])
 
+    scripts = srctree / "scripts"
+    with open(scripts / "Makefile") as f:
+        makefile = f.read()
+    for path in scripts.glob("*.rs"):
+        name = path.name.replace(".rs", "")
+        if f"{name}-rust" not in makefile:
+            continue
+        _script = append_crate(
+            name,
+            path,
+            [host_std],
+        )
+
     def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
         try:
             with open(build_file) as f:

-- 
2.48.1

[PATCH v4 11/11] scripts: generate_rust_analyzer.py: use `cfg_groups`

[Tamir Duberstein]

Declare common `cfg`s just once to reduce the size of rust-analyzer.json
from 30619 to 2624 lines.

Link: https://github.com/rust-lang/rust-analyzer/commit/2607c09fddef36da0d6f0a84625db5e20a5ebde3
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 35 +++++++++++++++++++++++------------
 1 file changed, 23 insertions(+), 12 deletions(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index 957b413fe0b6..3d89c0198db4 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -37,6 +37,7 @@ class Crate(TypedDict):
     root_module: str
     is_workspace_member: bool
     deps: List[Dependency]
+    cfg_groups: List[str]
     cfg: List[str]
     edition: Literal["2021"]
     env: Dict[str, str]
@@ -59,15 +60,8 @@ def generate_crates(
     sysroot_src: pathlib.Path,
     external_src: pathlib.Path,
     cfgs: List[str],
+    cfg_groups: List[str],
 ) -> List[Crate]:
-    # Generate the configuration list.
-    cfg = []
-    with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
-        for line in fd:
-            line = line.replace("--cfg=", "")
-            line = line.replace("\n", "")
-            cfg.append(line)
-
     # Now fill the crates list.
     crates: List[Crate] = []
     crates_cfgs = args_crates_cfgs(cfgs)
@@ -77,6 +71,7 @@ def generate_crates(
         root_module: pathlib.Path,
         deps: List[Dependency],
         cfg: List[str] = [],
+        cfg_groups: List[str] = [],
         is_workspace_member: bool = True,
     ) -> Crate:
         return {
@@ -85,6 +80,7 @@ def generate_crates(
             "is_workspace_member": is_workspace_member,
             "deps": deps,
             "cfg": cfg,
+            "cfg_groups": cfg_groups,
             "edition": "2021",
             "env": {
                 "RUST_MODFILE": "This is only for rust-analyzer"
@@ -101,10 +97,13 @@ def generate_crates(
         root_module: pathlib.Path,
         deps: List[Dependency],
         cfg: List[str] = [],
+        cfg_groups: List[str] = [],
         is_workspace_member: bool = True,
     ) -> Dependency:
         return register_crate(
-            build_crate(display_name, root_module, deps, cfg, is_workspace_member)
+            build_crate(
+                display_name, root_module, deps, cfg, cfg_groups, is_workspace_member
+            )
         )
 
     def append_proc_macro_crate(
@@ -190,7 +189,7 @@ def generate_crates(
             display_name,
             srctree / "rust" / display_name / "lib.rs",
             deps,
-            cfg=cfg,
+            cfg_groups=cfg_groups,
         )
         crate["env"]["OBJTREE"] = str(objtree.resolve(True))
         crate_with_generated: CrateWithGenerated = {
@@ -252,7 +251,7 @@ def generate_crates(
                 name,
                 path,
                 [core, kernel],
-                cfg=cfg,
+                cfg_groups=cfg_groups,
             )
 
     return crates
@@ -277,9 +276,21 @@ def main() -> None:
     # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.
     assert args.sysroot in args.sysroot_src.parents
 
+    # Generate the configuration list.
+    with open(args.objtree / "include" / "generated" / "rustc_cfg") as fd:
+        cfg_groups = {"rustc_cfg": [line.lstrip("--cfg=").rstrip("\n") for line in fd]}
+
     rust_project = {
-        "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src, args.exttree, args.cfgs),
+        "crates": generate_crates(
+            args.srctree,
+            args.objtree,
+            args.sysroot_src,
+            args.exttree,
+            args.cfgs,
+            list(cfg_groups.keys()),
+        ),
         "sysroot": str(args.sysroot),
+        "cfg_groups": cfg_groups,
     }
 
     json.dump(rust_project, sys.stdout, sort_keys=True, indent=4)

-- 
2.48.1

Re: [PATCH v4 11/11] scripts: generate_rust_analyzer.py: use `cfg_groups`

[Miguel Ojeda]

On Sat, Mar 22, 2025 at 2:24 PM Tamir Duberstein <tamird@gmail.com> wrote:
>
> Declare common `cfg`s just once to reduce the size of rust-analyzer.json
> from 30619 to 2624 lines.
>
> Link: https://github.com/rust-lang/rust-analyzer/commit/2607c09fddef36da0d6f0a84625db5e20a5ebde3
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

A note in case it matters for someone out there: that commit appeared
in tag 2024-08-26, but our minimum Rust version is dated 2024-05-02.

I think many developers are using a rust-analyzer that is newer (i.e.
installed independently), but if others install the component that
gets distributed alongside Rust, then I think it appeared in Rust
1.82.0.

So I wonder if we should wait for the MSRV bump to start using this.

Thanks!

Cheers,
Miguel

Re: [PATCH v4 01/11] scripts: generate_rust_analyzer.py: add missing whitespace

[Daniel Almeida]

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Add a space before the `/` operator for consistency with surrounding
> code and code formatting tools. Add a second newline between top-level
> items in accordance with PEP 8[1]:
> 
>> Surround top-level function and class definitions with two blank
> lines.
> 
> This change was made by a code formatting tool.
> 
> Link: https://peps.python.org/pep-0008/ [1]
> Reviewed-by: Fiona Behrens <me@kloenk.dev>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index a0e5a0aef444..fc1788764b31 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -118,7 +118,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>     ):
>         append_crate(
>             display_name,
> -            srctree / "rust"/ display_name / "lib.rs",
> +            srctree / "rust" / display_name / "lib.rs",
>             deps,
>             cfg=cfg,
>         )
> @@ -193,5 +193,6 @@ def main():
> 
>     json.dump(rust_project, sys.stdout, sort_keys=True, indent=4)
> 
> +
> if __name__ == "__main__":
>     main()
> 
> -- 
> 2.48.1
> 
> 

Reviewed-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 02/11] scripts: generate_rust_analyzer.py: use double quotes

[Daniel Almeida]

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Replace inconsistent use of single quotes with double quotes.
> 
> This change was made by a code formatting tool.
> 
> Reviewed-by: Fiona Behrens <me@kloenk.dev>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index fc1788764b31..e2bc4a717f87 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -169,8 +169,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> 
> def main():
>     parser = argparse.ArgumentParser()
> -    parser.add_argument('--verbose', '-v', action='store_true')
> -    parser.add_argument('--cfgs', action='append', default=[])
> +    parser.add_argument("--verbose", "-v", action="store_true")
> +    parser.add_argument("--cfgs", action="append", default=[])
>     parser.add_argument("srctree", type=pathlib.Path)
>     parser.add_argument("objtree", type=pathlib.Path)
>     parser.add_argument("sysroot", type=pathlib.Path)
> 
> -- 
> 2.48.1
> 
> 

Reviewed-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 03/11] scripts: generate_rust_analyzer.py: add trailing comma

[Daniel Almeida]

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Add missing trailing comma on multi-line function call as suggested by
> PEP-8:
> 
>> The pattern is to put each value (etc.) on a line by itself, always
>> adding a trailing comma, and add the close parenthesis/bracket/brace
>> on the next line.
> 
> This change was made by a code formatting tool.
> 
> Reviewed-by: Fiona Behrens <me@kloenk.dev>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index e2bc4a717f87..e997d923268d 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -180,7 +180,7 @@ def main():
> 
>     logging.basicConfig(
>         format="[%(asctime)s] [%(levelname)s] %(message)s",
> -        level=logging.INFO if args.verbose else logging.WARNING
> +        level=logging.INFO if args.verbose else logging.WARNING,
>     )
> 
>     # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.
> 
> -- 
> 2.48.1
> 
> 

Reviewed-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 04/11] scripts: generate_rust_analyzer.py: extract `{build,register}_crate`

[Daniel Almeida]

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Extract helpers from `append_crate` to avoid the need to peek into
> `crates[-1]`. This improves readability.
> 
> Suggested-by: Trevor Gross <tmgross@umich.edu>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 35 ++++++++++++++++++++++++++++++-----
> 1 file changed, 30 insertions(+), 5 deletions(-)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index e997d923268d..03f55cce673c 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -35,7 +35,14 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>     crates_indexes = {}
>     crates_cfgs = args_crates_cfgs(cfgs)
> 
> -    def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False):
> +    def build_crate(
> +        display_name,
> +        root_module,
> +        deps,
> +        cfg=[],
> +        is_workspace_member=True,
> +        is_proc_macro=False,
> +    ):
>         crate = {
>             "display_name": display_name,
>             "root_module": str(root_module),
> @@ -54,9 +61,26 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>                 stdin=subprocess.DEVNULL,
>             ).decode('utf-8').strip()
>             crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
> -        crates_indexes[display_name] = len(crates)
> +        return crate
> +
> +    def register_crate(crate):
> +        crates_indexes[crate["display_name"]] = len(crates)
>         crates.append(crate)
> 
> +    def append_crate(
> +        display_name,
> +        root_module,
> +        deps,
> +        cfg=[],
> +        is_workspace_member=True,
> +        is_proc_macro=False,
> +    ):
> +        register_crate(
> +            build_crate(
> +                display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro
> +            )
> +        )
> +
>     def append_sysroot_crate(
>         display_name,
>         deps,
> @@ -116,20 +140,21 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>         display_name,
>         deps,
>     ):
> -        append_crate(
> +        crate = build_crate(
>             display_name,
>             srctree / "rust" / display_name / "lib.rs",
>             deps,
>             cfg=cfg,
>         )
> -        crates[-1]["env"]["OBJTREE"] = str(objtree.resolve(True))
> -        crates[-1]["source"] = {
> +        crate["env"]["OBJTREE"] = str(objtree.resolve(True))
> +        crate["source"] = {
>             "include_dirs": [
>                 str(srctree / "rust" / display_name),
>                 str(objtree / "rust")
>             ],
>             "exclude_dirs": [],
>         }
> +        register_crate(crate)
> 
>     append_crate_with_generated("bindings", ["core"])
>     append_crate_with_generated("uapi", ["core"])
> 
> -- 
> 2.48.1
> 
> 

Reviewed-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints

[Daniel Almeida]

Hi Tamir,

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Python type hints allow static analysis tools like mypy to detect type
> errors during development, improving the developer experience.
> 
> Python type hints have been present in the kernel since 2019 at the
> latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for
> running KUnit tests").
> 
> Run `mypy --strict scripts/generate_rust_analyzer.py --python-version
> 3.8` to verify. Note that `mypy` no longer supports python < 3.8.
> 
> This removes `"is_proc_macro": false` from `rust-project.json` in
> exchange for stricter types. This field is interpreted as false if
> absent[1] so this doesn't change the behavior of rust-analyzer.

Can this be a separate patch? Not sure how this is related to Python type
hints, but it makes the current patch harder to review.

> 
> Link: https://github.com/rust-lang/rust-analyzer/blob/8d01570b5e812a49daa1f08404269f6ea5dd73a1/crates/project-model/src/project_json.rs#L372-L373 [1]
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 172 +++++++++++++++++++++++++-------------
> 1 file changed, 112 insertions(+), 60 deletions(-)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index 03f55cce673c..0772ea309f94 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -10,8 +10,10 @@ import os
> import pathlib
> import subprocess
> import sys
> +from typing import Dict, Iterable, List, Literal, Optional, TypedDict
> 
> -def args_crates_cfgs(cfgs):
> +
> +def args_crates_cfgs(cfgs: Iterable[str]) -> Dict[str, List[str]]:
>     crates_cfgs = {}
>     for cfg in cfgs:
>         crate, vals = cfg.split("=", 1)
> @@ -19,7 +21,45 @@ def args_crates_cfgs(cfgs):
> 
>     return crates_cfgs
> 
> -def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> +
> +class Dependency(TypedDict):
> +    crate: int
> +    name: str
> +
> +
> +class Source(TypedDict):
> +    include_dirs: List[str]
> +    exclude_dirs: List[str]
> +
> +
> +class Crate(TypedDict):
> +    display_name: str
> +    root_module: str
> +    is_workspace_member: bool
> +    deps: List[Dependency]
> +    cfg: List[str]
> +    edition: Literal["2021"]
> +    env: Dict[str, str]
> +
> +
> +# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11.
> +class ProcMacroCrate(Crate):
> +    is_proc_macro: Literal[True]
> +    proc_macro_dylib_path: Optional[str]  # `pathlib.Path` is not JSON serializable.
> +
> +
> +# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11.
> +class CrateWithGenerated(Crate):
> +    source: Optional[Source]
> +
> +
> +def generate_crates(
> +    srctree: pathlib.Path,
> +    objtree: pathlib.Path,
> +    sysroot_src: pathlib.Path,
> +    external_src: pathlib.Path,
> +    cfgs: List[str],
> +) -> List[Crate]:
>     # Generate the configuration list.
>     cfg = []
>     with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
> @@ -31,67 +71,75 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>     # Now fill the crates list -- dependencies need to come first.
>     #
>     # Avoid O(n^2) iterations by keeping a map of indexes.
> -    crates = []
> -    crates_indexes = {}
> +    crates: List[Crate] = []
> +    crates_indexes: Dict[str, int] = {}
>     crates_cfgs = args_crates_cfgs(cfgs)
> 
>     def build_crate(
> -        display_name,
> -        root_module,
> -        deps,
> -        cfg=[],
> -        is_workspace_member=True,
> -        is_proc_macro=False,
> -    ):
> -        crate = {
> +        display_name: str,
> +        root_module: pathlib.Path,
> +        deps: List[str],
> +        cfg: List[str] = [],
> +        is_workspace_member: bool = True,
> +    ) -> Crate:
> +        return {
>             "display_name": display_name,
>             "root_module": str(root_module),
>             "is_workspace_member": is_workspace_member,
> -            "is_proc_macro": is_proc_macro,
>             "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps],
>             "cfg": cfg,
>             "edition": "2021",
>             "env": {
>                 "RUST_MODFILE": "This is only for rust-analyzer"
> -            }
> +            },
>         }
> -        if is_proc_macro:
> -            proc_macro_dylib_name = subprocess.check_output(
> -                [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
> -                stdin=subprocess.DEVNULL,
> -            ).decode('utf-8').strip()
> -            crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
> -        return crate
> -
> -    def register_crate(crate):
> +
> +    def register_crate(crate: Crate) -> None:
>         crates_indexes[crate["display_name"]] = len(crates)
>         crates.append(crate)
> 
>     def append_crate(
> -        display_name,
> -        root_module,
> -        deps,
> -        cfg=[],
> -        is_workspace_member=True,
> -        is_proc_macro=False,
> -    ):
> +        display_name: str,
> +        root_module: pathlib.Path,
> +        deps: List[str],
> +        cfg: List[str] = [],
> +        is_workspace_member: bool = True,
> +    ) -> None:
>         register_crate(
> -            build_crate(
> -                display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro
> -            )
> +            build_crate(display_name, root_module, deps, cfg, is_workspace_member)
>         )
> 
> +    def append_proc_macro_crate(
> +        display_name: str,
> +        root_module: pathlib.Path,
> +        deps: List[str],
> +        cfg: List[str] = [],
> +    ) -> None:
> +        crate = build_crate(display_name, root_module, deps, cfg)
> +        proc_macro_dylib_name = subprocess.check_output(
> +            [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
> +            stdin=subprocess.DEVNULL,
> +        ).decode('utf-8').strip()
> +        proc_macro_crate: ProcMacroCrate = {
> +            **crate,
> +            "is_proc_macro": True,
> +            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
> +        }
> +        register_crate(proc_macro_crate)
> +

Same here. Is this supposed to be related to Python type hints somehow?

>     def append_sysroot_crate(
> -        display_name,
> -        deps,
> -        cfg=[],
> -    ):
> -        append_crate(
> -            display_name,
> -            sysroot_src / display_name / "src" / "lib.rs",
> -            deps,
> -            cfg,
> -            is_workspace_member=False,
> +        display_name: str,
> +        deps: List[str],
> +        cfg: List[str] = [],
> +    ) -> None:
> +        register_crate(

Why is register_crate here now? Maybe this change belongs to the preceding patch?

> +            build_crate(
> +                display_name,
> +                sysroot_src / display_name / "src" / "lib.rs",
> +                deps,
> +                cfg,
> +                is_workspace_member=False,
> +            )
>         )
> 
>     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
> @@ -108,11 +156,10 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>         [],
>     )
> 
> -    append_crate(
> +    append_proc_macro_crate(
>         "macros",
>         srctree / "rust" / "macros" / "lib.rs",
>         ["std", "proc_macro"],
> -        is_proc_macro=True,
>     )
> 
>     append_crate(
> @@ -121,12 +168,11 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>         ["core", "compiler_builtins"],
>     )
> 
> -    append_crate(
> +    append_proc_macro_crate(
>         "pin_init_internal",
>         srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs",
>         [],
>         cfg=["kernel"],
> -        is_proc_macro=True,
>     )
> 
>     append_crate(
> @@ -137,9 +183,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>     )
> 
>     def append_crate_with_generated(
> -        display_name,
> -        deps,
> -    ):
> +        display_name: str,
> +        deps: List[str],
> +    ) -> None:
>         crate = build_crate(
>             display_name,
>             srctree / "rust" / display_name / "lib.rs",
> @@ -147,20 +193,23 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>             cfg=cfg,
>         )
>         crate["env"]["OBJTREE"] = str(objtree.resolve(True))
> -        crate["source"] = {
> -            "include_dirs": [
> -                str(srctree / "rust" / display_name),
> -                str(objtree / "rust")
> -            ],
> -            "exclude_dirs": [],
> +        crate_with_generated: CrateWithGenerated = {
> +            **crate,
> +            "source": {
> +                "include_dirs": [
> +                    str(srctree / "rust" / display_name),
> +                    str(objtree / "rust")
> +                ],
> +                "exclude_dirs": [],
> +            }
>         }
> -        register_crate(crate)
> +        register_crate(crate_with_generated)
> 
>     append_crate_with_generated("bindings", ["core"])
>     append_crate_with_generated("uapi", ["core"])
>     append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "pin_init", "uapi"])
> 
> -    def is_root_crate(build_file, target):
> +    def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
>         try:
>             return f"{target}.o" in open(build_file).read()
>         except FileNotFoundError:
> @@ -169,7 +218,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>     # Then, the rest outside of `rust/`.
>     #
>     # We explicitly mention the top-level folders we want to cover.
> -    extra_dirs = map(lambda dir: srctree / dir, ("samples", "drivers"))
> +    extra_dirs: Iterable[pathlib.Path] = map(
> +        lambda dir: srctree / dir, ("samples", "drivers")
> +    )
>     if external_src is not None:
>         extra_dirs = [external_src]
>     for folder in extra_dirs:
> @@ -192,7 +243,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> 
>     return crates
> 
> -def main():
> +

What is this extra blank line for? Automatically generated by a formatter?

> +def main() -> None:
>     parser = argparse.ArgumentParser()
>     parser.add_argument("--verbose", "-v", action="store_true")
>     parser.add_argument("--cfgs", action="append", default=[])
> 
> -- 
> 2.48.1
> 
> 

Other than what I said above, the type hints themselves are fine.

— Daniel

Re: [PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints

[Daniel Almeida]

Ah, by the way:


> On 25 Mar 2025, at 10:37, Daniel Almeida <daniel.almeida@collabora.com> wrote:
> 
> Hi Tamir,
> 
>> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
>> 
>> Python type hints allow static analysis tools like mypy to detect type
>> errors during development, improving the developer experience.
>> 
>> Python type hints have been present in the kernel since 2019 at the
>> latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for
>> running KUnit tests").
>> 
>> Run `mypy --strict scripts/generate_rust_analyzer.py --python-version
>> 3.8` to verify. Note that `mypy` no longer supports python < 3.8.

$ mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8
Success: no issues found in 1 source file

— Daniel

Re: [PATCH v4 06/11] scripts: generate_rust_analyzer.py: use str(pathlib.Path)

[Daniel Almeida]

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Use the `/` operator on `pathlib.Path` rather than directly crafting a
> string. This is consistent with all other path manipulation in this
> script.
> 
> Reviewed-by: Fiona Behrens <me@kloenk.dev>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index 0772ea309f94..1ee079c6d916 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -123,7 +123,7 @@ def generate_crates(
>         proc_macro_crate: ProcMacroCrate = {
>             **crate,
>             "is_proc_macro": True,
> -            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
> +            "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name),
>         }
>         register_crate(proc_macro_crate)
> 
> 
> -- 
> 2.48.1
> 
> 

Reviewed-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 07/11] scripts: generate_rust_analyzer.py: identify crates explicitly

[Daniel Almeida]

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Use the return of `append_crate` to declare dependency on that crate.
> This allows multiple crates with the same display_name be defined, which
> we'll use to define host crates separately from target crates.
> 
> Reviewed-by: Fiona Behrens <me@kloenk.dev>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 70 +++++++++++++++++++--------------------
> 1 file changed, 34 insertions(+), 36 deletions(-)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index 1ee079c6d916..de1193117161 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -68,17 +68,14 @@ def generate_crates(
>             line = line.replace("\n", "")
>             cfg.append(line)
> 
> -    # Now fill the crates list -- dependencies need to come first.
> -    #
> -    # Avoid O(n^2) iterations by keeping a map of indexes.
> +    # Now fill the crates list.
>     crates: List[Crate] = []
> -    crates_indexes: Dict[str, int] = {}
>     crates_cfgs = args_crates_cfgs(cfgs)
> 
>     def build_crate(
>         display_name: str,
>         root_module: pathlib.Path,
> -        deps: List[str],
> +        deps: List[Dependency],
>         cfg: List[str] = [],
>         is_workspace_member: bool = True,
>     ) -> Crate:
> @@ -86,7 +83,7 @@ def generate_crates(
>             "display_name": display_name,
>             "root_module": str(root_module),
>             "is_workspace_member": is_workspace_member,
> -            "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps],
> +            "deps": deps,
>             "cfg": cfg,
>             "edition": "2021",
>             "env": {
> @@ -94,27 +91,28 @@ def generate_crates(
>             },
>         }
> 
> -    def register_crate(crate: Crate) -> None:
> -        crates_indexes[crate["display_name"]] = len(crates)
> +    def register_crate(crate: Crate) -> Dependency:
> +        index = len(crates)
>         crates.append(crate)
> +        return {"crate": index, "name": crate["display_name"]}
> 
>     def append_crate(
>         display_name: str,
>         root_module: pathlib.Path,
> -        deps: List[str],
> +        deps: List[Dependency],
>         cfg: List[str] = [],
>         is_workspace_member: bool = True,
> -    ) -> None:
> -        register_crate(
> +    ) -> Dependency:
> +        return register_crate(
>             build_crate(display_name, root_module, deps, cfg, is_workspace_member)
>         )
> 
>     def append_proc_macro_crate(
>         display_name: str,
>         root_module: pathlib.Path,
> -        deps: List[str],
> +        deps: List[Dependency],
>         cfg: List[str] = [],
> -    ) -> None:
> +    ) -> Dependency:
>         crate = build_crate(display_name, root_module, deps, cfg)
>         proc_macro_dylib_name = subprocess.check_output(
>             [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
> @@ -125,14 +123,14 @@ def generate_crates(
>             "is_proc_macro": True,
>             "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name),
>         }
> -        register_crate(proc_macro_crate)
> +        return register_crate(proc_macro_crate)
> 
>     def append_sysroot_crate(
>         display_name: str,
> -        deps: List[str],
> +        deps: List[Dependency],
>         cfg: List[str] = [],
> -    ) -> None:
> -        register_crate(
> +    ) -> Dependency:
> +        return register_crate(
>             build_crate(
>                 display_name,
>                 sysroot_src / display_name / "src" / "lib.rs",
> @@ -145,47 +143,47 @@ def generate_crates(
>     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
>     # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
>     # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
> -    append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
> -    append_sysroot_crate("alloc", ["core"])
> -    append_sysroot_crate("std", ["alloc", "core"])
> -    append_sysroot_crate("proc_macro", ["core", "std"])
> +    core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
> +    alloc = append_sysroot_crate("alloc", [core])
> +    std = append_sysroot_crate("std", [alloc, core])
> +    proc_macro = append_sysroot_crate("proc_macro", [core, std])
> 
> -    append_crate(
> +    compiler_builtins = append_crate(
>         "compiler_builtins",
>         srctree / "rust" / "compiler_builtins.rs",
>         [],
>     )
> 
> -    append_proc_macro_crate(
> +    macros = append_proc_macro_crate(
>         "macros",
>         srctree / "rust" / "macros" / "lib.rs",
> -        ["std", "proc_macro"],
> +        [std, proc_macro],
>     )
> 
> -    append_crate(
> +    build_error = append_crate(
>         "build_error",
>         srctree / "rust" / "build_error.rs",
> -        ["core", "compiler_builtins"],
> +        [core, compiler_builtins],
>     )
> 
> -    append_proc_macro_crate(
> +    pin_init_internal = append_proc_macro_crate(
>         "pin_init_internal",
>         srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs",
>         [],
>         cfg=["kernel"],
>     )
> 
> -    append_crate(
> +    pin_init = append_crate(
>         "pin_init",
>         srctree / "rust" / "pin-init" / "src" / "lib.rs",
> -        ["core", "pin_init_internal", "macros"],
> +        [core, pin_init_internal, macros],
>         cfg=["kernel"],
>     )
> 
>     def append_crate_with_generated(
>         display_name: str,
> -        deps: List[str],
> -    ) -> None:
> +        deps: List[Dependency],
> +    ) -> Dependency:
>         crate = build_crate(
>             display_name,
>             srctree / "rust" / display_name / "lib.rs",
> @@ -203,11 +201,11 @@ def generate_crates(
>                 "exclude_dirs": [],
>             }
>         }
> -        register_crate(crate_with_generated)
> +        return register_crate(crate_with_generated)
> 
> -    append_crate_with_generated("bindings", ["core"])
> -    append_crate_with_generated("uapi", ["core"])
> -    append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "pin_init", "uapi"])
> +    bindings = append_crate_with_generated("bindings", [core])
> +    uapi = append_crate_with_generated("uapi", [core])
> +    kernel = append_crate_with_generated("kernel", [core, macros, build_error, bindings, pin_init, uapi])
> 
>     def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
>         try:
> @@ -237,7 +235,7 @@ def generate_crates(
>             append_crate(
>                 name,
>                 path,
> -                ["core", "kernel"],
> +                [core, kernel],
>                 cfg=cfg,
>             )
> 
> 
> -- 
> 2.48.1
> 
> 

Reviewed-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 08/11] scripts: generate_rust_analyzer.py: define host crates

[Daniel Almeida]

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Define host crates used by the `macros` crate separately from target
> crates, now that we can uniquely identify crates with the same name.
> 
> This avoids rust-analyzer thinking the host `core` crate has our target
> configs applied to it.
> 
> Suggested-by: Miguel Ojeda <ojeda@kernel.org>
> Link: https://lore.kernel.org/all/CANiq72mw83RmLYeFFoJW6mUUygoyiA_f1ievSC2pmBESsQew+w@mail.gmail.com/
> Reviewed-by: Fiona Behrens <me@kloenk.dev>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 10 ++++++----
> 1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index de1193117161..bd6e321a6aa5 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -143,10 +143,12 @@ def generate_crates(
>     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
>     # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
>     # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
> +    host_core = append_sysroot_crate("core", [])
> +    host_alloc = append_sysroot_crate("alloc", [host_core])
> +    host_std = append_sysroot_crate("std", [host_alloc, host_core])
> +    host_proc_macro = append_sysroot_crate("proc_macro", [host_core, host_std])
> +
>     core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
> -    alloc = append_sysroot_crate("alloc", [core])
> -    std = append_sysroot_crate("std", [alloc, core])
> -    proc_macro = append_sysroot_crate("proc_macro", [core, std])
> 
>     compiler_builtins = append_crate(
>         "compiler_builtins",
> @@ -157,7 +159,7 @@ def generate_crates(
>     macros = append_proc_macro_crate(
>         "macros",
>         srctree / "rust" / "macros" / "lib.rs",
> -        [std, proc_macro],
> +        [host_std, host_proc_macro],
>     )
> 
>     build_error = append_crate(
> 
> -- 
> 2.48.1
> 
> 

Reviewed-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 09/11] scripts: generate_rust_analyzer.py: avoid FD leak

[Daniel Almeida]

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Use a context manager to avoid leaking file descriptors.
> 
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index bd6e321a6aa5..ccb15aa66929 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -211,7 +211,8 @@ def generate_crates(
> 
>     def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
>         try:
> -            return f"{target}.o" in open(build_file).read()
> +            with open(build_file) as f:
> +                return f"{target}.o" in f.read()
>         except FileNotFoundError:
>             return False
> 
> 
> -- 
> 2.48.1
> 
> 

Oh, nice!

Reviewed-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 10/11] scripts: generate_rust_analyzer.py: define scripts

[Daniel Almeida]

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Generate rust-project.json entries for scripts written in Rust. This is
> possible now that we have a definition for `std` built for the host.
> 
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index ccb15aa66929..957b413fe0b6 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -209,6 +209,19 @@ def generate_crates(
>     uapi = append_crate_with_generated("uapi", [core])
>     kernel = append_crate_with_generated("kernel", [core, macros, build_error, bindings, pin_init, uapi])
> 
> +    scripts = srctree / "scripts"
> +    with open(scripts / "Makefile") as f:
> +        makefile = f.read()
> +    for path in scripts.glob("*.rs"):
> +        name = path.name.replace(".rs", "")
> +        if f"{name}-rust" not in makefile:
> +            continue
> +        _script = append_crate(
> +            name,
> +            path,
> +            [host_std],
> +        )
> +
>     def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
>         try:
>             with open(build_file) as f:
> 
> -- 
> 2.48.1
> 
> 

Reviewed-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints

[Miguel Ojeda]

On Tue, Mar 25, 2025 at 2:39 PM Daniel Almeida
<daniel.almeida@collabora.com> wrote:
>
> $ mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8
> Success: no issues found in 1 source file

Should this be a Tested-by? Or am I confused and did you mean something else?

Cheers,
Miguel

Re: [PATCH v4 11/11] scripts: generate_rust_analyzer.py: use `cfg_groups`

[Daniel Almeida]

> On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> 
> Declare common `cfg`s just once to reduce the size of rust-analyzer.json
> from 30619 to 2624 lines.
> 
> Link: https://github.com/rust-lang/rust-analyzer/commit/2607c09fddef36da0d6f0a84625db5e20a5ebde3
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
> scripts/generate_rust_analyzer.py | 35 +++++++++++++++++++++++------------
> 1 file changed, 23 insertions(+), 12 deletions(-)
> 
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index 957b413fe0b6..3d89c0198db4 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -37,6 +37,7 @@ class Crate(TypedDict):
>     root_module: str
>     is_workspace_member: bool
>     deps: List[Dependency]
> +    cfg_groups: List[str]
>     cfg: List[str]
>     edition: Literal["2021"]
>     env: Dict[str, str]
> @@ -59,15 +60,8 @@ def generate_crates(
>     sysroot_src: pathlib.Path,
>     external_src: pathlib.Path,
>     cfgs: List[str],
> +    cfg_groups: List[str],
> ) -> List[Crate]:
> -    # Generate the configuration list.
> -    cfg = []
> -    with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
> -        for line in fd:
> -            line = line.replace("--cfg=", "")
> -            line = line.replace("\n", "")
> -            cfg.append(line)
> -
>     # Now fill the crates list.
>     crates: List[Crate] = []
>     crates_cfgs = args_crates_cfgs(cfgs)
> @@ -77,6 +71,7 @@ def generate_crates(
>         root_module: pathlib.Path,
>         deps: List[Dependency],
>         cfg: List[str] = [],
> +        cfg_groups: List[str] = [],
>         is_workspace_member: bool = True,
>     ) -> Crate:
>         return {
> @@ -85,6 +80,7 @@ def generate_crates(
>             "is_workspace_member": is_workspace_member,
>             "deps": deps,
>             "cfg": cfg,
> +            "cfg_groups": cfg_groups,
>             "edition": "2021",
>             "env": {
>                 "RUST_MODFILE": "This is only for rust-analyzer"
> @@ -101,10 +97,13 @@ def generate_crates(
>         root_module: pathlib.Path,
>         deps: List[Dependency],
>         cfg: List[str] = [],
> +        cfg_groups: List[str] = [],
>         is_workspace_member: bool = True,
>     ) -> Dependency:
>         return register_crate(
> -            build_crate(display_name, root_module, deps, cfg, is_workspace_member)
> +            build_crate(
> +                display_name, root_module, deps, cfg, cfg_groups, is_workspace_member
> +            )
>         )
> 
>     def append_proc_macro_crate(
> @@ -190,7 +189,7 @@ def generate_crates(
>             display_name,
>             srctree / "rust" / display_name / "lib.rs",
>             deps,
> -            cfg=cfg,
> +            cfg_groups=cfg_groups,
>         )
>         crate["env"]["OBJTREE"] = str(objtree.resolve(True))
>         crate_with_generated: CrateWithGenerated = {
> @@ -252,7 +251,7 @@ def generate_crates(
>                 name,
>                 path,
>                 [core, kernel],
> -                cfg=cfg,
> +                cfg_groups=cfg_groups,
>             )
> 
>     return crates
> @@ -277,9 +276,21 @@ def main() -> None:
>     # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.
>     assert args.sysroot in args.sysroot_src.parents
> 
> +    # Generate the configuration list.
> +    with open(args.objtree / "include" / "generated" / "rustc_cfg") as fd:
> +        cfg_groups = {"rustc_cfg": [line.lstrip("--cfg=").rstrip("\n") for line in fd]}
> +
>     rust_project = {
> -        "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src, args.exttree, args.cfgs),
> +        "crates": generate_crates(
> +            args.srctree,
> +            args.objtree,
> +            args.sysroot_src,
> +            args.exttree,
> +            args.cfgs,
> +            list(cfg_groups.keys()),
> +        ),
>         "sysroot": str(args.sysroot),
> +        "cfg_groups": cfg_groups,
>     }
> 
>     json.dump(rust_project, sys.stdout, sort_keys=True, indent=4)
> 
> -- 
> 2.48.1
> 
> 

Reviewed-by: Daniel Almeida <daniel.almeida@collabora.com>
Tested-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints

[Daniel Almeida]

Hi Miguel,

> On 25 Mar 2025, at 11:19, Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> wrote:
> 
> On Tue, Mar 25, 2025 at 2:39 PM Daniel Almeida
> <daniel.almeida@collabora.com> wrote:
>> 
>> $ mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8
>> Success: no issues found in 1 source file
> 
> Should this be a Tested-by? Or am I confused and did you mean something else?
> 
> Cheers,
> Miguel

Yes, that was a bit convoluted, sorry.

Tested-by: Daniel Almeida <daniel.almeida@collabora.com>

Re: [PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints

[Tamir Duberstein]

On Tue, Mar 25, 2025 at 9:38 AM Daniel Almeida
<daniel.almeida@collabora.com> wrote:
>
> Hi Tamir,
>
> > On 22 Mar 2025, at 10:23, Tamir Duberstein <tamird@gmail.com> wrote:
> >
> > Python type hints allow static analysis tools like mypy to detect type
> > errors during development, improving the developer experience.
> >
> > Python type hints have been present in the kernel since 2019 at the
> > latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for
> > running KUnit tests").
> >
> > Run `mypy --strict scripts/generate_rust_analyzer.py --python-version
> > 3.8` to verify. Note that `mypy` no longer supports python < 3.8.
> >
> > This removes `"is_proc_macro": false` from `rust-project.json` in
> > exchange for stricter types. This field is interpreted as false if
> > absent[1] so this doesn't change the behavior of rust-analyzer.
>
> Can this be a separate patch? Not sure how this is related to Python type
> hints, but it makes the current patch harder to review.

Yeah, I'll pop it out into another patch.

> >
> > Link: https://github.com/rust-lang/rust-analyzer/blob/8d01570b5e812a49daa1f08404269f6ea5dd73a1/crates/project-model/src/project_json.rs#L372-L373 [1]
> > Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> > ---
> > scripts/generate_rust_analyzer.py | 172 +++++++++++++++++++++++++-------------
> > 1 file changed, 112 insertions(+), 60 deletions(-)
> >
> > diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> > index 03f55cce673c..0772ea309f94 100755
> > --- a/scripts/generate_rust_analyzer.py
> > +++ b/scripts/generate_rust_analyzer.py
> > @@ -10,8 +10,10 @@ import os
> > import pathlib
> > import subprocess
> > import sys
> > +from typing import Dict, Iterable, List, Literal, Optional, TypedDict
> >
> > -def args_crates_cfgs(cfgs):
> > +
> > +def args_crates_cfgs(cfgs: Iterable[str]) -> Dict[str, List[str]]:
> >     crates_cfgs = {}
> >     for cfg in cfgs:
> >         crate, vals = cfg.split("=", 1)
> > @@ -19,7 +21,45 @@ def args_crates_cfgs(cfgs):
> >
> >     return crates_cfgs
> >
> > -def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> > +
> > +class Dependency(TypedDict):
> > +    crate: int
> > +    name: str
> > +
> > +
> > +class Source(TypedDict):
> > +    include_dirs: List[str]
> > +    exclude_dirs: List[str]
> > +
> > +
> > +class Crate(TypedDict):
> > +    display_name: str
> > +    root_module: str
> > +    is_workspace_member: bool
> > +    deps: List[Dependency]
> > +    cfg: List[str]
> > +    edition: Literal["2021"]
> > +    env: Dict[str, str]
> > +
> > +
> > +# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11.
> > +class ProcMacroCrate(Crate):
> > +    is_proc_macro: Literal[True]
> > +    proc_macro_dylib_path: Optional[str]  # `pathlib.Path` is not JSON serializable.
> > +
> > +
> > +# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11.
> > +class CrateWithGenerated(Crate):
> > +    source: Optional[Source]
> > +
> > +
> > +def generate_crates(
> > +    srctree: pathlib.Path,
> > +    objtree: pathlib.Path,
> > +    sysroot_src: pathlib.Path,
> > +    external_src: pathlib.Path,
> > +    cfgs: List[str],
> > +) -> List[Crate]:
> >     # Generate the configuration list.
> >     cfg = []
> >     with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
> > @@ -31,67 +71,75 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> >     # Now fill the crates list -- dependencies need to come first.
> >     #
> >     # Avoid O(n^2) iterations by keeping a map of indexes.
> > -    crates = []
> > -    crates_indexes = {}
> > +    crates: List[Crate] = []
> > +    crates_indexes: Dict[str, int] = {}
> >     crates_cfgs = args_crates_cfgs(cfgs)
> >
> >     def build_crate(
> > -        display_name,
> > -        root_module,
> > -        deps,
> > -        cfg=[],
> > -        is_workspace_member=True,
> > -        is_proc_macro=False,
> > -    ):
> > -        crate = {
> > +        display_name: str,
> > +        root_module: pathlib.Path,
> > +        deps: List[str],
> > +        cfg: List[str] = [],
> > +        is_workspace_member: bool = True,
> > +    ) -> Crate:
> > +        return {
> >             "display_name": display_name,
> >             "root_module": str(root_module),
> >             "is_workspace_member": is_workspace_member,
> > -            "is_proc_macro": is_proc_macro,
> >             "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps],
> >             "cfg": cfg,
> >             "edition": "2021",
> >             "env": {
> >                 "RUST_MODFILE": "This is only for rust-analyzer"
> > -            }
> > +            },
> >         }
> > -        if is_proc_macro:
> > -            proc_macro_dylib_name = subprocess.check_output(
> > -                [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
> > -                stdin=subprocess.DEVNULL,
> > -            ).decode('utf-8').strip()
> > -            crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
> > -        return crate
> > -
> > -    def register_crate(crate):
> > +
> > +    def register_crate(crate: Crate) -> None:
> >         crates_indexes[crate["display_name"]] = len(crates)
> >         crates.append(crate)
> >
> >     def append_crate(
> > -        display_name,
> > -        root_module,
> > -        deps,
> > -        cfg=[],
> > -        is_workspace_member=True,
> > -        is_proc_macro=False,
> > -    ):
> > +        display_name: str,
> > +        root_module: pathlib.Path,
> > +        deps: List[str],
> > +        cfg: List[str] = [],
> > +        is_workspace_member: bool = True,
> > +    ) -> None:
> >         register_crate(
> > -            build_crate(
> > -                display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro
> > -            )
> > +            build_crate(display_name, root_module, deps, cfg, is_workspace_member)
> >         )
> >
> > +    def append_proc_macro_crate(
> > +        display_name: str,
> > +        root_module: pathlib.Path,
> > +        deps: List[str],
> > +        cfg: List[str] = [],
> > +    ) -> None:
> > +        crate = build_crate(display_name, root_module, deps, cfg)
> > +        proc_macro_dylib_name = subprocess.check_output(
> > +            [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
> > +            stdin=subprocess.DEVNULL,
> > +        ).decode('utf-8').strip()
> > +        proc_macro_crate: ProcMacroCrate = {
> > +            **crate,
> > +            "is_proc_macro": True,
> > +            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
> > +        }
> > +        register_crate(proc_macro_crate)
> > +
>
> Same here. Is this supposed to be related to Python type hints somehow?

Same here - moved into a separate patch.

> >     def append_sysroot_crate(
> > -        display_name,
> > -        deps,
> > -        cfg=[],
> > -    ):
> > -        append_crate(
> > -            display_name,
> > -            sysroot_src / display_name / "src" / "lib.rs",
> > -            deps,
> > -            cfg,
> > -            is_workspace_member=False,
> > +        display_name: str,
> > +        deps: List[str],
> > +        cfg: List[str] = [],
> > +    ) -> None:
> > +        register_crate(
>
> Why is register_crate here now? Maybe this change belongs to the preceding patch?

Good spot, this didn't need to change. Reverted in this patch.

>
> > +            build_crate(
> > +                display_name,
> > +                sysroot_src / display_name / "src" / "lib.rs",
> > +                deps,
> > +                cfg,
> > +                is_workspace_member=False,
> > +            )
> >         )
> >
> >     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
> > @@ -108,11 +156,10 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> >         [],
> >     )
> >
> > -    append_crate(
> > +    append_proc_macro_crate(
> >         "macros",
> >         srctree / "rust" / "macros" / "lib.rs",
> >         ["std", "proc_macro"],
> > -        is_proc_macro=True,
> >     )
> >
> >     append_crate(
> > @@ -121,12 +168,11 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> >         ["core", "compiler_builtins"],
> >     )
> >
> > -    append_crate(
> > +    append_proc_macro_crate(
> >         "pin_init_internal",
> >         srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs",
> >         [],
> >         cfg=["kernel"],
> > -        is_proc_macro=True,
> >     )
> >
> >     append_crate(
> > @@ -137,9 +183,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> >     )
> >
> >     def append_crate_with_generated(
> > -        display_name,
> > -        deps,
> > -    ):
> > +        display_name: str,
> > +        deps: List[str],
> > +    ) -> None:
> >         crate = build_crate(
> >             display_name,
> >             srctree / "rust" / display_name / "lib.rs",
> > @@ -147,20 +193,23 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> >             cfg=cfg,
> >         )
> >         crate["env"]["OBJTREE"] = str(objtree.resolve(True))
> > -        crate["source"] = {
> > -            "include_dirs": [
> > -                str(srctree / "rust" / display_name),
> > -                str(objtree / "rust")
> > -            ],
> > -            "exclude_dirs": [],
> > +        crate_with_generated: CrateWithGenerated = {
> > +            **crate,
> > +            "source": {
> > +                "include_dirs": [
> > +                    str(srctree / "rust" / display_name),
> > +                    str(objtree / "rust")
> > +                ],
> > +                "exclude_dirs": [],
> > +            }
> >         }
> > -        register_crate(crate)
> > +        register_crate(crate_with_generated)
> >
> >     append_crate_with_generated("bindings", ["core"])
> >     append_crate_with_generated("uapi", ["core"])
> >     append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "pin_init", "uapi"])
> >
> > -    def is_root_crate(build_file, target):
> > +    def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
> >         try:
> >             return f"{target}.o" in open(build_file).read()
> >         except FileNotFoundError:
> > @@ -169,7 +218,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> >     # Then, the rest outside of `rust/`.
> >     #
> >     # We explicitly mention the top-level folders we want to cover.
> > -    extra_dirs = map(lambda dir: srctree / dir, ("samples", "drivers"))
> > +    extra_dirs: Iterable[pathlib.Path] = map(
> > +        lambda dir: srctree / dir, ("samples", "drivers")
> > +    )
> >     if external_src is not None:
> >         extra_dirs = [external_src]
> >     for folder in extra_dirs:
> > @@ -192,7 +243,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
> >
> >     return crates
> >
> > -def main():
> > +
>
> What is this extra blank line for? Automatically generated by a formatter?

Yeah, this comes from PEP 8. Moved to "scripts:
generate_rust_analyzer.py: add missing whitespace".

>
> > +def main() -> None:
> >     parser = argparse.ArgumentParser()
> >     parser.add_argument("--verbose", "-v", action="store_true")
> >     parser.add_argument("--cfgs", action="append", default=[])
> >
> > --
> > 2.48.1
> >
> >
>
> Other than what I said above, the type hints themselves are fine.
>
> — Daniel

Thanks for the reviews!

Re: [PATCH v4 04/11] scripts: generate_rust_analyzer.py: extract `{build,register}_crate`

[Fiona Behrens]

Tamir Duberstein <tamird@gmail.com> writes:

> Extract helpers from `append_crate` to avoid the need to peek into
> `crates[-1]`. This improves readability.
>
> Suggested-by: Trevor Gross <tmgross@umich.edu>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

Reviewed-by: Fiona Behrens <me@kloenk.dev>

> ---
>  scripts/generate_rust_analyzer.py | 35 ++++++++++++++++++++++++++++++-----
>  1 file changed, 30 insertions(+), 5 deletions(-)
>
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index e997d923268d..03f55cce673c 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -35,7 +35,14 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>      crates_indexes = {}
>      crates_cfgs = args_crates_cfgs(cfgs)
>  
> -    def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False):
> +    def build_crate(
> +        display_name,
> +        root_module,
> +        deps,
> +        cfg=[],
> +        is_workspace_member=True,
> +        is_proc_macro=False,
> +    ):
>          crate = {
>              "display_name": display_name,
>              "root_module": str(root_module),
> @@ -54,9 +61,26 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>                  stdin=subprocess.DEVNULL,
>              ).decode('utf-8').strip()
>              crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
> -        crates_indexes[display_name] = len(crates)
> +        return crate
> +
> +    def register_crate(crate):
> +        crates_indexes[crate["display_name"]] = len(crates)
>          crates.append(crate)
>  
> +    def append_crate(
> +        display_name,
> +        root_module,
> +        deps,
> +        cfg=[],
> +        is_workspace_member=True,
> +        is_proc_macro=False,
> +    ):
> +        register_crate(
> +            build_crate(
> +                display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro
> +            )
> +        )
> +
>      def append_sysroot_crate(
>          display_name,
>          deps,
> @@ -116,20 +140,21 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>          display_name,
>          deps,
>      ):
> -        append_crate(
> +        crate = build_crate(
>              display_name,
>              srctree / "rust" / display_name / "lib.rs",
>              deps,
>              cfg=cfg,
>          )
> -        crates[-1]["env"]["OBJTREE"] = str(objtree.resolve(True))
> -        crates[-1]["source"] = {
> +        crate["env"]["OBJTREE"] = str(objtree.resolve(True))
> +        crate["source"] = {
>              "include_dirs": [
>                  str(srctree / "rust" / display_name),
>                  str(objtree / "rust")
>              ],
>              "exclude_dirs": [],
>          }
> +        register_crate(crate)
>  
>      append_crate_with_generated("bindings", ["core"])
>      append_crate_with_generated("uapi", ["core"])

Re: [PATCH v4 09/11] scripts: generate_rust_analyzer.py: avoid FD leak

[Fiona Behrens]

Tamir Duberstein <tamird@gmail.com> writes:

> Use a context manager to avoid leaking file descriptors.
>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

Reviewed-by: Fiona Behrens <me@kloenk.dev>

> ---
>  scripts/generate_rust_analyzer.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index bd6e321a6aa5..ccb15aa66929 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -211,7 +211,8 @@ def generate_crates(
>  
>      def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
>          try:
> -            return f"{target}.o" in open(build_file).read()
> +            with open(build_file) as f:
> +                return f"{target}.o" in f.read()
>          except FileNotFoundError:
>              return False

Re: [PATCH v4 10/11] scripts: generate_rust_analyzer.py: define scripts

[Fiona Behrens]

Tamir Duberstein <tamird@gmail.com> writes:

> Generate rust-project.json entries for scripts written in Rust. This is
> possible now that we have a definition for `std` built for the host.
>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

Reviewed-by: Fiona Behrens <me@kloenk.dev>

> ---
>  scripts/generate_rust_analyzer.py | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index ccb15aa66929..957b413fe0b6 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -209,6 +209,19 @@ def generate_crates(
>      uapi = append_crate_with_generated("uapi", [core])
>      kernel = append_crate_with_generated("kernel", [core, macros, build_error, bindings, pin_init, uapi])
>  
> +    scripts = srctree / "scripts"
> +    with open(scripts / "Makefile") as f:
> +        makefile = f.read()
> +    for path in scripts.glob("*.rs"):
> +        name = path.name.replace(".rs", "")
> +        if f"{name}-rust" not in makefile:
> +            continue
> +        _script = append_crate(
> +            name,
> +            path,
> +            [host_std],
> +        )
> +
>      def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
>          try:
>              with open(build_file) as f: