[PATCH v2 0/7] rust: generate_rust_analyzer.py: define host crates

[PATCH v2 0/7] rust: generate_rust_analyzer.py: define host crates

[Tamir Duberstein]

This series updates rust-project.json to differentiate between host and
target crates, where the former are used as dependencies of the `macros`
crate. Please see individual commit messages for details.

The first 3 commits contain mechanical formatting changes and are
optional. The series can be taken without them.

I avoided more significant formatting or changes where possible to
reduce the diff. Unfortunately `scripts/generate_rust_analyzer.py` is
not consistently formatted before nor after this series.

The 5th commit ("scripts: generate_rust_analyzer.py: use
str(pathlib.Path)") can also be considered optional. It removes an
inconsistency I noticed while working on this series and which occurs on
a line which churns in this series anyway.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
Changes in v2:
- Rebased on "rust: fix rust-analyzer configuration for generated files" [1]
  Link: https://lore.kernel.org/all/CANiq72nv7nQ+1BinCHe2qsvwdUb-y9t7x=RGSppi_n9TBXNHpw@mail.gmail.com/ [1]
- Link to v1: https://lore.kernel.org/r/20250209-rust-analyzer-host-v1-0-a2286a2a2fa3@gmail.com

---
Tamir Duberstein (7):
      scripts: generate_rust_analyzer.py: add missing whitespace
      scripts: generate_rust_analyzer.py: use double quotes
      scripts: generate_rust_analyzer.py: add trailing comma
      scripts: generate_rust_analyzer.py: add type hints
      scripts: generate_rust_analyzer.py: use str(pathlib.Path)
      scripts: generate_rust_analyzer.py: identify crates explicitly
      scripts: generate_rust_analyzer.py: define host crates

 scripts/generate_rust_analyzer.py | 177 ++++++++++++++++++++++++++------------
 1 file changed, 121 insertions(+), 56 deletions(-)
---
base-commit: 510570138476387532129cba425ea7a375308d87
change-id: 20250209-rust-analyzer-host-43b108655578

Best regards,
-- 
Tamir Duberstein <tamird@gmail.com>

[PATCH v2 1/7] scripts: generate_rust_analyzer.py: add missing whitespace

[Tamir Duberstein]

Add a space before the `/` operator for consistency with surrounding
code and code formatting tools. Add a second newline between top-level
items in accordance with PEP 8[1]:

> Surround top-level function and class definitions with two blank
lines.

This change was made by a code formatting tool.

Link: https://peps.python.org/pep-0008/ [1]
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index adae71544cbd..15a690ae2edb 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -103,7 +103,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     ):
         append_crate(
             display_name,
-            srctree / "rust"/ display_name / "lib.rs",
+            srctree / "rust" / display_name / "lib.rs",
             deps,
             cfg=cfg,
         )
@@ -178,5 +178,6 @@ def main():
 
     json.dump(rust_project, sys.stdout, sort_keys=True, indent=4)
 
+
 if __name__ == "__main__":
     main()

-- 
2.48.1

Re: [PATCH v2 1/7] scripts: generate_rust_analyzer.py: add missing whitespace

[Fiona Behrens]

Tamir Duberstein <tamird@gmail.com> writes:

> Add a space before the `/` operator for consistency with surrounding
> code and code formatting tools. Add a second newline between top-level
> items in accordance with PEP 8[1]:
>
>> Surround top-level function and class definitions with two blank
> lines.
>
> This change was made by a code formatting tool.
>
> Link: https://peps.python.org/pep-0008/ [1]
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

Reviewed-by: Fiona Behrens <me@kloenk.dev>

> ---
>  scripts/generate_rust_analyzer.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index adae71544cbd..15a690ae2edb 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -103,7 +103,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>      ):
>          append_crate(
>              display_name,
> -            srctree / "rust"/ display_name / "lib.rs",
> +            srctree / "rust" / display_name / "lib.rs",
>              deps,
>              cfg=cfg,
>          )
> @@ -178,5 +178,6 @@ def main():
>  
>      json.dump(rust_project, sys.stdout, sort_keys=True, indent=4)
>  
> +
>  if __name__ == "__main__":
>      main()

[PATCH v2 2/7] scripts: generate_rust_analyzer.py: use double quotes

[Tamir Duberstein]

Replace inconsistent use of single quotes with double quotes.

This change was made by a code formatting tool.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index 15a690ae2edb..5913934ecb5a 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -154,8 +154,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument('--verbose', '-v', action='store_true')
-    parser.add_argument('--cfgs', action='append', default=[])
+    parser.add_argument("--verbose", "-v", action="store_true")
+    parser.add_argument("--cfgs", action="append", default=[])
     parser.add_argument("srctree", type=pathlib.Path)
     parser.add_argument("objtree", type=pathlib.Path)
     parser.add_argument("sysroot", type=pathlib.Path)

-- 
2.48.1

Re: [PATCH v2 2/7] scripts: generate_rust_analyzer.py: use double quotes

[Fiona Behrens]

Tamir Duberstein <tamird@gmail.com> writes:

> Replace inconsistent use of single quotes with double quotes.
>
> This change was made by a code formatting tool.
>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

Reviewed-by: Fiona Behrens <me@kloenk.dev>

> ---
>  scripts/generate_rust_analyzer.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index 15a690ae2edb..5913934ecb5a 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -154,8 +154,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
>  
>  def main():
>      parser = argparse.ArgumentParser()
> -    parser.add_argument('--verbose', '-v', action='store_true')
> -    parser.add_argument('--cfgs', action='append', default=[])
> +    parser.add_argument("--verbose", "-v", action="store_true")
> +    parser.add_argument("--cfgs", action="append", default=[])
>      parser.add_argument("srctree", type=pathlib.Path)
>      parser.add_argument("objtree", type=pathlib.Path)
>      parser.add_argument("sysroot", type=pathlib.Path)

[PATCH v2 3/7] scripts: generate_rust_analyzer.py: add trailing comma

[Tamir Duberstein]

Add missing trailing comma on multi-line function call as suggested by
PEP-8:

> The pattern is to put each value (etc.) on a line by itself, always
> adding a trailing comma, and add the close parenthesis/bracket/brace
> on the next line.

This change was made by a code formatting tool.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index 5913934ecb5a..7e78b926e61f 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -165,7 +165,7 @@ def main():
 
     logging.basicConfig(
         format="[%(asctime)s] [%(levelname)s] %(message)s",
-        level=logging.INFO if args.verbose else logging.WARNING
+        level=logging.INFO if args.verbose else logging.WARNING,
     )
 
     # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.

-- 
2.48.1

Re: [PATCH v2 3/7] scripts: generate_rust_analyzer.py: add trailing comma

[Fiona Behrens]

Tamir Duberstein <tamird@gmail.com> writes:

> Add missing trailing comma on multi-line function call as suggested by
> PEP-8:
>
>> The pattern is to put each value (etc.) on a line by itself, always
>> adding a trailing comma, and add the close parenthesis/bracket/brace
>> on the next line.
>
> This change was made by a code formatting tool.
>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

Reviewed-by: Fiona Behrens <me@kloenk.dev>

> ---
>  scripts/generate_rust_analyzer.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index 5913934ecb5a..7e78b926e61f 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -165,7 +165,7 @@ def main():
>  
>      logging.basicConfig(
>          format="[%(asctime)s] [%(levelname)s] %(message)s",
> -        level=logging.INFO if args.verbose else logging.WARNING
> +        level=logging.INFO if args.verbose else logging.WARNING,
>      )
>  
>      # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.

[PATCH v2 4/7] scripts: generate_rust_analyzer.py: add type hints

[Tamir Duberstein]

Python type hints allow static analysis tools like mypy to detect type
errors during development, improving the developer experience.

Python type hints have been present in the kernel since 2019 at the
latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for
running KUnit tests").

Run `uv tool run mypy --strict scripts/generate_rust_analyzer.py` to
verify.

This removes `"is_proc_macro": false` from `rust-project.json` in
exchange for stricter types. This field is interpreted as false if
absent[1] so this doesn't change the behavior of rust-analyzer.

Link: https://github.com/rust-lang/rust-analyzer/blob/8d01570b5e812a49daa1f08404269f6ea5dd73a1/crates/project-model/src/project_json.rs#L372-L373 [1]
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 130 ++++++++++++++++++++++++++++----------
 1 file changed, 96 insertions(+), 34 deletions(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index 7e78b926e61f..c73ea8d116a4 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -10,8 +10,10 @@ import os
 import pathlib
 import subprocess
 import sys
+import typing as T
 
-def args_crates_cfgs(cfgs):
+
+def args_crates_cfgs(cfgs: T.Iterable[str]) -> dict[str, list[str]]:
     crates_cfgs = {}
     for cfg in cfgs:
         crate, vals = cfg.split("=", 1)
@@ -19,7 +21,45 @@ def args_crates_cfgs(cfgs):
 
     return crates_cfgs
 
-def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
+
+class Dependency(T.TypedDict):
+    crate: int
+    name: str
+
+
+class Source(T.TypedDict):
+    include_dirs: list[str]
+    exclude_dirs: list[str]
+
+
+class Crate(T.TypedDict):
+    display_name: str
+    root_module: str
+    is_workspace_member: bool
+    deps: list[Dependency]
+    cfg: list[str]
+    edition: T.Literal["2021"]
+    env: dict[str, str]
+
+
+# `T.NotRequired` fields on `Crate` would be better but `T.NotRequired` was added in 3.11.
+class ProcMacroCrate(Crate):
+    is_proc_macro: T.Literal[True]
+    proc_macro_dylib_path: T.Optional[str]  # `pathlib.Path` is not JSON serializable.
+
+
+# `T.NotRequired` fields on `Crate` would be better but `T.NotRequired` was added in 3.11.
+class CrateWithGenerated(Crate):
+    source: T.Optional[Source]
+
+
+def generate_crates(
+    srctree: pathlib.Path,
+    objtree: pathlib.Path,
+    sysroot_src: pathlib.Path,
+    external_src: pathlib.Path,
+    cfgs: list[str],
+) -> list[Crate]:
     # Generate the configuration list.
     cfg = []
     with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
@@ -31,37 +71,53 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     # Now fill the crates list -- dependencies need to come first.
     #
     # Avoid O(n^2) iterations by keeping a map of indexes.
-    crates = []
-    crates_indexes = {}
+    crates: list[Crate] = []
+    crates_indexes: dict[str, int] = {}
     crates_cfgs = args_crates_cfgs(cfgs)
 
-    def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False):
-        crate = {
+    def append_crate(
+        display_name: str,
+        root_module: pathlib.Path,
+        deps: list[str],
+        cfg: list[str] = [],
+        is_workspace_member: bool = True,
+    ) -> None:
+        crates_indexes[display_name] = len(crates)
+        crates.append({
             "display_name": display_name,
             "root_module": str(root_module),
             "is_workspace_member": is_workspace_member,
-            "is_proc_macro": is_proc_macro,
             "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps],
             "cfg": cfg,
             "edition": "2021",
             "env": {
                 "RUST_MODFILE": "This is only for rust-analyzer"
-            }
+            },
+        })
+
+    def append_proc_macro_crate(
+        display_name: str,
+        root_module: pathlib.Path,
+        deps: list[str],
+        cfg: list[str] = [],
+    ) -> None:
+        append_crate(display_name, root_module, deps, cfg)
+        proc_macro_dylib_name = subprocess.check_output(
+            [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
+            stdin=subprocess.DEVNULL,
+        ).decode('utf-8').strip()
+        crate: ProcMacroCrate = {
+            **crates[-1],
+            "is_proc_macro": True,
+            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
         }
-        if is_proc_macro:
-            proc_macro_dylib_name = subprocess.check_output(
-                [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
-                stdin=subprocess.DEVNULL,
-            ).decode('utf-8').strip()
-            crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
-        crates_indexes[display_name] = len(crates)
-        crates.append(crate)
+        crates[-1] = crate
 
     def append_sysroot_crate(
-        display_name,
-        deps,
-        cfg=[],
-    ):
+        display_name: str,
+        deps: list[str],
+        cfg: list[str] = [],
+    ) -> None:
         append_crate(
             display_name,
             sysroot_src / display_name / "src" / "lib.rs",
@@ -84,11 +140,10 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         [],
     )
 
-    append_crate(
+    append_proc_macro_crate(
         "macros",
         srctree / "rust" / "macros" / "lib.rs",
         ["std", "proc_macro"],
-        is_proc_macro=True,
     )
 
     append_crate(
@@ -98,9 +153,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     )
 
     def append_crate_with_generated(
-        display_name,
-        deps,
-    ):
+        display_name: str,
+        deps: list[str],
+    ) -> None:
         append_crate(
             display_name,
             srctree / "rust" / display_name / "lib.rs",
@@ -108,19 +163,23 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
             cfg=cfg,
         )
         crates[-1]["env"]["OBJTREE"] = str(objtree.resolve(True))
-        crates[-1]["source"] = {
-            "include_dirs": [
-                str(srctree / "rust" / display_name),
-                str(objtree / "rust")
-            ],
-            "exclude_dirs": [],
+        crate: CrateWithGenerated = {
+            **crates[-1],
+            "source": {
+                "include_dirs": [
+                    str(srctree / "rust" / display_name),
+                    str(objtree / "rust")
+                ],
+                "exclude_dirs": [],
+            }
         }
+        crates[-1] = crate
 
     append_crate_with_generated("bindings", ["core"])
     append_crate_with_generated("uapi", ["core"])
     append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "uapi"])
 
-    def is_root_crate(build_file, target):
+    def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
         try:
             return f"{target}.o" in open(build_file).read()
         except FileNotFoundError:
@@ -129,7 +188,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     # Then, the rest outside of `rust/`.
     #
     # We explicitly mention the top-level folders we want to cover.
-    extra_dirs = map(lambda dir: srctree / dir, ("samples", "drivers"))
+    extra_dirs: T.Iterable[pathlib.Path] = map(
+        lambda dir: srctree / dir, ("samples", "drivers")
+    )
     if external_src is not None:
         extra_dirs = [external_src]
     for folder in extra_dirs:
@@ -152,7 +213,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
 
     return crates
 
-def main():
+
+def main() -> None:
     parser = argparse.ArgumentParser()
     parser.add_argument("--verbose", "-v", action="store_true")
     parser.add_argument("--cfgs", action="append", default=[])

-- 
2.48.1

Re: [PATCH v2 4/7] scripts: generate_rust_analyzer.py: add type hints

[Tamir Duberstein]

On Tue, Mar 11, 2025 at 9:17 PM Tamir Duberstein <tamird@gmail.com> wrote:
>
> Python type hints allow static analysis tools like mypy to detect type
> errors during development, improving the developer experience.
>
> Python type hints have been present in the kernel since 2019 at the
> latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for
> running KUnit tests").
>
> Run `uv tool run mypy --strict scripts/generate_rust_analyzer.py` to
> verify.
>
> This removes `"is_proc_macro": false` from `rust-project.json` in
> exchange for stricter types. This field is interpreted as false if
> absent[1] so this doesn't change the behavior of rust-analyzer.
>
> Link: https://github.com/rust-lang/rust-analyzer/blob/8d01570b5e812a49daa1f08404269f6ea5dd73a1/crates/project-model/src/project_json.rs#L372-L373 [1]
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

Gentle ping. Please have a look if you aren't intimidated by python
type hints :)

Re: [PATCH v2 4/7] scripts: generate_rust_analyzer.py: add type hints

[Trevor Gross]

On Tue, Mar 11, 2025 at 9:18 PM Tamir Duberstein <tamird@gmail.com> wrote:
>
> Python type hints allow static analysis tools like mypy to detect type
> errors during development, improving the developer experience.
>
> Python type hints have been present in the kernel since 2019 at the
> latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for
> running KUnit tests").
>
> Run `uv tool run mypy --strict scripts/generate_rust_analyzer.py` to
> verify.

From the discussion, it may be better to instead mention the direct
invocation (without uv).

Could you also mention the target min version? Since apparently the
kernel has a spread. It looks like maybe 3.8 based on what is used
here.

> This removes `"is_proc_macro": false` from `rust-project.json` in
> exchange for stricter types. This field is interpreted as false if
> absent[1] so this doesn't change the behavior of rust-analyzer.
>
> Link: https://github.com/rust-lang/rust-analyzer/blob/8d01570b5e812a49daa1f08404269f6ea5dd73a1/crates/project-model/src/project_json.rs#L372-L373 [1]
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
>  scripts/generate_rust_analyzer.py | 130 ++++++++++++++++++++++++++++----------
>  1 file changed, 96 insertions(+), 34 deletions(-)
>
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index 7e78b926e61f..c73ea8d116a4 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -10,8 +10,10 @@ import os
>  import pathlib
>  import subprocess
>  import sys
> +import typing as T

Nit: is there any need to keep everything namespaced? I think it
should be fine to import `Iterable` `TypedDict` etc directly since
they aren't confusable.

Same for `pathlib.Path` since there is no other `Path` (some of that
is preexisting).

> +    def append_proc_macro_crate(
> +        display_name: str,
> +        root_module: pathlib.Path,
> +        deps: list[str],
> +        cfg: list[str] = [],
> +    ) -> None:
> +        append_crate(display_name, root_module, deps, cfg)
> +        proc_macro_dylib_name = subprocess.check_output(
> +            [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],

Nit, may as well use this opportunity to wrap the line.

> +            stdin=subprocess.DEVNULL,
> +        ).decode('utf-8').strip()
> +        crate: ProcMacroCrate = {
> +            **crates[-1],
> +            "is_proc_macro": True,
> +            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
>          }
> -        if is_proc_macro:
> -            proc_macro_dylib_name = subprocess.check_output(
> -                [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
> -                stdin=subprocess.DEVNULL,
> -            ).decode('utf-8').strip()
> -            crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
> -        crates_indexes[display_name] = len(crates)
> -        crates.append(crate)
> +        crates[-1] = crate

The unpacking is a bit confusing here, can `crates[-1]` just be set
rather than duplicating and replacing it?

Maybe the body of `append_crate` should be `build_crate(...) -> Crate`
(which could then be a top-level function), then `append_crate`,
`append_crate_with_generated`, etc call that and handle modification /
appending themselves.

> +        crate: CrateWithGenerated = {
> +            **crates[-1],
> +            "source": {
> +                "include_dirs": [
> +                    str(srctree / "rust" / display_name),
> +                    str(objtree / "rust")
> +                ],
> +                "exclude_dirs": [],
> +            }
>          }
> +        crates[-1] = crate

Same note as above regarding rebuilding the last item.

- Trevor

Re: [PATCH v2 4/7] scripts: generate_rust_analyzer.py: add type hints

[Tamir Duberstein]

On Wed, Mar 19, 2025 at 5:25 PM Trevor Gross <tmgross@umich.edu> wrote:
>
> On Tue, Mar 11, 2025 at 9:18 PM Tamir Duberstein <tamird@gmail.com> wrote:
> >
> > Python type hints allow static analysis tools like mypy to detect type
> > errors during development, improving the developer experience.
> >
> > Python type hints have been present in the kernel since 2019 at the
> > latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for
> > running KUnit tests").
> >
> > Run `uv tool run mypy --strict scripts/generate_rust_analyzer.py` to
> > verify.
>
> From the discussion, it may be better to instead mention the direct
> invocation (without uv).

👍 done.

> Could you also mention the target min version? Since apparently the
> kernel has a spread. It looks like maybe 3.8 based on what is used
> here.

Added, it is now 3.8 because mypy doesn't support anything lower.

>
> > This removes `"is_proc_macro": false` from `rust-project.json` in
> > exchange for stricter types. This field is interpreted as false if
> > absent[1] so this doesn't change the behavior of rust-analyzer.
> >
> > Link: https://github.com/rust-lang/rust-analyzer/blob/8d01570b5e812a49daa1f08404269f6ea5dd73a1/crates/project-model/src/project_json.rs#L372-L373 [1]
> > Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> > ---
> >  scripts/generate_rust_analyzer.py | 130 ++++++++++++++++++++++++++++----------
> >  1 file changed, 96 insertions(+), 34 deletions(-)
> >
> > diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> > index 7e78b926e61f..c73ea8d116a4 100755
> > --- a/scripts/generate_rust_analyzer.py
> > +++ b/scripts/generate_rust_analyzer.py
> > @@ -10,8 +10,10 @@ import os
> >  import pathlib
> >  import subprocess
> >  import sys
> > +import typing as T
>
> Nit: is there any need to keep everything namespaced? I think it
> should be fine to import `Iterable` `TypedDict` etc directly since
> they aren't confusable.

I saw this style in another project and liked it, but I don't feel
strongly. Changed.

>
> Same for `pathlib.Path` since there is no other `Path` (some of that
> is preexisting).

I'll leave this one as is to avoid more churn.

>
> > +    def append_proc_macro_crate(
> > +        display_name: str,
> > +        root_module: pathlib.Path,
> > +        deps: list[str],
> > +        cfg: list[str] = [],
> > +    ) -> None:
> > +        append_crate(display_name, root_module, deps, cfg)
> > +        proc_macro_dylib_name = subprocess.check_output(
> > +            [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
>
> Nit, may as well use this opportunity to wrap the line.

I considered that, but when you use git with `-w` this line doesn't
change, so I'd rather leave it.

>
> > +            stdin=subprocess.DEVNULL,
> > +        ).decode('utf-8').strip()
> > +        crate: ProcMacroCrate = {
> > +            **crates[-1],
> > +            "is_proc_macro": True,
> > +            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
> >          }
> > -        if is_proc_macro:
> > -            proc_macro_dylib_name = subprocess.check_output(
> > -                [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
> > -                stdin=subprocess.DEVNULL,
> > -            ).decode('utf-8').strip()
> > -            crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
> > -        crates_indexes[display_name] = len(crates)
> > -        crates.append(crate)
> > +        crates[-1] = crate
>
> The unpacking is a bit confusing here, can `crates[-1]` just be set
> rather than duplicating and replacing it?
>
> Maybe the body of `append_crate` should be `build_crate(...) -> Crate`
> (which could then be a top-level function), then `append_crate`,
> `append_crate_with_generated`, etc call that and handle modification /
> appending themselves.

This is a nice improvement. Done.

> > +        crate: CrateWithGenerated = {
> > +            **crates[-1],
> > +            "source": {
> > +                "include_dirs": [
> > +                    str(srctree / "rust" / display_name),
> > +                    str(objtree / "rust")
> > +                ],
> > +                "exclude_dirs": [],
> > +            }
> >          }
> > +        crates[-1] = crate

👍

>
> - Trevor

Thanks for the review!

[PATCH v2 5/7] scripts: generate_rust_analyzer.py: use str(pathlib.Path)

[Tamir Duberstein]

Use the `/` operator on `pathlib.Path` rather than directly crafting a
string. This is consistent with all other path manipulation in this
script.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index c73ea8d116a4..1bb185ae2e87 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -109,7 +109,7 @@ def generate_crates(
         crate: ProcMacroCrate = {
             **crates[-1],
             "is_proc_macro": True,
-            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
+            "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name),
         }
         crates[-1] = crate
 

-- 
2.48.1

Re: [PATCH v2 5/7] scripts: generate_rust_analyzer.py: use str(pathlib.Path)

[Fiona Behrens]

Tamir Duberstein <tamird@gmail.com> writes:

> Use the `/` operator on `pathlib.Path` rather than directly crafting a
> string. This is consistent with all other path manipulation in this
> script.
>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

Reviewed-by: Fiona Behrens <me@kloenk.dev>

> ---
>  scripts/generate_rust_analyzer.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index c73ea8d116a4..1bb185ae2e87 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -109,7 +109,7 @@ def generate_crates(
>          crate: ProcMacroCrate = {
>              **crates[-1],
>              "is_proc_macro": True,
> -            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
> +            "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name),
>          }
>          crates[-1] = crate

[PATCH v2 6/7] scripts: generate_rust_analyzer.py: identify crates explicitly

[Tamir Duberstein]

Use the return of `append_crate` to declare dependency on that crate.
This allows multiple crates with the same display_name be defined, which
we'll use to define host crates separately from target crates.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 60 +++++++++++++++++++--------------------
 1 file changed, 30 insertions(+), 30 deletions(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index 1bb185ae2e87..e1002867735b 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -68,40 +68,38 @@ def generate_crates(
             line = line.replace("\n", "")
             cfg.append(line)
 
-    # Now fill the crates list -- dependencies need to come first.
-    #
-    # Avoid O(n^2) iterations by keeping a map of indexes.
+    # Now fill the crates list.
     crates: list[Crate] = []
-    crates_indexes: dict[str, int] = {}
     crates_cfgs = args_crates_cfgs(cfgs)
 
     def append_crate(
         display_name: str,
         root_module: pathlib.Path,
-        deps: list[str],
+        deps: list[Dependency],
         cfg: list[str] = [],
         is_workspace_member: bool = True,
-    ) -> None:
-        crates_indexes[display_name] = len(crates)
+    ) -> Dependency:
+        index = len(crates)
         crates.append({
             "display_name": display_name,
             "root_module": str(root_module),
             "is_workspace_member": is_workspace_member,
-            "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps],
+            "deps": deps,
             "cfg": cfg,
             "edition": "2021",
             "env": {
                 "RUST_MODFILE": "This is only for rust-analyzer"
             },
         })
+        return {"crate": index, "name": display_name}
 
     def append_proc_macro_crate(
         display_name: str,
         root_module: pathlib.Path,
-        deps: list[str],
+        deps: list[Dependency],
         cfg: list[str] = [],
-    ) -> None:
-        append_crate(display_name, root_module, deps, cfg)
+    ) -> Dependency:
+        dep = append_crate(display_name, root_module, deps, cfg)
         proc_macro_dylib_name = subprocess.check_output(
             [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
             stdin=subprocess.DEVNULL,
@@ -112,13 +110,14 @@ def generate_crates(
             "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name),
         }
         crates[-1] = crate
+        return dep
 
     def append_sysroot_crate(
         display_name: str,
-        deps: list[str],
+        deps: list[Dependency],
         cfg: list[str] = [],
-    ) -> None:
-        append_crate(
+    ) -> Dependency:
+        return append_crate(
             display_name,
             sysroot_src / display_name / "src" / "lib.rs",
             deps,
@@ -129,34 +128,34 @@ def generate_crates(
     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
     # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
     # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
-    append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
-    append_sysroot_crate("alloc", ["core"])
-    append_sysroot_crate("std", ["alloc", "core"])
-    append_sysroot_crate("proc_macro", ["core", "std"])
+    core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
+    alloc = append_sysroot_crate("alloc", [core])
+    std = append_sysroot_crate("std", [alloc, core])
+    proc_macro = append_sysroot_crate("proc_macro", [core, std])
 
-    append_crate(
+    compiler_builtins = append_crate(
         "compiler_builtins",
         srctree / "rust" / "compiler_builtins.rs",
         [],
     )
 
-    append_proc_macro_crate(
+    macros = append_proc_macro_crate(
         "macros",
         srctree / "rust" / "macros" / "lib.rs",
-        ["std", "proc_macro"],
+        [std, proc_macro],
     )
 
-    append_crate(
+    build_error = append_crate(
         "build_error",
         srctree / "rust" / "build_error.rs",
-        ["core", "compiler_builtins"],
+        [core, compiler_builtins],
     )
 
     def append_crate_with_generated(
         display_name: str,
-        deps: list[str],
-    ) -> None:
-        append_crate(
+        deps: list[Dependency],
+    ) -> Dependency:
+        dep = append_crate(
             display_name,
             srctree / "rust" / display_name / "lib.rs",
             deps,
@@ -174,10 +173,11 @@ def generate_crates(
             }
         }
         crates[-1] = crate
+        return dep
 
-    append_crate_with_generated("bindings", ["core"])
-    append_crate_with_generated("uapi", ["core"])
-    append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "uapi"])
+    bindings = append_crate_with_generated("bindings", [core])
+    uapi = append_crate_with_generated("uapi", [core])
+    kernel = append_crate_with_generated("kernel", [bindings, build_error, core, macros, uapi])
 
     def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
         try:
@@ -207,7 +207,7 @@ def generate_crates(
             append_crate(
                 name,
                 path,
-                ["core", "kernel"],
+                [core, kernel],
                 cfg=cfg,
             )
 

-- 
2.48.1

Re: [PATCH v2 6/7] scripts: generate_rust_analyzer.py: identify crates explicitly

[Fiona Behrens]

Tamir Duberstein <tamird@gmail.com> writes:

> Use the return of `append_crate` to declare dependency on that crate.
> This allows multiple crates with the same display_name be defined, which
> we'll use to define host crates separately from target crates.
>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

Reviewed-by: Fiona Behrens <me@kloenk.dev>

> ---
>  scripts/generate_rust_analyzer.py | 60 +++++++++++++++++++--------------------
>  1 file changed, 30 insertions(+), 30 deletions(-)
>
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index 1bb185ae2e87..e1002867735b 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -68,40 +68,38 @@ def generate_crates(
>              line = line.replace("\n", "")
>              cfg.append(line)
>  
> -    # Now fill the crates list -- dependencies need to come first.
> -    #
> -    # Avoid O(n^2) iterations by keeping a map of indexes.
> +    # Now fill the crates list.
>      crates: list[Crate] = []
> -    crates_indexes: dict[str, int] = {}
>      crates_cfgs = args_crates_cfgs(cfgs)
>  
>      def append_crate(
>          display_name: str,
>          root_module: pathlib.Path,
> -        deps: list[str],
> +        deps: list[Dependency],
>          cfg: list[str] = [],
>          is_workspace_member: bool = True,
> -    ) -> None:
> -        crates_indexes[display_name] = len(crates)
> +    ) -> Dependency:
> +        index = len(crates)
>          crates.append({
>              "display_name": display_name,
>              "root_module": str(root_module),
>              "is_workspace_member": is_workspace_member,
> -            "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps],
> +            "deps": deps,
>              "cfg": cfg,
>              "edition": "2021",
>              "env": {
>                  "RUST_MODFILE": "This is only for rust-analyzer"
>              },
>          })
> +        return {"crate": index, "name": display_name}
>  
>      def append_proc_macro_crate(
>          display_name: str,
>          root_module: pathlib.Path,
> -        deps: list[str],
> +        deps: list[Dependency],
>          cfg: list[str] = [],
> -    ) -> None:
> -        append_crate(display_name, root_module, deps, cfg)
> +    ) -> Dependency:
> +        dep = append_crate(display_name, root_module, deps, cfg)
>          proc_macro_dylib_name = subprocess.check_output(
>              [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
>              stdin=subprocess.DEVNULL,
> @@ -112,13 +110,14 @@ def generate_crates(
>              "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name),
>          }
>          crates[-1] = crate
> +        return dep
>  
>      def append_sysroot_crate(
>          display_name: str,
> -        deps: list[str],
> +        deps: list[Dependency],
>          cfg: list[str] = [],
> -    ) -> None:
> -        append_crate(
> +    ) -> Dependency:
> +        return append_crate(
>              display_name,
>              sysroot_src / display_name / "src" / "lib.rs",
>              deps,
> @@ -129,34 +128,34 @@ def generate_crates(
>      # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
>      # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
>      # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
> -    append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
> -    append_sysroot_crate("alloc", ["core"])
> -    append_sysroot_crate("std", ["alloc", "core"])
> -    append_sysroot_crate("proc_macro", ["core", "std"])
> +    core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
> +    alloc = append_sysroot_crate("alloc", [core])
> +    std = append_sysroot_crate("std", [alloc, core])
> +    proc_macro = append_sysroot_crate("proc_macro", [core, std])
>  
> -    append_crate(
> +    compiler_builtins = append_crate(
>          "compiler_builtins",
>          srctree / "rust" / "compiler_builtins.rs",
>          [],
>      )
>  
> -    append_proc_macro_crate(
> +    macros = append_proc_macro_crate(
>          "macros",
>          srctree / "rust" / "macros" / "lib.rs",
> -        ["std", "proc_macro"],
> +        [std, proc_macro],
>      )
>  
> -    append_crate(
> +    build_error = append_crate(
>          "build_error",
>          srctree / "rust" / "build_error.rs",
> -        ["core", "compiler_builtins"],
> +        [core, compiler_builtins],
>      )
>  
>      def append_crate_with_generated(
>          display_name: str,
> -        deps: list[str],
> -    ) -> None:
> -        append_crate(
> +        deps: list[Dependency],
> +    ) -> Dependency:
> +        dep = append_crate(
>              display_name,
>              srctree / "rust" / display_name / "lib.rs",
>              deps,
> @@ -174,10 +173,11 @@ def generate_crates(
>              }
>          }
>          crates[-1] = crate
> +        return dep
>  
> -    append_crate_with_generated("bindings", ["core"])
> -    append_crate_with_generated("uapi", ["core"])
> -    append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "uapi"])
> +    bindings = append_crate_with_generated("bindings", [core])
> +    uapi = append_crate_with_generated("uapi", [core])
> +    kernel = append_crate_with_generated("kernel", [bindings, build_error, core, macros, uapi])
>  
>      def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
>          try:
> @@ -207,7 +207,7 @@ def generate_crates(
>              append_crate(
>                  name,
>                  path,
> -                ["core", "kernel"],
> +                [core, kernel],
>                  cfg=cfg,
>              )

[PATCH v2 7/7] scripts: generate_rust_analyzer.py: define host crates

[Tamir Duberstein]

Define host crates used by the `macros` crate separately from target
crates, now that we can uniquely identify crates with the same name.

This avoids rust-analyzer thinking the host `core` crate has our target
configs applied to it.

Suggested-by: Miguel Ojeda <ojeda@kernel.org>
Link: https://lore.kernel.org/all/CANiq72mw83RmLYeFFoJW6mUUygoyiA_f1ievSC2pmBESsQew+w@mail.gmail.com/
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 scripts/generate_rust_analyzer.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index e1002867735b..4832a4901c94 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -128,10 +128,12 @@ def generate_crates(
     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
     # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
     # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
+    host_core = append_sysroot_crate("core", [])
+    host_alloc = append_sysroot_crate("alloc", [host_core])
+    host_std = append_sysroot_crate("std", [host_alloc, host_core])
+    host_proc_macro = append_sysroot_crate("proc_macro", [host_core, host_std])
+
     core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
-    alloc = append_sysroot_crate("alloc", [core])
-    std = append_sysroot_crate("std", [alloc, core])
-    proc_macro = append_sysroot_crate("proc_macro", [core, std])
 
     compiler_builtins = append_crate(
         "compiler_builtins",
@@ -142,7 +144,7 @@ def generate_crates(
     macros = append_proc_macro_crate(
         "macros",
         srctree / "rust" / "macros" / "lib.rs",
-        [std, proc_macro],
+        [host_std, host_proc_macro],
     )
 
     build_error = append_crate(

-- 
2.48.1

Re: [PATCH v2 7/7] scripts: generate_rust_analyzer.py: define host crates

[Fiona Behrens]

Tamir Duberstein <tamird@gmail.com> writes:

> Define host crates used by the `macros` crate separately from target
> crates, now that we can uniquely identify crates with the same name.
>
> This avoids rust-analyzer thinking the host `core` crate has our target
> configs applied to it.
>
> Suggested-by: Miguel Ojeda <ojeda@kernel.org>
> Link: https://lore.kernel.org/all/CANiq72mw83RmLYeFFoJW6mUUygoyiA_f1ievSC2pmBESsQew+w@mail.gmail.com/
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>

Reviewed-by: Fiona Behrens <me@kloenk.dev>

> ---
>  scripts/generate_rust_analyzer.py | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index e1002867735b..4832a4901c94 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -128,10 +128,12 @@ def generate_crates(
>      # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
>      # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
>      # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
> +    host_core = append_sysroot_crate("core", [])
> +    host_alloc = append_sysroot_crate("alloc", [host_core])
> +    host_std = append_sysroot_crate("std", [host_alloc, host_core])
> +    host_proc_macro = append_sysroot_crate("proc_macro", [host_core, host_std])
> +
>      core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
> -    alloc = append_sysroot_crate("alloc", [core])
> -    std = append_sysroot_crate("std", [alloc, core])
> -    proc_macro = append_sysroot_crate("proc_macro", [core, std])
>  
>      compiler_builtins = append_crate(
>          "compiler_builtins",
> @@ -142,7 +144,7 @@ def generate_crates(
>      macros = append_proc_macro_crate(
>          "macros",
>          srctree / "rust" / "macros" / "lib.rs",
> -        [std, proc_macro],
> +        [host_std, host_proc_macro],
>      )
>  
>      build_error = append_crate(