[PATCH v2] scripts: read cfgs from Makefile for rust-analyzer

[PATCH v2] scripts: read cfgs from Makefile for rust-analyzer

[Martin Rodriguez Reboredo]

Both `core` and `alloc` had their `cfgs` missing in `rust-project.json`,
to remedy this `generate_rust_analyzer.py` scans the Makefile from
inside the `rust` directory for them to be added to a dictionary that
each key corresponds to a crate and each value, to an array of `cfgs`.

Signed-off-by: Martin Rodriguez Reboredo <yakoyoku@gmail.com>
---
v1 -> v2:
- Read values from command line arguments

 rust/Makefile                     |  4 +++-
 scripts/generate_rust_analyzer.py | 16 ++++++++++++++--
 2 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/rust/Makefile b/rust/Makefile
index 7c9d9f11aec5..6234f5b70aee 100644
--- a/rust/Makefile
+++ b/rust/Makefile
@@ -373,7 +373,9 @@ quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L
 	$(if $(rustc_objcopy),;$(OBJCOPY) $(rustc_objcopy) $@)
 
 rust-analyzer:
-	$(Q)$(srctree)/scripts/generate_rust_analyzer.py $(srctree) $(objtree) \
+	$(Q)$(srctree)/scripts/generate_rust_analyzer.py \
+		--cfgs='core=$(core-cfgs)' --cfgs='alloc=$(alloc-cfgs)' \
+		$(srctree) $(objtree) \
 		$(RUST_LIB_SRC) > $(objtree)/rust-project.json
 
 redirect-intrinsics = \
diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index 946e250c1b2a..5a850f055609 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -9,7 +9,15 @@ import logging
 import pathlib
 import sys
 
-def generate_crates(srctree, objtree, sysroot_src):
+def args_crates_cfgs(cfgs):
+    crates_cfgs = {}
+    for cfg in cfgs:
+        crate, vals = cfg.split("=", 1)
+        crates_cfgs[crate] = vals.replace("--cfg", "").split()
+
+    return crates_cfgs
+
+def generate_crates(srctree, objtree, sysroot_src, cfgs):
     # Generate the configuration list.
     cfg = []
     with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
@@ -23,6 +31,7 @@ def generate_crates(srctree, objtree, sysroot_src):
     # Avoid O(n^2) iterations by keeping a map of indexes.
     crates = []
     crates_indexes = {}
+    crates_cfgs = args_crates_cfgs(cfgs)
 
     def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False):
         crates_indexes[display_name] = len(crates)
@@ -44,6 +53,7 @@ def generate_crates(srctree, objtree, sysroot_src):
         "core",
         sysroot_src / "core" / "src" / "lib.rs",
         [],
+        cfg=crates_cfgs.get("core", []),
         is_workspace_member=False,
     )
 
@@ -57,6 +67,7 @@ def generate_crates(srctree, objtree, sysroot_src):
         "alloc",
         srctree / "rust" / "alloc" / "lib.rs",
         ["core", "compiler_builtins"],
+        cfg=crates_cfgs.get("alloc", []),
     )
 
     append_crate(
@@ -123,6 +134,7 @@ def generate_crates(srctree, objtree, sysroot_src):
 def main():
     parser = argparse.ArgumentParser()
     parser.add_argument('--verbose', '-v', action='store_true')
+    parser.add_argument('--cfgs', action='append', default=[])
     parser.add_argument("srctree", type=pathlib.Path)
     parser.add_argument("objtree", type=pathlib.Path)
     parser.add_argument("sysroot_src", type=pathlib.Path)
@@ -134,7 +146,7 @@ def main():
     )
 
     rust_project = {
-        "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src),
+        "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src, args.cfgs),
         "sysroot_src": str(args.sysroot_src),
     }
 
-- 
2.40.1

Re: [PATCH v2] scripts: read cfgs from Makefile for rust-analyzer

[Martin Rodriguez Reboredo]

Bump 🧐

Re: [PATCH v2] scripts: read cfgs from Makefile for rust-analyzer

[Benno Lossin]

> Both `core` and `alloc` had their `cfgs` missing in `rust-project.json`,
> to remedy this `generate_rust_analyzer.py` scans the Makefile from
> inside the `rust` directory for them to be added to a dictionary that
> each key corresponds to a crate and each value, to an array of `cfgs`.
> 
> Signed-off-by: Martin Rodriguez Reboredo yakoyoku@gmail.com

When I try this patch *all* configuration options are input into the `cfg`
field of the `core` crate (others are also affected):
```json
            "cfg": [
                "CONFIG_RING_BUFFER",
                "CONFIG_RING_BUFFER=\"y\"",
                "CONFIG_HAVE_ARCH_SECCOMP_FILTER",
                "CONFIG_HAVE_ARCH_SECCOMP_FILTER=\"y\"",
                "CONFIG_KERNEL_GZIP",
                "CONFIG_KERNEL_GZIP=\"y\"",
                "CONFIG_CC_HAS_SANCOV_TRACE_PC",
                "CONFIG_CC_HAS_SANCOV_TRACE_PC=\"y\"",
                "CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP",
```
The snippet would continue on for about 1000 lines.
I think this is wrong and not intended.

--
Cheers,
Benno

> 
> ---
> v1 -> v2:
> 
> - Read values from command line arguments
> 
> rust/Makefile | 4 +++-
> scripts/generate_rust_analyzer.py | 16 ++++++++++++++--
> 2 files changed, 17 insertions(+), 3 deletions(-)
> 
> diff --git a/rust/Makefile b/rust/Makefile
> index 7c9d9f11aec5..6234f5b70aee 100644
> --- a/rust/Makefile
> +++ b/rust/Makefile
> @@ -373,7 +373,9 @@ quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L
> $(if $(rustc_objcopy),;$(OBJCOPY) $(rustc_objcopy) $@)
> 
> rust-analyzer:
> - $(Q)$(srctree)/scripts/generate_rust_analyzer.py $(srctree) $(objtree) \
> + $(Q)$(srctree)/scripts/generate_rust_analyzer.py \
> + --cfgs='core=$(core-cfgs)' --cfgs='alloc=$(alloc-cfgs)' \
> + $(srctree) $(objtree) \
> $(RUST_LIB_SRC) > $(objtree)/rust-project.json
> 
> 
> redirect-intrinsics = \
> diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
> index 946e250c1b2a..5a850f055609 100755
> --- a/scripts/generate_rust_analyzer.py
> +++ b/scripts/generate_rust_analyzer.py
> @@ -9,7 +9,15 @@ import logging
> import pathlib
> import sys
> 
> -def generate_crates(srctree, objtree, sysroot_src):
> +def args_crates_cfgs(cfgs):
> + crates_cfgs = {}
> + for cfg in cfgs:
> + crate, vals = cfg.split("=", 1)
> + crates_cfgs[crate] = vals.replace("--cfg", "").split()
> +
> + return crates_cfgs
> +
> +def generate_crates(srctree, objtree, sysroot_src, cfgs):
> # Generate the configuration list.
> cfg = []
> with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
> @@ -23,6 +31,7 @@ def generate_crates(srctree, objtree, sysroot_src):
> # Avoid O(n^2) iterations by keeping a map of indexes.
> crates = []
> crates_indexes = {}
> + crates_cfgs = args_crates_cfgs(cfgs)
> 
> def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False):
> crates_indexes[display_name] = len(crates)
> @@ -44,6 +53,7 @@ def generate_crates(srctree, objtree, sysroot_src):
> "core",
> sysroot_src / "core" / "src" / "lib.rs",
> [],
> + cfg=crates_cfgs.get("core", []),
> is_workspace_member=False,
> )
> 
> @@ -57,6 +67,7 @@ def generate_crates(srctree, objtree, sysroot_src):
> "alloc",
> srctree / "rust" / "alloc" / "lib.rs",
> ["core", "compiler_builtins"],
> + cfg=crates_cfgs.get("alloc", []),
> )
> 
> append_crate(
> @@ -123,6 +134,7 @@ def generate_crates(srctree, objtree, sysroot_src):
> def main():
> parser = argparse.ArgumentParser()
> parser.add_argument('--verbose', '-v', action='store_true')
> + parser.add_argument('--cfgs', action='append', default=[])
> parser.add_argument("srctree", type=pathlib.Path)
> parser.add_argument("objtree", type=pathlib.Path)
> parser.add_argument("sysroot_src", type=pathlib.Path)
> @@ -134,7 +146,7 @@ def main():
> )
> 
> rust_project = {
> - "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src),
> + "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src, args.cfgs),
> "sysroot_src": str(args.sysroot_src),
> }
> 
> --
> 2.40.1

Re: [PATCH v2] scripts: read cfgs from Makefile for rust-analyzer

[Martin Rodriguez Reboredo]

On 7/14/23 12:55, Benno Lossin wrote:
>> Both `core` and `alloc` had their `cfgs` missing in `rust-project.json`,
>> to remedy this `generate_rust_analyzer.py` scans the Makefile from
>> inside the `rust` directory for them to be added to a dictionary that
>> each key corresponds to a crate and each value, to an array of `cfgs`.
>>
>> Signed-off-by: Martin Rodriguez Reboredo yakoyoku@gmail.com
> 
> When I try this patch *all* configuration options are input into the `cfg`
> field of the `core` crate (others are also affected):
> ```json
>              "cfg": [
>                  "CONFIG_RING_BUFFER",
>                  "CONFIG_RING_BUFFER=\"y\"",
>                  "CONFIG_HAVE_ARCH_SECCOMP_FILTER",
>                  "CONFIG_HAVE_ARCH_SECCOMP_FILTER=\"y\"",
>                  "CONFIG_KERNEL_GZIP",
>                  "CONFIG_KERNEL_GZIP=\"y\"",
>                  "CONFIG_CC_HAS_SANCOV_TRACE_PC",
>                  "CONFIG_CC_HAS_SANCOV_TRACE_PC=\"y\"",
>                  "CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP",
> ```
> The snippet would continue on for about 1000 lines.
> I think this is wrong and not intended.
> 
> --
> Cheers,
> Benno
> 

Wait, can you tell me which branch are you on? As it works on
`rust-dev` and `rust-next` with `make rust-analyzer`. Also, do you
have any other details?

Re: [PATCH v2] scripts: read cfgs from Makefile for rust-analyzer

[Benno Lossin]

On 15.07.23 05:33, Martin Rodriguez Reboredo wrote:
> On 7/14/23 12:55, Benno Lossin wrote:
>>> Both `core` and `alloc` had their `cfgs` missing in `rust-project.json`,
>>> to remedy this `generate_rust_analyzer.py` scans the Makefile from
>>> inside the `rust` directory for them to be added to a dictionary that
>>> each key corresponds to a crate and each value, to an array of `cfgs`.
>>>
>>> Signed-off-by: Martin Rodriguez Reboredo yakoyoku@gmail.com
>>
>> When I try this patch *all* configuration options are input into the `cfg`
>> field of the `core` crate (others are also affected):
>> ```json
>>               "cfg": [
>>                   "CONFIG_RING_BUFFER",
>>                   "CONFIG_RING_BUFFER=\"y\"",
>>                   "CONFIG_HAVE_ARCH_SECCOMP_FILTER",
>>                   "CONFIG_HAVE_ARCH_SECCOMP_FILTER=\"y\"",
>>                   "CONFIG_KERNEL_GZIP",
>>                   "CONFIG_KERNEL_GZIP=\"y\"",
>>                   "CONFIG_CC_HAS_SANCOV_TRACE_PC",
>>                   "CONFIG_CC_HAS_SANCOV_TRACE_PC=\"y\"",
>>                   "CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP",
>> ```
>> The snippet would continue on for about 1000 lines.
>> I think this is wrong and not intended.
>>
>> --
>> Cheers,
>> Benno
>>
> 
> Wait, can you tell me which branch are you on? As it works on
> `rust-dev` and `rust-next` with `make rust-analyzer`. Also, do you
> have any other details?

I tried it on both `rust-dev` and on `rust-next` and on both I have
this problem. Here is what I executed:

```sh
git checkout $branch
make clean
make mrproper
make menuconfig
make LLVM=1 rust-analyzer
```

In the `make menuconfig` step I disabled `DEBUG_INFO_BTF` and enabled
`RUST`.

If you want, I can upload the `rust-project.json` file to GH, since
it is 55k lines long.

-- 
Cheers,
Benno

Re: [PATCH v2] scripts: read cfgs from Makefile for rust-analyzer

[Martin Rodriguez Reboredo]

On 7/15/23 05:07, Benno Lossin wrote:
> On 15.07.23 05:33, Martin Rodriguez Reboredo wrote:
>> On 7/14/23 12:55, Benno Lossin wrote:
>>>> Both `core` and `alloc` had their `cfgs` missing in `rust-project.json`,
>>>> to remedy this `generate_rust_analyzer.py` scans the Makefile from
>>>> inside the `rust` directory for them to be added to a dictionary that
>>>> each key corresponds to a crate and each value, to an array of `cfgs`.
>>>>
>>>> Signed-off-by: Martin Rodriguez Reboredo yakoyoku@gmail.com
>>>
>>> When I try this patch *all* configuration options are input into the `cfg`
>>> field of the `core` crate (others are also affected):
>>> ```json
>>>                "cfg": [
>>>                    "CONFIG_RING_BUFFER",
>>>                    "CONFIG_RING_BUFFER=\"y\"",
>>>                    "CONFIG_HAVE_ARCH_SECCOMP_FILTER",
>>>                    "CONFIG_HAVE_ARCH_SECCOMP_FILTER=\"y\"",
>>>                    "CONFIG_KERNEL_GZIP",
>>>                    "CONFIG_KERNEL_GZIP=\"y\"",
>>>                    "CONFIG_CC_HAS_SANCOV_TRACE_PC",
>>>                    "CONFIG_CC_HAS_SANCOV_TRACE_PC=\"y\"",
>>>                    "CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP",
>>> ```
>>> The snippet would continue on for about 1000 lines.
>>> I think this is wrong and not intended.
>>>
>>> --
>>> Cheers,
>>> Benno
>>>
>>
>> Wait, can you tell me which branch are you on? As it works on
>> `rust-dev` and `rust-next` with `make rust-analyzer`. Also, do you
>> have any other details?
> 
> I tried it on both `rust-dev` and on `rust-next` and on both I have
> this problem. Here is what I executed:
> 
> ```sh
> git checkout $branch
> make clean
> make mrproper
> make menuconfig
> make LLVM=1 rust-analyzer
> ```
> 
> In the `make menuconfig` step I disabled `DEBUG_INFO_BTF` and enabled
> `RUST`.
> 
> If you want, I can upload the `rust-project.json` file to GH, since
> it is 55k lines long.
> 

I'm thinking that you may be looking at the wrong side. So, can you
show me the output of these `jq` commands?

```sh
jq '.crates[] | select(.display_name == "core")' rust-project.json
jq '.crates[] | select(.display_name == "alloc")' rust-project.json
```

Re: [PATCH v2] scripts: read cfgs from Makefile for rust-analyzer

[Benno Lossin]

On 15.07.23 16:40, Martin Rodriguez Reboredo wrote:
>> If you want, I can upload the `rust-project.json` file to GH, since
>> it is 55k lines long.
>>
> 
> I'm thinking that you may be looking at the wrong side. 

Seems so, I looked at `bindings` and `kernel`, but got confused by deps.
I also verified that without your patch all of those configs are present.

Still I find it a bit confusing that all of these configs are present
there. How does rust-analyzer use those?

Also sorry for the noise.

> So, can you show me the output of these `jq` commands?
> 
> ```sh
> jq '.crates[] | select(.display_name == "core")' rust-project.json
> jq '.crates[] | select(.display_name == "alloc")' rust-project.json
> ```

```
{
  "cfg": [
    "no_fp_fmt_parse"
  ],
  "deps": [],
  "display_name": "core",
  "edition": "2021",
  "env": {
    "RUST_MODFILE": "This is only for rust-analyzer"
  },
  "is_proc_macro": false,
  "is_workspace_member": false,
  "root_module": "~/.rustup/toolchains/1.68.2-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/lib.rs"
}
```
and
```
{
  "cfg": [
    "no_borrow",
    "no_fmt",
    "no_global_oom_handling",
    "no_macros",
    "no_rc",
    "no_str",
    "no_string",
    "no_sync",
    "no_thin"
  ],
  "deps": [
    {
      "crate": 0,
      "name": "core"
    },
    {
      "crate": 1,
      "name": "compiler_builtins"
    }
  ],
  "display_name": "alloc",
  "edition": "2021",
  "env": {
    "RUST_MODFILE": "This is only for rust-analyzer"
  },
  "is_proc_macro": false,
  "is_workspace_member": true,
  "root_module": "rust/alloc/lib.rs"
}
```

-- 
Cheers,
Benno

Re: [PATCH v2] scripts: read cfgs from Makefile for rust-analyzer

[Martin Rodriguez Reboredo]

On 7/15/23 12:10, Benno Lossin wrote:
> On 15.07.23 16:40, Martin Rodriguez Reboredo wrote:
>>> If you want, I can upload the `rust-project.json` file to GH, since
>>> it is 55k lines long.
>>>
>>
>> I'm thinking that you may be looking at the wrong side.
> 
> Seems so, I looked at `bindings` and `kernel`, but got confused by deps.
> I also verified that without your patch all of those configs are present.
> 
> Still I find it a bit confusing that all of these configs are present
> there. How does rust-analyzer use those?

`rust-analyzer` sees those configs so when you type code it'll show you
suggestions based upon them in your IDE. For example, if you have
`CONFIG_NET` enabled then when you type "kernel::net::" your IDE will
suggest you completions from that module, otherwise it will not.

> Also sorry for the noise.

No problem, you'll easily get confused if you look upon that massive file.
Happened to me.

>> So, can you show me the output of these `jq` commands?
>>
>> ```sh
>> jq '.crates[] | select(.display_name == "core")' rust-project.json
>> jq '.crates[] | select(.display_name == "alloc")' rust-project.json
>> ```
> 
> [...]

Output looks as it was expected, so if in your IDE you go to the
definition of a function or type from either `core` or `alloc` then it's
going to show you the correct result, otherwise it might get confused or
throw you an error.

Re: [PATCH v2] scripts: read cfgs from Makefile for rust-analyzer

[Miguel Ojeda]

On Sun, May 21, 2023 at 1:17 AM Martin Rodriguez Reboredo
<yakoyoku@gmail.com> wrote:
>
> Both `core` and `alloc` had their `cfgs` missing in `rust-project.json`,
> to remedy this `generate_rust_analyzer.py` scans the Makefile from
> inside the `rust` directory for them to be added to a dictionary that
> each key corresponds to a crate and each value, to an array of `cfgs`.
>
> Signed-off-by: Martin Rodriguez Reboredo <yakoyoku@gmail.com>

This is a much better approach than reading the `Makefile`, thanks! It
works on my side.

The commit message is outdated, though: we are not reading from the
`Makefile` anymore in v2. What about something like...

    scripts: generate_rust_analyzer: provide `cfg`s for `core` and `alloc`

    Both `core` and `alloc` have their `cfgs` (such as `no_rc`) missing
    in `rust-project.json`.

    To remedy this, pass the flags to `generate_rust_analyzer.py` for
    them to be added to a dictionary where each key corresponds to
    a crate and each value to a list of `cfg`s. The dictionary is then
    used to pass the `cfg`s to each crate in the generated file (for
    `core` and `alloc` only).

Also we have Vinay's patch on `rust-next`, which conflicts a bit, so
if you can please send a quick v3 with the commit message fixed and
rebase on top of it, it would be great.

Thanks!

Cheers,
Miguel