* [PATCH] gen_compile_commands: honor the build directory for modules
@ 2026-10-15 13:57 vngocvan
2026-10-15 15:33 ` Nick Desaulniers
2026-10-16 1:21 ` Nathan Chancellor
0 siblings, 2 replies; 6+ messages in thread
From: vngocvan @ 2026-10-15 13:57 UTC (permalink / raw)
To: nathan, nsc
Cc: ndesaulniers, morbo, justinstitt, masahiroy, llvm, linux-kbuild,
linux-kernel
When invoked from outside the build tree with -d and a modules.order
argument, the script opens .mod files relative to the current directory
instead of the specified build directory. This can fail with
FileNotFoundError or collect commands from unrelated artifacts in the
current directory.
Pass the build directory to cmdfiles_for_modorder() and resolve both
the .mod files and their associated .cmd files relative to it. Absolute
paths continue to work, and the output location is unchanged.
Fixes: ecca4fea1ede ("gen_compile_commands: support *.o, *.a, modules.order in positional argument")
Signed-off-by: vngocvan <vngocvan@gmail.com>
---
scripts/clang-tools/gen_compile_commands.py | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
index 8d14b81efd73..149135f85f19 100755
--- a/scripts/clang-tools/gen_compile_commands.py
+++ b/scripts/clang-tools/gen_compile_commands.py
@@ -124,7 +124,7 @@ def cmdfiles_for_a(archive, ar):
yield to_cmdfile(obj)
-def cmdfiles_for_modorder(modorder):
+def cmdfiles_for_modorder(modorder, root_directory):
"""Generate the iterator of .cmd files associated with the modules.order.
Parse the given modules.order, and yield every .cmd file used to build the
@@ -132,6 +132,7 @@ def cmdfiles_for_modorder(modorder):
Args:
modorder: The modules.order file to parse
+ root_directory: The directory where the objects were built
Yields:
The path to every .cmd file found
@@ -142,11 +143,11 @@ def cmdfiles_for_modorder(modorder):
base, ext = os.path.splitext(obj)
if ext != '.o':
sys.exit('{}: module path must end with .o'.format(obj))
- mod = base + '.mod'
+ mod = os.path.join(root_directory, base + '.mod')
# Read from *.mod, to get a list of objects that compose the module.
with open(mod) as m:
for mod_line in m:
- yield to_cmdfile(mod_line.rstrip())
+ yield os.path.join(root_directory, to_cmdfile(mod_line.rstrip()))
def process_line(root_directory, command_prefix, file_path):
@@ -206,7 +207,7 @@ def main():
elif path.endswith('.a'):
cmdfiles = cmdfiles_for_a(path, ar)
elif path.endswith('modules.order'):
- cmdfiles = cmdfiles_for_modorder(path)
+ cmdfiles = cmdfiles_for_modorder(path, directory)
else:
sys.exit('{}: unknown file type'.format(path))
base-commit: 0fcb05b72344f40806cbe810de8872f1698bf053
--
2.56.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] gen_compile_commands: honor the build directory for modules
2026-10-15 13:57 [PATCH] gen_compile_commands: honor the build directory for modules vngocvan
@ 2026-10-15 15:33 ` Nick Desaulniers
2026-10-16 1:12 ` Nathan Chancellor
2026-10-16 1:21 ` Nathan Chancellor
1 sibling, 1 reply; 6+ messages in thread
From: Nick Desaulniers @ 2026-10-15 15:33 UTC (permalink / raw)
To: vngocvan
Cc: nathan, nsc, morbo, justinstitt, masahiroy, llvm, linux-kbuild,
linux-kernel
On Thu, Oct 15, 2026 at 6:58 AM vngocvan <vngocvan@gmail.com> wrote:
>
> When invoked from outside the build tree with -d and a modules.order
> argument, the script opens .mod files relative to the current directory
> instead of the specified build directory. This can fail with
> FileNotFoundError or collect commands from unrelated artifacts in the
> current directory.
>
> Pass the build directory to cmdfiles_for_modorder() and resolve both
> the .mod files and their associated .cmd files relative to it. Absolute
> paths continue to work, and the output location is unchanged.
>
> Fixes: ecca4fea1ede ("gen_compile_commands: support *.o, *.a, modules.order in positional argument")
> Signed-off-by: vngocvan <vngocvan@gmail.com>
> ---
> scripts/clang-tools/gen_compile_commands.py | 9 +++++----
> 1 file changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
> index 8d14b81efd73..149135f85f19 100755
> --- a/scripts/clang-tools/gen_compile_commands.py
> +++ b/scripts/clang-tools/gen_compile_commands.py
> @@ -124,7 +124,7 @@ def cmdfiles_for_a(archive, ar):
> yield to_cmdfile(obj)
>
>
> -def cmdfiles_for_modorder(modorder):
> +def cmdfiles_for_modorder(modorder, root_directory):
> """Generate the iterator of .cmd files associated with the modules.order.
>
> Parse the given modules.order, and yield every .cmd file used to build the
> @@ -132,6 +132,7 @@ def cmdfiles_for_modorder(modorder):
>
> Args:
> modorder: The modules.order file to parse
> + root_directory: The directory where the objects were built
>
> Yields:
> The path to every .cmd file found
> @@ -142,11 +143,11 @@ def cmdfiles_for_modorder(modorder):
> base, ext = os.path.splitext(obj)
> if ext != '.o':
> sys.exit('{}: module path must end with .o'.format(obj))
> - mod = base + '.mod'
> + mod = os.path.join(root_directory, base + '.mod')
> # Read from *.mod, to get a list of objects that compose the module.
> with open(mod) as m:
> for mod_line in m:
> - yield to_cmdfile(mod_line.rstrip())
> + yield os.path.join(root_directory, to_cmdfile(mod_line.rstrip()))
Thanks for the patch. Can we avoid os.path in favor of Path from
Pathlib? Here and throughout.
>
>
> def process_line(root_directory, command_prefix, file_path):
> @@ -206,7 +207,7 @@ def main():
> elif path.endswith('.a'):
> cmdfiles = cmdfiles_for_a(path, ar)
> elif path.endswith('modules.order'):
> - cmdfiles = cmdfiles_for_modorder(path)
> + cmdfiles = cmdfiles_for_modorder(path, directory)
> else:
> sys.exit('{}: unknown file type'.format(path))
>
>
> base-commit: 0fcb05b72344f40806cbe810de8872f1698bf053
> --
> 2.56.0
>
--
Thanks,
~Nick Desaulniers
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gen_compile_commands: honor the build directory for modules
2026-10-15 15:33 ` Nick Desaulniers
@ 2026-10-16 1:12 ` Nathan Chancellor
0 siblings, 0 replies; 6+ messages in thread
From: Nathan Chancellor @ 2026-10-16 1:12 UTC (permalink / raw)
To: Nick Desaulniers
Cc: vngocvan, nsc, morbo, justinstitt, masahiroy, llvm, linux-kbuild,
linux-kernel
On Thu, Oct 15, 2026 at 08:33:11AM -0700, Nick Desaulniers wrote:
> Thanks for the patch. Can we avoid os.path in favor of Path from
> Pathlib? Here and throughout.
I would not mind seeing some modernization in this script such as
os.path -> pathlib.Path but it should definitely be done as a separate
patch.
--
Cheers,
Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gen_compile_commands: honor the build directory for modules
2026-10-15 13:57 [PATCH] gen_compile_commands: honor the build directory for modules vngocvan
2026-10-15 15:33 ` Nick Desaulniers
@ 2026-10-16 1:21 ` Nathan Chancellor
[not found] ` <CAJavoFk+zgySyNps5tuKmntmvuAyev=kKL640TFrfCc-njeWBw@mail.gmail.com>
1 sibling, 1 reply; 6+ messages in thread
From: Nathan Chancellor @ 2026-10-16 1:21 UTC (permalink / raw)
To: vngocvan
Cc: nsc, ndesaulniers, morbo, justinstitt, masahiroy, llvm,
linux-kbuild, linux-kernel
Hi,
On Thu, Oct 15, 2026 at 08:57:52PM +0700, vngocvan wrote:
> When invoked from outside the build tree with -d and a modules.order
> argument, the script opens .mod files relative to the current directory
> instead of the specified build directory. This can fail with
> FileNotFoundError or collect commands from unrelated artifacts in the
> current directory.
How was this discovered? This cannot happen within Kbuild because this
script is always invoked from within the build folder, so was it a
manual invocation or something else?
> Pass the build directory to cmdfiles_for_modorder() and resolve both
> the .mod files and their associated .cmd files relative to it. Absolute
> paths continue to work, and the output location is unchanged.
I think you can omit the final sentence here, it does not add much value
in my opinion.
> Fixes: ecca4fea1ede ("gen_compile_commands: support *.o, *.a, modules.order in positional argument")
> Signed-off-by: vngocvan <vngocvan@gmail.com>
Please fix your name, this needs to be a known identity.
> ---
> scripts/clang-tools/gen_compile_commands.py | 9 +++++----
> 1 file changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
> index 8d14b81efd73..149135f85f19 100755
> --- a/scripts/clang-tools/gen_compile_commands.py
> +++ b/scripts/clang-tools/gen_compile_commands.py
> @@ -124,7 +124,7 @@ def cmdfiles_for_a(archive, ar):
> yield to_cmdfile(obj)
>
>
> -def cmdfiles_for_modorder(modorder):
> +def cmdfiles_for_modorder(modorder, root_directory):
> """Generate the iterator of .cmd files associated with the modules.order.
>
> Parse the given modules.order, and yield every .cmd file used to build the
> @@ -132,6 +132,7 @@ def cmdfiles_for_modorder(modorder):
>
> Args:
> modorder: The modules.order file to parse
> + root_directory: The directory where the objects were built
>
> Yields:
> The path to every .cmd file found
> @@ -142,11 +143,11 @@ def cmdfiles_for_modorder(modorder):
> base, ext = os.path.splitext(obj)
> if ext != '.o':
> sys.exit('{}: module path must end with .o'.format(obj))
> - mod = base + '.mod'
> + mod = os.path.join(root_directory, base + '.mod')
> # Read from *.mod, to get a list of objects that compose the module.
> with open(mod) as m:
> for mod_line in m:
> - yield to_cmdfile(mod_line.rstrip())
> + yield os.path.join(root_directory, to_cmdfile(mod_line.rstrip()))
>
>
> def process_line(root_directory, command_prefix, file_path):
> @@ -206,7 +207,7 @@ def main():
> elif path.endswith('.a'):
> cmdfiles = cmdfiles_for_a(path, ar)
> elif path.endswith('modules.order'):
> - cmdfiles = cmdfiles_for_modorder(path)
> + cmdfiles = cmdfiles_for_modorder(path, directory)
> else:
> sys.exit('{}: unknown file type'.format(path))
>
>
> base-commit: 0fcb05b72344f40806cbe810de8872f1698bf053
> --
> 2.56.0
>
--
Cheers,
Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gen_compile_commands: honor the build directory for modules
[not found] ` <CAJavoFk+zgySyNps5tuKmntmvuAyev=kKL640TFrfCc-njeWBw@mail.gmail.com>
@ 2026-10-16 4:44 ` Van Vo
2026-10-16 6:25 ` Van Vo
1 sibling, 0 replies; 6+ messages in thread
From: Van Vo @ 2026-10-16 4:44 UTC (permalink / raw)
To: llvm
Hi Nathan,
> How was this discovered?
This came from code inspection of standalone use of -d with an explicit
modules.order argument, followed initially by a constructed reproducer.
It was not a failure encountered through Kbuild's compile_commands.json
target.
I have now reproduced it using real Kbuild-generated artifacts from
0fcb05b72344f40806cbe810de8872f1698bf053 on Linux/aarch64, with Python
3.11.13 and GCC 12.2.0. This was a partial lib/ build with CONFIG_TEST_LKM=m.
The following uses a disposable source tree at that base. The guards
prevent overwriting an existing output directory or result file:
(
set -eu
test ! -e out
test ! -e outside.json
test ! -e inside.json
make O=out -j1 allnoconfig
bash scripts/config --file out/.config --enable MODULES \
--enable RUNTIME_TESTING_MENU --module TEST_LKM
make O=out -j1 olddefconfig
make O=out -j1 V=1 lib/
outside_exit=0
python3 -B scripts/clang-tools/gen_compile_commands.py -a ar \
-d out -o outside.json out/lib/modules.order ||
outside_exit=$?
printf 'outside_exit=%s\n' "$outside_exit"
inside_exit=0
(cd out &&
python3 -B ../scripts/clang-tools/gen_compile_commands.py -a ar \
-o ../inside.json lib/modules.order) ||
inside_exit=$?
printf 'inside_exit=%s\n' "$inside_exit"
test "$outside_exit" -eq 1
test "$inside_exit" -eq 0
python3 -c \
'import json; assert len(json.load(open("inside.json"))) == 1'
)
Kbuild generated lib/test_module.o, lib/test_module.mod,
lib/.test_module.o.cmd and lib/modules.order under out/. Both .mod and
modules.order contain lib/test_module.o.
The old script's invocation from the source directory fails with:
FileNotFoundError: [Errno 2] No such file or directory:
'lib/test_module.mod'
outside_exit=1
The same artifacts work when the old script is invoked from out/:
inside_exit=0
The successful invocation produces one JSON entry for lib/test_module.c.
The patched script succeeds from the source directory and produces an
identical entry. The old script also fails with absolute paths for -d
and modules.order; the patched script succeeds with those paths too.
Only the standalone generator invocation is outside the build directory;
compilation itself runs under out/. This is not a full kernel build or
a boot/runtime test.
> I think you can omit the final sentence here, it does not add much value
> in my opinion.
I'll drop that sentence and clarify the standalone scope in v2.
> Please fix your name, this needs to be a known identity.
I'll use Võ Ngọc Văn for the author and sign-off in v2.
> os.path -> pathlib.Path but it should definitely be done as a separate
> patch.
I'll keep the pathlib conversion as a separate follow-up patch.
Thanks,
Võ Ngọc Văn
Thanks & best regards,
V
On Fri, Oct 16, 2026 at 10:53 AM Van Vo <vngocvan@gmail.com> wrote:
>
> Hi Nathan,
>
> > How was this discovered?
>
> This came from code inspection of standalone use of -d with an explicit
> modules.order argument, followed initially by a constructed reproducer.
> It was not a failure encountered through Kbuild's compile_commands.json
> target.
>
> I have now reproduced it using real Kbuild-generated artifacts from
> 0fcb05b72344f40806cbe810de8872f1698bf053 on Linux/aarch64, with Python
> 3.11.13 and GCC 12.2.0. This was a partial lib/ build with CONFIG_TEST_LKM=m.
>
> The following uses a disposable source tree at that base. The guards
> prevent overwriting an existing output directory or result file:
>
> (
> set -eu
> test ! -e out
> test ! -e outside.json
> test ! -e inside.json
> make O=out -j1 allnoconfig
> bash scripts/config --file out/.config --enable MODULES \
> --enable RUNTIME_TESTING_MENU --module TEST_LKM
> make O=out -j1 olddefconfig
> make O=out -j1 V=1 lib/
>
> outside_exit=0
> python3 -B scripts/clang-tools/gen_compile_commands.py -a ar \
> -d out -o outside.json out/lib/modules.order ||
> outside_exit=$?
> printf 'outside_exit=%s\n' "$outside_exit"
>
> inside_exit=0
> (cd out &&
> python3 -B ../scripts/clang-tools/gen_compile_commands.py -a ar \
> -o ../inside.json lib/modules.order) ||
> inside_exit=$?
> printf 'inside_exit=%s\n' "$inside_exit"
> test "$outside_exit" -eq 1
> test "$inside_exit" -eq 0
> python3 -c \
> 'import json; assert len(json.load(open("inside.json"))) == 1'
> )
>
> Kbuild generated lib/test_module.o, lib/test_module.mod,
> lib/.test_module.o.cmd and lib/modules.order under out/. Both .mod and
> modules.order contain lib/test_module.o.
>
> The old script's invocation from the source directory fails with:
>
> FileNotFoundError: [Errno 2] No such file or directory: 'lib/test_module.mod'
> outside_exit=1
>
> The same artifacts work when the old script is invoked from out/:
>
> inside_exit=0
>
> The successful invocation produces one JSON entry for lib/test_module.c.
> The patched script succeeds from the source directory and produces an
> identical entry. The old script also fails with absolute paths for -d
> and modules.order; the patched script succeeds with those paths too.
>
> Only the standalone generator invocation is outside the build directory;
> compilation itself runs under out/. This is not a full kernel build or
> a boot/runtime test.
>
> > I think you can omit the final sentence here, it does not add much value
> > in my opinion.
>
> I'll drop that sentence and clarify the standalone scope in v2.
>
> > Please fix your name, this needs to be a known identity.
>
> I'll use Võ Ngọc Văn for the author and sign-off in v2.
>
> > os.path -> pathlib.Path but it should definitely be done as a separate
> > patch.
>
> I'll keep the pathlib conversion as a separate follow-up patch.
>
> Thanks,
> Võ Ngọc Văn
>
>
> On Fri, Oct 16, 2026 at 8:21 AM Nathan Chancellor <nathan@kernel.org> wrote:
>>
>> Hi,
>>
>> On Thu, Oct 15, 2026 at 08:57:52PM +0700, vngocvan wrote:
>> > When invoked from outside the build tree with -d and a modules.order
>> > argument, the script opens .mod files relative to the current directory
>> > instead of the specified build directory. This can fail with
>> > FileNotFoundError or collect commands from unrelated artifacts in the
>> > current directory.
>>
>> How was this discovered? This cannot happen within Kbuild because this
>> script is always invoked from within the build folder, so was it a
>> manual invocation or something else?
>>
>> > Pass the build directory to cmdfiles_for_modorder() and resolve both
>> > the .mod files and their associated .cmd files relative to it. Absolute
>> > paths continue to work, and the output location is unchanged.
>>
>> I think you can omit the final sentence here, it does not add much value
>> in my opinion.
>>
>> > Fixes: ecca4fea1ede ("gen_compile_commands: support *.o, *.a, modules.order in positional argument")
>> > Signed-off-by: vngocvan <vngocvan@gmail.com>
>>
>> Please fix your name, this needs to be a known identity.
>>
>> > ---
>> > scripts/clang-tools/gen_compile_commands.py | 9 +++++----
>> > 1 file changed, 5 insertions(+), 4 deletions(-)
>> >
>> > diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
>> > index 8d14b81efd73..149135f85f19 100755
>> > --- a/scripts/clang-tools/gen_compile_commands.py
>> > +++ b/scripts/clang-tools/gen_compile_commands.py
>> > @@ -124,7 +124,7 @@ def cmdfiles_for_a(archive, ar):
>> > yield to_cmdfile(obj)
>> >
>> >
>> > -def cmdfiles_for_modorder(modorder):
>> > +def cmdfiles_for_modorder(modorder, root_directory):
>> > """Generate the iterator of .cmd files associated with the modules.order.
>> >
>> > Parse the given modules.order, and yield every .cmd file used to build the
>> > @@ -132,6 +132,7 @@ def cmdfiles_for_modorder(modorder):
>> >
>> > Args:
>> > modorder: The modules.order file to parse
>> > + root_directory: The directory where the objects were built
>> >
>> > Yields:
>> > The path to every .cmd file found
>> > @@ -142,11 +143,11 @@ def cmdfiles_for_modorder(modorder):
>> > base, ext = os.path.splitext(obj)
>> > if ext != '.o':
>> > sys.exit('{}: module path must end with .o'.format(obj))
>> > - mod = base + '.mod'
>> > + mod = os.path.join(root_directory, base + '.mod')
>> > # Read from *.mod, to get a list of objects that compose the module.
>> > with open(mod) as m:
>> > for mod_line in m:
>> > - yield to_cmdfile(mod_line.rstrip())
>> > + yield os.path.join(root_directory, to_cmdfile(mod_line.rstrip()))
>> >
>> >
>> > def process_line(root_directory, command_prefix, file_path):
>> > @@ -206,7 +207,7 @@ def main():
>> > elif path.endswith('.a'):
>> > cmdfiles = cmdfiles_for_a(path, ar)
>> > elif path.endswith('modules.order'):
>> > - cmdfiles = cmdfiles_for_modorder(path)
>> > + cmdfiles = cmdfiles_for_modorder(path, directory)
>> > else:
>> > sys.exit('{}: unknown file type'.format(path))
>> >
>> >
>> > base-commit: 0fcb05b72344f40806cbe810de8872f1698bf053
>> > --
>> > 2.56.0
>> >
>>
>> --
>> Cheers,
>> Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gen_compile_commands: honor the build directory for modules
[not found] ` <CAJavoFk+zgySyNps5tuKmntmvuAyev=kKL640TFrfCc-njeWBw@mail.gmail.com>
2026-10-16 4:44 ` Van Vo
@ 2026-10-16 6:25 ` Van Vo
1 sibling, 0 replies; 6+ messages in thread
From: Van Vo @ 2026-10-16 6:25 UTC (permalink / raw)
To: Nathan Chancellor
Cc: nsc, ndesaulniers, morbo, justinstitt, masahiroy, llvm,
linux-kbuild, linux-kernel
Hi Nathan,
> How was this discovered?
This came from code inspection of standalone use of -d with an explicit
modules.order argument, followed initially by a constructed reproducer.
It was not a failure encountered through Kbuild's compile_commands.json
target.
I have now reproduced it using real Kbuild-generated artifacts from
0fcb05b72344f40806cbe810de8872f1698bf053 on Linux/aarch64, with Python
3.11.13 and GCC 12.2.0. This was a partial lib/ build with CONFIG_TEST_LKM=m.
The following uses a disposable source tree at that base. The guards
prevent overwriting an existing output directory or result file:
(
set -eu
test ! -e out
test ! -e outside.json
test ! -e inside.json
make O=out -j1 allnoconfig
bash scripts/config --file out/.config --enable MODULES \
--enable RUNTIME_TESTING_MENU --module TEST_LKM
make O=out -j1 olddefconfig
make O=out -j1 V=1 lib/
outside_exit=0
python3 -B scripts/clang-tools/gen_compile_commands.py -a ar \
-d out -o outside.json out/lib/modules.order ||
outside_exit=$?
printf 'outside_exit=%s\n' "$outside_exit"
inside_exit=0
(cd out &&
python3 -B ../scripts/clang-tools/gen_compile_commands.py -a ar \
-o ../inside.json lib/modules.order) ||
inside_exit=$?
printf 'inside_exit=%s\n' "$inside_exit"
test "$outside_exit" -eq 1
test "$inside_exit" -eq 0
python3 -c \
'import json; assert len(json.load(open("inside.json"))) == 1'
)
Kbuild generated lib/test_module.o, lib/test_module.mod,
lib/.test_module.o.cmd and lib/modules.order under out/. Both .mod and
modules.order contain lib/test_module.o.
The old script's invocation from the source directory fails with:
FileNotFoundError: [Errno 2] No such file or directory:
'lib/test_module.mod'
outside_exit=1
The same artifacts work when the old script is invoked from out/:
inside_exit=0
The successful invocation produces one JSON entry for lib/test_module.c.
The patched script succeeds from the source directory and produces an
identical entry. The old script also fails with absolute paths for -d
and modules.order; the patched script succeeds with those paths too.
Only the standalone generator invocation is outside the build directory;
compilation itself runs under out/. This is not a full kernel build or
a boot/runtime test.
> I think you can omit the final sentence here, it does not add much value
> in my opinion.
I'll drop that sentence and clarify the standalone scope in v2.
> Please fix your name, this needs to be a known identity.
I'll use Võ Ngọc Văn for the author and sign-off in v2.
> os.path -> pathlib.Path but it should definitely be done as a separate
> patch.
I'll keep the pathlib conversion as a separate follow-up patch.
Thanks,
Võ Ngọc Văn
On Fri, Oct 16, 2026 at 10:53 AM Van Vo <vngocvan@gmail.com> wrote:
>
> Hi Nathan,
>
> > How was this discovered?
>
> This came from code inspection of standalone use of -d with an explicit
> modules.order argument, followed initially by a constructed reproducer.
> It was not a failure encountered through Kbuild's compile_commands.json
> target.
>
> I have now reproduced it using real Kbuild-generated artifacts from
> 0fcb05b72344f40806cbe810de8872f1698bf053 on Linux/aarch64, with Python
> 3.11.13 and GCC 12.2.0. This was a partial lib/ build with CONFIG_TEST_LKM=m.
>
> The following uses a disposable source tree at that base. The guards
> prevent overwriting an existing output directory or result file:
>
> (
> set -eu
> test ! -e out
> test ! -e outside.json
> test ! -e inside.json
> make O=out -j1 allnoconfig
> bash scripts/config --file out/.config --enable MODULES \
> --enable RUNTIME_TESTING_MENU --module TEST_LKM
> make O=out -j1 olddefconfig
> make O=out -j1 V=1 lib/
>
> outside_exit=0
> python3 -B scripts/clang-tools/gen_compile_commands.py -a ar \
> -d out -o outside.json out/lib/modules.order ||
> outside_exit=$?
> printf 'outside_exit=%s\n' "$outside_exit"
>
> inside_exit=0
> (cd out &&
> python3 -B ../scripts/clang-tools/gen_compile_commands.py -a ar \
> -o ../inside.json lib/modules.order) ||
> inside_exit=$?
> printf 'inside_exit=%s\n' "$inside_exit"
> test "$outside_exit" -eq 1
> test "$inside_exit" -eq 0
> python3 -c \
> 'import json; assert len(json.load(open("inside.json"))) == 1'
> )
>
> Kbuild generated lib/test_module.o, lib/test_module.mod,
> lib/.test_module.o.cmd and lib/modules.order under out/. Both .mod and
> modules.order contain lib/test_module.o.
>
> The old script's invocation from the source directory fails with:
>
> FileNotFoundError: [Errno 2] No such file or directory: 'lib/test_module.mod'
> outside_exit=1
>
> The same artifacts work when the old script is invoked from out/:
>
> inside_exit=0
>
> The successful invocation produces one JSON entry for lib/test_module.c.
> The patched script succeeds from the source directory and produces an
> identical entry. The old script also fails with absolute paths for -d
> and modules.order; the patched script succeeds with those paths too.
>
> Only the standalone generator invocation is outside the build directory;
> compilation itself runs under out/. This is not a full kernel build or
> a boot/runtime test.
>
> > I think you can omit the final sentence here, it does not add much value
> > in my opinion.
>
> I'll drop that sentence and clarify the standalone scope in v2.
>
> > Please fix your name, this needs to be a known identity.
>
> I'll use Võ Ngọc Văn for the author and sign-off in v2.
>
> > os.path -> pathlib.Path but it should definitely be done as a separate
> > patch.
>
> I'll keep the pathlib conversion as a separate follow-up patch.
>
> Thanks,
> Võ Ngọc Văn
>
>
> On Fri, Oct 16, 2026 at 8:21 AM Nathan Chancellor <nathan@kernel.org> wrote:
>>
>> Hi,
>>
>> On Thu, Oct 15, 2026 at 08:57:52PM +0700, vngocvan wrote:
>> > When invoked from outside the build tree with -d and a modules.order
>> > argument, the script opens .mod files relative to the current directory
>> > instead of the specified build directory. This can fail with
>> > FileNotFoundError or collect commands from unrelated artifacts in the
>> > current directory.
>>
>> How was this discovered? This cannot happen within Kbuild because this
>> script is always invoked from within the build folder, so was it a
>> manual invocation or something else?
>>
>> > Pass the build directory to cmdfiles_for_modorder() and resolve both
>> > the .mod files and their associated .cmd files relative to it. Absolute
>> > paths continue to work, and the output location is unchanged.
>>
>> I think you can omit the final sentence here, it does not add much value
>> in my opinion.
>>
>> > Fixes: ecca4fea1ede ("gen_compile_commands: support *.o, *.a, modules.order in positional argument")
>> > Signed-off-by: vngocvan <vngocvan@gmail.com>
>>
>> Please fix your name, this needs to be a known identity.
>>
>> > ---
>> > scripts/clang-tools/gen_compile_commands.py | 9 +++++----
>> > 1 file changed, 5 insertions(+), 4 deletions(-)
>> >
>> > diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
>> > index 8d14b81efd73..149135f85f19 100755
>> > --- a/scripts/clang-tools/gen_compile_commands.py
>> > +++ b/scripts/clang-tools/gen_compile_commands.py
>> > @@ -124,7 +124,7 @@ def cmdfiles_for_a(archive, ar):
>> > yield to_cmdfile(obj)
>> >
>> >
>> > -def cmdfiles_for_modorder(modorder):
>> > +def cmdfiles_for_modorder(modorder, root_directory):
>> > """Generate the iterator of .cmd files associated with the modules.order.
>> >
>> > Parse the given modules.order, and yield every .cmd file used to build the
>> > @@ -132,6 +132,7 @@ def cmdfiles_for_modorder(modorder):
>> >
>> > Args:
>> > modorder: The modules.order file to parse
>> > + root_directory: The directory where the objects were built
>> >
>> > Yields:
>> > The path to every .cmd file found
>> > @@ -142,11 +143,11 @@ def cmdfiles_for_modorder(modorder):
>> > base, ext = os.path.splitext(obj)
>> > if ext != '.o':
>> > sys.exit('{}: module path must end with .o'.format(obj))
>> > - mod = base + '.mod'
>> > + mod = os.path.join(root_directory, base + '.mod')
>> > # Read from *.mod, to get a list of objects that compose the module.
>> > with open(mod) as m:
>> > for mod_line in m:
>> > - yield to_cmdfile(mod_line.rstrip())
>> > + yield os.path.join(root_directory, to_cmdfile(mod_line.rstrip()))
>> >
>> >
>> > def process_line(root_directory, command_prefix, file_path):
>> > @@ -206,7 +207,7 @@ def main():
>> > elif path.endswith('.a'):
>> > cmdfiles = cmdfiles_for_a(path, ar)
>> > elif path.endswith('modules.order'):
>> > - cmdfiles = cmdfiles_for_modorder(path)
>> > + cmdfiles = cmdfiles_for_modorder(path, directory)
>> > else:
>> > sys.exit('{}: unknown file type'.format(path))
>> >
>> >
>> > base-commit: 0fcb05b72344f40806cbe810de8872f1698bf053
>> > --
>> > 2.56.0
>> >
>>
>> --
>> Cheers,
>> Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-10-16 6:25 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-15 13:57 [PATCH] gen_compile_commands: honor the build directory for modules vngocvan
2026-10-15 15:33 ` Nick Desaulniers
2026-10-16 1:12 ` Nathan Chancellor
2026-10-16 1:21 ` Nathan Chancellor
[not found] ` <CAJavoFk+zgySyNps5tuKmntmvuAyev=kKL640TFrfCc-njeWBw@mail.gmail.com>
2026-10-16 4:44 ` Van Vo
2026-10-16 6:25 ` Van Vo
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.