* Re: [PATCH] gen_compile_commands: honor the build directory for modules
[not found] ` <CAJavoFk+zgySyNps5tuKmntmvuAyev=kKL640TFrfCc-njeWBw@mail.gmail.com>
@ 2026-10-16 4:44 ` Van Vo
2026-10-16 6:25 ` Van Vo
1 sibling, 0 replies; 6+ messages in thread
From: Van Vo @ 2026-10-16 4:44 UTC (permalink / raw)
To: llvm
Hi Nathan,
> How was this discovered?
This came from code inspection of standalone use of -d with an explicit
modules.order argument, followed initially by a constructed reproducer.
It was not a failure encountered through Kbuild's compile_commands.json
target.
I have now reproduced it using real Kbuild-generated artifacts from
0fcb05b72344f40806cbe810de8872f1698bf053 on Linux/aarch64, with Python
3.11.13 and GCC 12.2.0. This was a partial lib/ build with CONFIG_TEST_LKM=m.
The following uses a disposable source tree at that base. The guards
prevent overwriting an existing output directory or result file:
(
set -eu
test ! -e out
test ! -e outside.json
test ! -e inside.json
make O=out -j1 allnoconfig
bash scripts/config --file out/.config --enable MODULES \
--enable RUNTIME_TESTING_MENU --module TEST_LKM
make O=out -j1 olddefconfig
make O=out -j1 V=1 lib/
outside_exit=0
python3 -B scripts/clang-tools/gen_compile_commands.py -a ar \
-d out -o outside.json out/lib/modules.order ||
outside_exit=$?
printf 'outside_exit=%s\n' "$outside_exit"
inside_exit=0
(cd out &&
python3 -B ../scripts/clang-tools/gen_compile_commands.py -a ar \
-o ../inside.json lib/modules.order) ||
inside_exit=$?
printf 'inside_exit=%s\n' "$inside_exit"
test "$outside_exit" -eq 1
test "$inside_exit" -eq 0
python3 -c \
'import json; assert len(json.load(open("inside.json"))) == 1'
)
Kbuild generated lib/test_module.o, lib/test_module.mod,
lib/.test_module.o.cmd and lib/modules.order under out/. Both .mod and
modules.order contain lib/test_module.o.
The old script's invocation from the source directory fails with:
FileNotFoundError: [Errno 2] No such file or directory:
'lib/test_module.mod'
outside_exit=1
The same artifacts work when the old script is invoked from out/:
inside_exit=0
The successful invocation produces one JSON entry for lib/test_module.c.
The patched script succeeds from the source directory and produces an
identical entry. The old script also fails with absolute paths for -d
and modules.order; the patched script succeeds with those paths too.
Only the standalone generator invocation is outside the build directory;
compilation itself runs under out/. This is not a full kernel build or
a boot/runtime test.
> I think you can omit the final sentence here, it does not add much value
> in my opinion.
I'll drop that sentence and clarify the standalone scope in v2.
> Please fix your name, this needs to be a known identity.
I'll use Võ Ngọc Văn for the author and sign-off in v2.
> os.path -> pathlib.Path but it should definitely be done as a separate
> patch.
I'll keep the pathlib conversion as a separate follow-up patch.
Thanks,
Võ Ngọc Văn
Thanks & best regards,
V
On Fri, Oct 16, 2026 at 10:53 AM Van Vo <vngocvan@gmail.com> wrote:
>
> Hi Nathan,
>
> > How was this discovered?
>
> This came from code inspection of standalone use of -d with an explicit
> modules.order argument, followed initially by a constructed reproducer.
> It was not a failure encountered through Kbuild's compile_commands.json
> target.
>
> I have now reproduced it using real Kbuild-generated artifacts from
> 0fcb05b72344f40806cbe810de8872f1698bf053 on Linux/aarch64, with Python
> 3.11.13 and GCC 12.2.0. This was a partial lib/ build with CONFIG_TEST_LKM=m.
>
> The following uses a disposable source tree at that base. The guards
> prevent overwriting an existing output directory or result file:
>
> (
> set -eu
> test ! -e out
> test ! -e outside.json
> test ! -e inside.json
> make O=out -j1 allnoconfig
> bash scripts/config --file out/.config --enable MODULES \
> --enable RUNTIME_TESTING_MENU --module TEST_LKM
> make O=out -j1 olddefconfig
> make O=out -j1 V=1 lib/
>
> outside_exit=0
> python3 -B scripts/clang-tools/gen_compile_commands.py -a ar \
> -d out -o outside.json out/lib/modules.order ||
> outside_exit=$?
> printf 'outside_exit=%s\n' "$outside_exit"
>
> inside_exit=0
> (cd out &&
> python3 -B ../scripts/clang-tools/gen_compile_commands.py -a ar \
> -o ../inside.json lib/modules.order) ||
> inside_exit=$?
> printf 'inside_exit=%s\n' "$inside_exit"
> test "$outside_exit" -eq 1
> test "$inside_exit" -eq 0
> python3 -c \
> 'import json; assert len(json.load(open("inside.json"))) == 1'
> )
>
> Kbuild generated lib/test_module.o, lib/test_module.mod,
> lib/.test_module.o.cmd and lib/modules.order under out/. Both .mod and
> modules.order contain lib/test_module.o.
>
> The old script's invocation from the source directory fails with:
>
> FileNotFoundError: [Errno 2] No such file or directory: 'lib/test_module.mod'
> outside_exit=1
>
> The same artifacts work when the old script is invoked from out/:
>
> inside_exit=0
>
> The successful invocation produces one JSON entry for lib/test_module.c.
> The patched script succeeds from the source directory and produces an
> identical entry. The old script also fails with absolute paths for -d
> and modules.order; the patched script succeeds with those paths too.
>
> Only the standalone generator invocation is outside the build directory;
> compilation itself runs under out/. This is not a full kernel build or
> a boot/runtime test.
>
> > I think you can omit the final sentence here, it does not add much value
> > in my opinion.
>
> I'll drop that sentence and clarify the standalone scope in v2.
>
> > Please fix your name, this needs to be a known identity.
>
> I'll use Võ Ngọc Văn for the author and sign-off in v2.
>
> > os.path -> pathlib.Path but it should definitely be done as a separate
> > patch.
>
> I'll keep the pathlib conversion as a separate follow-up patch.
>
> Thanks,
> Võ Ngọc Văn
>
>
> On Fri, Oct 16, 2026 at 8:21 AM Nathan Chancellor <nathan@kernel.org> wrote:
>>
>> Hi,
>>
>> On Thu, Oct 15, 2026 at 08:57:52PM +0700, vngocvan wrote:
>> > When invoked from outside the build tree with -d and a modules.order
>> > argument, the script opens .mod files relative to the current directory
>> > instead of the specified build directory. This can fail with
>> > FileNotFoundError or collect commands from unrelated artifacts in the
>> > current directory.
>>
>> How was this discovered? This cannot happen within Kbuild because this
>> script is always invoked from within the build folder, so was it a
>> manual invocation or something else?
>>
>> > Pass the build directory to cmdfiles_for_modorder() and resolve both
>> > the .mod files and their associated .cmd files relative to it. Absolute
>> > paths continue to work, and the output location is unchanged.
>>
>> I think you can omit the final sentence here, it does not add much value
>> in my opinion.
>>
>> > Fixes: ecca4fea1ede ("gen_compile_commands: support *.o, *.a, modules.order in positional argument")
>> > Signed-off-by: vngocvan <vngocvan@gmail.com>
>>
>> Please fix your name, this needs to be a known identity.
>>
>> > ---
>> > scripts/clang-tools/gen_compile_commands.py | 9 +++++----
>> > 1 file changed, 5 insertions(+), 4 deletions(-)
>> >
>> > diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
>> > index 8d14b81efd73..149135f85f19 100755
>> > --- a/scripts/clang-tools/gen_compile_commands.py
>> > +++ b/scripts/clang-tools/gen_compile_commands.py
>> > @@ -124,7 +124,7 @@ def cmdfiles_for_a(archive, ar):
>> > yield to_cmdfile(obj)
>> >
>> >
>> > -def cmdfiles_for_modorder(modorder):
>> > +def cmdfiles_for_modorder(modorder, root_directory):
>> > """Generate the iterator of .cmd files associated with the modules.order.
>> >
>> > Parse the given modules.order, and yield every .cmd file used to build the
>> > @@ -132,6 +132,7 @@ def cmdfiles_for_modorder(modorder):
>> >
>> > Args:
>> > modorder: The modules.order file to parse
>> > + root_directory: The directory where the objects were built
>> >
>> > Yields:
>> > The path to every .cmd file found
>> > @@ -142,11 +143,11 @@ def cmdfiles_for_modorder(modorder):
>> > base, ext = os.path.splitext(obj)
>> > if ext != '.o':
>> > sys.exit('{}: module path must end with .o'.format(obj))
>> > - mod = base + '.mod'
>> > + mod = os.path.join(root_directory, base + '.mod')
>> > # Read from *.mod, to get a list of objects that compose the module.
>> > with open(mod) as m:
>> > for mod_line in m:
>> > - yield to_cmdfile(mod_line.rstrip())
>> > + yield os.path.join(root_directory, to_cmdfile(mod_line.rstrip()))
>> >
>> >
>> > def process_line(root_directory, command_prefix, file_path):
>> > @@ -206,7 +207,7 @@ def main():
>> > elif path.endswith('.a'):
>> > cmdfiles = cmdfiles_for_a(path, ar)
>> > elif path.endswith('modules.order'):
>> > - cmdfiles = cmdfiles_for_modorder(path)
>> > + cmdfiles = cmdfiles_for_modorder(path, directory)
>> > else:
>> > sys.exit('{}: unknown file type'.format(path))
>> >
>> >
>> > base-commit: 0fcb05b72344f40806cbe810de8872f1698bf053
>> > --
>> > 2.56.0
>> >
>>
>> --
>> Cheers,
>> Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] gen_compile_commands: honor the build directory for modules
[not found] ` <CAJavoFk+zgySyNps5tuKmntmvuAyev=kKL640TFrfCc-njeWBw@mail.gmail.com>
2026-10-16 4:44 ` Van Vo
@ 2026-10-16 6:25 ` Van Vo
1 sibling, 0 replies; 6+ messages in thread
From: Van Vo @ 2026-10-16 6:25 UTC (permalink / raw)
To: Nathan Chancellor
Cc: nsc, ndesaulniers, morbo, justinstitt, masahiroy, llvm,
linux-kbuild, linux-kernel
Hi Nathan,
> How was this discovered?
This came from code inspection of standalone use of -d with an explicit
modules.order argument, followed initially by a constructed reproducer.
It was not a failure encountered through Kbuild's compile_commands.json
target.
I have now reproduced it using real Kbuild-generated artifacts from
0fcb05b72344f40806cbe810de8872f1698bf053 on Linux/aarch64, with Python
3.11.13 and GCC 12.2.0. This was a partial lib/ build with CONFIG_TEST_LKM=m.
The following uses a disposable source tree at that base. The guards
prevent overwriting an existing output directory or result file:
(
set -eu
test ! -e out
test ! -e outside.json
test ! -e inside.json
make O=out -j1 allnoconfig
bash scripts/config --file out/.config --enable MODULES \
--enable RUNTIME_TESTING_MENU --module TEST_LKM
make O=out -j1 olddefconfig
make O=out -j1 V=1 lib/
outside_exit=0
python3 -B scripts/clang-tools/gen_compile_commands.py -a ar \
-d out -o outside.json out/lib/modules.order ||
outside_exit=$?
printf 'outside_exit=%s\n' "$outside_exit"
inside_exit=0
(cd out &&
python3 -B ../scripts/clang-tools/gen_compile_commands.py -a ar \
-o ../inside.json lib/modules.order) ||
inside_exit=$?
printf 'inside_exit=%s\n' "$inside_exit"
test "$outside_exit" -eq 1
test "$inside_exit" -eq 0
python3 -c \
'import json; assert len(json.load(open("inside.json"))) == 1'
)
Kbuild generated lib/test_module.o, lib/test_module.mod,
lib/.test_module.o.cmd and lib/modules.order under out/. Both .mod and
modules.order contain lib/test_module.o.
The old script's invocation from the source directory fails with:
FileNotFoundError: [Errno 2] No such file or directory:
'lib/test_module.mod'
outside_exit=1
The same artifacts work when the old script is invoked from out/:
inside_exit=0
The successful invocation produces one JSON entry for lib/test_module.c.
The patched script succeeds from the source directory and produces an
identical entry. The old script also fails with absolute paths for -d
and modules.order; the patched script succeeds with those paths too.
Only the standalone generator invocation is outside the build directory;
compilation itself runs under out/. This is not a full kernel build or
a boot/runtime test.
> I think you can omit the final sentence here, it does not add much value
> in my opinion.
I'll drop that sentence and clarify the standalone scope in v2.
> Please fix your name, this needs to be a known identity.
I'll use Võ Ngọc Văn for the author and sign-off in v2.
> os.path -> pathlib.Path but it should definitely be done as a separate
> patch.
I'll keep the pathlib conversion as a separate follow-up patch.
Thanks,
Võ Ngọc Văn
On Fri, Oct 16, 2026 at 10:53 AM Van Vo <vngocvan@gmail.com> wrote:
>
> Hi Nathan,
>
> > How was this discovered?
>
> This came from code inspection of standalone use of -d with an explicit
> modules.order argument, followed initially by a constructed reproducer.
> It was not a failure encountered through Kbuild's compile_commands.json
> target.
>
> I have now reproduced it using real Kbuild-generated artifacts from
> 0fcb05b72344f40806cbe810de8872f1698bf053 on Linux/aarch64, with Python
> 3.11.13 and GCC 12.2.0. This was a partial lib/ build with CONFIG_TEST_LKM=m.
>
> The following uses a disposable source tree at that base. The guards
> prevent overwriting an existing output directory or result file:
>
> (
> set -eu
> test ! -e out
> test ! -e outside.json
> test ! -e inside.json
> make O=out -j1 allnoconfig
> bash scripts/config --file out/.config --enable MODULES \
> --enable RUNTIME_TESTING_MENU --module TEST_LKM
> make O=out -j1 olddefconfig
> make O=out -j1 V=1 lib/
>
> outside_exit=0
> python3 -B scripts/clang-tools/gen_compile_commands.py -a ar \
> -d out -o outside.json out/lib/modules.order ||
> outside_exit=$?
> printf 'outside_exit=%s\n' "$outside_exit"
>
> inside_exit=0
> (cd out &&
> python3 -B ../scripts/clang-tools/gen_compile_commands.py -a ar \
> -o ../inside.json lib/modules.order) ||
> inside_exit=$?
> printf 'inside_exit=%s\n' "$inside_exit"
> test "$outside_exit" -eq 1
> test "$inside_exit" -eq 0
> python3 -c \
> 'import json; assert len(json.load(open("inside.json"))) == 1'
> )
>
> Kbuild generated lib/test_module.o, lib/test_module.mod,
> lib/.test_module.o.cmd and lib/modules.order under out/. Both .mod and
> modules.order contain lib/test_module.o.
>
> The old script's invocation from the source directory fails with:
>
> FileNotFoundError: [Errno 2] No such file or directory: 'lib/test_module.mod'
> outside_exit=1
>
> The same artifacts work when the old script is invoked from out/:
>
> inside_exit=0
>
> The successful invocation produces one JSON entry for lib/test_module.c.
> The patched script succeeds from the source directory and produces an
> identical entry. The old script also fails with absolute paths for -d
> and modules.order; the patched script succeeds with those paths too.
>
> Only the standalone generator invocation is outside the build directory;
> compilation itself runs under out/. This is not a full kernel build or
> a boot/runtime test.
>
> > I think you can omit the final sentence here, it does not add much value
> > in my opinion.
>
> I'll drop that sentence and clarify the standalone scope in v2.
>
> > Please fix your name, this needs to be a known identity.
>
> I'll use Võ Ngọc Văn for the author and sign-off in v2.
>
> > os.path -> pathlib.Path but it should definitely be done as a separate
> > patch.
>
> I'll keep the pathlib conversion as a separate follow-up patch.
>
> Thanks,
> Võ Ngọc Văn
>
>
> On Fri, Oct 16, 2026 at 8:21 AM Nathan Chancellor <nathan@kernel.org> wrote:
>>
>> Hi,
>>
>> On Thu, Oct 15, 2026 at 08:57:52PM +0700, vngocvan wrote:
>> > When invoked from outside the build tree with -d and a modules.order
>> > argument, the script opens .mod files relative to the current directory
>> > instead of the specified build directory. This can fail with
>> > FileNotFoundError or collect commands from unrelated artifacts in the
>> > current directory.
>>
>> How was this discovered? This cannot happen within Kbuild because this
>> script is always invoked from within the build folder, so was it a
>> manual invocation or something else?
>>
>> > Pass the build directory to cmdfiles_for_modorder() and resolve both
>> > the .mod files and their associated .cmd files relative to it. Absolute
>> > paths continue to work, and the output location is unchanged.
>>
>> I think you can omit the final sentence here, it does not add much value
>> in my opinion.
>>
>> > Fixes: ecca4fea1ede ("gen_compile_commands: support *.o, *.a, modules.order in positional argument")
>> > Signed-off-by: vngocvan <vngocvan@gmail.com>
>>
>> Please fix your name, this needs to be a known identity.
>>
>> > ---
>> > scripts/clang-tools/gen_compile_commands.py | 9 +++++----
>> > 1 file changed, 5 insertions(+), 4 deletions(-)
>> >
>> > diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
>> > index 8d14b81efd73..149135f85f19 100755
>> > --- a/scripts/clang-tools/gen_compile_commands.py
>> > +++ b/scripts/clang-tools/gen_compile_commands.py
>> > @@ -124,7 +124,7 @@ def cmdfiles_for_a(archive, ar):
>> > yield to_cmdfile(obj)
>> >
>> >
>> > -def cmdfiles_for_modorder(modorder):
>> > +def cmdfiles_for_modorder(modorder, root_directory):
>> > """Generate the iterator of .cmd files associated with the modules.order.
>> >
>> > Parse the given modules.order, and yield every .cmd file used to build the
>> > @@ -132,6 +132,7 @@ def cmdfiles_for_modorder(modorder):
>> >
>> > Args:
>> > modorder: The modules.order file to parse
>> > + root_directory: The directory where the objects were built
>> >
>> > Yields:
>> > The path to every .cmd file found
>> > @@ -142,11 +143,11 @@ def cmdfiles_for_modorder(modorder):
>> > base, ext = os.path.splitext(obj)
>> > if ext != '.o':
>> > sys.exit('{}: module path must end with .o'.format(obj))
>> > - mod = base + '.mod'
>> > + mod = os.path.join(root_directory, base + '.mod')
>> > # Read from *.mod, to get a list of objects that compose the module.
>> > with open(mod) as m:
>> > for mod_line in m:
>> > - yield to_cmdfile(mod_line.rstrip())
>> > + yield os.path.join(root_directory, to_cmdfile(mod_line.rstrip()))
>> >
>> >
>> > def process_line(root_directory, command_prefix, file_path):
>> > @@ -206,7 +207,7 @@ def main():
>> > elif path.endswith('.a'):
>> > cmdfiles = cmdfiles_for_a(path, ar)
>> > elif path.endswith('modules.order'):
>> > - cmdfiles = cmdfiles_for_modorder(path)
>> > + cmdfiles = cmdfiles_for_modorder(path, directory)
>> > else:
>> > sys.exit('{}: unknown file type'.format(path))
>> >
>> >
>> > base-commit: 0fcb05b72344f40806cbe810de8872f1698bf053
>> > --
>> > 2.56.0
>> >
>>
>> --
>> Cheers,
>> Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread