* Intermittent build failure with TRIM_UNUSED_KSYMS and related problems @ 2018-03-05 13:07 Thomas Lindroth 2018-03-12 15:47 ` Masahiro Yamada 0 siblings, 1 reply; 16+ messages in thread From: Thomas Lindroth @ 2018-03-05 13:07 UTC (permalink / raw) To: linux-kbuild I upgraded to 4.14.23 from an earlier kernel series a while ago and turned on some new options. Soon after I noticed one of my virtual machines didn't work right. It's a kvm based VM using vfio for assigning a pci device to the VM. The guest OS could no longer initialize that pci device. After a lot of trial and error I narrowed down the problem to TRIM_UNUSED_KSYMS, which I enabled in the upgrade. If and only if TRIM_UNUSED_KSYMS is enabled the guest gets the error "code 43" which is a generic error code meaning failure to initialize driver in windows based OS. I don't notice any other problems besides that. As I understand it TRIM_UNUSED_KSYMS will build the kernel and modules, then check which symbols are used by the modules and remove all unused EXPORT_SYMBOL_* from the kernel and rebuild it again. When I build the kernel I get a line like "KSYMS symbols: before=1872, after=1871, changed=17" followed by rebuild of a few files. One of the rebuilt files is always drivers/pci/access.c which looks suspicions based on the error I get. EXPORT_SYMBOL_GPL(pci_user_read_config_##size); EXPORT_SYMBOL_GPL(pci_user_write_config_##size); drivers/pci/access.c got these two exports. They stand out because they are macros instead of functions. The only place they are used in the kernel is vfio. All other uses are for accessing pci config space from userspace. I don't think anything in my userspace tries to access pci config space so that could explain why I only see a problem with the vfio based VM. I don't know why TRIM_UNUSED_KSYMS cause problems with vfio but I suspect those macros are related. When testing various config options I would change an option, run make clean followed by make. Turns out make clean doesn't clean include/generated/autoksyms.h. That's why the KSYMS line reported before=1872 instead of before=0. I guessed the kernel build might be confused about which files needed rebuilding so I tried to use a clean build path instead. That did not help to resolve the VM problem but it did result in build failures. The build failure is intermittent and only happens about once every 10 builds. Here is the full "make V=1 j1" output from a failed build: https://gist.githubusercontent.com/anonymous/3ee68c7936248c6f0772bcac8c5b6257/raw/b62df75c5329ec8f3bf556da1145bdf69d5d69f8/gistfile1.txt Here is the same output from a build that succeeds: https://gist.githubusercontent.com/anonymous/85331c68f448781ba64bbaafcd5cb47f/raw/55a86eff8a5e42fe93c26ce1df2aa7c96d1ae803/gistfile1.txt Here is the .config I used: https://gist.githubusercontent.com/anonymous/0d5eceb5ae65ffc5e853fb2664bb3acb/raw/8ca8f1a35468b5aac5b6485a12e71362e8d83ff3/gistfile1.txt Sorry for using gist links but the output is probably too big for the mailing list and regular pastebins. The build failure always looks something like this but the undefined symbols varies: Building modules, stage 2. MODPOST 146 modules ERROR: "__put_user_2" [net/ipv4/netfilter/ip_tables.ko] undefined! ERROR: "__put_user_2" [net/ipv4/netfilter/arp_tables.ko] undefined! ERROR: "__put_user_8" [fs/udf/udf.ko] undefined! ERROR: "__put_user_4" [fs/udf/udf.ko] undefined! ERROR: "__put_user_8" [fs/fat/fat.ko] undefined! ERROR: "__put_user_1" [fs/fat/fat.ko] undefined! ERROR: "__put_user_4" [fs/fat/fat.ko] undefined! ERROR: "__put_user_2" [fs/fat/fat.ko] undefined! ERROR: "__put_user_4" [drivers/net/tap.ko] undefined! ERROR: "__put_user_2" [drivers/net/tap.ko] undefined! ERROR: "__put_user_8" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__put_user_1" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__put_user_4" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__put_user_8" [drivers/input/joydev.ko] undefined! ERROR: "__put_user_1" [drivers/input/joydev.ko] undefined! ERROR: "__put_user_4" [drivers/input/joydev.ko] undefined! ERROR: "__fill_rsb" [arch/x86/kvm/kvm-intel.ko] undefined! make[2]: *** [/usr/src/linux-4.14.23/scripts/Makefile.modpost:92: __modpost] Error 1 make[1]: *** [/usr/src/linux-4.14.23/Makefile:1218: modules] Error 2 make[1]: Leaving directory '/home/cocobo/repository/kernel_build' The only difference between the two pasted build logs is that the failing build doesn't rebuild arch/x86/lib/retpoline.S. I don't know what cause the build failures but it seems like the build system can get confused about which files needs to be rebuild when trimming symbols. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-05 13:07 Intermittent build failure with TRIM_UNUSED_KSYMS and related problems Thomas Lindroth @ 2018-03-12 15:47 ` Masahiro Yamada 2018-03-12 19:15 ` Nicolas Pitre 2018-03-12 20:54 ` Nicolas Pitre 0 siblings, 2 replies; 16+ messages in thread From: Masahiro Yamada @ 2018-03-12 15:47 UTC (permalink / raw) To: Thomas Lindroth; +Cc: Linux Kbuild mailing list, Nicolas Pitre +CC Nicolas Pitre 2018-03-05 22:07 GMT+09:00 Thomas Lindroth <thomas.lindroth@gmail.com>: > I upgraded to 4.14.23 from an earlier kernel series a while ago and turned on some new options. Soon after I noticed one of my virtual machines didn't work right. It's a kvm based VM using vfio for assigning a pci device to the VM. The guest OS could no longer initialize that pci device. After a lot of trial and error I narrowed down the problem to TRIM_UNUSED_KSYMS, which I enabled in the upgrade. > > If and only if TRIM_UNUSED_KSYMS is enabled the guest gets the error "code 43" which is a generic error code meaning failure to initialize driver in windows based OS. I don't notice any other problems besides that. > > As I understand it TRIM_UNUSED_KSYMS will build the kernel and modules, then check which symbols are used by the modules and remove all unused EXPORT_SYMBOL_* from the kernel and rebuild it again. When I build the kernel I get a line like "KSYMS symbols: before=1872, after=1871, changed=17" followed by rebuild of a few files. One of the rebuilt files is always drivers/pci/access.c which looks suspicions based on the error I get. > > EXPORT_SYMBOL_GPL(pci_user_read_config_##size); > EXPORT_SYMBOL_GPL(pci_user_write_config_##size); > drivers/pci/access.c got these two exports. They stand out because they are macros instead of functions. The only place they are used in the kernel is vfio. All other uses are for accessing pci config space from userspace. I don't think anything in my userspace tries to access pci config space so that could explain why I only see a problem with the vfio based VM. I don't know why TRIM_UNUSED_KSYMS cause problems with vfio but I suspect those macros are related. > > When testing various config options I would change an option, run make clean followed by make. Turns out make clean doesn't clean include/generated/autoksyms.h. That's why the KSYMS line reported before=1872 instead of before=0. I guessed the kernel build might be confused about which files needed rebuilding so I tried to use a clean build path instead. That did not help to resolve the VM problem but it did result in build failures. > > The build failure is intermittent and only happens about once every 10 builds. > Here is the full "make V=1 j1" output from a failed build: > https://gist.githubusercontent.com/anonymous/3ee68c7936248c6f0772bcac8c5b6257/raw/b62df75c5329ec8f3bf556da1145bdf69d5d69f8/gistfile1.txt > Here is the same output from a build that succeeds: > https://gist.githubusercontent.com/anonymous/85331c68f448781ba64bbaafcd5cb47f/raw/55a86eff8a5e42fe93c26ce1df2aa7c96d1ae803/gistfile1.txt > Here is the .config I used: > https://gist.githubusercontent.com/anonymous/0d5eceb5ae65ffc5e853fb2664bb3acb/raw/8ca8f1a35468b5aac5b6485a12e71362e8d83ff3/gistfile1.txt > > Sorry for using gist links but the output is probably too big for the mailing list and regular pastebins. > > The build failure always looks something like this but the undefined symbols varies: > Building modules, stage 2. > MODPOST 146 modules > ERROR: "__put_user_2" [net/ipv4/netfilter/ip_tables.ko] undefined! > ERROR: "__put_user_2" [net/ipv4/netfilter/arp_tables.ko] undefined! > ERROR: "__put_user_8" [fs/udf/udf.ko] undefined! > ERROR: "__put_user_4" [fs/udf/udf.ko] undefined! > ERROR: "__put_user_8" [fs/fat/fat.ko] undefined! > ERROR: "__put_user_1" [fs/fat/fat.ko] undefined! > ERROR: "__put_user_4" [fs/fat/fat.ko] undefined! > ERROR: "__put_user_2" [fs/fat/fat.ko] undefined! > ERROR: "__put_user_4" [drivers/net/tap.ko] undefined! > ERROR: "__put_user_2" [drivers/net/tap.ko] undefined! > ERROR: "__put_user_8" [drivers/media/v4l2-core/videodev.ko] undefined! > ERROR: "__put_user_1" [drivers/media/v4l2-core/videodev.ko] undefined! > ERROR: "__put_user_4" [drivers/media/v4l2-core/videodev.ko] undefined! > ERROR: "__put_user_8" [drivers/input/joydev.ko] undefined! > ERROR: "__put_user_1" [drivers/input/joydev.ko] undefined! > ERROR: "__put_user_4" [drivers/input/joydev.ko] undefined! > ERROR: "__fill_rsb" [arch/x86/kvm/kvm-intel.ko] undefined! > make[2]: *** [/usr/src/linux-4.14.23/scripts/Makefile.modpost:92: __modpost] Error 1 > make[1]: *** [/usr/src/linux-4.14.23/Makefile:1218: modules] Error 2 > make[1]: Leaving directory '/home/cocobo/repository/kernel_build' > > The only difference between the two pasted build logs is that the failing build doesn't rebuild arch/x86/lib/retpoline.S. Indeed. retpoline.o is not recompiled in the first log. Is the content of arch/x86/lib/.retpoline.o.cmd between the success case and the failure? > I don't know what cause the build failures but it seems like the build system can get confused about which files needs to be rebuild when trimming symbols. I tried 20+ iterations on v4.14.23 with your .config file, but all succeeded on my machine. I CCed Nicolas Pitre in case this rings his bell. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-12 15:47 ` Masahiro Yamada @ 2018-03-12 19:15 ` Nicolas Pitre 2018-03-13 20:57 ` Thomas Lindroth 2018-03-12 20:54 ` Nicolas Pitre 1 sibling, 1 reply; 16+ messages in thread From: Nicolas Pitre @ 2018-03-12 19:15 UTC (permalink / raw) To: Masahiro Yamada; +Cc: Thomas Lindroth, Linux Kbuild mailing list On Tue, 13 Mar 2018, Masahiro Yamada wrote: > +CC Nicolas Pitre > > > 2018-03-05 22:07 GMT+09:00 Thomas Lindroth <thomas.lindroth@gmail.com>: > > I upgraded to 4.14.23 from an earlier kernel series a while ago and > > turned on some new options. Soon after I noticed one of my virtual > > machines didn't work right. It's a kvm based VM using vfio for > > assigning a pci device to the VM. The guest OS could no longer > > initialize that pci device. After a lot of trial and error I > > narrowed down the problem to TRIM_UNUSED_KSYMS, which I enabled in > > the upgrade. > > > > If and only if TRIM_UNUSED_KSYMS is enabled the guest gets the error > > "code 43" which is a generic error code meaning failure to > > initialize driver in windows based OS. I don't notice any other > > problems besides that. > > > > As I understand it TRIM_UNUSED_KSYMS will build the kernel and > > modules, then check which symbols are used by the modules and remove > > all unused EXPORT_SYMBOL_* from the kernel and rebuild it again. Yes, or it may add symbols that were missing before rebuilding again. Sometimes both, especially if the difference between "before" and "after" numbers doesn't match the "changed" one like your example below. > > When I build the kernel I get a line like "KSYMS symbols: > > before=1872, after=1871, changed=17" followed by rebuild of a few > > files. One of the rebuilt files is always drivers/pci/access.c which > > looks suspicions based on the error I get. What exactly did you do between the build that gave you the above KSYMS line and the build that preceded it? Did you modify your kernel config? > > EXPORT_SYMBOL_GPL(pci_user_read_config_##size); > > EXPORT_SYMBOL_GPL(pci_user_write_config_##size); > > > > drivers/pci/access.c got these two exports. They stand out because > > they are macros instead of functions. If you look in drivers/pci/.access.o.cmd you should see lines like: $(wildcard include/config/ksym/pci/bus/read/config/byte.h) \ $(wildcard include/config/ksym/pci/bus/read/config/word.h) \ $(wildcard include/config/ksym/pci/bus/read/config/dword.h) \ $(wildcard include/config/ksym/pci/bus/write/config/byte.h) \ $(wildcard include/config/ksym/pci/bus/write/config/word.h) \ $(wildcard include/config/ksym/pci/bus/write/config/dword.h) \ Those are the result of the above two EXPORT_SYMBOL lines when the macro containing them are expanded. If you don't have those then drivers/pci/access.c would not be rebuilt when needed in some cases. > > The only place they are used > > in the kernel is vfio. All other uses are for accessing pci config > > space from userspace. I don't think anything in my userspace tries > > to access pci config space so that could explain why I only see a > > problem with the vfio based VM. I don't know why TRIM_UNUSED_KSYMS > > cause problems with vfio but I suspect those macros are related. > > > > When testing various config options I would change an option, run > > make clean followed by make. Turns out make clean doesn't clean > > include/generated/autoksyms.h. That's why the KSYMS line reported > > before=1872 instead of before=0. I guessed the kernel build might be > > confused about which files needed rebuilding so I tried to use a > > clean build path instead. That did not help to resolve the VM > > problem but it did result in build failures. > > > > The build failure is intermittent and only happens about once every 10 builds. So far I can't reproduce any build failure here. > > Here is the full "make V=1 j1" output from a failed build: > > https://gist.githubusercontent.com/anonymous/3ee68c7936248c6f0772bcac8c5b6257/raw/b62df75c5329ec8f3bf556da1145bdf69d5d69f8/gistfile1.txt > > Here is the same output from a build that succeeds: > > https://gist.githubusercontent.com/anonymous/85331c68f448781ba64bbaafcd5cb47f/raw/55a86eff8a5e42fe93c26ce1df2aa7c96d1ae803/gistfile1.txt > > Here is the .config I used: > > https://gist.githubusercontent.com/anonymous/0d5eceb5ae65ffc5e853fb2664bb3acb/raw/8ca8f1a35468b5aac5b6485a12e71362e8d83ff3/gistfile1.txt > > > > Sorry for using gist links but the output is probably too big for the mailing list and regular pastebins. > > > > The build failure always looks something like this but the undefined symbols varies: > > Building modules, stage 2. > > MODPOST 146 modules > > ERROR: "__put_user_2" [net/ipv4/netfilter/ip_tables.ko] undefined! > > ERROR: "__put_user_2" [net/ipv4/netfilter/arp_tables.ko] undefined! > > ERROR: "__put_user_8" [fs/udf/udf.ko] undefined! > > ERROR: "__put_user_4" [fs/udf/udf.ko] undefined! > > ERROR: "__put_user_8" [fs/fat/fat.ko] undefined! > > ERROR: "__put_user_1" [fs/fat/fat.ko] undefined! [...] Clearly something failed to rebuild arch/x86/lib/putuser.S that provides those symbols. When that happens, make sure your arch/x86/lib/.putuser.o.cmd has the following lines: $(wildcard include/config/ksym///put/user/1.h) \ $(wildcard include/config/ksym///put/user/2.h) \ $(wildcard include/config/ksym///put/user/4.h) \ $(wildcard include/config/ksym///put/user/8.h) \ You just have to replace _ with / to search for those symbols. Then look at the file date for those actual files i.e.: $ ls --full-time include/config/ksym///put/user/1.h -rw-rw-r-- 1 nico nico 0 2018-03-12 13:51:07.176136516 -0400 include/config/ksym///put/user/1.h And that ought to be older than the kernel file: $ ls --full-time vmlinux.o -rw-rw-r-- 1 nico nico 547474544 2018-03-12 13:52:35.728357983 -0400 vmlinux.o If so then verify that include/generated/autoksyms.h has the corresponding defines: $ grep __put_user_ include/generated/autoksyms.h #define __KSYM___put_user_1 1 #define __KSYM___put_user_2 1 #define __KSYM___put_user_4 1 #define __KSYM___put_user_8 1 If all the above is true then something really weird is happening. > > The only difference between the two pasted build logs is that the failing build doesn't rebuild arch/x86/lib/retpoline.S. > > Indeed. retpoline.o is not recompiled in the first log. > Is the content of arch/x86/lib/.retpoline.o.cmd between the success > case and the failure? Would be good to see the difference if any. The list of symbols it provides is: $(wildcard include/config/ksym///x86/indirect/thunk/rax.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/rbx.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/rcx.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/rdx.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/rsi.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/rdi.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/rbp.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/r8.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/r9.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/r10.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/r11.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/r12.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/r13.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/r14.h) \ $(wildcard include/config/ksym///x86/indirect/thunk/r15.h) \ $(wildcard include/config/ksym///fill/rsb.h) \ $(wildcard include/config/ksym///clear/rsb.h) \ Do you get any of those in your modpost error list? Also... is the build always failing because of symbols starting with one or more underscores? What filesystem are you using on your build system? Nicolas ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-12 19:15 ` Nicolas Pitre @ 2018-03-13 20:57 ` Thomas Lindroth 2018-03-14 1:43 ` Nicolas Pitre 0 siblings, 1 reply; 16+ messages in thread From: Thomas Lindroth @ 2018-03-13 20:57 UTC (permalink / raw) To: Nicolas Pitre, Masahiro Yamada; +Cc: Linux Kbuild mailing list On 03/12/2018 08:15 PM, Nicolas Pitre wrote: >>> When I build the kernel I get a line like "KSYMS symbols: >>> before=1872, after=1871, changed=17" followed by rebuild of a few >>> files. One of the rebuilt files is always drivers/pci/access.c which >>> looks suspicions based on the error I get. > > What exactly did you do between the build that gave you the above KSYMS > line and the build that preceded it? Did you modify your kernel config? Yes. I basically ran something like this: $ make xconfig (toggled things that looked relevant to my VM problem) $ make clean $ make -j8 # make install # make modules_install Then tested the new kernel and repeat again until I found which option caused my VM problems. >>> EXPORT_SYMBOL_GPL(pci_user_read_config_##size); >>> EXPORT_SYMBOL_GPL(pci_user_write_config_##size); >>> >>> drivers/pci/access.c got these two exports. They stand out because >>> they are macros instead of functions. > > If you look in drivers/pci/.access.o.cmd you should see lines like: > > $(wildcard include/config/ksym/pci/bus/read/config/byte.h) \ > $(wildcard include/config/ksym/pci/bus/read/config/word.h) \ > $(wildcard include/config/ksym/pci/bus/read/config/dword.h) \ > $(wildcard include/config/ksym/pci/bus/write/config/byte.h) \ > $(wildcard include/config/ksym/pci/bus/write/config/word.h) \ > $(wildcard include/config/ksym/pci/bus/write/config/dword.h) \ > > Those are the result of the above two EXPORT_SYMBOL lines when the macro > containing them are expanded. If you don't have those then > drivers/pci/access.c would not be rebuilt when needed in some cases. Here are all the ksym includes from drivers/pci/.access.o.cmd: $(wildcard include/config/ksym/pci/bus/read/config/byte.h) \ $(wildcard include/config/ksym/pci/bus/read/config/word.h) \ $(wildcard include/config/ksym/pci/bus/read/config/dword.h) \ $(wildcard include/config/ksym/pci/bus/write/config/byte.h) \ $(wildcard include/config/ksym/pci/bus/write/config/word.h) \ $(wildcard include/config/ksym/pci/bus/write/config/dword.h) \ $(wildcard include/config/ksym/pci/generic/config/read.h) \ $(wildcard include/config/ksym/pci/generic/config/write.h) \ $(wildcard include/config/ksym/pci/generic/config/read32.h) \ $(wildcard include/config/ksym/pci/generic/config/write32.h) \ $(wildcard include/config/ksym/pci/bus/set/ops.h) \ $(wildcard include/config/ksym/pci/user/read/config/byte.h) \ $(wildcard include/config/ksym/pci/user/read/config/word.h) \ $(wildcard include/config/ksym/pci/user/read/config/dword.h) \ $(wildcard include/config/ksym/pci/user/write/config/byte.h) \ $(wildcard include/config/ksym/pci/user/write/config/word.h) \ $(wildcard include/config/ksym/pci/user/write/config/dword.h) \ $(wildcard include/config/ksym/pci/read/vpd.h) \ $(wildcard include/config/ksym/pci/write/vpd.h) \ $(wildcard include/config/ksym/pci/set/vpd/size.h) \ $(wildcard include/config/ksym/pci/cfg/access/lock.h) \ $(wildcard include/config/ksym/pci/cfg/access/trylock.h) \ $(wildcard include/config/ksym/pci/cfg/access/unlock.h) \ $(wildcard include/config/ksym/pcie/capability/read/word.h) \ $(wildcard include/config/ksym/pcie/capability/read/dword.h) \ $(wildcard include/config/ksym/pcie/capability/write/word.h) \ $(wildcard include/config/ksym/pcie/capability/write/dword.h) \ $(wildcard include/config/ksym/pcie/capability/clear/and/set/word.h) \ $(wildcard include/config/ksym/pcie/capability/clear/and/set/dword.h) \ $(wildcard include/config/ksym/pci/read/config/byte.h) \ $(wildcard include/config/ksym/pci/read/config/word.h) \ $(wildcard include/config/ksym/pci/read/config/dword.h) \ $(wildcard include/config/ksym/pci/write/config/byte.h) \ $(wildcard include/config/ksym/pci/write/config/word.h) \ $(wildcard include/config/ksym/pci/write/config/dword.h) \ I'm guessing you meant these: $(wildcard include/config/ksym/pci/user/read/config/byte.h) \ $(wildcard include/config/ksym/pci/user/read/config/word.h) \ $(wildcard include/config/ksym/pci/user/read/config/dword.h) \ $(wildcard include/config/ksym/pci/user/write/config/byte.h) \ $(wildcard include/config/ksym/pci/user/write/config/word.h) \ $(wildcard include/config/ksym/pci/user/write/config/dword.h) \ They exist but most of the corresponding files like include/config/ksym/pci/user/read/config/byte.h do not. I guess that's normal since vfio is built into the kernel and no other module use those exports. They should get trimmed and include/config/ksym entries don't exist for trimmed symbols? >>> The build failure always looks something like this but the undefined symbols varies: >>> Building modules, stage 2. >>> MODPOST 146 modules >>> ERROR: "__put_user_2" [net/ipv4/netfilter/ip_tables.ko] undefined! >>> ERROR: "__put_user_2" [net/ipv4/netfilter/arp_tables.ko] undefined! >>> ERROR: "__put_user_8" [fs/udf/udf.ko] undefined! >>> ERROR: "__put_user_4" [fs/udf/udf.ko] undefined! >>> ERROR: "__put_user_8" [fs/fat/fat.ko] undefined! >>> ERROR: "__put_user_1" [fs/fat/fat.ko] undefined! > [...] > > Clearly something failed to rebuild arch/x86/lib/putuser.S that provides > those symbols. When that happens, make sure your > arch/x86/lib/.putuser.o.cmd has the following lines: > > $(wildcard include/config/ksym///put/user/1.h) \ > $(wildcard include/config/ksym///put/user/2.h) \ > $(wildcard include/config/ksym///put/user/4.h) \ > $(wildcard include/config/ksym///put/user/8.h) \ > > You just have to replace _ with / to search for those symbols. Then look > at the file date for those actual files i.e.: I don't have the build directory for that failure any more but I ran some more builds until I got a build failure like this: ERROR: "clear_user" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__clear_user" [arch/x86/kvm/kvm.ko] undefined! Those missing exports are part of arch/x86/lib/usercopy_64.c and the relevant ksym parts of arch/x86/lib/.usercopy_64.o.cmd look like this: $(wildcard include/config/ksym///clear/user.h) \ $(wildcard include/config/ksym/clear/user.h) \ $(wildcard include/config/ksym/arch/wb/cache/pmem.h) \ $(wildcard include/config/ksym/memcpy/flushcache.h) \ That's identical to how it looks when the build succeeds. > $ ls --full-time include/config/ksym///put/user/1.h > -rw-rw-r-- 1 nico nico 0 2018-03-12 13:51:07.176136516 -0400 include/config/ksym///put/user/1.h > > And that ought to be older than the kernel file: > > $ ls --full-time vmlinux.o > -rw-rw-r-- 1 nico nico 547474544 2018-03-12 13:52:35.728357983 -0400 vmlinux.o $ ls --full-time linux-fail/include/config/ksym///clear/user.h -rw-r--r-- 1 cocobo cocobo 0 2018-03-13 17:16:53.000000000 +0100 linux-fail/include/config/ksym///clear/user.h $ ls --full-time linux-fail/vmlinux.o -rw-r--r-- 1 cocobo cocobo 545613144 2018-03-13 17:20:57.000000000 +0100 linux-fail/vmlinux.o Looks fine. > If so then verify that include/generated/autoksyms.h has the > corresponding defines: > > $ grep __put_user_ include/generated/autoksyms.h > #define __KSYM___put_user_1 1 > #define __KSYM___put_user_2 1 > #define __KSYM___put_user_4 1 > #define __KSYM___put_user_8 1 linux-fail/include/generated/autoksyms.h: #define __KSYM___clear_user 1 #define __KSYM_clear_user 1 That file's content is also identical to how it looks when the build succeeds. >>> The only difference between the two pasted build logs is that the failing build doesn't rebuild arch/x86/lib/retpoline.S. >> >> Indeed. retpoline.o is not recompiled in the first log. >> Is the content of arch/x86/lib/.retpoline.o.cmd between the success >> case and the failure? > > Would be good to see the difference if any. I don't have that build folder but since arch/x86/lib/.usercopy_64.o.cmd was identical when the build failed with "clear_user" arch/x86/lib/.retpoline.o.cmd was probably identical when it failed with "__put_user_2" > The list of symbols it provides is: > > $(wildcard include/config/ksym///x86/indirect/thunk/rax.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/rbx.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/rcx.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/rdx.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/rsi.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/rdi.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/rbp.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/r8.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/r9.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/r10.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/r11.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/r12.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/r13.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/r14.h) \ > $(wildcard include/config/ksym///x86/indirect/thunk/r15.h) \ > $(wildcard include/config/ksym///fill/rsb.h) \ > $(wildcard include/config/ksym///clear/rsb.h) \ > > Do you get any of those in your modpost error list? > > Also... is the build always failing because of symbols starting with one > or more underscores? The type of failures I've seen so far are: ERROR: "clear_user" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__clear_user" [arch/x86/kvm/kvm.ko] undefined! ERROR: "__fill_rsb" [arch/x86/kvm/kvm-intel.ko] undefined! ERROR: "__put_user_2" [net/ipv4/netfilter/ip_tables.ko] undefined! ERROR: "__put_user_2" [net/ipv4/netfilter/arp_tables.ko] undefined! ERROR: "__put_user_8" [fs/udf/udf.ko] undefined! ERROR: "__put_user_4" [fs/udf/udf.ko] undefined! ERROR: "__put_user_8" [fs/fat/fat.ko] undefined! ERROR: "__put_user_1" [fs/fat/fat.ko] undefined! ERROR: "__put_user_4" [fs/fat/fat.ko] undefined! ERROR: "__put_user_2" [fs/fat/fat.ko] undefined! ERROR: "__put_user_4" [drivers/net/tap.ko] undefined! ERROR: "__put_user_2" [drivers/net/tap.ko] undefined! ERROR: "__put_user_8" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__put_user_1" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__put_user_4" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__put_user_8" [drivers/input/joydev.ko] undefined! ERROR: "__put_user_1" [drivers/input/joydev.ko] undefined! ERROR: "__put_user_4" [drivers/input/joydev.ko] undefined! ERROR: "__fill_rsb" [arch/x86/kvm/kvm-intel.ko] undefined! There might have been others but I didn't save every error message. > What filesystem are you using on your build system? Ext4 but it's an old filesystem with 128B inode size created before mkfs.ext4 defaulted to 256B inode size. Without 256B inodes there is no support for nanosecond resolution file timestamps. Perhaps there is some race that only occurs because of the low resolution timestamps? Here is what dumpe2fs -h looks like: Filesystem volume name: <none> Last mounted on: / Filesystem UUID: aba63f98-03a3-4a45-8f33-806484403b42 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 6553600 Block count: 26214400 Reserved block count: 1310720 Free blocks: 3654940 Free inodes: 4653181 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 1017 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 256 Flex block group size: 16 Filesystem created: Sat Dec 22 23:19:28 2012 Last mount time: Mon Mar 12 09:59:50 2018 Last write time: Fri Feb 9 20:31:41 2018 Mount count: 1426 Maximum mount count: -1 Last checked: Thu Aug 14 09:53:55 2014 Check interval: 0 (<none>) Lifetime writes: 19 TB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode: 8 First orphan inode: 6171922 Default directory hash: half_md4 Directory Hash Seed: 005d4e21-62e4-451e-b0a8-d1c99e6a4fcd Journal backup: inode blocks Journal features: journal_incompat_revoke Journal size: 128M Journal length: 32768 Journal sequence: 0x019af2a1 Journal start: 22507 To debug some more I untared a fresh 4.14.23 kernel, built it with that .config and got the "clear_user" error. Then I did the same thing again in another folder and the build was a success. Then I did some comparison between those build folders. The build success folder got more files since it finished building everything. Here is a list of the unique files: Only in linux-success/arch/x86/crypto: chacha20-x86_64.ko Only in linux-success/arch/x86/crypto: .chacha20-x86_64.ko.cmd Only in linux-success/arch/x86/crypto: chacha20-x86_64.mod.o Only in linux-success/arch/x86/crypto: .chacha20-x86_64.mod.o.cmd Only in linux-success/arch/x86/crypto: crc32-pclmul.ko Only in linux-success/arch/x86/crypto: .crc32-pclmul.ko.cmd Only in linux-success/arch/x86/crypto: crc32-pclmul.mod.o Only in linux-success/arch/x86/crypto: .crc32-pclmul.mod.o.cmd Only in linux-success/arch/x86/crypto: poly1305-x86_64.ko Only in linux-success/arch/x86/crypto: .poly1305-x86_64.ko.cmd Only in linux-success/arch/x86/crypto: poly1305-x86_64.mod.o Only in linux-success/arch/x86/crypto: .poly1305-x86_64.mod.o.cmd Only in linux-success/arch/x86/crypto: sha1-ssse3.ko Only in linux-success/arch/x86/crypto: .sha1-ssse3.ko.cmd Only in linux-success/arch/x86/crypto: sha1-ssse3.mod.o Only in linux-success/arch/x86/crypto: .sha1-ssse3.mod.o.cmd Only in linux-success/arch/x86/crypto: sha256-ssse3.ko Only in linux-success/arch/x86/crypto: .sha256-ssse3.ko.cmd Only in linux-success/arch/x86/crypto: sha256-ssse3.mod.o Only in linux-success/arch/x86/crypto: .sha256-ssse3.mod.o.cmd Only in linux-success/arch/x86/crypto: sha512-ssse3.ko Only in linux-success/arch/x86/crypto: .sha512-ssse3.ko.cmd Only in linux-success/arch/x86/crypto: sha512-ssse3.mod.o Only in linux-success/arch/x86/crypto: .sha512-ssse3.mod.o.cmd Only in linux-success/arch/x86/kvm: kvm-intel.ko Only in linux-success/arch/x86/kvm: .kvm-intel.ko.cmd Only in linux-success/arch/x86/kvm: kvm-intel.mod.o Only in linux-success/arch/x86/kvm: .kvm-intel.mod.o.cmd Only in linux-success/arch/x86/kvm: kvm.ko Only in linux-success/arch/x86/kvm: .kvm.ko.cmd Only in linux-success/arch/x86/kvm: kvm.mod.o Only in linux-success/arch/x86/kvm: .kvm.mod.o.cmd Only in linux-success/crypto: af_alg.ko Only in linux-success/crypto: .af_alg.ko.cmd Only in linux-success/crypto: af_alg.mod.o Only in linux-success/crypto: .af_alg.mod.o.cmd Only in linux-success/crypto: algif_aead.ko Only in linux-success/crypto: .algif_aead.ko.cmd Only in linux-success/crypto: algif_aead.mod.o Only in linux-success/crypto: .algif_aead.mod.o.cmd Only in linux-success/crypto: algif_hash.ko Only in linux-success/crypto: .algif_hash.ko.cmd Only in linux-success/crypto: algif_hash.mod.o Only in linux-success/crypto: .algif_hash.mod.o.cmd Only in linux-success/crypto: algif_rng.ko Only in linux-success/crypto: .algif_rng.ko.cmd Only in linux-success/crypto: algif_rng.mod.o Only in linux-success/crypto: .algif_rng.mod.o.cmd Only in linux-success/crypto: algif_skcipher.ko Only in linux-success/crypto: .algif_skcipher.ko.cmd Only in linux-success/crypto: algif_skcipher.mod.o Only in linux-success/crypto: .algif_skcipher.mod.o.cmd Only in linux-success/crypto: arc4.ko Only in linux-success/crypto: .arc4.ko.cmd Only in linux-success/crypto: arc4.mod.o Only in linux-success/crypto: .arc4.mod.o.cmd Only in linux-success/crypto: ccm.ko Only in linux-success/crypto: .ccm.ko.cmd Only in linux-success/crypto: ccm.mod.o Only in linux-success/crypto: .ccm.mod.o.cmd Only in linux-success/crypto: chacha20_generic.ko Only in linux-success/crypto: .chacha20_generic.ko.cmd Only in linux-success/crypto: chacha20_generic.mod.o Only in linux-success/crypto: .chacha20_generic.mod.o.cmd Only in linux-success/crypto: chacha20poly1305.ko Only in linux-success/crypto: .chacha20poly1305.ko.cmd Only in linux-success/crypto: chacha20poly1305.mod.o Only in linux-success/crypto: .chacha20poly1305.mod.o.cmd Only in linux-success/crypto: cmac.ko Only in linux-success/crypto: .cmac.ko.cmd Only in linux-success/crypto: cmac.mod.o Only in linux-success/crypto: .cmac.mod.o.cmd Only in linux-success/crypto: crc32_generic.ko Only in linux-success/crypto: .crc32_generic.ko.cmd Only in linux-success/crypto: crc32_generic.mod.o Only in linux-success/crypto: .crc32_generic.mod.o.cmd Only in linux-success/crypto: ctr.ko Only in linux-success/crypto: .ctr.ko.cmd Only in linux-success/crypto: ctr.mod.o Only in linux-success/crypto: .ctr.mod.o.cmd Only in linux-success/crypto: cts.ko Only in linux-success/crypto: .cts.ko.cmd Only in linux-success/crypto: cts.mod.o Only in linux-success/crypto: .cts.mod.o.cmd Only in linux-success/crypto: echainiv.ko Only in linux-success/crypto: .echainiv.ko.cmd Only in linux-success/crypto: echainiv.mod.o Only in linux-success/crypto: .echainiv.mod.o.cmd Only in linux-success/crypto: gcm.ko Only in linux-success/crypto: .gcm.ko.cmd Only in linux-success/crypto: gcm.mod.o Only in linux-success/crypto: .gcm.mod.o.cmd Only in linux-success/crypto: ghash-generic.ko Only in linux-success/crypto: .ghash-generic.ko.cmd Only in linux-success/crypto: ghash-generic.mod.o Only in linux-success/crypto: .ghash-generic.mod.o.cmd Only in linux-success/crypto: keywrap.ko Only in linux-success/crypto: .keywrap.ko.cmd Only in linux-success/crypto: keywrap.mod.o Only in linux-success/crypto: .keywrap.mod.o.cmd Only in linux-success/crypto: md4.ko Only in linux-success/crypto: .md4.ko.cmd Only in linux-success/crypto: md4.mod.o Only in linux-success/crypto: .md4.mod.o.cmd Only in linux-success/crypto: md5.ko Only in linux-success/crypto: .md5.ko.cmd Only in linux-success/crypto: md5.mod.o Only in linux-success/crypto: .md5.mod.o.cmd Only in linux-success/crypto: pcbc.ko Only in linux-success/crypto: .pcbc.ko.cmd Only in linux-success/crypto: pcbc.mod.o Only in linux-success/crypto: .pcbc.mod.o.cmd Only in linux-success/crypto: poly1305_generic.ko Only in linux-success/crypto: .poly1305_generic.ko.cmd Only in linux-success/crypto: poly1305_generic.mod.o Only in linux-success/crypto: .poly1305_generic.mod.o.cmd Only in linux-success/crypto: rmd128.ko Only in linux-success/crypto: .rmd128.ko.cmd Only in linux-success/crypto: rmd128.mod.o Only in linux-success/crypto: .rmd128.mod.o.cmd Only in linux-success/crypto: rmd160.ko Only in linux-success/crypto: .rmd160.ko.cmd Only in linux-success/crypto: rmd160.mod.o Only in linux-success/crypto: .rmd160.mod.o.cmd Only in linux-success/crypto: rmd256.ko Only in linux-success/crypto: .rmd256.ko.cmd Only in linux-success/crypto: rmd256.mod.o Only in linux-success/crypto: .rmd256.mod.o.cmd Only in linux-success/crypto: rmd320.ko Only in linux-success/crypto: .rmd320.ko.cmd Only in linux-success/crypto: rmd320.mod.o Only in linux-success/crypto: .rmd320.mod.o.cmd Only in linux-success/crypto: seqiv.ko Only in linux-success/crypto: .seqiv.ko.cmd Only in linux-success/crypto: seqiv.mod.o Only in linux-success/crypto: .seqiv.mod.o.cmd Only in linux-success/crypto: sha1_generic.ko Only in linux-success/crypto: .sha1_generic.ko.cmd Only in linux-success/crypto: sha1_generic.mod.o Only in linux-success/crypto: .sha1_generic.mod.o.cmd Only in linux-success/crypto: vmac.ko Only in linux-success/crypto: .vmac.ko.cmd Only in linux-success/crypto: vmac.mod.o Only in linux-success/crypto: .vmac.mod.o.cmd Only in linux-success/crypto: xcbc.ko Only in linux-success/crypto: .xcbc.ko.cmd Only in linux-success/crypto: xcbc.mod.o Only in linux-success/crypto: .xcbc.mod.o.cmd Only in linux-success/drivers/hid: hid-logitech.ko Only in linux-success/drivers/hid: .hid-logitech.ko.cmd Only in linux-success/drivers/hid: hid-logitech.mod.o Only in linux-success/drivers/hid: .hid-logitech.mod.o.cmd Only in linux-success/drivers/i2c: i2c-mux.ko Only in linux-success/drivers/i2c: .i2c-mux.ko.cmd Only in linux-success/drivers/i2c: i2c-mux.mod.o Only in linux-success/drivers/i2c: .i2c-mux.mod.o.cmd Only in linux-success/drivers/input: joydev.ko Only in linux-success/drivers/input: .joydev.ko.cmd Only in linux-success/drivers/input: joydev.mod.o Only in linux-success/drivers/input: .joydev.mod.o.cmd Only in linux-success/drivers/media/usb/gspca: gspca_main.ko Only in linux-success/drivers/media/usb/gspca: .gspca_main.ko.cmd Only in linux-success/drivers/media/usb/gspca: gspca_main.mod.o Only in linux-success/drivers/media/usb/gspca: .gspca_main.mod.o.cmd Only in linux-success/drivers/media/usb/gspca: gspca_pac7311.ko Only in linux-success/drivers/media/usb/gspca: .gspca_pac7311.ko.cmd Only in linux-success/drivers/media/usb/gspca: gspca_pac7311.mod.o Only in linux-success/drivers/media/usb/gspca: .gspca_pac7311.mod.o.cmd Only in linux-success/drivers/media/v4l2-core: v4l2-common.ko Only in linux-success/drivers/media/v4l2-core: .v4l2-common.ko.cmd Only in linux-success/drivers/media/v4l2-core: v4l2-common.mod.o Only in linux-success/drivers/media/v4l2-core: .v4l2-common.mod.o.cmd Only in linux-success/drivers/media/v4l2-core: v4l2-dv-timings.ko Only in linux-success/drivers/media/v4l2-core: .v4l2-dv-timings.ko.cmd Only in linux-success/drivers/media/v4l2-core: v4l2-dv-timings.mod.o Only in linux-success/drivers/media/v4l2-core: .v4l2-dv-timings.mod.o.cmd Only in linux-success/drivers/media/v4l2-core: videodev.ko Only in linux-success/drivers/media/v4l2-core: .videodev.ko.cmd Only in linux-success/drivers/media/v4l2-core: videodev.mod.o Only in linux-success/drivers/media/v4l2-core: .videodev.mod.o.cmd Only in linux-success/drivers/mfd: lpc_ich.ko Only in linux-success/drivers/mfd: .lpc_ich.ko.cmd Only in linux-success/drivers/mfd: lpc_ich.mod.o Only in linux-success/drivers/mfd: .lpc_ich.mod.o.cmd Only in linux-success/drivers/mfd: lpc_sch.ko Only in linux-success/drivers/mfd: .lpc_sch.ko.cmd Only in linux-success/drivers/mfd: lpc_sch.mod.o Only in linux-success/drivers/mfd: .lpc_sch.mod.o.cmd Only in linux-success/drivers/mfd: mfd-core.ko Only in linux-success/drivers/mfd: .mfd-core.ko.cmd Only in linux-success/drivers/mfd: mfd-core.mod.o Only in linux-success/drivers/mfd: .mfd-core.mod.o.cmd Only in linux-success/drivers/net: macvlan.ko Only in linux-success/drivers/net: .macvlan.ko.cmd Only in linux-success/drivers/net: macvlan.mod.o Only in linux-success/drivers/net: .macvlan.mod.o.cmd Only in linux-success/drivers/net: macvtap.ko Only in linux-success/drivers/net: .macvtap.ko.cmd Only in linux-success/drivers/net: macvtap.mod.o Only in linux-success/drivers/net: .macvtap.mod.o.cmd Only in linux-success/drivers/net: tap.ko Only in linux-success/drivers/net: .tap.ko.cmd Only in linux-success/drivers/net: tap.mod.o Only in linux-success/drivers/net: .tap.mod.o.cmd Only in linux-success/drivers/net: veth.ko Only in linux-success/drivers/net: .veth.ko.cmd Only in linux-success/drivers/net: veth.mod.o Only in linux-success/drivers/net: .veth.mod.o.cmd Only in linux-success/drivers/net/wireless/ath/ath9k: ath9k_common.ko Only in linux-success/drivers/net/wireless/ath/ath9k: .ath9k_common.ko.cmd Only in linux-success/drivers/net/wireless/ath/ath9k: ath9k_common.mod.o Only in linux-success/drivers/net/wireless/ath/ath9k: .ath9k_common.mod.o.cmd Only in linux-success/drivers/net/wireless/ath/ath9k: ath9k_htc.ko Only in linux-success/drivers/net/wireless/ath/ath9k: .ath9k_htc.ko.cmd Only in linux-success/drivers/net/wireless/ath/ath9k: ath9k_htc.mod.o Only in linux-success/drivers/net/wireless/ath/ath9k: .ath9k_htc.mod.o.cmd Only in linux-success/drivers/net/wireless/ath/ath9k: ath9k_hw.ko Only in linux-success/drivers/net/wireless/ath/ath9k: .ath9k_hw.ko.cmd Only in linux-success/drivers/net/wireless/ath/ath9k: ath9k_hw.mod.o Only in linux-success/drivers/net/wireless/ath/ath9k: .ath9k_hw.mod.o.cmd Only in linux-success/drivers/net/wireless/ath: ath.ko Only in linux-success/drivers/net/wireless/ath: .ath.ko.cmd Only in linux-success/drivers/net/wireless/ath: ath.mod.o Only in linux-success/drivers/net/wireless/ath: .ath.mod.o.cmd Only in linux-success/drivers/usb/class: cdc-acm.ko Only in linux-success/drivers/usb/class: .cdc-acm.ko.cmd Only in linux-success/drivers/usb/class: cdc-acm.mod.o Only in linux-success/drivers/usb/class: .cdc-acm.mod.o.cmd Only in linux-success/drivers/usb/storage: uas.ko Only in linux-success/drivers/usb/storage: .uas.ko.cmd Only in linux-success/drivers/usb/storage: uas.mod.o Only in linux-success/drivers/usb/storage: .uas.mod.o.cmd Only in linux-success/drivers/usb/storage: usb-storage.ko Only in linux-success/drivers/usb/storage: .usb-storage.ko.cmd Only in linux-success/drivers/usb/storage: usb-storage.mod.o Only in linux-success/drivers/usb/storage: .usb-storage.mod.o.cmd Only in linux-success/drivers/vhost: vhost.ko Only in linux-success/drivers/vhost: .vhost.ko.cmd Only in linux-success/drivers/vhost: vhost.mod.o Only in linux-success/drivers/vhost: .vhost.mod.o.cmd Only in linux-success/drivers/vhost: vhost_net.ko Only in linux-success/drivers/vhost: .vhost_net.ko.cmd Only in linux-success/drivers/vhost: vhost_net.mod.o Only in linux-success/drivers/vhost: .vhost_net.mod.o.cmd Only in linux-success/drivers/virtio: virtio_balloon.ko Only in linux-success/drivers/virtio: .virtio_balloon.ko.cmd Only in linux-success/drivers/virtio: virtio_balloon.mod.o Only in linux-success/drivers/virtio: .virtio_balloon.mod.o.cmd Only in linux-success/drivers/virtio: virtio.ko Only in linux-success/drivers/virtio: .virtio.ko.cmd Only in linux-success/drivers/virtio: virtio.mod.o Only in linux-success/drivers/virtio: .virtio.mod.o.cmd Only in linux-success/drivers/virtio: virtio_pci.ko Only in linux-success/drivers/virtio: .virtio_pci.ko.cmd Only in linux-success/drivers/virtio: virtio_pci.mod.o Only in linux-success/drivers/virtio: .virtio_pci.mod.o.cmd Only in linux-success/drivers/virtio: virtio_ring.ko Only in linux-success/drivers/virtio: .virtio_ring.ko.cmd Only in linux-success/drivers/virtio: virtio_ring.mod.o Only in linux-success/drivers/virtio: .virtio_ring.mod.o.cmd Only in linux-success/fs/fat: fat.ko Only in linux-success/fs/fat: .fat.ko.cmd Only in linux-success/fs/fat: fat.mod.o Only in linux-success/fs/fat: .fat.mod.o.cmd Only in linux-success/fs/fat: vfat.ko Only in linux-success/fs/fat: .vfat.ko.cmd Only in linux-success/fs/fat: vfat.mod.o Only in linux-success/fs/fat: .vfat.mod.o.cmd Only in linux-success/fs/isofs: isofs.ko Only in linux-success/fs/isofs: .isofs.ko.cmd Only in linux-success/fs/isofs: isofs.mod.o Only in linux-success/fs/isofs: .isofs.mod.o.cmd Only in linux-success/fs/nls: nls_cp437.ko Only in linux-success/fs/nls: .nls_cp437.ko.cmd Only in linux-success/fs/nls: nls_cp437.mod.o Only in linux-success/fs/nls: .nls_cp437.mod.o.cmd Only in linux-success/fs/nls: nls_iso8859-1.ko Only in linux-success/fs/nls: .nls_iso8859-1.ko.cmd Only in linux-success/fs/nls: nls_iso8859-1.mod.o Only in linux-success/fs/nls: .nls_iso8859-1.mod.o.cmd Only in linux-success/fs/ntfs: ntfs.ko Only in linux-success/fs/ntfs: .ntfs.ko.cmd Only in linux-success/fs/ntfs: ntfs.mod.o Only in linux-success/fs/ntfs: .ntfs.mod.o.cmd Only in linux-success/fs/udf: udf.ko Only in linux-success/fs/udf: .udf.ko.cmd Only in linux-success/fs/udf: udf.mod.o Only in linux-success/fs/udf: .udf.mod.o.cmd Only in linux-success/lib: crc-itu-t.ko Only in linux-success/lib: .crc-itu-t.ko.cmd Only in linux-success/lib: crc-itu-t.mod.o Only in linux-success/lib: .crc-itu-t.mod.o.cmd Only in linux-success/lib: ts_bm.ko Only in linux-success/lib: .ts_bm.ko.cmd Only in linux-success/lib: ts_bm.mod.o Only in linux-success/lib: .ts_bm.mod.o.cmd Only in linux-success/lib: ts_fsm.ko Only in linux-success/lib: .ts_fsm.ko.cmd Only in linux-success/lib: ts_fsm.mod.o Only in linux-success/lib: .ts_fsm.mod.o.cmd Only in linux-success/lib: ts_kmp.ko Only in linux-success/lib: .ts_kmp.ko.cmd Only in linux-success/lib: ts_kmp.mod.o Only in linux-success/lib: .ts_kmp.mod.o.cmd Only in linux-success/net/8021q: 8021q.ko Only in linux-success/net/8021q: .8021q.ko.cmd Only in linux-success/net/8021q: 8021q.mod.o Only in linux-success/net/8021q: .8021q.mod.o.cmd Only in linux-success/net/ipv4/netfilter: arptable_filter.ko Only in linux-success/net/ipv4/netfilter: .arptable_filter.ko.cmd Only in linux-success/net/ipv4/netfilter: arptable_filter.mod.o Only in linux-success/net/ipv4/netfilter: .arptable_filter.mod.o.cmd Only in linux-success/net/ipv4/netfilter: arp_tables.ko Only in linux-success/net/ipv4/netfilter: .arp_tables.ko.cmd Only in linux-success/net/ipv4/netfilter: arp_tables.mod.o Only in linux-success/net/ipv4/netfilter: .arp_tables.mod.o.cmd Only in linux-success/net/ipv4/netfilter: arpt_mangle.ko Only in linux-success/net/ipv4/netfilter: .arpt_mangle.ko.cmd Only in linux-success/net/ipv4/netfilter: arpt_mangle.mod.o Only in linux-success/net/ipv4/netfilter: .arpt_mangle.mod.o.cmd Only in linux-success/net/ipv4/netfilter: iptable_filter.ko Only in linux-success/net/ipv4/netfilter: .iptable_filter.ko.cmd Only in linux-success/net/ipv4/netfilter: iptable_filter.mod.o Only in linux-success/net/ipv4/netfilter: .iptable_filter.mod.o.cmd Only in linux-success/net/ipv4/netfilter: iptable_mangle.ko Only in linux-success/net/ipv4/netfilter: .iptable_mangle.ko.cmd Only in linux-success/net/ipv4/netfilter: iptable_mangle.mod.o Only in linux-success/net/ipv4/netfilter: .iptable_mangle.mod.o.cmd Only in linux-success/net/ipv4/netfilter: iptable_nat.ko Only in linux-success/net/ipv4/netfilter: .iptable_nat.ko.cmd Only in linux-success/net/ipv4/netfilter: iptable_nat.mod.o Only in linux-success/net/ipv4/netfilter: .iptable_nat.mod.o.cmd Only in linux-success/net/ipv4/netfilter: iptable_raw.ko Only in linux-success/net/ipv4/netfilter: .iptable_raw.ko.cmd Only in linux-success/net/ipv4/netfilter: iptable_raw.mod.o Only in linux-success/net/ipv4/netfilter: .iptable_raw.mod.o.cmd Only in linux-success/net/ipv4/netfilter: ip_tables.ko Only in linux-success/net/ipv4/netfilter: .ip_tables.ko.cmd Only in linux-success/net/ipv4/netfilter: ip_tables.mod.o Only in linux-success/net/ipv4/netfilter: .ip_tables.mod.o.cmd Only in linux-success/net/ipv4/netfilter: ipt_ah.ko Only in linux-success/net/ipv4/netfilter: .ipt_ah.ko.cmd Only in linux-success/net/ipv4/netfilter: ipt_ah.mod.o Only in linux-success/net/ipv4/netfilter: .ipt_ah.mod.o.cmd Only in linux-success/net/ipv4/netfilter: ipt_CLUSTERIP.ko Only in linux-success/net/ipv4/netfilter: .ipt_CLUSTERIP.ko.cmd Only in linux-success/net/ipv4/netfilter: ipt_CLUSTERIP.mod.o Only in linux-success/net/ipv4/netfilter: .ipt_CLUSTERIP.mod.o.cmd Only in linux-success/net/ipv4/netfilter: ipt_ECN.ko Only in linux-success/net/ipv4/netfilter: .ipt_ECN.ko.cmd Only in linux-success/net/ipv4/netfilter: ipt_ECN.mod.o Only in linux-success/net/ipv4/netfilter: .ipt_ECN.mod.o.cmd Only in linux-success/net/ipv4/netfilter: ipt_MASQUERADE.ko Only in linux-success/net/ipv4/netfilter: .ipt_MASQUERADE.ko.cmd Only in linux-success/net/ipv4/netfilter: ipt_MASQUERADE.mod.o Only in linux-success/net/ipv4/netfilter: .ipt_MASQUERADE.mod.o.cmd Only in linux-success/net/ipv4/netfilter: ipt_REJECT.ko Only in linux-success/net/ipv4/netfilter: .ipt_REJECT.ko.cmd Only in linux-success/net/ipv4/netfilter: ipt_REJECT.mod.o Only in linux-success/net/ipv4/netfilter: .ipt_REJECT.mod.o.cmd Only in linux-success/net/ipv4/netfilter: nf_dup_ipv4.ko Only in linux-success/net/ipv4/netfilter: .nf_dup_ipv4.ko.cmd Only in linux-success/net/ipv4/netfilter: nf_dup_ipv4.mod.o Only in linux-success/net/ipv4/netfilter: .nf_dup_ipv4.mod.o.cmd Only in linux-success/net/ipv4/netfilter: nf_nat_ipv4.ko Only in linux-success/net/ipv4/netfilter: .nf_nat_ipv4.ko.cmd Only in linux-success/net/ipv4/netfilter: nf_nat_ipv4.mod.o Only in linux-success/net/ipv4/netfilter: .nf_nat_ipv4.mod.o.cmd Only in linux-success/net/ipv4/netfilter: nf_nat_masquerade_ipv4.ko Only in linux-success/net/ipv4/netfilter: .nf_nat_masquerade_ipv4.ko.cmd Only in linux-success/net/ipv4/netfilter: nf_nat_masquerade_ipv4.mod.o Only in linux-success/net/ipv4/netfilter: .nf_nat_masquerade_ipv4.mod.o.cmd Only in linux-success/net/ipv4/netfilter: nf_reject_ipv4.ko Only in linux-success/net/ipv4/netfilter: .nf_reject_ipv4.ko.cmd Only in linux-success/net/ipv4/netfilter: nf_reject_ipv4.mod.o Only in linux-success/net/ipv4/netfilter: .nf_reject_ipv4.mod.o.cmd Only in linux-success/net/mac80211: mac80211.ko Only in linux-success/net/mac80211: .mac80211.ko.cmd Only in linux-success/net/mac80211: mac80211.mod.o Only in linux-success/net/mac80211: .mac80211.mod.o.cmd Only in linux-success/net/netfilter: nf_nat_ftp.ko Only in linux-success/net/netfilter: .nf_nat_ftp.ko.cmd Only in linux-success/net/netfilter: nf_nat_ftp.mod.o Only in linux-success/net/netfilter: .nf_nat_ftp.mod.o.cmd Only in linux-success/net/netfilter: nf_nat_irc.ko Only in linux-success/net/netfilter: .nf_nat_irc.ko.cmd Only in linux-success/net/netfilter: nf_nat_irc.mod.o Only in linux-success/net/netfilter: .nf_nat_irc.mod.o.cmd Only in linux-success/net/netfilter: nf_nat.ko Only in linux-success/net/netfilter: .nf_nat.ko.cmd Only in linux-success/net/netfilter: nf_nat.mod.o Only in linux-success/net/netfilter: .nf_nat.mod.o.cmd Only in linux-success/net/netfilter: nf_nat_redirect.ko Only in linux-success/net/netfilter: .nf_nat_redirect.ko.cmd Only in linux-success/net/netfilter: nf_nat_redirect.mod.o Only in linux-success/net/netfilter: .nf_nat_redirect.mod.o.cmd Only in linux-success/net/netfilter: nfnetlink_acct.ko Only in linux-success/net/netfilter: .nfnetlink_acct.ko.cmd Only in linux-success/net/netfilter: nfnetlink_acct.mod.o Only in linux-success/net/netfilter: .nfnetlink_acct.mod.o.cmd Only in linux-success/net/netfilter: nfnetlink_log.ko Only in linux-success/net/netfilter: .nfnetlink_log.ko.cmd Only in linux-success/net/netfilter: nfnetlink_log.mod.o Only in linux-success/net/netfilter: .nfnetlink_log.mod.o.cmd Only in linux-success/net/netfilter: nfnetlink_queue.ko Only in linux-success/net/netfilter: .nfnetlink_queue.ko.cmd Only in linux-success/net/netfilter: nfnetlink_queue.mod.o Only in linux-success/net/netfilter: .nfnetlink_queue.mod.o.cmd Only in linux-success/net/netfilter: xt_CLASSIFY.ko Only in linux-success/net/netfilter: .xt_CLASSIFY.ko.cmd Only in linux-success/net/netfilter: xt_CLASSIFY.mod.o Only in linux-success/net/netfilter: .xt_CLASSIFY.mod.o.cmd Only in linux-success/net/netfilter: xt_cluster.ko Only in linux-success/net/netfilter: .xt_cluster.ko.cmd Only in linux-success/net/netfilter: xt_cluster.mod.o Only in linux-success/net/netfilter: .xt_cluster.mod.o.cmd Only in linux-success/net/netfilter: xt_comment.ko Only in linux-success/net/netfilter: .xt_comment.ko.cmd Only in linux-success/net/netfilter: xt_comment.mod.o Only in linux-success/net/netfilter: .xt_comment.mod.o.cmd Only in linux-success/net/netfilter: xt_connbytes.ko Only in linux-success/net/netfilter: .xt_connbytes.ko.cmd Only in linux-success/net/netfilter: xt_connbytes.mod.o Only in linux-success/net/netfilter: .xt_connbytes.mod.o.cmd Only in linux-success/net/netfilter: xt_connlimit.ko Only in linux-success/net/netfilter: .xt_connlimit.ko.cmd Only in linux-success/net/netfilter: xt_connlimit.mod.o Only in linux-success/net/netfilter: .xt_connlimit.mod.o.cmd Only in linux-success/net/netfilter: xt_connmark.ko Only in linux-success/net/netfilter: .xt_connmark.ko.cmd Only in linux-success/net/netfilter: xt_connmark.mod.o Only in linux-success/net/netfilter: .xt_connmark.mod.o.cmd Only in linux-success/net/netfilter: xt_conntrack.ko Only in linux-success/net/netfilter: .xt_conntrack.ko.cmd Only in linux-success/net/netfilter: xt_conntrack.mod.o Only in linux-success/net/netfilter: .xt_conntrack.mod.o.cmd Only in linux-success/net/netfilter: xt_cpu.ko Only in linux-success/net/netfilter: .xt_cpu.ko.cmd Only in linux-success/net/netfilter: xt_cpu.mod.o Only in linux-success/net/netfilter: .xt_cpu.mod.o.cmd Only in linux-success/net/netfilter: xt_ecn.ko Only in linux-success/net/netfilter: .xt_ecn.ko.cmd Only in linux-success/net/netfilter: xt_ecn.mod.o Only in linux-success/net/netfilter: .xt_ecn.mod.o.cmd Only in linux-success/net/netfilter: xt_esp.ko Only in linux-success/net/netfilter: .xt_esp.ko.cmd Only in linux-success/net/netfilter: xt_esp.mod.o Only in linux-success/net/netfilter: .xt_esp.mod.o.cmd Only in linux-success/net/netfilter: xt_hashlimit.ko Only in linux-success/net/netfilter: .xt_hashlimit.ko.cmd Only in linux-success/net/netfilter: xt_hashlimit.mod.o Only in linux-success/net/netfilter: .xt_hashlimit.mod.o.cmd Only in linux-success/net/netfilter: xt_helper.ko Only in linux-success/net/netfilter: .xt_helper.ko.cmd Only in linux-success/net/netfilter: xt_helper.mod.o Only in linux-success/net/netfilter: .xt_helper.mod.o.cmd Only in linux-success/net/netfilter: xt_hl.ko Only in linux-success/net/netfilter: xt_HL.ko Only in linux-success/net/netfilter: .xt_hl.ko.cmd Only in linux-success/net/netfilter: .xt_HL.ko.cmd Only in linux-success/net/netfilter: xt_hl.mod.o Only in linux-success/net/netfilter: xt_HL.mod.o Only in linux-success/net/netfilter: .xt_hl.mod.o.cmd Only in linux-success/net/netfilter: .xt_HL.mod.o.cmd Only in linux-success/net/netfilter: xt_IDLETIMER.ko Only in linux-success/net/netfilter: .xt_IDLETIMER.ko.cmd Only in linux-success/net/netfilter: xt_IDLETIMER.mod.o Only in linux-success/net/netfilter: .xt_IDLETIMER.mod.o.cmd Only in linux-success/net/netfilter: xt_iprange.ko Only in linux-success/net/netfilter: .xt_iprange.ko.cmd Only in linux-success/net/netfilter: xt_iprange.mod.o Only in linux-success/net/netfilter: .xt_iprange.mod.o.cmd Only in linux-success/net/netfilter: xt_length.ko Only in linux-success/net/netfilter: .xt_length.ko.cmd Only in linux-success/net/netfilter: xt_length.mod.o Only in linux-success/net/netfilter: .xt_length.mod.o.cmd Only in linux-success/net/netfilter: xt_limit.ko Only in linux-success/net/netfilter: .xt_limit.ko.cmd Only in linux-success/net/netfilter: xt_limit.mod.o Only in linux-success/net/netfilter: .xt_limit.mod.o.cmd Only in linux-success/net/netfilter: xt_mac.ko Only in linux-success/net/netfilter: .xt_mac.ko.cmd Only in linux-success/net/netfilter: xt_mac.mod.o Only in linux-success/net/netfilter: .xt_mac.mod.o.cmd Only in linux-success/net/netfilter: xt_mark.ko Only in linux-success/net/netfilter: .xt_mark.ko.cmd Only in linux-success/net/netfilter: xt_mark.mod.o Only in linux-success/net/netfilter: .xt_mark.mod.o.cmd Only in linux-success/net/netfilter: xt_multiport.ko Only in linux-success/net/netfilter: .xt_multiport.ko.cmd Only in linux-success/net/netfilter: xt_multiport.mod.o Only in linux-success/net/netfilter: .xt_multiport.mod.o.cmd Only in linux-success/net/netfilter: xt_nat.ko Only in linux-success/net/netfilter: .xt_nat.ko.cmd Only in linux-success/net/netfilter: xt_nat.mod.o Only in linux-success/net/netfilter: .xt_nat.mod.o.cmd Only in linux-success/net/netfilter: xt_NETMAP.ko Only in linux-success/net/netfilter: .xt_NETMAP.ko.cmd Only in linux-success/net/netfilter: xt_NETMAP.mod.o Only in linux-success/net/netfilter: .xt_NETMAP.mod.o.cmd Only in linux-success/net/netfilter: xt_nfacct.ko Only in linux-success/net/netfilter: .xt_nfacct.ko.cmd Only in linux-success/net/netfilter: xt_nfacct.mod.o Only in linux-success/net/netfilter: .xt_nfacct.mod.o.cmd Only in linux-success/net/netfilter: xt_NFLOG.ko Only in linux-success/net/netfilter: .xt_NFLOG.ko.cmd Only in linux-success/net/netfilter: xt_NFLOG.mod.o Only in linux-success/net/netfilter: .xt_NFLOG.mod.o.cmd Only in linux-success/net/netfilter: xt_NFQUEUE.ko Only in linux-success/net/netfilter: .xt_NFQUEUE.ko.cmd Only in linux-success/net/netfilter: xt_NFQUEUE.mod.o Only in linux-success/net/netfilter: .xt_NFQUEUE.mod.o.cmd Only in linux-success/net/netfilter: xt_osf.ko Only in linux-success/net/netfilter: .xt_osf.ko.cmd Only in linux-success/net/netfilter: xt_osf.mod.o Only in linux-success/net/netfilter: .xt_osf.mod.o.cmd Only in linux-success/net/netfilter: xt_owner.ko Only in linux-success/net/netfilter: .xt_owner.ko.cmd Only in linux-success/net/netfilter: xt_owner.mod.o Only in linux-success/net/netfilter: .xt_owner.mod.o.cmd Only in linux-success/net/netfilter: xt_pkttype.ko Only in linux-success/net/netfilter: .xt_pkttype.ko.cmd Only in linux-success/net/netfilter: xt_pkttype.mod.o Only in linux-success/net/netfilter: .xt_pkttype.mod.o.cmd Only in linux-success/net/netfilter: xt_quota.ko Only in linux-success/net/netfilter: .xt_quota.ko.cmd Only in linux-success/net/netfilter: xt_quota.mod.o Only in linux-success/net/netfilter: .xt_quota.mod.o.cmd Only in linux-success/net/netfilter: xt_rateest.ko Only in linux-success/net/netfilter: xt_RATEEST.ko Only in linux-success/net/netfilter: .xt_rateest.ko.cmd Only in linux-success/net/netfilter: .xt_RATEEST.ko.cmd Only in linux-success/net/netfilter: xt_rateest.mod.o Only in linux-success/net/netfilter: xt_RATEEST.mod.o Only in linux-success/net/netfilter: .xt_rateest.mod.o.cmd Only in linux-success/net/netfilter: .xt_RATEEST.mod.o.cmd Only in linux-success/net/netfilter: xt_realm.ko Only in linux-success/net/netfilter: .xt_realm.ko.cmd Only in linux-success/net/netfilter: xt_realm.mod.o Only in linux-success/net/netfilter: .xt_realm.mod.o.cmd Only in linux-success/net/netfilter: xt_recent.ko Only in linux-success/net/netfilter: .xt_recent.ko.cmd Only in linux-success/net/netfilter: xt_recent.mod.o Only in linux-success/net/netfilter: .xt_recent.mod.o.cmd Only in linux-success/net/netfilter: xt_REDIRECT.ko Only in linux-success/net/netfilter: .xt_REDIRECT.ko.cmd Only in linux-success/net/netfilter: xt_REDIRECT.mod.o Only in linux-success/net/netfilter: .xt_REDIRECT.mod.o.cmd Only in linux-success/net/netfilter: xt_state.ko Only in linux-success/net/netfilter: .xt_state.ko.cmd Only in linux-success/net/netfilter: xt_state.mod.o Only in linux-success/net/netfilter: .xt_state.mod.o.cmd Only in linux-success/net/netfilter: xt_statistic.ko Only in linux-success/net/netfilter: .xt_statistic.ko.cmd Only in linux-success/net/netfilter: xt_statistic.mod.o Only in linux-success/net/netfilter: .xt_statistic.mod.o.cmd Only in linux-success/net/netfilter: xt_string.ko Only in linux-success/net/netfilter: .xt_string.ko.cmd Only in linux-success/net/netfilter: xt_string.mod.o Only in linux-success/net/netfilter: .xt_string.mod.o.cmd Only in linux-success/net/netfilter: xt_tcpmss.ko Only in linux-success/net/netfilter: xt_TCPMSS.ko Only in linux-success/net/netfilter: .xt_tcpmss.ko.cmd Only in linux-success/net/netfilter: .xt_TCPMSS.ko.cmd Only in linux-success/net/netfilter: xt_tcpmss.mod.o Only in linux-success/net/netfilter: xt_TCPMSS.mod.o Only in linux-success/net/netfilter: .xt_tcpmss.mod.o.cmd Only in linux-success/net/netfilter: .xt_TCPMSS.mod.o.cmd Only in linux-success/net/netfilter: xt_TEE.ko Only in linux-success/net/netfilter: .xt_TEE.ko.cmd Only in linux-success/net/netfilter: xt_TEE.mod.o Only in linux-success/net/netfilter: .xt_TEE.mod.o.cmd Only in linux-success/net/netfilter: xt_time.ko Only in linux-success/net/netfilter: .xt_time.ko.cmd Only in linux-success/net/netfilter: xt_time.mod.o Only in linux-success/net/netfilter: .xt_time.mod.o.cmd Only in linux-success/net/netfilter: xt_u32.ko Only in linux-success/net/netfilter: .xt_u32.ko.cmd Only in linux-success/net/netfilter: xt_u32.mod.o Only in linux-success/net/netfilter: .xt_u32.mod.o.cmd Only in linux-success/net/wireless: cfg80211.ko Only in linux-success/net/wireless: .cfg80211.ko.cmd Only in linux-success/net/wireless: cfg80211.mod.o Only in linux-success/net/wireless: .cfg80211.mod.o.cmd It looks like the only missing files in the build failure folder are modules that didn't build because of the error. Doing a full "diff -urp linux-success linux-fail" shows a lot of changed binary files as expected. There are some irrelevant changes to files like include/generated/compile.h, arch/x86/boot/zoffset.h arch/x86/boot/compressed/piggy.S, certs/signing_key.pem, etc. The only relevant change I can spot is: diff -u linux-success/Module.symvers linux-fail/Module.symvers -0x00000000 __clear_user vmlinux EXPORT_SYMBOL -0x00000000 clear_user vmlinux EXPORT_SYMBOL diff -u <(cut -d " " -f 2- ./linux-success/System.map) <(cut -d " " -f 2- ./linux-fail/System.map) -r __ksymtab___clear_user -r __ksymtab_clear_user -r __kstrtab_clear_user -r __kstrtab___clear_user diff -u <(cut -d " " -f 2- ./linux-success/.tmp_System.map) <(cut -d " " -f 2- ./linux-fail/.tmp_System.map) -r __ksymtab___clear_user -r __ksymtab_clear_user -r __kstrtab_clear_user -r __kstrtab___clear_user I had to cut away the first field from the system.map files because it always differ. .tmp_kallsyms[12].S also differ a lot but I don't know what to make of those files. There is no difference to any file in include/config/ksym or any *.cmd file except some *.cmd files related to modules only exist in the success folder. I patched the kernel with "kbuild: show the list of exported symbols with V=1 for TRIM_UNUSED_KSYMS" and ran the build with V=1 until I got a failure. Here is the full output: success: https://gist.githubusercontent.com/anonymous/86325ce634568f1e1d31c23b5b9df295/raw/a897c77f34499a990505057f0221b4a935db4a16/gistfile1.txt fail: https://gist.githubusercontent.com/anonymous/279a7925f1a8cd05dcad78dca0720688/raw/9e407b262fdac284b15ae07f2d7a7824051d56cc/gistfile1.txt The error was: ERROR: "clear_user" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__clear_user" [arch/x86/kvm/kvm.ko] undefined! and the relevant part of the diff of the build: --- kernel_build_debug_patch_success 2018-03-13 14:47:24.000000000 +0100 +++ kernel_build_debug_patch_fail 2018-03-13 15:12:33.000000000 +0100 @@ -45298,13 +45298,6 @@ KSYM___x86_indirect_thunk_r15 KSYM___fill_rsb KSYM___clear_rsb - gcc -Wp,-MD,arch/x86/lib/.usercopy_64.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/6.4.0/include -I./arch/x86/include -I./arch/x86/include/generated -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -march=core2 -mno-red-zone -mcmodel=kernel -funit-at-a-time -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -Wno-frame-address -O2 --param=allow-store-data-races=0 -DCC_HAVE_ASM_GOTO -fplugin=./scripts/gcc-plugins/latent_entropy_plugin.so -fplugin=./scripts/gcc-plugins/structleak_plugin.so -DLATENT_ENTROPY_PLUGIN -fplugin-arg-structleak_plugin-byref-all -DSTRUCTLEAK_PLUGIN -Wframe-larger-than=1024 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -pg -mfentry -DCC_USING_FENTRY -fno-inline-functions-called-once -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fno-stack-check -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -DKBUILD_BASENAME='"usercopy_64"' -DKBUILD_MODNAME='"usercopy_64"' -c -o arch/x86/lib/usercopy_64.o arch/x86/lib/usercopy_64.c -KSYM___clear_user -KSYM_clear_user -KSYM_arch_wb_cache_pmem -KSYM_memcpy_flushcache - ./tools/objtool/objtool orc generate --no-fp "arch/x86/lib/usercopy_64.o"; - if [ "-pg" = "-pg" ]; then if [ arch/x86/lib/usercopy_64.o != "scripts/mod/empty.o" ]; then ./scripts/recordmcount "arch/x86/lib/usercopy_64.o"; fi; fi; rm -f arch/x86/lib/lib.a; ar rcsTPD arch/x86/lib/lib.a arch/x86/lib/clear_page_64.o arch/x86/lib/cmdline.o arch/x86/lib/cmpxchg16b_emu.o arch/x86/lib/copy_page_64.o arch/x86/lib/copy_user_64.o arch/x86/lib/cpu.o arch/x86/lib/csum-copy_64.o arch/x86/lib/csum-partial_64.o arch/x86/lib/csum-wrappers_64.o arch/x86/lib/delay.o arch/x86/lib/getuser.o arch/x86/lib/inat.o arch/x86/lib/insn.o arch/x86/lib/kaslr.o arch/x86/lib/memcpy_64.o arch/x86/lib/memmove_64.o arch/x86/lib/memset_64.o arch/x86/lib/misc.o arch/x86/lib/putuser.o arch/x86/lib/retpoline.o arch/x86/lib/rwsem.o arch/x86/lib/usercopy.o arch/x86/lib/usercopy_64.o objdump -h arch/x86/lib/lib.a | sed -ne '/___ksymtab/s/.*+\([^ ]*\).*/EXTERN(\1)/p' >arch/x86/lib/.lib-ksyms.o.lds; rm -f arch/x86/lib/.lib_exports.o; echo | gcc -Wp,-MD,arch/x86/lib/.lib-ksyms.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/6.4.0/include -I./arch/x86/include -I./arch/x86/include/generated -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -D__ASSEMBLY__ -fno-PIE -m64 -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -DCC_HAVE_ASM_GOTO -Wa,-gdwarf-2 -mfentry -DCC_USING_FENTRY -c -o arch/x86/lib/.lib_exports.o -x assembler -; ld -m elf_x86_64 -r -o arch/x86/lib/lib-ksyms.o -T arch/x86/lib/.lib-ksyms.o.lds arch/x86/lib/.lib_exports.o; rm arch/x86/lib/.lib_exports.o arch/x86/lib/.lib-ksyms.o.lds rm -f arch/x86/lib/built-in.o; ar rcSTPD arch/x86/lib/built-in.o arch/x86/lib/msr-smp.o arch/x86/lib/cache-smp.o arch/x86/lib/msr.o arch/x86/lib/msr-reg.o arch/x86/lib/msr-reg-export.o arch/x86/lib/hweight.o arch/x86/lib/iomap_copy_64.o arch/x86/lib/lib-ksyms.o ; scripts/mod/modpost arch/x86/lib/built-in.o ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-13 20:57 ` Thomas Lindroth @ 2018-03-14 1:43 ` Nicolas Pitre 2018-03-15 18:48 ` Thomas Lindroth 0 siblings, 1 reply; 16+ messages in thread From: Nicolas Pitre @ 2018-03-14 1:43 UTC (permalink / raw) To: Thomas Lindroth; +Cc: Masahiro Yamada, Linux Kbuild mailing list [-- Attachment #1: Type: text/plain, Size: 12441 bytes --] On Tue, 13 Mar 2018, Thomas Lindroth wrote: > I'm guessing you meant these: > $(wildcard include/config/ksym/pci/user/read/config/byte.h) \ > $(wildcard include/config/ksym/pci/user/read/config/word.h) \ > $(wildcard include/config/ksym/pci/user/read/config/dword.h) \ > $(wildcard include/config/ksym/pci/user/write/config/byte.h) \ > $(wildcard include/config/ksym/pci/user/write/config/word.h) \ > $(wildcard include/config/ksym/pci/user/write/config/dword.h) \ > > They exist but most of the corresponding files like > include/config/ksym/pci/user/read/config/byte.h do not. > I guess that's normal since vfio is built into the kernel and no other > module use those exports. They should get trimmed and include/config/ksym > entries don't exist for trimmed symbols? They could exist if those symbols neededto be exported in a previous build. In fact, what those files do is only holding a timestamp. When scripts/adjust_autoksyms.sh determines that some symbol changes state (newly needed or no longer needed) then the corresponding file is touched to update its timestamp to trigger the rebuild of affected source files whose dependency list contains that timestamp files. What those .o.cmd files contain is the actual dependency list. > I don't have the build directory for that failure any more but I ran some more builds > until I got a build failure like this: > ERROR: "clear_user" [drivers/media/v4l2-core/videodev.ko] undefined! > ERROR: "__clear_user" [arch/x86/kvm/kvm.ko] undefined! > > Those missing exports are part of arch/x86/lib/usercopy_64.c and the relevant ksym parts > of arch/x86/lib/.usercopy_64.o.cmd look like this: > $(wildcard include/config/ksym///clear/user.h) \ > $(wildcard include/config/ksym/clear/user.h) \ > $(wildcard include/config/ksym/arch/wb/cache/pmem.h) \ > $(wildcard include/config/ksym/memcpy/flushcache.h) \ So the dependency list is fine. > That's identical to how it looks when the build succeeds. Therefore the problem isn't there. > $ ls --full-time linux-fail/include/config/ksym///clear/user.h > -rw-r--r-- 1 cocobo cocobo 0 2018-03-13 17:16:53.000000000 +0100 linux-fail/include/config/ksym///clear/user.h > > $ ls --full-time linux-fail/vmlinux.o > -rw-r--r-- 1 cocobo cocobo 545613144 2018-03-13 17:20:57.000000000 +0100 linux-fail/vmlinux.o > > Looks fine. > > > If so then verify that include/generated/autoksyms.h has the > > corresponding defines: > > > > $ grep __put_user_ include/generated/autoksyms.h > > #define __KSYM___put_user_1 1 > > #define __KSYM___put_user_2 1 > > #define __KSYM___put_user_4 1 > > #define __KSYM___put_user_8 1 > > linux-fail/include/generated/autoksyms.h: > #define __KSYM___clear_user 1 > #define __KSYM_clear_user 1 > > That file's content is also identical to how it looks when the build succeeds. Now, what's the date of include/generated/autoksyms.h compared to arch/x86/lib/usercopy_64.o? If include/generated/autoksyms.h is older than arch/x86/lib/usercopy_64.o then the presence of __KSYM_clear_user in the former should have instanciated the corresponding EXPORT_SYMBOL() in the later. If it is the other way around then you should compare the time for arch/x86/lib/usercopy_64.o against include/config/ksym///clear/user.h. If the later is newer than the former then something failed to notice that usercopy_64.o wasn't up to date, in which case the makefile or make itself might need some investigation. If not then we'd have to look in the full build log to figure out if __KSYM_clear_user existed in the previous version of include/generated/autoksyms.h before it was refreshed by adjust_autoksyms.sh. If it was then we're back to if #1 above. If not then the timestamp for include/config/ksym///clear/user.h hasn't been updated as it should. > > Also... is the build always failing because of symbols starting with one > > or more underscores? > > The type of failures I've seen so far are: > > ERROR: "clear_user" [drivers/media/v4l2-core/videodev.ko] undefined! > ERROR: "__clear_user" [arch/x86/kvm/kvm.ko] undefined! > > ERROR: "__fill_rsb" [arch/x86/kvm/kvm-intel.ko] undefined! > > ERROR: "__put_user_2" [net/ipv4/netfilter/ip_tables.ko] undefined! > ERROR: "__put_user_2" [net/ipv4/netfilter/arp_tables.ko] undefined! > ERROR: "__put_user_8" [fs/udf/udf.ko] undefined! > ERROR: "__put_user_4" [fs/udf/udf.ko] undefined! > ERROR: "__put_user_8" [fs/fat/fat.ko] undefined! > ERROR: "__put_user_1" [fs/fat/fat.ko] undefined! > ERROR: "__put_user_4" [fs/fat/fat.ko] undefined! > ERROR: "__put_user_2" [fs/fat/fat.ko] undefined! > ERROR: "__put_user_4" [drivers/net/tap.ko] undefined! > ERROR: "__put_user_2" [drivers/net/tap.ko] undefined! > ERROR: "__put_user_8" [drivers/media/v4l2-core/videodev.ko] undefined! > ERROR: "__put_user_1" [drivers/media/v4l2-core/videodev.ko] undefined! > ERROR: "__put_user_4" [drivers/media/v4l2-core/videodev.ko] undefined! > ERROR: "__put_user_8" [drivers/input/joydev.ko] undefined! > ERROR: "__put_user_1" [drivers/input/joydev.ko] undefined! > ERROR: "__put_user_4" [drivers/input/joydev.ko] undefined! > ERROR: "__fill_rsb" [arch/x86/kvm/kvm-intel.ko] undefined! > > There might have been others but I didn't save every error message. Maybe it is just a coincidence, but there is a lot of underscore prefixed symbols in that list, except for one case. This translates to successive / in the path for the timestamp file. And that one case that doesn't fit the pattern does actually aliases a path that does. I wonder if the filesystem cache could get confused by successive / in paths here, given the non deterministic nature of the build failure you get. Could you please test with the following patch to validate this hypothesis: diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh index 513da1a4a2..2205114add 100755 --- a/scripts/adjust_autoksyms.sh +++ b/scripts/adjust_autoksyms.sh @@ -79,6 +79,7 @@ changed=$( count=0 sort "$cur_ksyms_file" "$new_ksyms_file" | uniq -u | sed -n 's/^#define __KSYM_\(.*\) 1/\1/p' | tr "A-Z_" "a-z/" | +sed -e 's|//*|/|g' -e 's|^/||' | while read sympath; do if [ -z "$sympath" ]; then continue; fi depfile="include/config/ksym/${sympath}.h" diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c index 449b68c4c9..57ae789f91 100644 --- a/scripts/basic/fixdep.c +++ b/scripts/basic/fixdep.c @@ -115,7 +115,7 @@ static void usage(void) */ static void print_config(const char *m, int slen) { - int c, i; + int c, prev_c = '/', i; printf(" $(wildcard include/config/"); for (i = 0; i < slen; i++) { @@ -124,7 +124,9 @@ static void print_config(const char *m, int slen) c = '/'; else c = tolower(c); - putchar(c); + if (c != '/' || prev_c != '/') + putchar(c); + prev_c = c; } printf(".h) \\\n"); } That would be very interesting if this patch fixed your build failures. > I patched the kernel with "kbuild: show the list of exported symbols with V=1 for TRIM_UNUSED_KSYMS" > and ran the build with V=1 until I got a failure. Here is the full output: > > success: https://gist.githubusercontent.com/anonymous/86325ce634568f1e1d31c23b5b9df295/raw/a897c77f34499a990505057f0221b4a935db4a16/gistfile1.txt > fail: https://gist.githubusercontent.com/anonymous/279a7925f1a8cd05dcad78dca0720688/raw/9e407b262fdac284b15ae07f2d7a7824051d56cc/gistfile1.txt > > The error was: > ERROR: "clear_user" [drivers/media/v4l2-core/videodev.ko] undefined! > ERROR: "__clear_user" [arch/x86/kvm/kvm.ko] undefined! > > and the relevant part of the diff of the build: > > --- kernel_build_debug_patch_success 2018-03-13 14:47:24.000000000 +0100 > +++ kernel_build_debug_patch_fail 2018-03-13 15:12:33.000000000 +0100 > @@ -45298,13 +45298,6 @@ > KSYM___x86_indirect_thunk_r15 > KSYM___fill_rsb > KSYM___clear_rsb > - gcc -Wp,-MD,arch/x86/lib/.usercopy_64.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/6.4.0/include -I./arch/x86/include -I./arch/x86/include/generated -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -march=core2 -mno-red-zone -mcmodel=kernel -funit-at-a-time -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -Wno-frame-address -O2 --param=allow-store-data-races=0 -DCC_HAVE_ASM_GOTO -fplugin=./scripts/gcc-plugins/latent_entropy_plugin.so -fplugin=./scripts/gcc-plugins/structleak_plugin.so -DLATENT_ENTROPY_PLUGIN -fplugin-arg-structleak_plugin-byref-all -DSTRUCTLEAK_PLUGIN -Wframe-larger-than=1024 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -g -pg -mfentry -DCC_USING_FENTRY -fno-inline-functions-called-once -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fno-stack-check -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -DKBUILD_BASENAME='"usercopy_64"' -DKBUILD_MODNAME='"usercopy_64"' -c -o arch/x86/lib/usercopy_64.o arch/x86/lib/usercopy_64.c > -KSYM___clear_user > -KSYM_clear_user > -KSYM_arch_wb_cache_pmem > -KSYM_memcpy_flushcache > - ./tools/objtool/objtool orc generate --no-fp "arch/x86/lib/usercopy_64.o"; > - if [ "-pg" = "-pg" ]; then if [ arch/x86/lib/usercopy_64.o != "scripts/mod/empty.o" ]; then ./scripts/recordmcount "arch/x86/lib/usercopy_64.o"; fi; fi; > rm -f arch/x86/lib/lib.a; ar rcsTPD arch/x86/lib/lib.a arch/x86/lib/clear_page_64.o arch/x86/lib/cmdline.o arch/x86/lib/cmpxchg16b_emu.o arch/x86/lib/copy_page_64.o arch/x86/lib/copy_user_64.o arch/x86/lib/cpu.o arch/x86/lib/csum-copy_64.o arch/x86/lib/csum-partial_64.o arch/x86/lib/csum-wrappers_64.o arch/x86/lib/delay.o arch/x86/lib/getuser.o arch/x86/lib/inat.o arch/x86/lib/insn.o arch/x86/lib/kaslr.o arch/x86/lib/memcpy_64.o arch/x86/lib/memmove_64.o arch/x86/lib/memset_64.o arch/x86/lib/misc.o arch/x86/lib/putuser.o arch/x86/lib/retpoline.o arch/x86/lib/rwsem.o arch/x86/lib/usercopy.o arch/x86/lib/usercopy_64.o > objdump -h arch/x86/lib/lib.a | sed -ne '/___ksymtab/s/.*+\([^ ]*\).*/EXTERN(\1)/p' >arch/x86/lib/.lib-ksyms.o.lds; rm -f arch/x86/lib/.lib_exports.o; echo | gcc -Wp,-MD,arch/x86/lib/.lib-ksyms.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/6.4.0/include -I./arch/x86/include -I./arch/x86/include/generated -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -D__ASSEMBLY__ -fno-PIE -m64 -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -DCC_HAVE_ASM_GOTO -Wa,-gdwarf-2 -mfentry -DCC_USING_FENTRY -c -o arch/x86/lib/.lib_exports.o -x assembler -; ld -m elf_x86_64 -r -o arch/x86/lib/lib-ksyms.o -T arch/x86/lib/.lib-ksyms.o.lds arch/x86/lib/.lib_exports.o; rm arch/x86/lib/.lib_exports.o arch/x86/lib/.lib-ksyms.o.lds > rm -f arch/x86/lib/built-in.o; ar rcSTPD arch/x86/lib/built-in.o arch/x86/lib/msr-smp.o arch/x86/lib/cache-smp.o arch/x86/lib/msr.o arch/x86/lib/msr-reg.o arch/x86/lib/msr-reg-export.o arch/x86/lib/hweight.o arch/x86/lib/iomap_copy_64.o arch/x86/lib/lib-ksyms.o ; scripts/mod/modpost arch/x86/lib/built-in.o That's it: usercopy_64.o not being rebuilt as it should, despite this on line 25369 of your log: ++ touch include/config/ksym///clear/user.h Or this on line 28057: ++ touch include/config/ksym/clear/user.h But since all the other failure cases implied symbols with "__" prefixes, I bet that the "clear_user" shows up only because it aliases "__clear_user". Nicolas ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-14 1:43 ` Nicolas Pitre @ 2018-03-15 18:48 ` Thomas Lindroth 2018-03-15 20:56 ` Nicolas Pitre 0 siblings, 1 reply; 16+ messages in thread From: Thomas Lindroth @ 2018-03-15 18:48 UTC (permalink / raw) To: Nicolas Pitre; +Cc: Masahiro Yamada, Linux Kbuild mailing list On 03/14/2018 02:43 AM, Nicolas Pitre wrote: > Now, what's the date of include/generated/autoksyms.h compared to > arch/x86/lib/usercopy_64.o? > > If include/generated/autoksyms.h is older than > arch/x86/lib/usercopy_64.o then the presence of __KSYM_clear_user in the > former should have instanciated the corresponding EXPORT_SYMBOL() in the > later. > > If it is the other way around then you should compare the time for > arch/x86/lib/usercopy_64.o against include/config/ksym///clear/user.h. > If the later is newer than the former then something failed to notice > that usercopy_64.o wasn't up to date, in which case the makefile or make > itself might need some investigation. > > If not then we'd have to look in the full build log to figure out if > __KSYM_clear_user existed in the previous version of > include/generated/autoksyms.h before it was refreshed by > adjust_autoksyms.sh. If it was then we're back to if #1 above. If not > then the timestamp for include/config/ksym///clear/user.h hasn't been > updated as it should. Here are the timestamps for the fail case: -rw-r--r-- 1 cocobo cocobo 66424 2018-03-13 17:20:18.000000000 +0100 linux-fail/include/generated/autoksyms.h -rw-r--r-- 1 cocobo cocobo 121064 2018-03-13 17:16:53.000000000 +0100 linux-fail/arch/x86/lib/usercopy_64.o -rw-r--r-- 1 cocobo cocobo 0 2018-03-13 17:16:53.000000000 +0100 linux-fail/include/config/ksym///clear/user.h It's suspicious that usercopy_64.o and ksym///clear/user.h got the same timestamp. My gut feeling says that ksym///clear/user.h was touched after usercopy_64.o was built but less than 1 sec had passed so they got the same timestamps due to the poor timestamp resolution on my old ext4 filesystem. Since the timestamps on ksym///clear/user.h wasn't newer than usercopy_64.o the rebuild was skipped. AS arch/x86/lib/putuser.o AS arch/x86/lib/retpoline.o <--- AS arch/x86/lib/rwsem.o CC arch/x86/lib/usercopy.o CC arch/x86/lib/usercopy_64.o <--- AR arch/x86/lib/lib.a EXPORTS arch/x86/lib/lib-ksyms.o AR arch/x86/lib/built-in.o CC virt/lib/irqbypass.o AR virt/lib/built-in.o AR virt/built-in.o CHK include/generated/autoksyms.h KSYMS symbols: before=0, after=1871, changed=1871 The problematic usercopy_64.o and retpoline.o are built just before ksym. The build and ksym generation probably happens in less than 1 sec. Here are the timestamps for the success case: -rw-r--r-- 1 cocobo cocobo 66424 2018-03-13 16:58:02.000000000 +0100 linux-success/include/generated/autoksyms.h -rw-r--r-- 1 cocobo cocobo 126912 2018-03-13 16:58:01.000000000 +0100 linux-success/arch/x86/lib/usercopy_64.o -rw-r--r-- 1 cocobo cocobo 0 2018-03-13 16:54:38.000000000 +0100 linux-success/include/config/ksym///clear/user.h usercopy_64.o was rebuilt here so it has a more recent timestamp than ksym///clear/user.h. To test this a bit more I copied the 4.14.23 source to tmpfs and ran the build there. Tmpfs supports nanosecond timestamps. The build succeeded 16 times in a row. Usually there is around 50/50 chance of success/failure on ext4. >>> Also... is the build always failing because of symbols starting with one >>> or more underscores? >> >> The type of failures I've seen so far are: >> >> ERROR: "clear_user" [drivers/media/v4l2-core/videodev.ko] undefined! >> ERROR: "__clear_user" [arch/x86/kvm/kvm.ko] undefined! >> >> ERROR: "__fill_rsb" [arch/x86/kvm/kvm-intel.ko] undefined! >> >> ERROR: "__put_user_2" [net/ipv4/netfilter/ip_tables.ko] undefined! >> ERROR: "__put_user_2" [net/ipv4/netfilter/arp_tables.ko] undefined! >> ERROR: "__put_user_8" [fs/udf/udf.ko] undefined! >> ERROR: "__put_user_4" [fs/udf/udf.ko] undefined! >> ERROR: "__put_user_8" [fs/fat/fat.ko] undefined! >> ERROR: "__put_user_1" [fs/fat/fat.ko] undefined! >> ERROR: "__put_user_4" [fs/fat/fat.ko] undefined! >> ERROR: "__put_user_2" [fs/fat/fat.ko] undefined! >> ERROR: "__put_user_4" [drivers/net/tap.ko] undefined! >> ERROR: "__put_user_2" [drivers/net/tap.ko] undefined! >> ERROR: "__put_user_8" [drivers/media/v4l2-core/videodev.ko] undefined! >> ERROR: "__put_user_1" [drivers/media/v4l2-core/videodev.ko] undefined! >> ERROR: "__put_user_4" [drivers/media/v4l2-core/videodev.ko] undefined! >> ERROR: "__put_user_8" [drivers/input/joydev.ko] undefined! >> ERROR: "__put_user_1" [drivers/input/joydev.ko] undefined! >> ERROR: "__put_user_4" [drivers/input/joydev.ko] undefined! >> ERROR: "__fill_rsb" [arch/x86/kvm/kvm-intel.ko] undefined! >> >> There might have been others but I didn't save every error message. > > Maybe it is just a coincidence, but there is a lot of underscore > prefixed symbols in that list, except for one case. This translates to > successive / in the path for the timestamp file. And that one case that > doesn't fit the pattern does actually aliases a path that does. I wonder > if the filesystem cache could get confused by successive / in paths > here, given the non deterministic nature of the build failure you get. > > Could you please test with the following patch to validate this > hypothesis: > > diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh > index 513da1a4a2..2205114add 100755 > --- a/scripts/adjust_autoksyms.sh > +++ b/scripts/adjust_autoksyms.sh > @@ -79,6 +79,7 @@ changed=$( > count=0 > sort "$cur_ksyms_file" "$new_ksyms_file" | uniq -u | > sed -n 's/^#define __KSYM_\(.*\) 1/\1/p' | tr "A-Z_" "a-z/" | > +sed -e 's|//*|/|g' -e 's|^/||' | > while read sympath; do > if [ -z "$sympath" ]; then continue; fi > depfile="include/config/ksym/${sympath}.h" > diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c > index 449b68c4c9..57ae789f91 100644 > --- a/scripts/basic/fixdep.c > +++ b/scripts/basic/fixdep.c > @@ -115,7 +115,7 @@ static void usage(void) > */ > static void print_config(const char *m, int slen) > { > - int c, i; > + int c, prev_c = '/', i; > > printf(" $(wildcard include/config/"); > for (i = 0; i < slen; i++) { > @@ -124,7 +124,9 @@ static void print_config(const char *m, int slen) > c = '/'; > else > c = tolower(c); > - putchar(c); > + if (c != '/' || prev_c != '/') > + putchar(c); > + prev_c = c; > } > printf(".h) \\\n"); > } > > That would be very interesting if this patch fixed your build failures. The patch applied with some fuzz to 4.14.23. Using the patch the first two builds I did succeeded and the third failed like: Kernel: arch/x86/boot/bzImage is ready (#2) Building modules, stage 2. MODPOST 146 modules ERROR: "__put_user_2" [net/ipv4/netfilter/ip_tables.ko] undefined! ERROR: "__put_user_2" [net/ipv4/netfilter/arp_tables.ko] undefined! ERROR: "__put_user_8" [fs/udf/udf.ko] undefined! ERROR: "__put_user_4" [fs/udf/udf.ko] undefined! ERROR: "__put_user_8" [fs/fat/fat.ko] undefined! ERROR: "__put_user_1" [fs/fat/fat.ko] undefined! ERROR: "__put_user_4" [fs/fat/fat.ko] undefined! ERROR: "__put_user_2" [fs/fat/fat.ko] undefined! ERROR: "__put_user_4" [drivers/net/tap.ko] undefined! ERROR: "__put_user_2" [drivers/net/tap.ko] undefined! ERROR: "__put_user_8" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__put_user_1" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__put_user_4" [drivers/media/v4l2-core/videodev.ko] undefined! ERROR: "__put_user_8" [drivers/input/joydev.ko] undefined! ERROR: "__put_user_1" [drivers/input/joydev.ko] undefined! ERROR: "__put_user_4" [drivers/input/joydev.ko] undefined! ERROR: "__fill_rsb" [arch/x86/kvm/kvm-intel.ko] undefined! make[1]: *** [scripts/Makefile.modpost:92: __modpost] Error 1 make: *** [Makefile:1218: modules] Error 2 ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-15 18:48 ` Thomas Lindroth @ 2018-03-15 20:56 ` Nicolas Pitre 2018-03-16 0:34 ` Masahiro Yamada 2018-03-20 15:16 ` Masahiro Yamada 0 siblings, 2 replies; 16+ messages in thread From: Nicolas Pitre @ 2018-03-15 20:56 UTC (permalink / raw) To: Thomas Lindroth; +Cc: Masahiro Yamada, Linux Kbuild mailing list On Thu, 15 Mar 2018, Thomas Lindroth wrote: > Here are the timestamps for the fail case: > -rw-r--r-- 1 cocobo cocobo 66424 2018-03-13 17:20:18.000000000 +0100 linux-fail/include/generated/autoksyms.h > -rw-r--r-- 1 cocobo cocobo 121064 2018-03-13 17:16:53.000000000 +0100 linux-fail/arch/x86/lib/usercopy_64.o > -rw-r--r-- 1 cocobo cocobo 0 2018-03-13 17:16:53.000000000 +0100 linux-fail/include/config/ksym///clear/user.h > > It's suspicious that usercopy_64.o and ksym///clear/user.h got the same timestamp. > My gut feeling says that ksym///clear/user.h was touched after usercopy_64.o was > built but less than 1 sec had passed so they got the same timestamps due to the > poor timestamp resolution on my old ext4 filesystem. Since the timestamps on > ksym///clear/user.h wasn't newer than usercopy_64.o the rebuild was skipped. > > AS arch/x86/lib/putuser.o > AS arch/x86/lib/retpoline.o <--- > AS arch/x86/lib/rwsem.o > CC arch/x86/lib/usercopy.o > CC arch/x86/lib/usercopy_64.o <--- > AR arch/x86/lib/lib.a > EXPORTS arch/x86/lib/lib-ksyms.o > AR arch/x86/lib/built-in.o > CC virt/lib/irqbypass.o > AR virt/lib/built-in.o > AR virt/built-in.o > CHK include/generated/autoksyms.h > KSYMS symbols: before=0, after=1871, changed=1871 > > The problematic usercopy_64.o and retpoline.o are built just before ksym. The build > and ksym generation probably happens in less than 1 sec. > > Here are the timestamps for the success case: > -rw-r--r-- 1 cocobo cocobo 66424 2018-03-13 16:58:02.000000000 +0100 linux-success/include/generated/autoksyms.h > -rw-r--r-- 1 cocobo cocobo 126912 2018-03-13 16:58:01.000000000 +0100 linux-success/arch/x86/lib/usercopy_64.o > -rw-r--r-- 1 cocobo cocobo 0 2018-03-13 16:54:38.000000000 +0100 linux-success/include/config/ksym///clear/user.h > > usercopy_64.o was rebuilt here so it has a more recent timestamp than ksym///clear/user.h. > > To test this a bit more I copied the 4.14.23 source to tmpfs and ran the build there. > Tmpfs supports nanosecond timestamps. The build succeeded 16 times in a row. Usually > there is around 50/50 chance of success/failure on ext4. OK. That must be it. Could you please test the following patch: ----- >8 Subject: [PATCH] kbuild: make scripts/adjust_autoksyms.sh robust against timestamp races Some filesystems have timestamps with coarse precision that may allow for a recently built object file to have the same timestamp as the updated time on one of its dependency files. When that happens, the object file doesn't get rebuilt as it should. This is especially the case on filesystems that don't have sub-second time precision, such as ext3 or Ext4 with 128B inodes. Let's prevent that by making sure updated dependency files have a newer timestamp than the first file we created (i.e. autoksyms.h.tmpnew). Reported-by: Thomas Lindroth <thomas.lindroth@gmail.com> Signed-off-by: Nicolas Pitre <nico@linaro.org> diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh index 513da1a4a2..d67830e6e3 100755 --- a/scripts/adjust_autoksyms.sh +++ b/scripts/adjust_autoksyms.sh @@ -84,6 +84,13 @@ while read sympath; do depfile="include/config/ksym/${sympath}.h" mkdir -p "$(dirname "$depfile")" touch "$depfile" + # Filesystems with coarse time precision may create timestamps + # equal to the one from a file that was very recently built and that + # needs to be rebuild. Let's guard against that by making sure our + # dep files are always newer than the first file we created here. + while [ ! "$depfile" -nt "$new_ksyms_file" ]; do + touch "$depfile" + done echo $((count += 1)) done | tail -1 ) changed=${changed:-0} ----- >8 > > Maybe it is just a coincidence, but there is a lot of underscore > > prefixed symbols in that list, except for one case. This translates to > > successive / in the path for the timestamp file. And that one case that > > doesn't fit the pattern does actually aliases a path that does. I wonder > > if the filesystem cache could get confused by successive / in paths > > here, given the non deterministic nature of the build failure you get. > > > > Could you please test with the following patch to validate this > > hypothesis: > The patch applied with some fuzz to 4.14.23. Using the patch the first two > builds I did succeeded and the third failed like: > Kernel: arch/x86/boot/bzImage is ready (#2) > Building modules, stage 2. > MODPOST 146 modules > ERROR: "__put_user_2" [net/ipv4/netfilter/ip_tables.ko] undefined! OK. Glad this hypothesis didn't verify. Nicolas ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-15 20:56 ` Nicolas Pitre @ 2018-03-16 0:34 ` Masahiro Yamada 2018-03-16 1:40 ` Nicolas Pitre 2018-03-20 15:16 ` Masahiro Yamada 1 sibling, 1 reply; 16+ messages in thread From: Masahiro Yamada @ 2018-03-16 0:34 UTC (permalink / raw) To: Nicolas Pitre; +Cc: Thomas Lindroth, Linux Kbuild mailing list 2018-03-16 5:56 GMT+09:00 Nicolas Pitre <nicolas.pitre@linaro.org>: > ----- >8 > Subject: [PATCH] kbuild: make scripts/adjust_autoksyms.sh robust against timestamp races > > Some filesystems have timestamps with coarse precision that may allow > for a recently built object file to have the same timestamp as the > updated time on one of its dependency files. When that happens, the > object file doesn't get rebuilt as it should. > > This is especially the case on filesystems that don't have sub-second > time precision, such as ext3 or Ext4 with 128B inodes. > > Let's prevent that by making sure updated dependency files have a newer > timestamp than the first file we created (i.e. autoksyms.h.tmpnew). > > Reported-by: Thomas Lindroth <thomas.lindroth@gmail.com> > Signed-off-by: Nicolas Pitre <nico@linaro.org> > > diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh > index 513da1a4a2..d67830e6e3 100755 > --- a/scripts/adjust_autoksyms.sh > +++ b/scripts/adjust_autoksyms.sh > @@ -84,6 +84,13 @@ while read sympath; do > depfile="include/config/ksym/${sympath}.h" > mkdir -p "$(dirname "$depfile")" > touch "$depfile" Just a nit. I think this 'touch' is unnecessary. If $depfile does not exist, the '-nt' is evaluated to false, so it will be touched inside the while loop anyway. > + # Filesystems with coarse time precision may create timestamps > + # equal to the one from a file that was very recently built and that > + # needs to be rebuild. Let's guard against that by making sure our > + # dep files are always newer than the first file we created here. > + while [ ! "$depfile" -nt "$new_ksyms_file" ]; do > + touch "$depfile" > + done > echo $((count += 1)) > done | tail -1 ) > changed=${changed:-0} > ----- >8 -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-16 0:34 ` Masahiro Yamada @ 2018-03-16 1:40 ` Nicolas Pitre 2018-03-16 3:37 ` Masahiro Yamada 0 siblings, 1 reply; 16+ messages in thread From: Nicolas Pitre @ 2018-03-16 1:40 UTC (permalink / raw) To: Masahiro Yamada; +Cc: Thomas Lindroth, Linux Kbuild mailing list On Fri, 16 Mar 2018, Masahiro Yamada wrote: > 2018-03-16 5:56 GMT+09:00 Nicolas Pitre <nicolas.pitre@linaro.org>: > > ----- >8 > > Subject: [PATCH] kbuild: make scripts/adjust_autoksyms.sh robust against timestamp races > > > > Some filesystems have timestamps with coarse precision that may allow > > for a recently built object file to have the same timestamp as the > > updated time on one of its dependency files. When that happens, the > > object file doesn't get rebuilt as it should. > > > > This is especially the case on filesystems that don't have sub-second > > time precision, such as ext3 or Ext4 with 128B inodes. > > > > Let's prevent that by making sure updated dependency files have a newer > > timestamp than the first file we created (i.e. autoksyms.h.tmpnew). > > > > Reported-by: Thomas Lindroth <thomas.lindroth@gmail.com> > > Signed-off-by: Nicolas Pitre <nico@linaro.org> > > > > diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh > > index 513da1a4a2..d67830e6e3 100755 > > --- a/scripts/adjust_autoksyms.sh > > +++ b/scripts/adjust_autoksyms.sh > > @@ -84,6 +84,13 @@ while read sympath; do > > depfile="include/config/ksym/${sympath}.h" > > mkdir -p "$(dirname "$depfile")" > > touch "$depfile" > > Just a nit. > > I think this 'touch' is unnecessary. > If $depfile does not exist, the '-nt' is evaluated to false, > so it will be touched inside the while loop anyway. Though it is more efficient with the first touch up front, as the common case requires only 2 stat system calls instead of 4. > > + # Filesystems with coarse time precision may create timestamps > > + # equal to the one from a file that was very recently built and that > > + # needs to be rebuild. Let's guard against that by making sure our > > + # dep files are always newer than the first file we created here. > > + while [ ! "$depfile" -nt "$new_ksyms_file" ]; do > > + touch "$depfile" > > + done > > echo $((count += 1)) > > done | tail -1 ) > > changed=${changed:-0} > > ----- >8 > > > -- > Best Regards > Masahiro Yamada > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-16 1:40 ` Nicolas Pitre @ 2018-03-16 3:37 ` Masahiro Yamada 0 siblings, 0 replies; 16+ messages in thread From: Masahiro Yamada @ 2018-03-16 3:37 UTC (permalink / raw) To: Nicolas Pitre; +Cc: Thomas Lindroth, Linux Kbuild mailing list 2018-03-16 10:40 GMT+09:00 Nicolas Pitre <nicolas.pitre@linaro.org>: > On Fri, 16 Mar 2018, Masahiro Yamada wrote: > >> 2018-03-16 5:56 GMT+09:00 Nicolas Pitre <nicolas.pitre@linaro.org>: >> > ----- >8 >> > Subject: [PATCH] kbuild: make scripts/adjust_autoksyms.sh robust against timestamp races >> > >> > Some filesystems have timestamps with coarse precision that may allow >> > for a recently built object file to have the same timestamp as the >> > updated time on one of its dependency files. When that happens, the >> > object file doesn't get rebuilt as it should. >> > >> > This is especially the case on filesystems that don't have sub-second >> > time precision, such as ext3 or Ext4 with 128B inodes. >> > >> > Let's prevent that by making sure updated dependency files have a newer >> > timestamp than the first file we created (i.e. autoksyms.h.tmpnew). >> > >> > Reported-by: Thomas Lindroth <thomas.lindroth@gmail.com> >> > Signed-off-by: Nicolas Pitre <nico@linaro.org> >> > >> > diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh >> > index 513da1a4a2..d67830e6e3 100755 >> > --- a/scripts/adjust_autoksyms.sh >> > +++ b/scripts/adjust_autoksyms.sh >> > @@ -84,6 +84,13 @@ while read sympath; do >> > depfile="include/config/ksym/${sympath}.h" >> > mkdir -p "$(dirname "$depfile")" >> > touch "$depfile" >> >> Just a nit. >> >> I think this 'touch' is unnecessary. >> If $depfile does not exist, the '-nt' is evaluated to false, >> so it will be touched inside the while loop anyway. > > Though it is more efficient with the first touch up front, as the common > case requires only 2 stat system calls instead of 4. Ah, you are right. I take my comment back. I am fine with this patch. Thanks! >> > + # Filesystems with coarse time precision may create timestamps >> > + # equal to the one from a file that was very recently built and that >> > + # needs to be rebuild. Let's guard against that by making sure our >> > + # dep files are always newer than the first file we created here. >> > + while [ ! "$depfile" -nt "$new_ksyms_file" ]; do >> > + touch "$depfile" >> > + done >> > echo $((count += 1)) >> > done | tail -1 ) >> > changed=${changed:-0} >> > ----- >8 >> >> >> -- >> Best Regards >> Masahiro Yamada >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-15 20:56 ` Nicolas Pitre 2018-03-16 0:34 ` Masahiro Yamada @ 2018-03-20 15:16 ` Masahiro Yamada 2018-03-20 17:28 ` Thomas Lindroth 1 sibling, 1 reply; 16+ messages in thread From: Masahiro Yamada @ 2018-03-20 15:16 UTC (permalink / raw) To: Thomas Lindroth; +Cc: Linux Kbuild mailing list, Nicolas Pitre Hi Thomas, 2018-03-16 5:56 GMT+09:00 Nicolas Pitre <nicolas.pitre@linaro.org>: > Subject: [PATCH] kbuild: make scripts/adjust_autoksyms.sh robust against timestamp races > > Some filesystems have timestamps with coarse precision that may allow > for a recently built object file to have the same timestamp as the > updated time on one of its dependency files. When that happens, the > object file doesn't get rebuilt as it should. > > This is especially the case on filesystems that don't have sub-second > time precision, such as ext3 or Ext4 with 128B inodes. > > Let's prevent that by making sure updated dependency files have a newer > timestamp than the first file we created (i.e. autoksyms.h.tmpnew). > > Reported-by: Thomas Lindroth <thomas.lindroth@gmail.com> > Signed-off-by: Nicolas Pitre <nico@linaro.org> > > diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh > index 513da1a4a2..d67830e6e3 100755 > --- a/scripts/adjust_autoksyms.sh > +++ b/scripts/adjust_autoksyms.sh > @@ -84,6 +84,13 @@ while read sympath; do > depfile="include/config/ksym/${sympath}.h" > mkdir -p "$(dirname "$depfile")" > touch "$depfile" > + # Filesystems with coarse time precision may create timestamps > + # equal to the one from a file that was very recently built and that > + # needs to be rebuild. Let's guard against that by making sure our > + # dep files are always newer than the first file we created here. > + while [ ! "$depfile" -nt "$new_ksyms_file" ]; do > + touch "$depfile" > + done > echo $((count += 1)) > done | tail -1 ) > changed=${changed:-0} Does this patch solve your problem? I guess so, but I want you to test it just in case. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-20 15:16 ` Masahiro Yamada @ 2018-03-20 17:28 ` Thomas Lindroth 2018-03-21 1:07 ` Nicolas Pitre 0 siblings, 1 reply; 16+ messages in thread From: Thomas Lindroth @ 2018-03-20 17:28 UTC (permalink / raw) To: Masahiro Yamada; +Cc: Linux Kbuild mailing list, Nicolas Pitre On 03/20/2018 04:16 PM, Masahiro Yamada wrote: >> Some filesystems have timestamps with coarse precision that may allow >> for a recently built object file to have the same timestamp as the >> updated time on one of its dependency files. When that happens, the >> object file doesn't get rebuilt as it should. >> >> This is especially the case on filesystems that don't have sub-second >> time precision, such as ext3 or Ext4 with 128B inodes. >> >> Let's prevent that by making sure updated dependency files have a newer >> timestamp than the first file we created (i.e. autoksyms.h.tmpnew). >> >> Reported-by: Thomas Lindroth <thomas.lindroth@gmail.com> >> Signed-off-by: Nicolas Pitre <nico@linaro.org> >> >> diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh >> index 513da1a4a2..d67830e6e3 100755 >> --- a/scripts/adjust_autoksyms.sh >> +++ b/scripts/adjust_autoksyms.sh >> @@ -84,6 +84,13 @@ while read sympath; do >> depfile="include/config/ksym/${sympath}.h" >> mkdir -p "$(dirname "$depfile")" >> touch "$depfile" >> + # Filesystems with coarse time precision may create timestamps >> + # equal to the one from a file that was very recently built and that >> + # needs to be rebuild. Let's guard against that by making sure our >> + # dep files are always newer than the first file we created here. >> + while [ ! "$depfile" -nt "$new_ksyms_file" ]; do >> + touch "$depfile" >> + done >> echo $((count += 1)) >> done | tail -1 ) >> changed=${changed:-0} > > > Does this patch solve your problem? > I guess so, but I want you to test it just in case. Yes, it solves the build problem but I'm still experiencing the original problem with vfio forwarded pci device even with this patch. The problem is the same as before. A kvm based VM fails to initialize a pci device assigned using vfio if TRIM_UNUSED_KSYMS is on but works fine with it off. I don't feel like spending time debugging that problem myself. TRIM_UNUSED_KSYMS sounded useful at first but the build time increased so much I probably won't use it anyway. If you manage to figure out what the problem is I'm willing to test patches. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-20 17:28 ` Thomas Lindroth @ 2018-03-21 1:07 ` Nicolas Pitre 2018-03-21 9:51 ` Thomas Lindroth 2018-03-21 13:34 ` Masahiro Yamada 0 siblings, 2 replies; 16+ messages in thread From: Nicolas Pitre @ 2018-03-21 1:07 UTC (permalink / raw) To: Thomas Lindroth; +Cc: Masahiro Yamada, Linux Kbuild mailing list On Tue, 20 Mar 2018, Thomas Lindroth wrote: > On 03/20/2018 04:16 PM, Masahiro Yamada wrote: > >> Some filesystems have timestamps with coarse precision that may allow > >> for a recently built object file to have the same timestamp as the > >> updated time on one of its dependency files. When that happens, the > >> object file doesn't get rebuilt as it should. > >> > >> This is especially the case on filesystems that don't have sub-second > >> time precision, such as ext3 or Ext4 with 128B inodes. > >> > >> Let's prevent that by making sure updated dependency files have a newer > >> timestamp than the first file we created (i.e. autoksyms.h.tmpnew). > >> > >> Reported-by: Thomas Lindroth <thomas.lindroth@gmail.com> > >> Signed-off-by: Nicolas Pitre <nico@linaro.org> > >> > >> diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh > >> index 513da1a4a2..d67830e6e3 100755 > >> --- a/scripts/adjust_autoksyms.sh > >> +++ b/scripts/adjust_autoksyms.sh > >> @@ -84,6 +84,13 @@ while read sympath; do > >> depfile="include/config/ksym/${sympath}.h" > >> mkdir -p "$(dirname "$depfile")" > >> touch "$depfile" > >> + # Filesystems with coarse time precision may create timestamps > >> + # equal to the one from a file that was very recently built and that > >> + # needs to be rebuild. Let's guard against that by making sure our > >> + # dep files are always newer than the first file we created here. > >> + while [ ! "$depfile" -nt "$new_ksyms_file" ]; do > >> + touch "$depfile" > >> + done > >> echo $((count += 1)) > >> done | tail -1 ) > >> changed=${changed:-0} > > > > > > Does this patch solve your problem? > > I guess so, but I want you to test it just in case. > > Yes, it solves the build problem but I'm still experiencing the original > problem with vfio forwarded pci device even with this patch. All this patch pretends to do is fix the build race issue. So it should probably be applied for that reason alone. > The problem is the same as before. A kvm based VM fails to initialize > a pci device assigned using vfio if TRIM_UNUSED_KSYMS is on but works > fine with it off. > > I don't feel like spending time debugging that problem myself. > TRIM_UNUSED_KSYMS sounded useful at first but the build time increased > so much I probably won't use it anyway. If you manage to figure out > what the problem is I'm willing to test patches. If you could give me simple step by step reproduction instructions for the issue then I'll have a look. Nicolas ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-21 1:07 ` Nicolas Pitre @ 2018-03-21 9:51 ` Thomas Lindroth 2018-03-21 13:34 ` Masahiro Yamada 1 sibling, 0 replies; 16+ messages in thread From: Thomas Lindroth @ 2018-03-21 9:51 UTC (permalink / raw) To: Nicolas Pitre; +Cc: Masahiro Yamada, Linux Kbuild mailing list On 03/21/2018 02:07 AM, Nicolas Pitre wrote: >> The problem is the same as before. A kvm based VM fails to initialize >> a pci device assigned using vfio if TRIM_UNUSED_KSYMS is on but works >> fine with it off. >> >> I don't feel like spending time debugging that problem myself. >> TRIM_UNUSED_KSYMS sounded useful at first but the build time increased >> so much I probably won't use it anyway. If you manage to figure out >> what the problem is I'm willing to test patches. > > If you could give me simple step by step reproduction instructions for > the issue then I'll have a look. I use vfio for passing a GPU to a windows VM for gaming. That's not so easy to setup and requires special hardware support. You will probably have a hard time reproducing my setup but if you want to try here are some info for how it's done. https://vfio.blogspot.se/2015/05/vfio-gpu-how-to-series-part-1-hardware.html https://vfio.blogspot.se/2015/05/vfio-gpu-how-to-series-part-2.html https://vfio.blogspot.se/2015/05/vfio-gpu-how-to-series-part-3-host.html https://vfio.blogspot.se/2015/05/vfio-gpu-how-to-series-part-4-our-first.html Arch linux wiki also got a useful page https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF You already have my .config. I build vfio into the kernel directly. Here are my kernel parameters intel_iommu=on memtest=1 kvm.ignore_msrs=1 vfio-pci.disable_idle_d3=1 vfio-pci.disable_vga=1 vfio-pci.ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff That will assign all nvidia GPUs to the vfio driver at boot. The VM run windows 7 using qemu-2.11.1 with libvirt-4.1.0 as frontend. Here is the libvirt xml definition of the VM <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win7</name> <uuid>f11b648e-c652-4c42-b2ba-02732533a15d</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <hugepages/> </memoryBacking> <vcpu placement='static'>6</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='5'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='6'/> <vcpupin vcpu='4' cpuset='3'/> <vcpupin vcpu='5' cpuset='7'/> <emulatorpin cpuset='4'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE_patched.fd</loader> <nvram>/var/lib/libvirt/qemu/nvram/win7_VARS.fd</nvram> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='SomeString'/> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='3' threads='2'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='yes'/> <suspend-to-disk enabled='yes'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <source dev='/dev/VM_default_VG/main_VM_volume'/> <target dev='sda' bus='sata'/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/virtual/share/virtio-win-0.1.126.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='user'> <mac address='00:11:22:33:44:55'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='bridge'> <mac address='00:11:22:33:44:66'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </interface> <input type='mouse' bus='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </input> <input type='keyboard' bus='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'> <listen type='address'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source startupPolicy='optional'> <vendor id='0x054c'/> <product id='0x05c4'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source startupPolicy='optional'> <vendor id='0x04a9'/> <product id='0x190d'/> </source> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-object'/> <qemu:arg value='input-linux,id=mouse1,evdev=/dev/input/by-id/usb-Logitech_G700_Laser_Mouse_E52DB3FA0035-event-mouse'/> <qemu:arg value='-object'/> <qemu:arg value='input-linux,id=kbd1,evdev=/dev/input/by-id/usb-195d_Gaming_keyboard-if02-event-kbd,grab_all=on,repeat=on'/> </qemu:commandline> </domain> The OVMF version I use is a specially patched version to make hyperv enlightenments work in windows 7. https://github.com/tholin/OVMF-win7-hyperv This is what lspci -nn looks like 00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06) 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06) 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) 00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1] 00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba] 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-V [8086:153b] 00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad] 00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0) 00:1c.2 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 [8086:8c94] (rev d0) 00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0) 00:1c.4 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 [8086:8c98] (rev d0) 00:1c.6 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 [8086:8c9c] (rev d0) 00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6] 00:1f.0 ISA bridge [0601]: Intel Corporation 9 Series Chipset Family Z97 LPC Controller [8086:8cc4] 00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82] 00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2] 01:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca) 02:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca) 02:10.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca) 04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1) 04:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1) 06:00.0 Ethernet controller [0200]: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller [1969:e091] (rev 10) 07:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch [10b5:8605] (rev ab) 08:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch [10b5:8605] (rev ab) 08:02.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch [10b5:8605] (rev ab) 08:03.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8605 PCI Express 4-port Gen2 Switch [10b5:8605] (rev ab) 0b:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge [10b5:8112] (rev aa) 0c:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788] 0d:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 12) 0e:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 12) I assign these devices to the VM 04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1) 04:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1) My CPU is a i7-4790K and I use the igpu for the host. My original hunch was that the kernel functions needed for accessing the pci config space was broken somehow. I think the sysfs driver for /sys/bus/pci/devices/*/config use the same functions as vfio and lspci -vvvxxxxnn give reasonable output even with TRIM_UNUSED_KSYMS. Each boot always give slightly different output so it's hard to compare. lspci doesn't write to the config space so perhaps only writing is broken or perhaps the problem has nothing to do with pci config space at all. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-21 1:07 ` Nicolas Pitre 2018-03-21 9:51 ` Thomas Lindroth @ 2018-03-21 13:34 ` Masahiro Yamada 1 sibling, 0 replies; 16+ messages in thread From: Masahiro Yamada @ 2018-03-21 13:34 UTC (permalink / raw) To: Nicolas Pitre; +Cc: Thomas Lindroth, Linux Kbuild mailing list 2018-03-21 10:07 GMT+09:00 Nicolas Pitre <nicolas.pitre@linaro.org>: > On Tue, 20 Mar 2018, Thomas Lindroth wrote: > >> On 03/20/2018 04:16 PM, Masahiro Yamada wrote: >> >> Some filesystems have timestamps with coarse precision that may allow >> >> for a recently built object file to have the same timestamp as the >> >> updated time on one of its dependency files. When that happens, the >> >> object file doesn't get rebuilt as it should. >> >> >> >> This is especially the case on filesystems that don't have sub-second >> >> time precision, such as ext3 or Ext4 with 128B inodes. >> >> >> >> Let's prevent that by making sure updated dependency files have a newer >> >> timestamp than the first file we created (i.e. autoksyms.h.tmpnew). >> >> >> >> Reported-by: Thomas Lindroth <thomas.lindroth@gmail.com> >> >> Signed-off-by: Nicolas Pitre <nico@linaro.org> >> >> >> >> diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh >> >> index 513da1a4a2..d67830e6e3 100755 >> >> --- a/scripts/adjust_autoksyms.sh >> >> +++ b/scripts/adjust_autoksyms.sh >> >> @@ -84,6 +84,13 @@ while read sympath; do >> >> depfile="include/config/ksym/${sympath}.h" >> >> mkdir -p "$(dirname "$depfile")" >> >> touch "$depfile" >> >> + # Filesystems with coarse time precision may create timestamps >> >> + # equal to the one from a file that was very recently built and that >> >> + # needs to be rebuild. Let's guard against that by making sure our >> >> + # dep files are always newer than the first file we created here. >> >> + while [ ! "$depfile" -nt "$new_ksyms_file" ]; do >> >> + touch "$depfile" >> >> + done >> >> echo $((count += 1)) >> >> done | tail -1 ) >> >> changed=${changed:-0} >> > >> > >> > Does this patch solve your problem? >> > I guess so, but I want you to test it just in case. >> >> Yes, it solves the build problem but I'm still experiencing the original >> problem with vfio forwarded pci device even with this patch. > > All this patch pretends to do is fix the build race issue. > So it should probably be applied for that reason alone. I agree. Applied to linux-kbuild/fixes. Thanks! -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Intermittent build failure with TRIM_UNUSED_KSYMS and related problems 2018-03-12 15:47 ` Masahiro Yamada 2018-03-12 19:15 ` Nicolas Pitre @ 2018-03-12 20:54 ` Nicolas Pitre 1 sibling, 0 replies; 16+ messages in thread From: Nicolas Pitre @ 2018-03-12 20:54 UTC (permalink / raw) To: Masahiro Yamada; +Cc: Thomas Lindroth, Linux Kbuild mailing list On Tue, 13 Mar 2018, Masahiro Yamada wrote: > 2018-03-05 22:07 GMT+09:00 Thomas Lindroth <thomas.lindroth@gmail.com>: > > As I understand it TRIM_UNUSED_KSYMS will build the kernel and > > modules, then check which symbols are used by the modules and remove > > all unused EXPORT_SYMBOL_* from the kernel and rebuild it again. > > > > EXPORT_SYMBOL_GPL(pci_user_read_config_##size); > > EXPORT_SYMBOL_GPL(pci_user_write_config_##size); > > > > drivers/pci/access.c got these two exports. They stand out because > > they are macros instead of functions. I propose the following patch to clear up doubts about the most obscure part of the build for which there is currently no direct visibility, which is the actual extraction of exported symbols. ---- >8 Subject: kbuild: show the list of exported symbols with V=1 for TRIM_UNUSED_KSYMS When CONFIG_TRIM_UNUSED_KSYMS=y the list of exported symbols is extracted from the source code and piped directly into the fixdep tool. Because of that, there is no direct way of obtaining that list to make sure it is correct when debugging related issues. Let's log that list of symbols when the verbosity level is set to 1. Signed-off-by: Nicolas Pitre <nico@linaro.org> diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include index 9ffd3dda38..141a63787b 100644 --- a/scripts/Kbuild.include +++ b/scripts/Kbuild.include @@ -298,6 +298,7 @@ ksym_dep_filter = \ cmd_and_fixdep = \ $(echo-cmd) $(cmd_$(1)); \ + $(if $(findstring 1,$(KBUILD_VERBOSE)), $(ksym_dep_filter);) \ $(ksym_dep_filter) | \ scripts/basic/fixdep -e $(depfile) $@ '$(make-cmd)' \ > $(dot-target).tmp; \ ^ permalink raw reply related [flat|nested] 16+ messages in thread
end of thread, other threads:[~2018-03-21 13:35 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-03-05 13:07 Intermittent build failure with TRIM_UNUSED_KSYMS and related problems Thomas Lindroth 2018-03-12 15:47 ` Masahiro Yamada 2018-03-12 19:15 ` Nicolas Pitre 2018-03-13 20:57 ` Thomas Lindroth 2018-03-14 1:43 ` Nicolas Pitre 2018-03-15 18:48 ` Thomas Lindroth 2018-03-15 20:56 ` Nicolas Pitre 2018-03-16 0:34 ` Masahiro Yamada 2018-03-16 1:40 ` Nicolas Pitre 2018-03-16 3:37 ` Masahiro Yamada 2018-03-20 15:16 ` Masahiro Yamada 2018-03-20 17:28 ` Thomas Lindroth 2018-03-21 1:07 ` Nicolas Pitre 2018-03-21 9:51 ` Thomas Lindroth 2018-03-21 13:34 ` Masahiro Yamada 2018-03-12 20:54 ` Nicolas Pitre
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox