Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Alexei Starovoitov]

On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> Introduce in-kernel headers and other artifacts which are made available
> as an archive through proc (/proc/kheaders.txz file). This archive makes
> it possible to build kernel modules, run eBPF programs, and other
> tracing programs that need to extend the kernel for tracing purposes
> without any dependency on the file system having headers and build
> artifacts.
> 
> On Android and embedded systems, it is common to switch kernels but not
> have kernel headers available on the file system. Raw kernel headers
> also cannot be copied into the filesystem like they can be on other
> distros, due to licensing and other issues. There's no linux-headers
> package on Android. Further once a different kernel is booted, any
> headers stored on the file system will no longer be useful. By storing
> the headers as a compressed archive within the kernel, we can avoid these
> issues that have been a hindrance for a long time.

The set looks good to me and since the main use case is building bpf progs
I can route it via bpf-next tree if there are no objections.
Masahiro, could you please ack it?

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote:
> On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > Introduce in-kernel headers and other artifacts which are made available
> > as an archive through proc (/proc/kheaders.txz file). This archive makes
> > it possible to build kernel modules, run eBPF programs, and other
> > tracing programs that need to extend the kernel for tracing purposes
> > without any dependency on the file system having headers and build
> > artifacts.
> > 
> > On Android and embedded systems, it is common to switch kernels but not
> > have kernel headers available on the file system. Raw kernel headers
> > also cannot be copied into the filesystem like they can be on other
> > distros, due to licensing and other issues. There's no linux-headers
> > package on Android. Further once a different kernel is booted, any
> > headers stored on the file system will no longer be useful. By storing
> > the headers as a compressed archive within the kernel, we can avoid these
> > issues that have been a hindrance for a long time.
> 
> The set looks good to me and since the main use case is building bpf progs
> I can route it via bpf-next tree if there are no objections.
> Masahiro, could you please ack it?
> 

Yes, eBPF is one of the usecases. After this, I am also planning to send
patches to BCC so that it can use this feature when compiling C to eBPF.

Thanks!

 - Joel

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Manoj]

Joel Fernandes writes:

> On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote:
>> On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
>> > Introduce in-kernel headers and other artifacts which are made available
>> > as an archive through proc (/proc/kheaders.txz file). This archive makes
>> > it possible to build kernel modules, run eBPF programs, and other
>> > tracing programs that need to extend the kernel for tracing purposes
>> > without any dependency on the file system having headers and build
>> > artifacts.
>> > 
>> > On Android and embedded systems, it is common to switch kernels but not
>> > have kernel headers available on the file system. Raw kernel headers
>> > also cannot be copied into the filesystem like they can be on other
>> > distros, due to licensing and other issues. There's no linux-headers
>> > package on Android. Further once a different kernel is booted, any
>> > headers stored on the file system will no longer be useful. By storing
>> > the headers as a compressed archive within the kernel, we can avoid these
>> > issues that have been a hindrance for a long time.
>> 
>> The set looks good to me and since the main use case is building bpf progs
>> I can route it via bpf-next tree if there are no objections.
>> Masahiro, could you please ack it?
>> 
>
> Yes, eBPF is one of the usecases. After this, I am also planning to send
> patches to BCC so that it can use this feature when compiling C to eBPF.
>

Tested-by: Manoj Rao <linux@manojrajarao.com>

I think this can definitely make it easier to use eBPF on
Android. Thanks for initiating this.

> Thanks!
>
>  - Joel


-- 
Manoj
http://www.mycpu.org

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Masahiro Yamada]

On Fri, Feb 15, 2019 at 11:48 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > Introduce in-kernel headers and other artifacts which are made available
> > as an archive through proc (/proc/kheaders.txz file).


The extension '.txz' is not used in kernel code.


'.tar.xz' is used for 'tarxz-pkg', 'perf-tarxz-src-pkg' etc.


$ git grep  '\.txz'
$ git grep  '\.tar\.xz'
Documentation/admin-guide/README.rst:     xz -cd linux-4.X.tar.xz | tar xvf -
arch/x86/crypto/camellia-aesni-avx-asm_64.S: *
http://koti.mbnet.fi/axh/crypto/camellia-BSD-1.2.0-aesni1.tar.xz
crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
scripts/package/Makefile:       @echo '  perf-tarxz-src-pkg  - Build
$(perf-tar).tar.xz source tarball'
tools/testing/selftests/gen_kselftest_tar.sh:
 ext=".tar.xz"



I prefer '.tar.xz' for consistency.






BTW, have you ever looked at scripts/extract-ikconfig?

You added IKHD_ST and IKHD_ED
just to mimic kernel/configs.c

It is currently pointless without the extracting tool,
but you might think it is useful to extract headers
from vmlinux or the module without mounting procfs.




> > This archive makes
> > it possible to build kernel modules, run eBPF programs, and other
> > tracing programs that need to extend the kernel for tracing purposes
> > without any dependency on the file system having headers and build
> > artifacts.
> >
> > On Android and embedded systems, it is common to switch kernels but not
> > have kernel headers available on the file system. Raw kernel headers
> > also cannot be copied into the filesystem like they can be on other
> > distros, due to licensing and other issues. There's no linux-headers
> > package on Android. Further once a different kernel is booted, any
> > headers stored on the file system will no longer be useful. By storing
> > the headers as a compressed archive within the kernel, we can avoid these
> > issues that have been a hindrance for a long time.
>
> The set looks good to me and since the main use case is building bpf progs
> I can route it via bpf-next tree if there are no objections.
> Masahiro, could you please ack it?


Honestly, I was not tracking this thread
since I did not know I was responsible for this.


I just started to take a closer look, then immediately got scared.

This version is not mature enough for the merge.



First of all, this patch cannot be compiled out-of-tree (O= option).


I do not know why 0-day bot did not catch this apparent breakage.


$ make -j8 O=hoge
make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge'
  GEN     Makefile
  Using .. as source for kernel
  DESCEND  objtool
  CALL    ../scripts/checksyscalls.sh
  CHK     include/generated/compile.h
make[2]: *** No rule to make target 'Module.symvers', needed by
'kernel/kheaders_data.txz'.  Stop.
make[2]: *** Waiting for unfinished jobs....
/home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target
'kernel' failed
make[1]: *** [kernel] Error 2
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge'
Makefile:152: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2





I was able to compile it in-tree
but it makes the incremental build extremely slow.

(Here, the incremental build means
"make" without changing any code after the full build.)




Before this patch, "make -j8" took 11 sec on my machine.

real 0m11.777s
user 0m16.608s
sys 0m5.164s



After this patch, x86_64_defconfig + CONFIG_IKHEADERS_PROC=y
takes 53 sec for me since kernel/kheaders_data.txz is regenerated
every time even when you did not touch any source file.


$ time make -j8
  DESCEND  objtool
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     kernel/kheaders_data.txz
  UPD     kernel/kheaders_data.h
  CC      kernel/kheaders.o
  AR      kernel/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  AR      built-in.a
  LD      vmlinux.o
  MODPOST vmlinux.o
  KSYM    .tmp_kallsyms1.o
  KSYM    .tmp_kallsyms2.o
  LD      vmlinux
  SORTEX  vmlinux
  SYSMAP  System.map
  Building modules, stage 2.
  CC      arch/x86/boot/version.o
  MODPOST 17 modules
  VOFFSET arch/x86/boot/compressed/../voffset.h
  OBJCOPY arch/x86/boot/compressed/vmlinux.bin
  RELOCS  arch/x86/boot/compressed/vmlinux.relocs
  CC      arch/x86/boot/compressed/kaslr.o
  GZIP    arch/x86/boot/compressed/vmlinux.bin.gz
  CC      arch/x86/boot/compressed/misc.o
  MKPIGGY arch/x86/boot/compressed/piggy.S
  AS      arch/x86/boot/compressed/piggy.o
  LD      arch/x86/boot/compressed/vmlinux
  ZOFFSET arch/x86/boot/zoffset.h
  OBJCOPY arch/x86/boot/vmlinux.bin
  AS      arch/x86/boot/header.o
  LD      arch/x86/boot/setup.elf
  OBJCOPY arch/x86/boot/setup.bin
  BUILD   arch/x86/boot/bzImage
Setup is 15612 bytes (padded to 15872 bytes).
System is 12673 kB
CRC 697aaf88
Kernel: arch/x86/boot/bzImage is ready  (#6)

real 0m53.024s
user 0m32.076s
sys 0m9.296s




Also, I notice $(ARCH) must be fixed to $(SRCARCH),
but that is one of minor issues.



We should take time for careful review and test.


Please give me more time for thorough review.



--
Best Regards
Masahiro Yamada

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Alexei Starovoitov]

On Tue, Feb 19, 2019 at 01:14:11PM +0900, Masahiro Yamada wrote:
> 
> I was able to compile it in-tree
> but it makes the incremental build extremely slow.
> 
> (Here, the incremental build means
> "make" without changing any code after the full build.)
> 
> Before this patch, "make -j8" took 11 sec on my machine.
> 
> After this patch, x86_64_defconfig + CONFIG_IKHEADERS_PROC=y
> takes 53 sec for me since kernel/kheaders_data.txz is regenerated
> every time even when you did not touch any source file.
> 
> We should take time for careful review and test.
> 
> Please give me more time for thorough review.

Thanks for taking a look. All are very valid points.
Incremental build is must have.

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Tue, Feb 19, 2019 at 01:14:11PM +0900, Masahiro Yamada wrote:
> On Fri, Feb 15, 2019 at 11:48 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > > Introduce in-kernel headers and other artifacts which are made available
> > > as an archive through proc (/proc/kheaders.txz file).
> 
> 
> The extension '.txz' is not used in kernel code.
> 
> 
> '.tar.xz' is used for 'tarxz-pkg', 'perf-tarxz-src-pkg' etc.
> 
> 
> $ git grep  '\.txz'
> $ git grep  '\.tar\.xz'
> Documentation/admin-guide/README.rst:     xz -cd linux-4.X.tar.xz | tar xvf -
> arch/x86/crypto/camellia-aesni-avx-asm_64.S: *
> http://koti.mbnet.fi/axh/crypto/camellia-BSD-1.2.0-aesni1.tar.xz
> crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> scripts/package/Makefile:       @echo '  perf-tarxz-src-pkg  - Build
> $(perf-tar).tar.xz source tarball'
> tools/testing/selftests/gen_kselftest_tar.sh:
>  ext=".tar.xz"
> 
> 
> 
> I prefer '.tar.xz' for consistency.

Ok, I will change it to that.

> BTW, have you ever looked at scripts/extract-ikconfig?
> 
> You added IKHD_ST and IKHD_ED
> just to mimic kernel/configs.c
> 
> It is currently pointless without the extracting tool,
> but you might think it is useful to extract headers
> from vmlinux or the module without mounting procfs.

I am planing add to extraction support in the future, so I prefer to leave the
markers as it is for now. Hope that's Ok with you.


> > > This archive makes
> > > it possible to build kernel modules, run eBPF programs, and other
> > > tracing programs that need to extend the kernel for tracing purposes
> > > without any dependency on the file system having headers and build
> > > artifacts.
> > >
> > > On Android and embedded systems, it is common to switch kernels but not
> > > have kernel headers available on the file system. Raw kernel headers
> > > also cannot be copied into the filesystem like they can be on other
> > > distros, due to licensing and other issues. There's no linux-headers
> > > package on Android. Further once a different kernel is booted, any
> > > headers stored on the file system will no longer be useful. By storing
> > > the headers as a compressed archive within the kernel, we can avoid these
> > > issues that have been a hindrance for a long time.
> >
> > The set looks good to me and since the main use case is building bpf progs
> > I can route it via bpf-next tree if there are no objections.
> > Masahiro, could you please ack it?
> 
> 
> Honestly, I was not tracking this thread
> since I did not know I was responsible for this.
> 
> 
> I just started to take a closer look, then immediately got scared.
> 
> This version is not mature enough for the merge.
> First of all, this patch cannot be compiled out-of-tree (O= option).

Oh, ok. This is not how I build the kernel. So I am sorry for missing that. I
will try this out-of-tree build option and fix it.


> I do not know why 0-day bot did not catch this apparent breakage.
> 
> 
> $ make -j8 O=hoge
> make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge'
>   GEN     Makefile
>   Using .. as source for kernel
>   DESCEND  objtool
>   CALL    ../scripts/checksyscalls.sh
>   CHK     include/generated/compile.h
> make[2]: *** No rule to make target 'Module.symvers', needed by
> 'kernel/kheaders_data.txz'.  Stop.
> make[2]: *** Waiting for unfinished jobs....
> /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target
> 'kernel' failed
> make[1]: *** [kernel] Error 2
> make[1]: *** Waiting for unfinished jobs....
> make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge'
> Makefile:152: recipe for target 'sub-make' failed
> make: *** [sub-make] Error 2
> 
> 
> 
> 
> 
> I was able to compile it in-tree
> but it makes the incremental build extremely slow.
> 
> (Here, the incremental build means
> "make" without changing any code after the full build.)
> 
> 
> 
> 
> Before this patch, "make -j8" took 11 sec on my machine.
> 
> real 0m11.777s
> user 0m16.608s
> sys 0m5.164s
> 
> 
> 
> After this patch, x86_64_defconfig + CONFIG_IKHEADERS_PROC=y
> takes 53 sec for me since kernel/kheaders_data.txz is regenerated
> every time even when you did not touch any source file.

Hmm, true. Let me know look into that as well..


> $ time make -j8
>   DESCEND  objtool
>   CALL    scripts/checksyscalls.sh
>   CHK     include/generated/compile.h
>   GEN     kernel/kheaders_data.txz
>   UPD     kernel/kheaders_data.h
>   CC      kernel/kheaders.o
>   AR      kernel/built-in.a
>   GEN     .version
>   CHK     include/generated/compile.h
>   UPD     include/generated/compile.h
>   CC      init/version.o
>   AR      init/built-in.a
>   AR      built-in.a
>   LD      vmlinux.o
>   MODPOST vmlinux.o
>   KSYM    .tmp_kallsyms1.o
>   KSYM    .tmp_kallsyms2.o
>   LD      vmlinux
>   SORTEX  vmlinux
>   SYSMAP  System.map
>   Building modules, stage 2.
>   CC      arch/x86/boot/version.o
>   MODPOST 17 modules
>   VOFFSET arch/x86/boot/compressed/../voffset.h
>   OBJCOPY arch/x86/boot/compressed/vmlinux.bin
>   RELOCS  arch/x86/boot/compressed/vmlinux.relocs
>   CC      arch/x86/boot/compressed/kaslr.o
>   GZIP    arch/x86/boot/compressed/vmlinux.bin.gz
>   CC      arch/x86/boot/compressed/misc.o
>   MKPIGGY arch/x86/boot/compressed/piggy.S
>   AS      arch/x86/boot/compressed/piggy.o
>   LD      arch/x86/boot/compressed/vmlinux
>   ZOFFSET arch/x86/boot/zoffset.h
>   OBJCOPY arch/x86/boot/vmlinux.bin
>   AS      arch/x86/boot/header.o
>   LD      arch/x86/boot/setup.elf
>   OBJCOPY arch/x86/boot/setup.bin
>   BUILD   arch/x86/boot/bzImage
> Setup is 15612 bytes (padded to 15872 bytes).
> System is 12673 kB
> CRC 697aaf88
> Kernel: arch/x86/boot/bzImage is ready  (#6)
> 
> real 0m53.024s
> user 0m32.076s
> sys 0m9.296s
> 
> 
> 
> 
> Also, I notice $(ARCH) must be fixed to $(SRCARCH),
> but that is one of minor issues.

Ok.

> We should take time for careful review and test.

Yes, I agree.

> Please give me more time for thorough review.

Sure. Thanks a lot for the feedback provided so far. I will send another
revision soon with all the suggestions addressed.

thanks,

 - Joel

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Masahiro Yamada]

On Tue, Feb 19, 2019 at 1:14 PM Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
>
> On Fri, Feb 15, 2019 at 11:48 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > > Introduce in-kernel headers and other artifacts which are made available
> > > as an archive through proc (/proc/kheaders.txz file).
>
>
> The extension '.txz' is not used in kernel code.
>
>
> '.tar.xz' is used for 'tarxz-pkg', 'perf-tarxz-src-pkg' etc.
>
>
> $ git grep  '\.txz'
> $ git grep  '\.tar\.xz'
> Documentation/admin-guide/README.rst:     xz -cd linux-4.X.tar.xz | tar xvf -
> arch/x86/crypto/camellia-aesni-avx-asm_64.S: *
> http://koti.mbnet.fi/axh/crypto/camellia-BSD-1.2.0-aesni1.tar.xz
> crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> scripts/package/Makefile:       @echo '  perf-tarxz-src-pkg  - Build
> $(perf-tar).tar.xz source tarball'
> tools/testing/selftests/gen_kselftest_tar.sh:
>  ext=".tar.xz"
>
>
>
> I prefer '.tar.xz' for consistency.
>
>
>
>
>
>
> BTW, have you ever looked at scripts/extract-ikconfig?
>
> You added IKHD_ST and IKHD_ED
> just to mimic kernel/configs.c
>
> It is currently pointless without the extracting tool,
> but you might think it is useful to extract headers
> from vmlinux or the module without mounting procfs.
>
>
>
>
> > > This archive makes
> > > it possible to build kernel modules, run eBPF programs, and other
> > > tracing programs that need to extend the kernel for tracing purposes
> > > without any dependency on the file system having headers and build
> > > artifacts.
> > >
> > > On Android and embedded systems, it is common to switch kernels but not
> > > have kernel headers available on the file system. Raw kernel headers
> > > also cannot be copied into the filesystem like they can be on other
> > > distros, due to licensing and other issues. There's no linux-headers
> > > package on Android. Further once a different kernel is booted, any
> > > headers stored on the file system will no longer be useful. By storing
> > > the headers as a compressed archive within the kernel, we can avoid these
> > > issues that have been a hindrance for a long time.
> >
> > The set looks good to me and since the main use case is building bpf progs
> > I can route it via bpf-next tree if there are no objections.
> > Masahiro, could you please ack it?
>
>
> Honestly, I was not tracking this thread
> since I did not know I was responsible for this.
>
>
> I just started to take a closer look, then immediately got scared.
>
> This version is not mature enough for the merge.
>
>
>
> First of all, this patch cannot be compiled out-of-tree (O= option).
>
>
> I do not know why 0-day bot did not catch this apparent breakage.
>
>
> $ make -j8 O=hoge
> make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge'
>   GEN     Makefile
>   Using .. as source for kernel
>   DESCEND  objtool
>   CALL    ../scripts/checksyscalls.sh
>   CHK     include/generated/compile.h
> make[2]: *** No rule to make target 'Module.symvers', needed by
> 'kernel/kheaders_data.txz'.  Stop.
> make[2]: *** Waiting for unfinished jobs....
> /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target
> 'kernel' failed
> make[1]: *** [kernel] Error 2
> make[1]: *** Waiting for unfinished jobs....
> make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge'
> Makefile:152: recipe for target 'sub-make' failed
> make: *** [sub-make] Error 2




I saw this build error for in-tree building as well.

We cannot build this from a pristine source tree.

For example, I observed the build error in the following procedure.

$ make mrproper
$ make defconfig
 Set CONFIG_IKHEADERS_PROC=y
$ make




Module.symvers is generated in the modpost stage
(the very last stage of build).

But, vmlinux depends on kernel/kheaders_data.txz,
which includes Module.symvers.

So, this is not so simple since it is a circular dependency...




-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Tue, Feb 19, 2019 at 01:42:13PM +0900, Masahiro Yamada wrote:
[..]
> > > > This archive makes
> > > > it possible to build kernel modules, run eBPF programs, and other
> > > > tracing programs that need to extend the kernel for tracing purposes
> > > > without any dependency on the file system having headers and build
> > > > artifacts.
> > > >
> > > > On Android and embedded systems, it is common to switch kernels but not
> > > > have kernel headers available on the file system. Raw kernel headers
> > > > also cannot be copied into the filesystem like they can be on other
> > > > distros, due to licensing and other issues. There's no linux-headers
> > > > package on Android. Further once a different kernel is booted, any
> > > > headers stored on the file system will no longer be useful. By storing
> > > > the headers as a compressed archive within the kernel, we can avoid these
> > > > issues that have been a hindrance for a long time.
> > >
> > > The set looks good to me and since the main use case is building bpf progs
> > > I can route it via bpf-next tree if there are no objections.
> > > Masahiro, could you please ack it?
> >
> >
> > Honestly, I was not tracking this thread
> > since I did not know I was responsible for this.
> >
> >
> > I just started to take a closer look, then immediately got scared.
> >
> > This version is not mature enough for the merge.
> >
> >
> >
> > First of all, this patch cannot be compiled out-of-tree (O= option).
> >
> >
> > I do not know why 0-day bot did not catch this apparent breakage.
> >
> >
> > $ make -j8 O=hoge
> > make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge'
> >   GEN     Makefile
> >   Using .. as source for kernel
> >   DESCEND  objtool
> >   CALL    ../scripts/checksyscalls.sh
> >   CHK     include/generated/compile.h
> > make[2]: *** No rule to make target 'Module.symvers', needed by
> > 'kernel/kheaders_data.txz'.  Stop.
> > make[2]: *** Waiting for unfinished jobs....
> > /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target
> > 'kernel' failed
> > make[1]: *** [kernel] Error 2
> > make[1]: *** Waiting for unfinished jobs....
> > make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge'
> > Makefile:152: recipe for target 'sub-make' failed
> > make: *** [sub-make] Error 2
> 
> 
> 
> 
> I saw this build error for in-tree building as well.
> 
> We cannot build this from a pristine source tree.
> 
> For example, I observed the build error in the following procedure.
> 
> $ make mrproper
> $ make defconfig
>  Set CONFIG_IKHEADERS_PROC=y
> $ make
> 
> 
> 
> 
> Module.symvers is generated in the modpost stage
> (the very last stage of build).
> 
> But, vmlinux depends on kernel/kheaders_data.txz,
> which includes Module.symvers.
> 
> So, this is not so simple since it is a circular dependency...

I guess I was not building a pristine tree either and missed this circular
dependency :-/ . Any ideas on how we can fix the Module.symvers issue? One
idea is to reserve the space in the binaries, but only populate the space
reserved in the binary *after* the modpost stage, once the archive is ready..

thanks,

 - Joel

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Tue, Feb 19, 2019 at 01:42:13PM +0900, Masahiro Yamada wrote:
> On Tue, Feb 19, 2019 at 1:14 PM Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
> >
> > On Fri, Feb 15, 2019 at 11:48 PM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> > >
> > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > > > Introduce in-kernel headers and other artifacts which are made available
> > > > as an archive through proc (/proc/kheaders.txz file).
> >
> >
> > The extension '.txz' is not used in kernel code.
> >
> >
> > '.tar.xz' is used for 'tarxz-pkg', 'perf-tarxz-src-pkg' etc.
> >
> >
> > $ git grep  '\.txz'
> > $ git grep  '\.tar\.xz'
> > Documentation/admin-guide/README.rst:     xz -cd linux-4.X.tar.xz | tar xvf -
> > arch/x86/crypto/camellia-aesni-avx-asm_64.S: *
> > http://koti.mbnet.fi/axh/crypto/camellia-BSD-1.2.0-aesni1.tar.xz
> > crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> > crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> > crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> > crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> > crypto/testmgr.h: *   https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
> > scripts/package/Makefile:       @echo '  perf-tarxz-src-pkg  - Build
> > $(perf-tar).tar.xz source tarball'
> > tools/testing/selftests/gen_kselftest_tar.sh:
> >  ext=".tar.xz"
> >
> >
> >
> > I prefer '.tar.xz' for consistency.
> >
> >
> >
> >
> >
> >
> > BTW, have you ever looked at scripts/extract-ikconfig?
> >
> > You added IKHD_ST and IKHD_ED
> > just to mimic kernel/configs.c
> >
> > It is currently pointless without the extracting tool,
> > but you might think it is useful to extract headers
> > from vmlinux or the module without mounting procfs.
> >
> >
> >
> >
> > > > This archive makes
> > > > it possible to build kernel modules, run eBPF programs, and other
> > > > tracing programs that need to extend the kernel for tracing purposes
> > > > without any dependency on the file system having headers and build
> > > > artifacts.
> > > >
> > > > On Android and embedded systems, it is common to switch kernels but not
> > > > have kernel headers available on the file system. Raw kernel headers
> > > > also cannot be copied into the filesystem like they can be on other
> > > > distros, due to licensing and other issues. There's no linux-headers
> > > > package on Android. Further once a different kernel is booted, any
> > > > headers stored on the file system will no longer be useful. By storing
> > > > the headers as a compressed archive within the kernel, we can avoid these
> > > > issues that have been a hindrance for a long time.
> > >
> > > The set looks good to me and since the main use case is building bpf progs
> > > I can route it via bpf-next tree if there are no objections.
> > > Masahiro, could you please ack it?
> >
> >
> > Honestly, I was not tracking this thread
> > since I did not know I was responsible for this.
> >
> >
> > I just started to take a closer look, then immediately got scared.
> >
> > This version is not mature enough for the merge.
> >
> >
> >
> > First of all, this patch cannot be compiled out-of-tree (O= option).
> >
> >
> > I do not know why 0-day bot did not catch this apparent breakage.
> >
> >
> > $ make -j8 O=hoge
> > make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge'
> >   GEN     Makefile
> >   Using .. as source for kernel
> >   DESCEND  objtool
> >   CALL    ../scripts/checksyscalls.sh
> >   CHK     include/generated/compile.h
> > make[2]: *** No rule to make target 'Module.symvers', needed by
> > 'kernel/kheaders_data.txz'.  Stop.
> > make[2]: *** Waiting for unfinished jobs....
> > /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target
> > 'kernel' failed
> > make[1]: *** [kernel] Error 2
> > make[1]: *** Waiting for unfinished jobs....
> > make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge'
> > Makefile:152: recipe for target 'sub-make' failed
> > make: *** [sub-make] Error 2
> 
> 
> 
> 
> I saw this build error for in-tree building as well.
> 
> We cannot build this from a pristine source tree.
> 
> For example, I observed the build error in the following procedure.
> 
> $ make mrproper
> $ make defconfig
>  Set CONFIG_IKHEADERS_PROC=y
> $ make
> 
> 
> 
> 
> Module.symvers is generated in the modpost stage
> (the very last stage of build).
> 
> But, vmlinux depends on kernel/kheaders_data.txz,
> which includes Module.symvers.

Firstly, I want to apologize for not testing this and other corner cases you
brought up. I should have known better. Since my build was working, I assumed
that the feature is working. For that, I am very sorry.

Secondly, it turns out Module.symvers circularly dependency problem also
exists with another use case.
If one does 'make modules_prepare' in a base kernel tree and then tries to
build modules with that tree, a warning like this is printed but the module
still gets built:

  WARNING: Symbol version dump ./Module.symvers
           is missing; modules will have no dependencies and modversions.

  CC [M]  /tmp/testmod/test.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /tmp/testmod/test.mod.o
  LD [M]  /tmp/testmod/test.ko

So, I am thinking that at least for first pass I will just drop the inclusion
of Module.symvers in the archive and allow any modules built using
/proc/kheaders.tar.xz to not use it.

Kbuild will print a warning anyway when anyone tries to build using
/proc/kheaders.tar.xz, so if the user really wants module symbol versioning
then they should probably use a full kernel source tree with Module.symvers
available. For our usecase, kernel symbol versioning is a bit useless when
using /proc/kheaders.tar.gz because the proc file is generated with the same
kernel that the module is being built against, and subsequently loaded into
the kernel. So it is not likely that the CRC of a kernel symbol will be
different from what the module expects.

I can't think any other ways at the moment to break the circular dependency
so I'm thinking this is good enough for now especially since Kbuild will
print a proper warning. Let me know what you think?

thanks,

 - Joel

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Masahiro Yamada]

On Wed, Feb 20, 2019 at 12:17 AM Joel Fernandes <joel@joelfernandes.org> wrote:

>
> Firstly, I want to apologize for not testing this and other corner cases you
> brought up. I should have known better. Since my build was working, I assumed
> that the feature is working. For that, I am very sorry.


You do not need to apologize. 0day bot usually catches build errors.
I guess 0day bot performs compile-tests only incrementally
and that is why we did not get any report.



> Secondly, it turns out Module.symvers circularly dependency problem also
> exists with another use case.
> If one does 'make modules_prepare' in a base kernel tree and then tries to
> build modules with that tree, a warning like this is printed but the module
> still gets built:
>
>   WARNING: Symbol version dump ./Module.symvers
>            is missing; modules will have no dependencies and modversions.
>
>   CC [M]  /tmp/testmod/test.o
>   Building modules, stage 2.
>   MODPOST 1 modules
>   CC      /tmp/testmod/test.mod.o
>   LD [M]  /tmp/testmod/test.ko
>
> So, I am thinking that at least for first pass I will just drop the inclusion
> of Module.symvers in the archive and allow any modules built using
> /proc/kheaders.tar.xz to not use it.
>
> Kbuild will print a warning anyway when anyone tries to build using
> /proc/kheaders.tar.xz, so if the user really wants module symbol versioning
> then they should probably use a full kernel source tree with Module.symvers
> available. For our usecase, kernel symbol versioning is a bit useless when
> using /proc/kheaders.tar.gz because the proc file is generated with the same
> kernel that the module is being built against, and subsequently loaded into
> the kernel. So it is not likely that the CRC of a kernel symbol will be
> different from what the module expects.


Without Module.symver, modpost cannot check whether references are
resolvable or not.

You will see "WARNING ... undefined" for every symbol referenced from
the module.


I am not an Android developer.
So, I will leave this judge to other people.




One more request if you have a chance to submit the next version.
Please do not hide error messages.

I wondered why you redirected stdout/stderr from the script.

I applied the following patch, and I tested.  Then I see why.

Please fix your code instead of hiding underlying problems.


diff --git a/kernel/Makefile b/kernel/Makefile
index 1d13a7a..a76ccbd 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -148,7 +148,7 @@ $(obj)/kheaders.o: $(obj)/kheaders_data.h
 targets += kheaders_data.txz

 quiet_cmd_genikh = GEN     $(obj)/kheaders_data.txz
-cmd_genikh = $(srctree)/scripts/gen_ikh_data.sh $@ $^ >/dev/null 2>&1
+cmd_genikh = $(srctree)/scripts/gen_ikh_data.sh $@ $^
 $(obj)/kheaders_data.txz: $(ikh_file_list) FORCE
        $(call cmd,genikh)






masahiro@grover:~/workspace/linux-yamada$ make
  CALL    scripts/checksyscalls.sh
  DESCEND  objtool
  CHK     include/generated/compile.h
  GEN     kernel/kheaders_data.txz
find: ‘FORCE’: No such file or directory
70106 blocks
Can't do inplace edit: kernel/kheaders_data.txz.tmp is not a regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch is not a regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86 is not a
regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include
is not a regular file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/uapi is not a regular
file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/uapi/asm is not a
regular file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/generated is not a
regular file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/generated/uapi is not a
regular file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/generated/uapi/asm is
not a regular file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/generated/asm is not a
regular file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/asm is not a regular
file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/asm/xen is not a regular
file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/asm/uv is not a regular
file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/asm/numachip is not a
regular file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/asm/e820 is not a
regular file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/asm/fpu is not a regular
file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/asm/crypto is not a
regular file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/arch/x86/include/asm/trace is not a
regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts is not a
regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/genksyms
is not a regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/ksymoops
is not a regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/gdb is not
a regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/gdb/linux
is not a regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/basic is
not a regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/dtc is not
a regular file.
Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/dtc/libfdt
is not a regular file.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes is not a
regular file.
Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/arm64:
No such file or directory.
Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/xtensa:
No such file or directory.
Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/openrisc:
No such file or directory.
Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/nios2:
No such file or directory.
Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/mips:
No such file or directory.
Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/arm:
No such file or directory.
Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/microblaze:
No such file or directory.
Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/arc:
No such file or directory.
Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/sh:
No such file or directory.
Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/powerpc:
No such file or directory.
Can't do inplace edit:
kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/dt-bindings
is not a regular file.

  [ massive amount of error messages continues ]






> I can't think any other ways at the moment to break the circular dependency
> so I'm thinking this is good enough for now especially since Kbuild will
> print a proper warning. Let me know what you think?
>
> thanks,
>
>  - Joel
>
--
Best Regards
Masahiro Yamada

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Thu, Feb 21, 2019 at 11:34:41PM +0900, Masahiro Yamada wrote:
> On Wed, Feb 20, 2019 at 12:17 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> 
> >
> > Firstly, I want to apologize for not testing this and other corner cases you
> > brought up. I should have known better. Since my build was working, I assumed
> > that the feature is working. For that, I am very sorry.
> 
> 
> You do not need to apologize. 0day bot usually catches build errors.
> I guess 0day bot performs compile-tests only incrementally
> and that is why we did not get any report.

Oh ok :) thanks.

> > Secondly, it turns out Module.symvers circularly dependency problem also
> > exists with another use case.
> > If one does 'make modules_prepare' in a base kernel tree and then tries to
> > build modules with that tree, a warning like this is printed but the module
> > still gets built:
> >
> >   WARNING: Symbol version dump ./Module.symvers
> >            is missing; modules will have no dependencies and modversions.
> >
> >   CC [M]  /tmp/testmod/test.o
> >   Building modules, stage 2.
> >   MODPOST 1 modules
> >   CC      /tmp/testmod/test.mod.o
> >   LD [M]  /tmp/testmod/test.ko
> >
> > So, I am thinking that at least for first pass I will just drop the inclusion
> > of Module.symvers in the archive and allow any modules built using
> > /proc/kheaders.tar.xz to not use it.
> >
> > Kbuild will print a warning anyway when anyone tries to build using
> > /proc/kheaders.tar.xz, so if the user really wants module symbol versioning
> > then they should probably use a full kernel source tree with Module.symvers
> > available. For our usecase, kernel symbol versioning is a bit useless when
> > using /proc/kheaders.tar.gz because the proc file is generated with the same
> > kernel that the module is being built against, and subsequently loaded into
> > the kernel. So it is not likely that the CRC of a kernel symbol will be
> > different from what the module expects.
> 
> 
> Without Module.symver, modpost cannot check whether references are
> resolvable or not.
> 
> You will see "WARNING ... undefined" for every symbol referenced from
> the module.
> 
> 
> I am not an Android developer.
> So, I will leave this judge to other people.

IMO I don't see a way around this limiation but it would be nice if there was
a way to make it work. Since the kernel modules being built by this mechanism
are for tracing/debugging purposes, it is not a major concern for us.


> One more request if you have a chance to submit the next version.
> Please do not hide error messages.

Actually it was intended to suppress noise, not hide errors as such. I have
fixed all the errors in the next version and will be submitting it soon.

Thanks a lot for the review!

 - Joel


> I wondered why you redirected stdout/stderr from the script.
> 
> I applied the following patch, and I tested.  Then I see why.
> 
> Please fix your code instead of hiding underlying problems.
> 
> 
> diff --git a/kernel/Makefile b/kernel/Makefile
> index 1d13a7a..a76ccbd 100644
> --- a/kernel/Makefile
> +++ b/kernel/Makefile
> @@ -148,7 +148,7 @@ $(obj)/kheaders.o: $(obj)/kheaders_data.h
>  targets += kheaders_data.txz
> 
>  quiet_cmd_genikh = GEN     $(obj)/kheaders_data.txz
> -cmd_genikh = $(srctree)/scripts/gen_ikh_data.sh $@ $^ >/dev/null 2>&1
> +cmd_genikh = $(srctree)/scripts/gen_ikh_data.sh $@ $^
>  $(obj)/kheaders_data.txz: $(ikh_file_list) FORCE
>         $(call cmd,genikh)
> 
> 
> 
> 
> 
> 
> masahiro@grover:~/workspace/linux-yamada$ make
>   CALL    scripts/checksyscalls.sh
>   DESCEND  objtool
>   CHK     include/generated/compile.h
>   GEN     kernel/kheaders_data.txz
> find: ‘FORCE’: No such file or directory
> 70106 blocks
> Can't do inplace edit: kernel/kheaders_data.txz.tmp is not a regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch is not a regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86 is not a
> regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/arch/x86/include
> is not a regular file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/uapi is not a regular
> file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/uapi/asm is not a
> regular file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/generated is not a
> regular file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/generated/uapi is not a
> regular file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/generated/uapi/asm is
> not a regular file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/generated/asm is not a
> regular file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/asm is not a regular
> file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/asm/xen is not a regular
> file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/asm/uv is not a regular
> file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/asm/numachip is not a
> regular file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/asm/e820 is not a
> regular file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/asm/fpu is not a regular
> file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/asm/crypto is not a
> regular file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/arch/x86/include/asm/trace is not a
> regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts is not a
> regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/genksyms
> is not a regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/ksymoops
> is not a regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/gdb is not
> a regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/gdb/linux
> is not a regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/basic is
> not a regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/dtc is not
> a regular file.
> Can't do inplace edit: kernel/kheaders_data.txz.tmp/scripts/dtc/libfdt
> is not a regular file.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes is not a
> regular file.
> Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/arm64:
> No such file or directory.
> Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/xtensa:
> No such file or directory.
> Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/openrisc:
> No such file or directory.
> Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/nios2:
> No such file or directory.
> Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/mips:
> No such file or directory.
> Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/arm:
> No such file or directory.
> Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/microblaze:
> No such file or directory.
> Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/arc:
> No such file or directory.
> Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/sh:
> No such file or directory.
> Can't open kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/powerpc:
> No such file or directory.
> Can't do inplace edit:
> kernel/kheaders_data.txz.tmp/scripts/dtc/include-prefixes/dt-bindings
> is not a regular file.
> 
>   [ massive amount of error messages continues ]
> 
> 
> 
> 
> 
> 
> > I can't think any other ways at the moment to break the circular dependency
> > so I'm thinking this is good enough for now especially since Kbuild will
> > print a proper warning. Let me know what you think?
> >
> > thanks,
> >
> >  - Joel
> >
> --
> Best Regards
> Masahiro Yamada

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote:
> On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > Introduce in-kernel headers and other artifacts which are made available
> > as an archive through proc (/proc/kheaders.txz file). This archive makes
> > it possible to build kernel modules, run eBPF programs, and other
> > tracing programs that need to extend the kernel for tracing purposes
> > without any dependency on the file system having headers and build
> > artifacts.
> > 
> > On Android and embedded systems, it is common to switch kernels but not
> > have kernel headers available on the file system. Raw kernel headers
> > also cannot be copied into the filesystem like they can be on other
> > distros, due to licensing and other issues. There's no linux-headers
> > package on Android. Further once a different kernel is booted, any
> > headers stored on the file system will no longer be useful. By storing
> > the headers as a compressed archive within the kernel, we can avoid these
> > issues that have been a hindrance for a long time.
> 
> The set looks good to me and since the main use case is building bpf progs
> I can route it via bpf-next tree if there are no objections.
> Masahiro, could you please ack it?

FYI, Masahiro's comments were all address by v5:
https://lore.kernel.org/patchwork/project/lkml/list/?series=387311

I believe aren't more outstanding concerns. Could we consider it for v5.2?

thanks,

 - Joel

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Mon, Mar 25, 2019 at 09:49:47AM -0400, Joel Fernandes wrote:
> On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote:
> > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > > Introduce in-kernel headers and other artifacts which are made available
> > > as an archive through proc (/proc/kheaders.txz file). This archive makes
> > > it possible to build kernel modules, run eBPF programs, and other
> > > tracing programs that need to extend the kernel for tracing purposes
> > > without any dependency on the file system having headers and build
> > > artifacts.
> > > 
> > > On Android and embedded systems, it is common to switch kernels but not
> > > have kernel headers available on the file system. Raw kernel headers
> > > also cannot be copied into the filesystem like they can be on other
> > > distros, due to licensing and other issues. There's no linux-headers
> > > package on Android. Further once a different kernel is booted, any
> > > headers stored on the file system will no longer be useful. By storing
> > > the headers as a compressed archive within the kernel, we can avoid these
> > > issues that have been a hindrance for a long time.
> > 
> > The set looks good to me and since the main use case is building bpf progs
> > I can route it via bpf-next tree if there are no objections.
> > Masahiro, could you please ack it?
> 
> FYI, Masahiro's comments were all address by v5:
> https://lore.kernel.org/patchwork/project/lkml/list/?series=387311
> 
> I believe aren't more outstanding concerns. Could we consider it for v5.2?

Just to highlight the problem, today I booted v5.0 on an emulated Android
device for some testing, that didn't have a set of prebuilt headers that we
have been packaging on well known kernels, to get around this issue. This
caused great pain and issues with what I was doing. I know me and many others
really want this. With this I can boot an emulated Android device with
IKCONFIG_PROC=y and run BCC with that that. Also I want to do the BCC side of
the development, but first want to know if we can merge this upstream.

Masahiro, I believe due diligence has been done in solidifying it as posted
in the v5.  Anything else we need to do here? Are you with the patches?

thanks!

 - Joel

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Masahiro Yamada]

On Thu, Mar 28, 2019 at 2:32 AM Joel Fernandes <joel@joelfernandes.org> wrote:
>
> On Mon, Mar 25, 2019 at 09:49:47AM -0400, Joel Fernandes wrote:
> > On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote:
> > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > > > Introduce in-kernel headers and other artifacts which are made available
> > > > as an archive through proc (/proc/kheaders.txz file). This archive makes
> > > > it possible to build kernel modules, run eBPF programs, and other
> > > > tracing programs that need to extend the kernel for tracing purposes
> > > > without any dependency on the file system having headers and build
> > > > artifacts.
> > > >
> > > > On Android and embedded systems, it is common to switch kernels but not
> > > > have kernel headers available on the file system. Raw kernel headers
> > > > also cannot be copied into the filesystem like they can be on other
> > > > distros, due to licensing and other issues. There's no linux-headers
> > > > package on Android. Further once a different kernel is booted, any
> > > > headers stored on the file system will no longer be useful. By storing
> > > > the headers as a compressed archive within the kernel, we can avoid these
> > > > issues that have been a hindrance for a long time.
> > >
> > > The set looks good to me and since the main use case is building bpf progs
> > > I can route it via bpf-next tree if there are no objections.
> > > Masahiro, could you please ack it?
> >
> > FYI, Masahiro's comments were all address by v5:
> > https://lore.kernel.org/patchwork/project/lkml/list/?series=387311
> >
> > I believe aren't more outstanding concerns. Could we consider it for v5.2?
>
> Just to highlight the problem, today I booted v5.0 on an emulated Android
> device for some testing, that didn't have a set of prebuilt headers that we
> have been packaging on well known kernels, to get around this issue. This
> caused great pain and issues with what I was doing. I know me and many others
> really want this. With this I can boot an emulated Android device with
> IKCONFIG_PROC=y and run BCC with that that. Also I want to do the BCC side of
> the development, but first want to know if we can merge this upstream.
>
> Masahiro, I believe due diligence has been done in solidifying it as posted
> in the v5.  Anything else we need to do here? Are you with the patches?


As you said, these updates are "cosmetic".
Nobody is worried about (or interested in) them.
Even if we miss something, they are fixable later.

I accept embedding headers in the kernel,
but I do not like the part about external module build.
 - kernel embeds build artifacts that depend on a
   particular host-arch
 - users do not know which compiler to use

Perhaps, you may remember my concerns.
https://lore.kernel.org/patchwork/patch/1046307/#1239501

I reviewed this, and already expressed my opinion,
so I finished all job I can do.

Anyway, you believe this approach is solid.
All that is left is somebody makes a decision.
Already you are asking this to Andrew.
Good luck!



--
Best Regards
Masahiro Yamada

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Wed, Apr 03, 2019 at 04:48:37PM +0900, Masahiro Yamada wrote:
> On Thu, Mar 28, 2019 at 2:32 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> >
> > On Mon, Mar 25, 2019 at 09:49:47AM -0400, Joel Fernandes wrote:
> > > On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote:
> > > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > > > > Introduce in-kernel headers and other artifacts which are made available
> > > > > as an archive through proc (/proc/kheaders.txz file). This archive makes
> > > > > it possible to build kernel modules, run eBPF programs, and other
> > > > > tracing programs that need to extend the kernel for tracing purposes
> > > > > without any dependency on the file system having headers and build
> > > > > artifacts.
> > > > >
> > > > > On Android and embedded systems, it is common to switch kernels but not
> > > > > have kernel headers available on the file system. Raw kernel headers
> > > > > also cannot be copied into the filesystem like they can be on other
> > > > > distros, due to licensing and other issues. There's no linux-headers
> > > > > package on Android. Further once a different kernel is booted, any
> > > > > headers stored on the file system will no longer be useful. By storing
> > > > > the headers as a compressed archive within the kernel, we can avoid these
> > > > > issues that have been a hindrance for a long time.
> > > >
> > > > The set looks good to me and since the main use case is building bpf progs
> > > > I can route it via bpf-next tree if there are no objections.
> > > > Masahiro, could you please ack it?
> > >
> > > FYI, Masahiro's comments were all address by v5:
> > > https://lore.kernel.org/patchwork/project/lkml/list/?series=387311
> > >
> > > I believe aren't more outstanding concerns. Could we consider it for v5.2?
> >
> > Just to highlight the problem, today I booted v5.0 on an emulated Android
> > device for some testing, that didn't have a set of prebuilt headers that we
> > have been packaging on well known kernels, to get around this issue. This
> > caused great pain and issues with what I was doing. I know me and many others
> > really want this. With this I can boot an emulated Android device with
> > IKCONFIG_PROC=y and run BCC with that that. Also I want to do the BCC side of
> > the development, but first want to know if we can merge this upstream.
> >
> > Masahiro, I believe due diligence has been done in solidifying it as posted
> > in the v5.  Anything else we need to do here? Are you with the patches?
> 
> 
> As you said, these updates are "cosmetic".
> Nobody is worried about (or interested in) them.
> Even if we miss something, they are fixable later.
> 
> I accept embedding headers in the kernel,
> but I do not like the part about external module build.
>  - kernel embeds build artifacts that depend on a
>    particular host-arch
>  - users do not know which compiler to use
> Perhaps, you may remember my concerns.
> https://lore.kernel.org/patchwork/patch/1046307/#1239501

Masahiro,
I think we already discussed this right. The compiler issue is a user-issue -
it is not a problem that has arisen because this patch. Anyone can build a
kernel module today without this patch - using a compiler that is different
from the running kernel. So I did not fully understand your concern. This
patch does not ship a compiler in the archive. The compiler is upto the user
based on user environment. They can already shoot themself in foot without
this patch.

Greg,
Would be great to get your thoughts here as well about Masami's concern.

Honestly, the "kernel module building artifacts" bit is a bonus with this
patch. The main thing we are adding or that we need is really the headers
(for BCC). I just thought shipping the kernel build artifacts would also be
awesome because people can now build kernel modules without needing a kernel
tree at all.

But I also want this stuff merged so if folks are really unhappy with the
module build artifacts and only want headers - then that's also fine with me
and I can yank them - that would also mean though that this work cannot be a
replacement for linux-headers package on distros, so that would be sad.

thanks!

- Joel

> 
> --
> Best Regards
> Masahiro Yamada

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Daniel Colascione]

On Wed, Apr 3, 2019 at 10:20 AM Joel Fernandes <joel@joelfernandes.org> wrote:
>
> On Wed, Apr 03, 2019 at 04:48:37PM +0900, Masahiro Yamada wrote:
> > On Thu, Mar 28, 2019 at 2:32 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> > >
> > > On Mon, Mar 25, 2019 at 09:49:47AM -0400, Joel Fernandes wrote:
> > > > On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote:
> > > > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > > > > > Introduce in-kernel headers and other artifacts which are made available
> > > > > > as an archive through proc (/proc/kheaders.txz file). This archive makes
> > > > > > it possible to build kernel modules, run eBPF programs, and other
> > > > > > tracing programs that need to extend the kernel for tracing purposes
> > > > > > without any dependency on the file system having headers and build
> > > > > > artifacts.
> > > > > >
> > > > > > On Android and embedded systems, it is common to switch kernels but not
> > > > > > have kernel headers available on the file system. Raw kernel headers
> > > > > > also cannot be copied into the filesystem like they can be on other
> > > > > > distros, due to licensing and other issues. There's no linux-headers
> > > > > > package on Android. Further once a different kernel is booted, any
> > > > > > headers stored on the file system will no longer be useful. By storing
> > > > > > the headers as a compressed archive within the kernel, we can avoid these
> > > > > > issues that have been a hindrance for a long time.
> > > > >
> > > > > The set looks good to me and since the main use case is building bpf progs
> > > > > I can route it via bpf-next tree if there are no objections.
> > > > > Masahiro, could you please ack it?
> > > >
> > > > FYI, Masahiro's comments were all address by v5:
> > > > https://lore.kernel.org/patchwork/project/lkml/list/?series=387311
> > > >
> > > > I believe aren't more outstanding concerns. Could we consider it for v5.2?
> > >
> > > Just to highlight the problem, today I booted v5.0 on an emulated Android
> > > device for some testing, that didn't have a set of prebuilt headers that we
> > > have been packaging on well known kernels, to get around this issue. This
> > > caused great pain and issues with what I was doing. I know me and many others
> > > really want this. With this I can boot an emulated Android device with
> > > IKCONFIG_PROC=y and run BCC with that that. Also I want to do the BCC side of
> > > the development, but first want to know if we can merge this upstream.
> > >
> > > Masahiro, I believe due diligence has been done in solidifying it as posted
> > > in the v5.  Anything else we need to do here? Are you with the patches?
> >
> >
> > As you said, these updates are "cosmetic".
> > Nobody is worried about (or interested in) them.
> > Even if we miss something, they are fixable later.
> >
> > I accept embedding headers in the kernel,
> > but I do not like the part about external module build.
> >  - kernel embeds build artifacts that depend on a
> >    particular host-arch
> >  - users do not know which compiler to use
> > Perhaps, you may remember my concerns.
> > https://lore.kernel.org/patchwork/patch/1046307/#1239501
>
> Masahiro,
> I think we already discussed this right. The compiler issue is a user-issue -
> it is not a problem that has arisen because this patch. Anyone can build a
> kernel module today without this patch - using a compiler that is different
> from the running kernel. So I did not fully understand your concern. This
> patch does not ship a compiler in the archive. The compiler is upto the user
> based on user environment. They can already shoot themself in foot without
> this patch.
>
> Greg,
> Would be great to get your thoughts here as well about Masami's concern.
>
> Honestly, the "kernel module building artifacts" bit is a bonus with this
> patch. The main thing we are adding or that we need is really the headers
> (for BCC). I just thought shipping the kernel build artifacts would also be
> awesome because people can now build kernel modules without needing a kernel
> tree at all.
>
> But I also want this stuff merged so if folks are really unhappy with the
> module build artifacts and only want headers - then that's also fine with me
> and I can yank them - that would also mean though that this work cannot be a
> replacement for linux-headers package on distros, so that would be sad.
>
> thanks!

IMHO, including the module build stuff is fine, and the stuff that
Masahiro mentions doesn't matter much. To be specific about the
concerns:

>> > >  - kernel embeds build artifacts that depend on a
> >    particular host-arch

What are these artifacts? We build the kernel as a standalone system.
It's not as if we statically link host glibc into it or something.

> >  - users do not know which compiler to use

Does that matter much? Do things ever break if the kernel proper is
built with GCC x.y.z and we build a module with GCC x.y.(z+1)? Why
would it? Structure layout is invariant across compiler version, or at
least ought to be.  And don't we have exactly the same problem with
any kernel headers package? I don't think it'd hurt to include the
output of $(CC) --version though.

Like Joel, I'd like to see this change merged. It'll make life easier
for a lot of people.

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Joel Fernandes]

On Wed, Apr 03, 2019 at 10:46:01AM -0700, Daniel Colascione wrote:
> On Wed, Apr 3, 2019 at 10:20 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> >
> > On Wed, Apr 03, 2019 at 04:48:37PM +0900, Masahiro Yamada wrote:
> > > On Thu, Mar 28, 2019 at 2:32 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> > > >
> > > > On Mon, Mar 25, 2019 at 09:49:47AM -0400, Joel Fernandes wrote:
> > > > > On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote:
> > > > > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > > > > > > Introduce in-kernel headers and other artifacts which are made available
> > > > > > > as an archive through proc (/proc/kheaders.txz file). This archive makes
> > > > > > > it possible to build kernel modules, run eBPF programs, and other
> > > > > > > tracing programs that need to extend the kernel for tracing purposes
> > > > > > > without any dependency on the file system having headers and build
> > > > > > > artifacts.
> > > > > > >
> > > > > > > On Android and embedded systems, it is common to switch kernels but not
> > > > > > > have kernel headers available on the file system. Raw kernel headers
> > > > > > > also cannot be copied into the filesystem like they can be on other
> > > > > > > distros, due to licensing and other issues. There's no linux-headers
> > > > > > > package on Android. Further once a different kernel is booted, any
> > > > > > > headers stored on the file system will no longer be useful. By storing
> > > > > > > the headers as a compressed archive within the kernel, we can avoid these
> > > > > > > issues that have been a hindrance for a long time.
> > > > > >
> > > > > > The set looks good to me and since the main use case is building bpf progs
> > > > > > I can route it via bpf-next tree if there are no objections.
> > > > > > Masahiro, could you please ack it?
> > > > >
> > > > > FYI, Masahiro's comments were all address by v5:
> > > > > https://lore.kernel.org/patchwork/project/lkml/list/?series=387311
> > > > >
> > > > > I believe aren't more outstanding concerns. Could we consider it for v5.2?
> > > >
> > > > Just to highlight the problem, today I booted v5.0 on an emulated Android
> > > > device for some testing, that didn't have a set of prebuilt headers that we
> > > > have been packaging on well known kernels, to get around this issue. This
> > > > caused great pain and issues with what I was doing. I know me and many others
> > > > really want this. With this I can boot an emulated Android device with
> > > > IKCONFIG_PROC=y and run BCC with that that. Also I want to do the BCC side of
> > > > the development, but first want to know if we can merge this upstream.
> > > >
> > > > Masahiro, I believe due diligence has been done in solidifying it as posted
> > > > in the v5.  Anything else we need to do here? Are you with the patches?
> > >
> > >
> > > As you said, these updates are "cosmetic".
> > > Nobody is worried about (or interested in) them.
> > > Even if we miss something, they are fixable later.
> > >
> > > I accept embedding headers in the kernel,
> > > but I do not like the part about external module build.
> > >  - kernel embeds build artifacts that depend on a
> > >    particular host-arch
> > >  - users do not know which compiler to use
> > > Perhaps, you may remember my concerns.
> > > https://lore.kernel.org/patchwork/patch/1046307/#1239501
> >
> > Masahiro,
> > I think we already discussed this right. The compiler issue is a user-issue -
> > it is not a problem that has arisen because this patch. Anyone can build a
> > kernel module today without this patch - using a compiler that is different
> > from the running kernel. So I did not fully understand your concern. This
> > patch does not ship a compiler in the archive. The compiler is upto the user
> > based on user environment. They can already shoot themself in foot without
> > this patch.
> >
> > Greg,
> > Would be great to get your thoughts here as well about Masami's concern.
> >
> > Honestly, the "kernel module building artifacts" bit is a bonus with this
> > patch. The main thing we are adding or that we need is really the headers
> > (for BCC). I just thought shipping the kernel build artifacts would also be
> > awesome because people can now build kernel modules without needing a kernel
> > tree at all.
> >
> > But I also want this stuff merged so if folks are really unhappy with the
> > module build artifacts and only want headers - then that's also fine with me
> > and I can yank them - that would also mean though that this work cannot be a
> > replacement for linux-headers package on distros, so that would be sad.
> >
> > thanks!
> 
> IMHO, including the module build stuff is fine, and the stuff that
> Masahiro mentions doesn't matter much. To be specific about the
> concerns:
> 
> >> > >  - kernel embeds build artifacts that depend on a
> > >    particular host-arch
> 
> What are these artifacts? We build the kernel as a standalone system.
> It's not as if we statically link host glibc into it or something.

There are some scripts/ that are built in the host-arch ABI. These are also put
in the archive because they are needed to build modules. So if you
cross-compile then the archive will have scripts/ that are in the host arch,
not the target arch. This makes it not possible to build modules on the
target using the archive. This is not much of an issue because I already
proved that such kernel modules can be built using a chroot in this thread:
https://lkml.org/lkml/2019/2/28/1313
And in the non cross-compile case, it will be immensely useful for distros..

> > >  - users do not know which compiler to use
> 
> Does that matter much? Do things ever break if the kernel proper is
> built with GCC x.y.z and we build a module with GCC x.y.(z+1)? Why
> would it? Structure layout is invariant across compiler version, or at
> least ought to be.  And don't we have exactly the same problem with
> any kernel headers package? I don't think it'd hurt to include the
> output of $(CC) --version though.

Apparently there were some issues in some posting where Greg said we don't
support anything but modules built with the same compiler as the kernel its
being loaded into, even if such modules do work in practice. I don't recall
the details, I can dig up that thread if you want. My point is, whether
that's an issue or not - it is not an issue introduced by this patch. That's
just a module building issue which we neither solve here, nor introduce.

> Like Joel, I'd like to see this change merged. It'll make life easier
> for a lot of people.

Thanks.

- Joel

Re: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel

[Masahiro Yamada]

On Thu, Apr 4, 2019 at 2:59 AM Joel Fernandes <joel@joelfernandes.org> wrote:
>
> On Wed, Apr 03, 2019 at 10:46:01AM -0700, Daniel Colascione wrote:
> > On Wed, Apr 3, 2019 at 10:20 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> > >
> > > On Wed, Apr 03, 2019 at 04:48:37PM +0900, Masahiro Yamada wrote:
> > > > On Thu, Mar 28, 2019 at 2:32 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> > > > >
> > > > > On Mon, Mar 25, 2019 at 09:49:47AM -0400, Joel Fernandes wrote:
> > > > > > On Thu, Feb 14, 2019 at 07:19:29PM -0800, Alexei Starovoitov wrote:
> > > > > > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
> > > > > > > > Introduce in-kernel headers and other artifacts which are made available
> > > > > > > > as an archive through proc (/proc/kheaders.txz file). This archive makes
> > > > > > > > it possible to build kernel modules, run eBPF programs, and other
> > > > > > > > tracing programs that need to extend the kernel for tracing purposes
> > > > > > > > without any dependency on the file system having headers and build
> > > > > > > > artifacts.
> > > > > > > >
> > > > > > > > On Android and embedded systems, it is common to switch kernels but not
> > > > > > > > have kernel headers available on the file system. Raw kernel headers
> > > > > > > > also cannot be copied into the filesystem like they can be on other
> > > > > > > > distros, due to licensing and other issues. There's no linux-headers
> > > > > > > > package on Android. Further once a different kernel is booted, any
> > > > > > > > headers stored on the file system will no longer be useful. By storing
> > > > > > > > the headers as a compressed archive within the kernel, we can avoid these
> > > > > > > > issues that have been a hindrance for a long time.
> > > > > > >
> > > > > > > The set looks good to me and since the main use case is building bpf progs
> > > > > > > I can route it via bpf-next tree if there are no objections.
> > > > > > > Masahiro, could you please ack it?
> > > > > >
> > > > > > FYI, Masahiro's comments were all address by v5:
> > > > > > https://lore.kernel.org/patchwork/project/lkml/list/?series=387311
> > > > > >
> > > > > > I believe aren't more outstanding concerns. Could we consider it for v5.2?
> > > > >
> > > > > Just to highlight the problem, today I booted v5.0 on an emulated Android
> > > > > device for some testing, that didn't have a set of prebuilt headers that we
> > > > > have been packaging on well known kernels, to get around this issue. This
> > > > > caused great pain and issues with what I was doing. I know me and many others
> > > > > really want this. With this I can boot an emulated Android device with
> > > > > IKCONFIG_PROC=y and run BCC with that that. Also I want to do the BCC side of
> > > > > the development, but first want to know if we can merge this upstream.
> > > > >
> > > > > Masahiro, I believe due diligence has been done in solidifying it as posted
> > > > > in the v5.  Anything else we need to do here? Are you with the patches?
> > > >
> > > >
> > > > As you said, these updates are "cosmetic".
> > > > Nobody is worried about (or interested in) them.
> > > > Even if we miss something, they are fixable later.
> > > >
> > > > I accept embedding headers in the kernel,
> > > > but I do not like the part about external module build.
> > > >  - kernel embeds build artifacts that depend on a
> > > >    particular host-arch
> > > >  - users do not know which compiler to use
> > > > Perhaps, you may remember my concerns.
> > > > https://lore.kernel.org/patchwork/patch/1046307/#1239501
> > >
> > > Masahiro,
> > > I think we already discussed this right. The compiler issue is a user-issue -
> > > it is not a problem that has arisen because this patch. Anyone can build a
> > > kernel module today without this patch - using a compiler that is different
> > > from the running kernel. So I did not fully understand your concern. This
> > > patch does not ship a compiler in the archive. The compiler is upto the user
> > > based on user environment. They can already shoot themself in foot without
> > > this patch.
> > >
> > > Greg,
> > > Would be great to get your thoughts here as well about Masami's concern.
> > >
> > > Honestly, the "kernel module building artifacts" bit is a bonus with this
> > > patch. The main thing we are adding or that we need is really the headers
> > > (for BCC).


Yeah, this part looks solid to me.


> I just thought shipping the kernel build artifacts would also be
> > > awesome because people can now build kernel modules without needing a kernel
> > > tree at all.
> > >
> > > But I also want this stuff merged so if folks are really unhappy with the
> > > module build artifacts and only want headers - then that's also fine with me
> > > and I can yank them - that would also mean though that this work cannot be a
> > > replacement for linux-headers package on distros, so that would be sad.
> > >
> > > thanks!
> >
> > IMHO, including the module build stuff is fine, and the stuff that
> > Masahiro mentions doesn't matter much. To be specific about the
> > concerns:
> >
> > >> > >  - kernel embeds build artifacts that depend on a
> > > >    particular host-arch
> >
> > What are these artifacts? We build the kernel as a standalone system.
> > It's not as if we statically link host glibc into it or something.
>
> There are some scripts/ that are built in the host-arch ABI. These are also put
> in the archive because they are needed to build modules. So if you
> cross-compile then the archive will have scripts/ that are in the host arch,
> not the target arch. This makes it not possible to build modules on the
> target using the archive. This is not much of an issue because I already
> proved that such kernel modules can be built using a chroot in this thread:
> https://lkml.org/lkml/2019/2/28/1313
> And in the non cross-compile case, it will be immensely useful for distros..


Now the kernel depends on the host-arch that built it.
It looks somewhat weird to me.


> > > >  - users do not know which compiler to use
> >
> > Does that matter much? Do things ever break if the kernel proper is
> > built with GCC x.y.z and we build a module with GCC x.y.(z+1)? Why
> > would it? Structure layout is invariant across compiler version, or at
> > least ought to be.


The difference in the minor version
will not be a problem in practice.

The difference in the major version could be a problem.

Difference compilers have a different set of compiler options in general.

In Kbuild Makefiles, unsupported compiler options are
disabled by $(call cc-option,...).

So, users may potentially compile external modules
wit a slightly different set of flags without noticing that.

It may still work, but some compiler flags insert stubs.
There are some cases where compiler flags are sensitive.

For example, if we miss the compiler flag that was given to vmlinux,
slightly different code structure may cause kernel panic:
https://bugzilla.kernel.org/show_bug.cgi?id=201891


>  And don't we have exactly the same problem with
> > any kernel headers package? I don't think it'd hurt to include the
> > output of $(CC) --version though.
>
> Apparently there were some issues in some posting where Greg said we don't
> support anything but modules built with the same compiler as the kernel its
> being loaded into, even if such modules do work in practice. I don't recall
> the details, I can dig up that thread if you want. My point is, whether
> that's an issue or not - it is not an issue introduced by this patch. That's
> just a module building issue which we neither solve here, nor introduce.

What I can tell is, distributions provide packages of the compiler,
kernel headers, and whatever needed to compile external modules.

I feel this is a double-edged sword.


> > Like Joel, I'd like to see this change merged. It'll make life easier
> > for a lot of people.
>
> Thanks.
>
> - Joel
>


-- 
Best Regards
Masahiro Yamada