[RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Jiri Slaby (SUSE)]

From: Jiri Slaby <jslaby@suse.cz>

== WARNING ==
This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM
are completely unhandled.

The simple version is just do there:
  ifeq ($(CONFIG_64BIT,y)
but it has its own deficiencies, of course.

So any ideas, inputs?
== WARNING ==

When pahole is run with -j on 32bit userspace (32bit pahole in
particular), it randomly fails with OOM:
> btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.
> btf_encoder__encode: failed to tag kfuncs!

or simply SIGSEGV (failed to allocate the btf encoder).

It very depends on how many threads are created.

So do not invoke pahole with -j on 32bit.

Signed-off-by: Jiri Slaby <jslaby@suse.cz>
Fixes: b4f72786429c ("scripts/pahole-flags.sh: Parse DWARF and generate BTF with multithreading.")
Closes: https://bugzilla.suse.com/show_bug.cgi?id=1229450
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Nicolas Schier <nicolas@fjasle.eu>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Martin KaFai Lau <martin.lau@linux.dev>
Cc: Eduard Zingerman <eddyz87@gmail.com>
Cc: Song Liu <song@kernel.org>
Cc: Yonghong Song <yonghong.song@linux.dev>
Cc: John Fastabend <john.fastabend@gmail.com>
Cc: KP Singh <kpsingh@kernel.org>
Cc: Stanislav Fomichev <sdf@fomichev.me>
Cc: Hao Luo <haoluo@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: linux-kernel@vger.kernel.org
Cc: linux-kbuild@vger.kernel.org
Cc: bpf@vger.kernel.org
Cc: shung-hsi.yu@suse.com
Cc: msuchanek@suse.com
---
 init/Kconfig            |  4 ++++
 scripts/Makefile.btf    |  2 ++
 scripts/pahole-class.sh | 21 +++++++++++++++++++++
 3 files changed, 27 insertions(+)
 create mode 100644 scripts/pahole-class.sh

diff --git a/init/Kconfig b/init/Kconfig
index f36ca8a0e209..f5e80497eef0 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -113,6 +113,10 @@ config PAHOLE_VERSION
 	int
 	default $(shell,$(srctree)/scripts/pahole-version.sh $(PAHOLE))
 
+config PAHOLE_CLASS
+	string
+	default $(shell,$(srctree)/scripts/pahole-class.sh $(PAHOLE))
+
 config CONSTRUCTORS
 	bool
 
diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
index b75f09f3f424..f7de8e922bce 100644
--- a/scripts/Makefile.btf
+++ b/scripts/Makefile.btf
@@ -12,7 +12,9 @@ endif
 
 pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
 
+ifeq ($(CONFIG_PAHOLE_CLASS),ELF64)
 pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
+endif
 
 pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
 
diff --git a/scripts/pahole-class.sh b/scripts/pahole-class.sh
new file mode 100644
index 000000000000..d15a92077f76
--- /dev/null
+++ b/scripts/pahole-class.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+#
+# Usage: $ ./pahole-class.sh pahole
+#
+# Prints pahole's ELF class, such as ELF64
+
+if [ ! -x "$(command -v "$@")" ]; then
+	echo 0
+	exit 1
+fi
+
+PAHOLE="$(which "$@")"
+CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')"
+
+# Scripts like scripts/dummy-tools/pahole
+if [ -n "$CLASS" ]; then
+	echo "$CLASS"
+else
+	echo ELF64
+fi
-- 
2.46.0

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Jiri Slaby]

On 20. 08. 24, 10:59, Jiri Slaby (SUSE) wrote:
> From: Jiri Slaby <jslaby@suse.cz>
> 
> == WARNING ==
> This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM
> are completely unhandled.
> 
> The simple version is just do there:
>    ifeq ($(CONFIG_64BIT,y)
> but it has its own deficiencies, of course.
> 
> So any ideas, inputs?

Also as Shung-Hsi Yu suggests, we can cap -j to 1 in pahole proper when 
sizeof(long) == 4.

> == WARNING ==
> 
> When pahole is run with -j on 32bit userspace (32bit pahole in
> particular), it randomly fails with OOM:
>> btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.
>> btf_encoder__encode: failed to tag kfuncs!
> 
> or simply SIGSEGV (failed to allocate the btf encoder).
> 
> It very depends on how many threads are created.

I forgot to add, that it depends on the kernel version too. It happens 
much often with 6.11-rc now (vmlinux got big enough, apparently).

thanks,
-- 
js
suse labs

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Jiri Olsa]

On Tue, Aug 20, 2024 at 10:59:50AM +0200, Jiri Slaby (SUSE) wrote:
> From: Jiri Slaby <jslaby@suse.cz>
> 
> == WARNING ==
> This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM
> are completely unhandled.
> 
> The simple version is just do there:
>   ifeq ($(CONFIG_64BIT,y)
> but it has its own deficiencies, of course.
> 
> So any ideas, inputs?
> == WARNING ==
> 
> When pahole is run with -j on 32bit userspace (32bit pahole in
> particular), it randomly fails with OOM:
> > btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.
> > btf_encoder__encode: failed to tag kfuncs!
> 
> or simply SIGSEGV (failed to allocate the btf encoder).
> 
> It very depends on how many threads are created.
> 
> So do not invoke pahole with -j on 32bit.

could you share more details about your setup?

does it need to run on pure 32bit to reproduce? I can't reproduce when
doing cross build and running 32 bit pahole on x86_64.. I do see some
errors though

  [667939] STRUCT bpf_prog_aux Error emitting BTF type
  Encountered error while encoding BTF.

thanks,
jirka

> 
> Signed-off-by: Jiri Slaby <jslaby@suse.cz>
> Fixes: b4f72786429c ("scripts/pahole-flags.sh: Parse DWARF and generate BTF with multithreading.")
> Closes: https://bugzilla.suse.com/show_bug.cgi?id=1229450
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Cc: Nathan Chancellor <nathan@kernel.org>
> Cc: Nicolas Schier <nicolas@fjasle.eu>
> Cc: Alexei Starovoitov <ast@kernel.org>
> Cc: Daniel Borkmann <daniel@iogearbox.net>
> Cc: Andrii Nakryiko <andrii@kernel.org>
> Cc: Martin KaFai Lau <martin.lau@linux.dev>
> Cc: Eduard Zingerman <eddyz87@gmail.com>
> Cc: Song Liu <song@kernel.org>
> Cc: Yonghong Song <yonghong.song@linux.dev>
> Cc: John Fastabend <john.fastabend@gmail.com>
> Cc: KP Singh <kpsingh@kernel.org>
> Cc: Stanislav Fomichev <sdf@fomichev.me>
> Cc: Hao Luo <haoluo@google.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: linux-kernel@vger.kernel.org
> Cc: linux-kbuild@vger.kernel.org
> Cc: bpf@vger.kernel.org
> Cc: shung-hsi.yu@suse.com
> Cc: msuchanek@suse.com
> ---
>  init/Kconfig            |  4 ++++
>  scripts/Makefile.btf    |  2 ++
>  scripts/pahole-class.sh | 21 +++++++++++++++++++++
>  3 files changed, 27 insertions(+)
>  create mode 100644 scripts/pahole-class.sh
> 
> diff --git a/init/Kconfig b/init/Kconfig
> index f36ca8a0e209..f5e80497eef0 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -113,6 +113,10 @@ config PAHOLE_VERSION
>  	int
>  	default $(shell,$(srctree)/scripts/pahole-version.sh $(PAHOLE))
>  
> +config PAHOLE_CLASS
> +	string
> +	default $(shell,$(srctree)/scripts/pahole-class.sh $(PAHOLE))
> +
>  config CONSTRUCTORS
>  	bool
>  
> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
> index b75f09f3f424..f7de8e922bce 100644
> --- a/scripts/Makefile.btf
> +++ b/scripts/Makefile.btf
> @@ -12,7 +12,9 @@ endif
>  
>  pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
>  
> +ifeq ($(CONFIG_PAHOLE_CLASS),ELF64)
>  pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
> +endif
>  
>  pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
>  
> diff --git a/scripts/pahole-class.sh b/scripts/pahole-class.sh
> new file mode 100644
> index 000000000000..d15a92077f76
> --- /dev/null
> +++ b/scripts/pahole-class.sh
> @@ -0,0 +1,21 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Usage: $ ./pahole-class.sh pahole
> +#
> +# Prints pahole's ELF class, such as ELF64
> +
> +if [ ! -x "$(command -v "$@")" ]; then
> +	echo 0
> +	exit 1
> +fi
> +
> +PAHOLE="$(which "$@")"
> +CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')"
> +
> +# Scripts like scripts/dummy-tools/pahole
> +if [ -n "$CLASS" ]; then
> +	echo "$CLASS"
> +else
> +	echo ELF64
> +fi
> -- 
> 2.46.0
> 

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Jiri Slaby]

On 20. 08. 24, 16:33, Jiri Olsa wrote:
> On Tue, Aug 20, 2024 at 10:59:50AM +0200, Jiri Slaby (SUSE) wrote:
>> From: Jiri Slaby <jslaby@suse.cz>
>>
>> == WARNING ==
>> This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM
>> are completely unhandled.
>>
>> The simple version is just do there:
>>    ifeq ($(CONFIG_64BIT,y)
>> but it has its own deficiencies, of course.
>>
>> So any ideas, inputs?
>> == WARNING ==
>>
>> When pahole is run with -j on 32bit userspace (32bit pahole in
>> particular), it randomly fails with OOM:
>>> btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.
>>> btf_encoder__encode: failed to tag kfuncs!
>>
>> or simply SIGSEGV (failed to allocate the btf encoder).
>>
>> It very depends on how many threads are created.
>>
>> So do not invoke pahole with -j on 32bit.
> 
> could you share more details about your setup?
> 
> does it need to run on pure 32bit to reproduce?

armv7l builds are 32bit only.

> I can't reproduce when
> doing cross build and running 32 bit pahole on x86_64..

i586 is built using 64bit kernel. It is enough to have 32bit userspace.
As written in the linked bug:
https://bugzilla.suse.com/show_bug.cgi?id=1229450#c6

FWIW, steps to reproduce locally:
docker pull jirislaby/pahole_crash
docker run -it jirislaby/pahole_crash

The VM space of pahole is exhausted:
process map: https://bugzilla.suse.com/attachment.cgi?id=876821
strace of mmaps: https://bugzilla.suse.com/attachment.cgi?id=876822

You need to run with large enough -j on a fast machine. Note that this 
happens on build hosts even with -j4, but they are under heavy load, so 
parallelism of held memory is high.

On my box with 16 cores, it is (likely far) enough to run with -j32.

> I do see some
> errors though
> 
>    [667939] STRUCT bpf_prog_aux Error emitting BTF type
>    Encountered error while encoding BTF.

It's possible that it is one of the errors. There are different ones. As 
I wrote above, sometimes it is a crash, sometimes it is the failure I 
mentioned above. But it always ends up with a failed build:
 > libbpf: failed to find '.BTF' ELF section in vmlinux
 > FAILED: load BTF from vmlinux: No data available
 > make[2]: *** [../scripts/Makefile.vmlinux:34: vmlinux] Error 255
 > make[2]: *** Deleting file 'vmlinux'
 > make[1]: *** 
[/home/abuild/rpmbuild/BUILD/kernel-vanilla-6.11~rc3.338.gc3f2d783a459/linux-6.11-rc3-338-gc3f2d783a459/Makefile:1158: 
vmlinux] Error 2
 > make: *** [../Makefile:224: __sub-make] Error 2
 > error: Bad exit status from /var/tmp/rpm-tmp.olf5Nu (%build)

thanks,
-- 
js
suse labs

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Jiri Slaby]

On 21. 08. 24, 7:32, Jiri Slaby wrote:
> On 20. 08. 24, 16:33, Jiri Olsa wrote:
>> On Tue, Aug 20, 2024 at 10:59:50AM +0200, Jiri Slaby (SUSE) wrote:
>>> From: Jiri Slaby <jslaby@suse.cz>
>>>
>>> == WARNING ==
>>> This is only a PoC. There are deficiencies like CROSS_COMPILE or LLVM
>>> are completely unhandled.
>>>
>>> The simple version is just do there:
>>>    ifeq ($(CONFIG_64BIT,y)
>>> but it has its own deficiencies, of course.
>>>
>>> So any ideas, inputs?
>>> == WARNING ==
>>>
>>> When pahole is run with -j on 32bit userspace (32bit pahole in
>>> particular), it randomly fails with OOM:
>>>> btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of 
>>>> memory.
>>>> btf_encoder__encode: failed to tag kfuncs!
>>>
>>> or simply SIGSEGV (failed to allocate the btf encoder).
>>>
>>> It very depends on how many threads are created.
>>>
>>> So do not invoke pahole with -j on 32bit.
>>
>> could you share more details about your setup?
>>
>> does it need to run on pure 32bit to reproduce?
> 
> armv7l builds are 32bit only.
> 
>> I can't reproduce when
>> doing cross build and running 32 bit pahole on x86_64..
> 
> i586 is built using 64bit kernel. It is enough to have 32bit userspace.
> As written in the linked bug:
> https://bugzilla.suse.com/show_bug.cgi?id=1229450#c6
> 
> FWIW, steps to reproduce locally:
> docker pull jirislaby/pahole_crash
> docker run -it jirislaby/pahole_crash
> 
> The VM space of pahole is exhausted:
> process map: https://bugzilla.suse.com/attachment.cgi?id=876821
> strace of mmaps: https://bugzilla.suse.com/attachment.cgi?id=876822
> 
> You need to run with large enough -j on a fast machine. Note that this 
> happens on build hosts even with -j4, but they are under heavy load, so 
> parallelism of held memory is high.

 From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
Run on 64bit:
pahole -j32 -> 4.102 GB
pahole -j16 -> 3.895 GB
pahole -j1 -> 3.706 GB

On 32bit (the same vmlinux):
pahole -j32 -> 2.870 GB (crash)
pahole -j16 -> 2.810 GB
pahole -j1 -> 2.444 GB

Look there for full massif report.

So now I think we should disable BTF generation with 32bit pahole 
completely. Or someone debugs it and improves debug info loading to not 
eat that much.

thanks,
-- 
js
suse labs

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Jiri Slaby]

On 21. 08. 24, 8:40, Jiri Slaby wrote:
>  From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
> Run on 64bit:
> pahole -j32 -> 4.102 GB
> pahole -j16 -> 3.895 GB
> pahole -j1 -> 3.706 GB
> 
> On 32bit (the same vmlinux):
> pahole -j32 -> 2.870 GB (crash)
> pahole -j16 -> 2.810 GB
> pahole -j1 -> 2.444 GB
> 
> Look there for full massif report.

 From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21:
(In reply to Jiri Slaby from comment #20)
 > | |   |   ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun 
(dwarf_loader.c:959)

So given this struct class_member is the largest consumer, running 
pahole on pahole. The below results in 4.102 GB -> 3.585 GB savings.

--- a/dwarves.h
+++ b/dwarves.h
@@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu,
   */
  struct tag {
         struct list_head node;
+       const char       *attribute;
+       void             *priv;
         type_id_t        type;
         uint16_t         tag;
+       uint16_t         recursivity_level;
         bool             visited;
         bool             top_level;
         bool             has_btf_type_tag;
-       uint16_t         recursivity_level;
-       const char       *attribute;
-       void             *priv;
  };

  // To use with things like type->type_enum == 
perf_event_type+perf_user_event_type
@@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct 
function *func)
  struct class_member {
         struct tag       tag;
         const char       *name;
+       uint64_t         const_value;
         uint32_t         bit_offset;
         uint32_t         bit_size;
         uint32_t         byte_offset;
         int              hole;
         size_t           byte_size;
+       uint32_t         alignment;
         int8_t           bitfield_offset;
         uint8_t          bitfield_size;
         uint8_t          bit_hole;
         uint8_t          bitfield_end:1;
-       uint64_t         const_value;
-       uint32_t         alignment;
         uint8_t          visited:1;
         uint8_t          is_static:1;
         uint8_t          has_bit_offset:1;

-- 
js
suse labs

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Shung-Hsi Yu]

(Add pahole maintainer and mailing list)

Hi Arnaldo,

We're running into kernel build failure on 32-bit (both full 32-bit and
32-bit userspace on 64-bit kernel) because pahole crashed due to virtual
memory exhaustion[1]. As a workaround we currently limit pahole's
parallel job count to 1 on such system[2]:

On Tue, 20 Aug 2024 10:59:50AM +0200, Jiri Slaby wrote:
[...]
> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
> index b75f09f3f424..f7de8e922bce 100644
> --- a/scripts/Makefile.btf
> +++ b/scripts/Makefile.btf
> @@ -12,7 +12,9 @@ endif
>  
>  pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
>  
> +ifeq ($(CONFIG_PAHOLE_CLASS),ELF64)
>  pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
> +endif
>  
>  pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
>  
> diff --git a/scripts/pahole-class.sh b/scripts/pahole-class.sh
> new file mode 100644
> index 000000000000..d15a92077f76
> --- /dev/null
> +++ b/scripts/pahole-class.sh
> @@ -0,0 +1,21 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Usage: $ ./pahole-class.sh pahole
> +#
> +# Prints pahole's ELF class, such as ELF64
> +
> +if [ ! -x "$(command -v "$@")" ]; then
> +	echo 0
> +	exit 1
> +fi
> +
> +PAHOLE="$(which "$@")"
> +CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')"
> +
> +# Scripts like scripts/dummy-tools/pahole
> +if [ -n "$CLASS" ]; then
> +	echo "$CLASS"
> +else
> +	echo ELF64
> +fi
> -- 

This helped lowered the memory usage enough so pahole no longer crash:

On Wed, Aug 21, 2024 at 09:29:57AM GMT, Jiri Slaby wrote:
> On 21. 08. 24, 8:40, Jiri Slaby wrote:
> >  From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
> > Run on 64bit:
> > pahole -j32 -> 4.102 GB
> > pahole -j16 -> 3.895 GB
> > pahole -j1 -> 3.706 GB
> > 
> > On 32bit (the same vmlinux):
> > pahole -j32 -> 2.870 GB (crash)
> > pahole -j16 -> 2.810 GB
> > pahole -j1 -> 2.444 GB

Jiri (Slaby) in the meanwhile has also proposed structure packing to
further reduce memory usage. (Note: I think the numbers below are from a
64-bit machine)

> From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21:
> (In reply to Jiri Slaby from comment #20)
> > | |   |   ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun
> (dwarf_loader.c:959)
> 
> So given this struct class_member is the largest consumer, running pahole on
> pahole. The below results in 4.102 GB -> 3.585 GB savings.
> 
> --- a/dwarves.h
> +++ b/dwarves.h
> @@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu,
>   */
>  struct tag {
>         struct list_head node;
> +       const char       *attribute;
> +       void             *priv;
>         type_id_t        type;
>         uint16_t         tag;
> +       uint16_t         recursivity_level;
>         bool             visited;
>         bool             top_level;
>         bool             has_btf_type_tag;
> -       uint16_t         recursivity_level;
> -       const char       *attribute;
> -       void             *priv;
>  };
> 
>  // To use with things like type->type_enum ==
> perf_event_type+perf_user_event_type
> @@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct
> function *func)
>  struct class_member {
>         struct tag       tag;
>         const char       *name;
> +       uint64_t         const_value;
>         uint32_t         bit_offset;
>         uint32_t         bit_size;
>         uint32_t         byte_offset;
>         int              hole;
>         size_t           byte_size;
> +       uint32_t         alignment;
>         int8_t           bitfield_offset;
>         uint8_t          bitfield_size;
>         uint8_t          bit_hole;
>         uint8_t          bitfield_end:1;
> -       uint64_t         const_value;
> -       uint32_t         alignment;
>         uint8_t          visited:1;
>         uint8_t          is_static:1;
>         uint8_t          has_bit_offset:1;
>--

What do you think?

IIUC pahole's memory usage is largely tied to the number of entries in
vmlinux/kmodule DWARF, and there probably isn't much we could do about
that.

Shung-Hsi

1: https://bugzilla.suse.com/show_bug.cgi?id=1229450
2: https://lore.kernel.org/all/20240820085950.200358-1-jirislaby@kernel.org/

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Arnaldo Carvalho de Melo]

On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
> (Add pahole maintainer and mailing list)
> 
> Hi Arnaldo,
> 
> We're running into kernel build failure on 32-bit (both full 32-bit and
> 32-bit userspace on 64-bit kernel) because pahole crashed due to virtual
> memory exhaustion[1]. As a workaround we currently limit pahole's
> parallel job count to 1 on such system[2]:
> 
> On Tue, 20 Aug 2024 10:59:50AM +0200, Jiri Slaby wrote:
> [...]
> > diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
> > index b75f09f3f424..f7de8e922bce 100644
> > --- a/scripts/Makefile.btf
> > +++ b/scripts/Makefile.btf
> > @@ -12,7 +12,9 @@ endif
> >  
> >  pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
> >  
> > +ifeq ($(CONFIG_PAHOLE_CLASS),ELF64)
> >  pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
> > +endif
> >  
> >  pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> >  
> > diff --git a/scripts/pahole-class.sh b/scripts/pahole-class.sh
> > new file mode 100644
> > index 000000000000..d15a92077f76
> > --- /dev/null
> > +++ b/scripts/pahole-class.sh
> > @@ -0,0 +1,21 @@
> > +#!/bin/sh
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# Usage: $ ./pahole-class.sh pahole
> > +#
> > +# Prints pahole's ELF class, such as ELF64
> > +
> > +if [ ! -x "$(command -v "$@")" ]; then
> > +	echo 0
> > +	exit 1
> > +fi
> > +
> > +PAHOLE="$(which "$@")"
> > +CLASS="$(readelf -h "$PAHOLE" 2>/dev/null | sed -n 's/.*Class: *// p')"
> > +
> > +# Scripts like scripts/dummy-tools/pahole
> > +if [ -n "$CLASS" ]; then
> > +	echo "$CLASS"
> > +else
> > +	echo ELF64
> > +fi
> > -- 
> 
> This helped lowered the memory usage enough so pahole no longer crash:
> 
> On Wed, Aug 21, 2024 at 09:29:57AM GMT, Jiri Slaby wrote:
> > On 21. 08. 24, 8:40, Jiri Slaby wrote:
> > >  From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20:
> > > Run on 64bit:
> > > pahole -j32 -> 4.102 GB
> > > pahole -j16 -> 3.895 GB
> > > pahole -j1 -> 3.706 GB
> > > 
> > > On 32bit (the same vmlinux):
> > > pahole -j32 -> 2.870 GB (crash)
> > > pahole -j16 -> 2.810 GB
> > > pahole -j1 -> 2.444 GB
> 
> Jiri (Slaby) in the meanwhile has also proposed structure packing to
> further reduce memory usage. (Note: I think the numbers below are from a
> 64-bit machine)

That is interesting, packing pahole data structures ;-) :-)

Also a coincidence is that I did some packing on what is in the next
branch:

3ef508ad94012933 dwarf_loader: Make 'struct dwarf_tag' more compact by getting rid of 'struct dwarf_off_ref
70febc8858588348 core: Reorganize 'struct class_member' to save 8 bytes
76bcb88a67556468 core: Make tag->recursivity_level a uint8_t
b8b9e04d177d8eb7 core: Make tag->top_level a single bit flag
539acefcdd5b0f71 core: Make tag->has_btf_type_tag a single bit flag
dba2c2c1aa5dfa05 core: Make tag->visited a single bit flag
7409cfadcae0253b core: Shrink 'struct namespace' a bit by using a hole in its embedded 'tag'

Also I did more work to reduce the number of allocations:

cbecc3785266f0c5 dwarf_loader: Do just one alloc for 'struct dwarf_tag + struct tag'

With it we get:

⬢[acme@toolbox pahole]$ pahole -C class_member build/libdwarves.so.1.0.0 
struct class_member {
	struct tag                 tag;                  /*     0    32 */

	/* XXX last struct has 1 bit hole */

	const char  *              name;                 /*    32     8 */
	uint32_t                   bit_offset;           /*    40     4 */
	uint32_t                   bit_size;             /*    44     4 */
	uint32_t                   byte_offset;          /*    48     4 */
	int                        hole;                 /*    52     4 */
	size_t                     byte_size;            /*    56     8 */
	/* --- cacheline 1 boundary (64 bytes) --- */
	int8_t                     bitfield_offset;      /*    64     1 */
	uint8_t                    bitfield_size;        /*    65     1 */
	uint8_t                    bit_hole;             /*    66     1 */
	uint8_t                    bitfield_end:1;       /*    67: 0  1 */
	uint8_t                    visited:1;            /*    67: 1  1 */
	uint8_t                    is_static:1;          /*    67: 2  1 */
	uint8_t                    has_bit_offset:1;     /*    67: 3  1 */
	uint8_t                    accessibility:2;      /*    67: 4  1 */
	uint8_t                    virtuality:2;         /*    67: 6  1 */
	uint32_t                   alignment;            /*    68     4 */
	uint64_t                   const_value;          /*    72     8 */

	/* size: 80, cachelines: 2, members: 18 */
	/* member types with bit holes: 1, total: 1 */
	/* last cacheline: 16 bytes */
};

⬢[acme@toolbox pahole]$

And also the dwarf_tag, that is allocated for each tag coming from DWARF
got smaller:

⬢[acme@toolbox pahole]$ pahole -C dwarf_tag build/libdwarves.so.1.0.0 
struct dwarf_tag {
	struct hlist_node          hash_node;            /*     0    16 */
	Dwarf_Off                  type;                 /*    16     8 */
	Dwarf_Off                  id;                   /*    24     8 */
	union {
		Dwarf_Off          abstract_origin;      /*    32     8 */
		Dwarf_Off          containing_type;      /*    32     8 */
	};                                               /*    32     8 */
	Dwarf_Off                  specification;        /*    40     8 */
	struct {
		_Bool              type:1;               /*    48: 0  1 */
		_Bool              abstract_origin:1;    /*    48: 1  1 */
		_Bool              containing_type:1;    /*    48: 2  1 */
		_Bool              specification:1;      /*    48: 3  1 */
	} from_types_section;                            /*    48     1 */

	/* XXX last struct has 4 bits of padding */
	/* XXX 1 byte hole, try to pack */

	uint16_t                   decl_line;            /*    50     2 */
	uint32_t                   small_id;             /*    52     4 */
	const char  *              decl_file;            /*    56     8 */

	/* size: 64, cachelines: 1, members: 9 */
	/* sum members: 63, holes: 1, sum holes: 1 */
	/* member types with bit paddings: 1, total: 4 bits */
};

⬢[acme@toolbox pahole]$

I stumbled on this limitation as well when trying to build the kernel on
a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just
created a swapfile and it managed to proceed, a bit slowly, but worked
as well.

Please let me know if what is in the 'next' branch of:

https://git.kernel.org/pub/scm/devel/pahole/pahole.git

Works for you, that will be extra motivation to move it to the master
branch and cut 1.28.

Thanks!

- Arnaldo
 
> > From https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21:
> > (In reply to Jiri Slaby from comment #20)
> > > | |   |   ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun
> > (dwarf_loader.c:959)
> > 
> > So given this struct class_member is the largest consumer, running pahole on
> > pahole. The below results in 4.102 GB -> 3.585 GB savings.
> > 
> > --- a/dwarves.h
> > +++ b/dwarves.h
> > @@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu,
> >   */
> >  struct tag {
> >         struct list_head node;
> > +       const char       *attribute;
> > +       void             *priv;
> >         type_id_t        type;
> >         uint16_t         tag;
> > +       uint16_t         recursivity_level;
> >         bool             visited;
> >         bool             top_level;
> >         bool             has_btf_type_tag;
> > -       uint16_t         recursivity_level;
> > -       const char       *attribute;
> > -       void             *priv;
> >  };
> > 
> >  // To use with things like type->type_enum ==
> > perf_event_type+perf_user_event_type
> > @@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct
> > function *func)
> >  struct class_member {
> >         struct tag       tag;
> >         const char       *name;
> > +       uint64_t         const_value;
> >         uint32_t         bit_offset;
> >         uint32_t         bit_size;
> >         uint32_t         byte_offset;
> >         int              hole;
> >         size_t           byte_size;
> > +       uint32_t         alignment;
> >         int8_t           bitfield_offset;
> >         uint8_t          bitfield_size;
> >         uint8_t          bit_hole;
> >         uint8_t          bitfield_end:1;
> > -       uint64_t         const_value;
> > -       uint32_t         alignment;
> >         uint8_t          visited:1;
> >         uint8_t          is_static:1;
> >         uint8_t          has_bit_offset:1;
> >--
> 
> What do you think?
> 
> IIUC pahole's memory usage is largely tied to the number of entries in
> vmlinux/kmodule DWARF, and there probably isn't much we could do about
> that.
> 
> Shung-Hsi
> 
> 1: https://bugzilla.suse.com/show_bug.cgi?id=1229450
> 2: https://lore.kernel.org/all/20240820085950.200358-1-jirislaby@kernel.org/

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Jiri Slaby]

On 22. 08. 24, 17:24, Arnaldo Carvalho de Melo wrote:
> On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
> I stumbled on this limitation as well when trying to build the kernel on
> a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just
> created a swapfile and it managed to proceed, a bit slowly, but worked
> as well.

Here, it hits the VM space limit (3 G).

> Please let me know if what is in the 'next' branch of:
> 
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git
> 
> Works for you, that will be extra motivation to move it to the master
> branch and cut 1.28.

on 64bit (-j1):
* master: 3.706 GB
(* master + my changes: 3.559 GB)
* next: 3.157 GB 


on 32bit:
  * master-j1: 2.445 GB
  * master-j16: 2.608 GB
  * master-j32: 2.811 GB
  * next-j1: 2.256 GB
  * next-j16: 2.401 GB
  * next-j32: 2.613 GB

It's definitely better. So I think it could work now, if the thread 
count was limited to 1 on 32bit. As building with -j10, -j20 randomly 
fails on random machines (32bit processes only of course). Unlike -j1.

thanks,
-- 
js
suse labs

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Arnaldo Carvalho de Melo]

On Mon, Aug 26, 2024 at 10:57:22AM +0200, Jiri Slaby wrote:
> On 22. 08. 24, 17:24, Arnaldo Carvalho de Melo wrote:
> > On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
> > I stumbled on this limitation as well when trying to build the kernel on
> > a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just
> > created a swapfile and it managed to proceed, a bit slowly, but worked
> > as well.
> 
> Here, it hits the VM space limit (3 G).

right, in my case it was on a 64-bit system, so just not enough memory,
not address space.
 
> > Please let me know if what is in the 'next' branch of:

> > https://git.kernel.org/pub/scm/devel/pahole/pahole.git

> > Works for you, that will be extra motivation to move it to the master
> > branch and cut 1.28.

> on 64bit (-j1):
> * master: 3.706 GB
> (* master + my changes: 3.559 GB)
> * next: 3.157 GB
 
> on 32bit:
>  * master-j1: 2.445 GB
>  * master-j16: 2.608 GB
>  * master-j32: 2.811 GB
>  * next-j1: 2.256 GB
>  * next-j16: 2.401 GB
>  * next-j32: 2.613 GB
> 
> It's definitely better. So I think it could work now, if the thread count
> was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on
> random machines (32bit processes only of course). Unlike -j1.

Cool, I just merged a patch from Alan Maguire that should help with the
parallel case, would be able to test it? It is in the 'next' branch:

⬢[acme@toolbox pahole]$ git log --oneline -5
f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs

Excerpt of the above:

    This leads to deleting ~90 CUs during parallel vmlinux BTF generation
    versus deleting just 1 prior to this change.

c7ec9200caa7d485 btf_encoder: Add "distilled_base" BTF feature to split BTF generation
bc4e6a9adfc72758 pahole: Sync with libbpf-1.5
5e3ed3ec2947c69f pahole: Do --lang_exclude CU filtering earlier
c46455bb0379fa38 dwarf_loader: Allow filtering CUs early in loading
⬢[acme@toolbox pahole]$

- Arnaldo

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Sedat Dilek]

On Mon, Aug 26, 2024 at 7:03 PM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> On Mon, Aug 26, 2024 at 10:57:22AM +0200, Jiri Slaby wrote:
> > On 22. 08. 24, 17:24, Arnaldo Carvalho de Melo wrote:
> > > On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
> > > I stumbled on this limitation as well when trying to build the kernel on
> > > a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just
> > > created a swapfile and it managed to proceed, a bit slowly, but worked
> > > as well.
> >
> > Here, it hits the VM space limit (3 G).
>
> right, in my case it was on a 64-bit system, so just not enough memory,
> not address space.
>
> > > Please let me know if what is in the 'next' branch of:
>
> > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git
>
> > > Works for you, that will be extra motivation to move it to the master
> > > branch and cut 1.28.
>
> > on 64bit (-j1):
> > * master: 3.706 GB
> > (* master + my changes: 3.559 GB)
> > * next: 3.157 GB
>
> > on 32bit:
> >  * master-j1: 2.445 GB
> >  * master-j16: 2.608 GB
> >  * master-j32: 2.811 GB
> >  * next-j1: 2.256 GB
> >  * next-j16: 2.401 GB
> >  * next-j32: 2.613 GB
> >
> > It's definitely better. So I think it could work now, if the thread count
> > was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on
> > random machines (32bit processes only of course). Unlike -j1.
>
> Cool, I just merged a patch from Alan Maguire that should help with the
> parallel case, would be able to test it? It is in the 'next' branch:
>
> ⬢[acme@toolbox pahole]$ git log --oneline -5
> f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs
>

*R*edzce? memory usage ...

-Sedat-

> Excerpt of the above:
>
>     This leads to deleting ~90 CUs during parallel vmlinux BTF generation
>     versus deleting just 1 prior to this change.
>
> c7ec9200caa7d485 btf_encoder: Add "distilled_base" BTF feature to split BTF generation
> bc4e6a9adfc72758 pahole: Sync with libbpf-1.5
> 5e3ed3ec2947c69f pahole: Do --lang_exclude CU filtering earlier
> c46455bb0379fa38 dwarf_loader: Allow filtering CUs early in loading
> ⬢[acme@toolbox pahole]$
>
> - Arnaldo
>

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Phil Auld]

On Mon, Aug 26, 2024 at 08:42:10PM +0200 Sedat Dilek wrote:
> On Mon, Aug 26, 2024 at 7:03 PM Arnaldo Carvalho de Melo
> <acme@kernel.org> wrote:
> >
> > On Mon, Aug 26, 2024 at 10:57:22AM +0200, Jiri Slaby wrote:
> > > On 22. 08. 24, 17:24, Arnaldo Carvalho de Melo wrote:
> > > > On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
> > > > I stumbled on this limitation as well when trying to build the kernel on
> > > > a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just
> > > > created a swapfile and it managed to proceed, a bit slowly, but worked
> > > > as well.
> > >
> > > Here, it hits the VM space limit (3 G).
> >
> > right, in my case it was on a 64-bit system, so just not enough memory,
> > not address space.
> >
> > > > Please let me know if what is in the 'next' branch of:
> >
> > > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git
> >
> > > > Works for you, that will be extra motivation to move it to the master
> > > > branch and cut 1.28.
> >
> > > on 64bit (-j1):
> > > * master: 3.706 GB
> > > (* master + my changes: 3.559 GB)
> > > * next: 3.157 GB
> >
> > > on 32bit:
> > >  * master-j1: 2.445 GB
> > >  * master-j16: 2.608 GB
> > >  * master-j32: 2.811 GB
> > >  * next-j1: 2.256 GB
> > >  * next-j16: 2.401 GB
> > >  * next-j32: 2.613 GB
> > >
> > > It's definitely better. So I think it could work now, if the thread count
> > > was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on
> > > random machines (32bit processes only of course). Unlike -j1.
> >
> > Cool, I just merged a patch from Alan Maguire that should help with the
> > parallel case, would be able to test it? It is in the 'next' branch:
> >
> > ⬢[acme@toolbox pahole]$ git log --oneline -5
> > f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs
> >
> 
> *R*edzce? memory usage ...
>

If you meant that further typo it's golden, and if not the irony is rich :)

Either way this is my favorite email of the day!


Cheers,
Phil


> -Sedat-
> 
> > Excerpt of the above:
> >
> >     This leads to deleting ~90 CUs during parallel vmlinux BTF generation
> >     versus deleting just 1 prior to this change.
> >
> > c7ec9200caa7d485 btf_encoder: Add "distilled_base" BTF feature to split BTF generation
> > bc4e6a9adfc72758 pahole: Sync with libbpf-1.5
> > 5e3ed3ec2947c69f pahole: Do --lang_exclude CU filtering earlier
> > c46455bb0379fa38 dwarf_loader: Allow filtering CUs early in loading
> > ⬢[acme@toolbox pahole]$
> >
> > - Arnaldo
> >
> 

-- 

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Arnaldo Carvalho de Melo]

On Mon, Aug 26, 2024 at 02:48:18PM -0400, Phil Auld wrote:
> On Mon, Aug 26, 2024 at 08:42:10PM +0200 Sedat Dilek wrote:
> > On Mon, Aug 26, 2024 at 7:03 PM Arnaldo Carvalho de Melo
> > <acme@kernel.org> wrote:
> > >
> > > On Mon, Aug 26, 2024 at 10:57:22AM +0200, Jiri Slaby wrote:
> > > > On 22. 08. 24, 17:24, Arnaldo Carvalho de Melo wrote:
> > > > > On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
> > > > > I stumbled on this limitation as well when trying to build the kernel on
> > > > > a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just
> > > > > created a swapfile and it managed to proceed, a bit slowly, but worked
> > > > > as well.
> > > >
> > > > Here, it hits the VM space limit (3 G).
> > >
> > > right, in my case it was on a 64-bit system, so just not enough memory,
> > > not address space.
> > >
> > > > > Please let me know if what is in the 'next' branch of:
> > >
> > > > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git
> > >
> > > > > Works for you, that will be extra motivation to move it to the master
> > > > > branch and cut 1.28.
> > >
> > > > on 64bit (-j1):
> > > > * master: 3.706 GB
> > > > (* master + my changes: 3.559 GB)
> > > > * next: 3.157 GB
> > >
> > > > on 32bit:
> > > >  * master-j1: 2.445 GB
> > > >  * master-j16: 2.608 GB
> > > >  * master-j32: 2.811 GB
> > > >  * next-j1: 2.256 GB
> > > >  * next-j16: 2.401 GB
> > > >  * next-j32: 2.613 GB
> > > >
> > > > It's definitely better. So I think it could work now, if the thread count
> > > > was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on
> > > > random machines (32bit processes only of course). Unlike -j1.
> > >
> > > Cool, I just merged a patch from Alan Maguire that should help with the
> > > parallel case, would be able to test it? It is in the 'next' branch:
> > >
> > > ⬢[acme@toolbox pahole]$ git log --oneline -5
> > > f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs
> > >
> > 
> > *R*edzce? memory usage ...
> >
> 
> If you meant that further typo it's golden, and if not the irony is rich :)
> 
> Either way this is my favorite email of the day!

Hahaha, I went to uppercase what comes after the colon and introduced
that typo ;-)

Faxing it....

- Arnaldo

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Andrii Nakryiko]

On Mon, Aug 26, 2024 at 1:04 PM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> On Mon, Aug 26, 2024 at 02:48:18PM -0400, Phil Auld wrote:
> > On Mon, Aug 26, 2024 at 08:42:10PM +0200 Sedat Dilek wrote:
> > > On Mon, Aug 26, 2024 at 7:03 PM Arnaldo Carvalho de Melo
> > > <acme@kernel.org> wrote:
> > > >
> > > > On Mon, Aug 26, 2024 at 10:57:22AM +0200, Jiri Slaby wrote:
> > > > > On 22. 08. 24, 17:24, Arnaldo Carvalho de Melo wrote:
> > > > > > On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
> > > > > > I stumbled on this limitation as well when trying to build the kernel on
> > > > > > a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just
> > > > > > created a swapfile and it managed to proceed, a bit slowly, but worked
> > > > > > as well.
> > > > >
> > > > > Here, it hits the VM space limit (3 G).
> > > >
> > > > right, in my case it was on a 64-bit system, so just not enough memory,
> > > > not address space.
> > > >
> > > > > > Please let me know if what is in the 'next' branch of:
> > > >
> > > > > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git
> > > >
> > > > > > Works for you, that will be extra motivation to move it to the master
> > > > > > branch and cut 1.28.
> > > >
> > > > > on 64bit (-j1):
> > > > > * master: 3.706 GB
> > > > > (* master + my changes: 3.559 GB)
> > > > > * next: 3.157 GB
> > > >
> > > > > on 32bit:
> > > > >  * master-j1: 2.445 GB
> > > > >  * master-j16: 2.608 GB
> > > > >  * master-j32: 2.811 GB
> > > > >  * next-j1: 2.256 GB
> > > > >  * next-j16: 2.401 GB
> > > > >  * next-j32: 2.613 GB
> > > > >
> > > > > It's definitely better. So I think it could work now, if the thread count
> > > > > was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on
> > > > > random machines (32bit processes only of course). Unlike -j1.
> > > >
> > > > Cool, I just merged a patch from Alan Maguire that should help with the
> > > > parallel case, would be able to test it? It is in the 'next' branch:
> > > >
> > > > ⬢[acme@toolbox pahole]$ git log --oneline -5
> > > > f37212d1611673a2 (HEAD -> master) pahole: Teduce memory usage by smarter deleting of CUs
> > > >
> > >
> > > *R*edzce? memory usage ...
> > >
> >
> > If you meant that further typo it's golden, and if not the irony is rich :)
> >
> > Either way this is my favorite email of the day!
>
> Hahaha, I went to uppercase what comes after the colon and introduced
> that typo ;-)
>
> Faxing it....

typo-fest continues ;)

but it's great to see that a new pahole release is coming (cc'ing
Usama), we are eagerly waiting for one of the bug fixes that will go
into 1.28 (one of the first commits after 1.27 was cut). Note, libbpf
CI's for pahole-staging is failing, but that seems a one-off test and
has nothing to do with pahole being broken.

>
> - Arnaldo
>

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Jiri Slaby]

On 26. 08. 24, 19:03, Arnaldo Carvalho de Melo wrote:
> On Mon, Aug 26, 2024 at 10:57:22AM +0200, Jiri Slaby wrote:
>> On 22. 08. 24, 17:24, Arnaldo Carvalho de Melo wrote:
>>> On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
>>> I stumbled on this limitation as well when trying to build the kernel on
>>> a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just
>>> created a swapfile and it managed to proceed, a bit slowly, but worked
>>> as well.
>>
>> Here, it hits the VM space limit (3 G).
> 
> right, in my case it was on a 64-bit system, so just not enough memory,
> not address space.
>   
>>> Please let me know if what is in the 'next' branch of:
> 
>>> https://git.kernel.org/pub/scm/devel/pahole/pahole.git
> 
>>> Works for you, that will be extra motivation to move it to the master
>>> branch and cut 1.28.
> 
>> on 64bit (-j1):
>> * master: 3.706 GB
>> (* master + my changes: 3.559 GB)
>> * next: 3.157 GB
>   
>> on 32bit:
>>   * master-j1: 2.445 GB
>>   * master-j16: 2.608 GB
>>   * master-j32: 2.811 GB
>>   * next-j1: 2.256 GB
>>   * next-j16: 2.401 GB
>>   * next-j32: 2.613 GB
>>
>> It's definitely better. So I think it could work now, if the thread count
>> was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on
>> random machines (32bit processes only of course). Unlike -j1.
> 
> Cool, I just merged a patch from Alan Maguire that should help with the
> parallel case, would be able to test it? It is in the 'next' branch:

Not much helping.

On my box (as all previous runs):
next-j1 2.242
next-j16 2.808
next-j32 2.646

On a build host:
next-j1: 2.242
next-j16: 2.824
next-j20: 2.902 (crash)
next-j32: 2.902 (crash)

-- 
js
suse labs

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Jiri Slaby]

On 27. 08. 24, 10:37, Jiri Slaby wrote:
> On 26. 08. 24, 19:03, Arnaldo Carvalho de Melo wrote:
>> On Mon, Aug 26, 2024 at 10:57:22AM +0200, Jiri Slaby wrote:
>>> On 22. 08. 24, 17:24, Arnaldo Carvalho de Melo wrote:
>>>> On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
>>>> I stumbled on this limitation as well when trying to build the 
>>>> kernel on
>>>> a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just
>>>> created a swapfile and it managed to proceed, a bit slowly, but worked
>>>> as well.
>>>
>>> Here, it hits the VM space limit (3 G).
>>
>> right, in my case it was on a 64-bit system, so just not enough memory,
>> not address space.
>>>> Please let me know if what is in the 'next' branch of:
>>
>>>> https://git.kernel.org/pub/scm/devel/pahole/pahole.git
>>
>>>> Works for you, that will be extra motivation to move it to the master
>>>> branch and cut 1.28.
>>
>>> on 64bit (-j1):
>>> * master: 3.706 GB
>>> (* master + my changes: 3.559 GB)
>>> * next: 3.157 GB
>>> on 32bit:
>>>   * master-j1: 2.445 GB
>>>   * master-j16: 2.608 GB
>>>   * master-j32: 2.811 GB
>>>   * next-j1: 2.256 GB
>>>   * next-j16: 2.401 GB
>>>   * next-j32: 2.613 GB
>>>
>>> It's definitely better. So I think it could work now, if the thread 
>>> count
>>> was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on
>>> random machines (32bit processes only of course). Unlike -j1.
>>
>> Cool, I just merged a patch from Alan Maguire that should help with the
>> parallel case, would be able to test it? It is in the 'next' branch:
> 
> Not much helping.
> 
> On my box (as all previous runs):
> next-j1 2.242
> next-j16 2.808
> next-j32 2.646
> 
> On a build host:
> next-j1: 2.242
> next-j16: 2.824
> next-j20: 2.902 (crash)
> next-j32: 2.902 (crash)

So I disabled BTF on all 32bits (arm+i586) in openSUSE Tumbleweed for 
now. Better kernel without BTF, than no kernel at all 8-).

If you have ideas to force -j1 on 32bit, I can reenable/retest...

thanks,
-- 
js
suse labs

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Arnaldo Carvalho de Melo]

On Mon, Aug 26, 2024 at 10:57:22AM +0200, Jiri Slaby wrote:
> On 22. 08. 24, 17:24, Arnaldo Carvalho de Melo wrote:
> > On Thu, Aug 22, 2024 at 11:55:05AM +0800, Shung-Hsi Yu wrote:
> > I stumbled on this limitation as well when trying to build the kernel on
> > a Libre Computer rk3399-pc board with only 4GiB of RAM, there I just
> > created a swapfile and it managed to proceed, a bit slowly, but worked
> > as well.
> 
> Here, it hits the VM space limit (3 G).
> 
> > Please let me know if what is in the 'next' branch of:
> > 
> > https://git.kernel.org/pub/scm/devel/pahole/pahole.git
> > 
> > Works for you, that will be extra motivation to move it to the master
> > branch and cut 1.28.
> 
> on 64bit (-j1):
> * master: 3.706 GB
> (* master + my changes: 3.559 GB)
> * next: 3.157 GB
> 
> 
> on 32bit:
>  * master-j1: 2.445 GB
>  * master-j16: 2.608 GB
>  * master-j32: 2.811 GB
>  * next-j1: 2.256 GB
>  * next-j16: 2.401 GB
>  * next-j32: 2.613 GB
> 
> It's definitely better. So I think it could work now, if the thread count
> was limited to 1 on 32bit. As building with -j10, -j20 randomly fails on
> random machines (32bit processes only of course). Unlike -j1.

Great! I'm now processing a patch by Alan Maguire that should help with
parallel loading, I'll add it o the next branch, together with his work
on distilled BTF.

If you could test it, it would be great to see how much it helps with
teh serial case and if it allows for at least some parallel processing
on 32-bit architectures.

Thanks for all your effort on this, appreciated.

- Arnaldo

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Sedat Dilek]

On Thu, Aug 22, 2024 at 5:24 PM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:

> Please let me know if what is in the 'next' branch of:
>
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git
>
> Works for you, that will be extra motivation to move it to the master
> branch and cut 1.28.

For pahole version 1.28 - Please, Go Go Go.

-Sedat-

pahole 1.27 segfaults when generating BTF for modules built with LTO #2032
https://github.com/ClangBuiltLinux/linux/issues/2032

Re: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

[Sedat Dilek]

On Mon, Aug 26, 2024 at 12:18 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Thu, Aug 22, 2024 at 5:24 PM Arnaldo Carvalho de Melo
> <acme@kernel.org> wrote:
>
> > Please let me know if what is in the 'next' branch of:
> >
> > https://git.kernel.org/pub/scm/devel/pahole/pahole.git
> >
> > Works for you, that will be extra motivation to move it to the master
> > branch and cut 1.28.
>
> For pahole version 1.28 - Please, Go Go Go.
>
> -Sedat-
>
> pahole 1.27 segfaults when generating BTF for modules built with LTO #2032
> https://github.com/ClangBuiltLinux/linux/issues/2032

Hi Arnaldo,

Any news for pahole version 1.28 release?

Thanks.

Best regards,
-Sedat-