[PATCHES 0/2] Introduce --btf_features=+extra_features syntax

[PATCHES 0/2] Introduce --btf_features=+extra_features syntax

[Arnaldo Carvalho de Melo]

Hi,

	Please take a look if you agree this is a more compact, less
confusing way of asking for the set of standard BTF features + some
extra features such as 'reproducible_build'.

	We have this in perf, for things like:

⬢[acme@toolbox pahole]$ perf report -h -F 

 Usage: perf report [<options>]

    -F, --fields <key[,keys...]>
                          output field(s): overhead period sample  overhead overhead_sys
                          overhead_us overhead_guest_sys overhead_guest_us overhead_children
                          sample period weight1 weight2 weight3 ins_lat retire_lat
                          p_stage_cyc pid comm dso symbol parent cpu socket
                          srcline srcfile local_weight weight transaction trace
                          symbol_size dso_size cgroup cgroup_id ipc_null time
                          code_page_size local_ins_lat ins_lat local_p_stage_cyc
                          p_stage_cyc addr local_retire_lat retire_lat simd
                          type typeoff symoff dso_from dso_to symbol_from symbol_to
                          mispredict abort in_tx cycles srcline_from srcline_to
                          ipc_lbr addr_from addr_to symbol_daddr dso_daddr locked
                          tlb mem snoop dcacheline symbol_iaddr phys_daddr data_page_size
                          blocked

⬢[acme@toolbox pahole]$

From the 'perf report' man page for '-F':

        If the keys starts with a prefix '+', then it will append the specified
        field(s) to the default field order. For example: perf report -F +period,sample.

- Arnaldo

Arnaldo Carvalho de Melo (2):
  pahole: Factor out routine to process "--btf_features=all"
  pahole: Allow asking for extra features using the '+' prefix in
    --btf_features

 man-pages/pahole.1          |  6 ++++++
 pahole.c                    | 23 ++++++++++++++++-------
 tests/reproducible_build.sh |  2 +-
 3 files changed, 23 insertions(+), 8 deletions(-)

-- 
2.44.0

[PATCH 1/2] pahole: Factor out routine to process "--btf_features=all"

[Arnaldo Carvalho de Melo]

From: Arnaldo Carvalho de Melo <acme@redhat.com>

As we'll use it to process "--btf_features=+reproducible_build" meaning
"all + reproducible_build".

Cc: Alan Maguire <alan.maguire@oracle.com>
Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Daniel Xu <dxu@dxuuu.xyz>
Cc: Eduard Zingerman <eddyz87@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 pahole.c | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/pahole.c b/pahole.c
index 38cc6362015fd95b..af94d2a45ee96cbe 100644
--- a/pahole.c
+++ b/pahole.c
@@ -1340,6 +1340,14 @@ static void show_supported_btf_features(FILE *output)
 	fprintf(output, "\n");
 }
 
+static void btf_features__enable_for_all(void)
+{
+	for (int i = 0; i < ARRAY_SIZE(btf_features); i++) {
+		if (btf_features[i].enable_for_all)
+			enable_btf_feature(&btf_features[i]);
+	}
+}
+
 /* Translate --btf_features=feature1[,feature2] into conf_load values.
  * Explicitly ignores unrecognized features to allow future specification
  * of new opt-in features.
@@ -1352,12 +1360,7 @@ static void parse_btf_features(const char *features, bool strict)
 	init_btf_features();
 
 	if (strcmp(features, "all") == 0) {
-		int i;
-
-		for (i = 0; i < ARRAY_SIZE(btf_features); i++) {
-			if (btf_features[i].enable_for_all)
-				enable_btf_feature(&btf_features[i]);
-		}
+		btf_features__enable_for_all();
 		return;
 	}
 
@@ -1371,7 +1374,7 @@ static void parse_btf_features(const char *features, bool strict)
 			 * allowed.
 			 */
 			if (strcmp(feature_name, "all") == 0) {
-				parse_btf_features(feature_name, strict);
+				btf_features__enable_for_all();
 			} else if (strict) {
 				fprintf(stderr, "Feature '%s' in '%s' is not supported.  Supported BTF features are:\n",
 					feature_name, features);
-- 
2.44.0

[PATCH 2/2] pahole: Allow asking for extra features using the '+' prefix in --btf_features

[Arnaldo Carvalho de Melo]

From: Arnaldo Carvalho de Melo <acme@redhat.com>

Instead of the somewhat confusing:

  --btf_features=all,reproducible_build

That means "'all' the standard BTF features plus the 'reproducible_build'
extra BTF feature", use + directly, making it more compact:

  --btf_features=+reproducible_build

In the future we may want the '-' counterpart as a way to _remove_ some
of the standard set of BTF features.

Cc: Alan Maguire <alan.maguire@oracle.com>
Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Daniel Xu <dxu@dxuuu.xyz>
Cc: Eduard Zingerman <eddyz87@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 man-pages/pahole.1          | 6 ++++++
 pahole.c                    | 6 ++++++
 tests/reproducible_build.sh | 2 +-
 3 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
index 64de3438b5f9a77a..2f4f42f8323efd6e 100644
--- a/man-pages/pahole.1
+++ b/man-pages/pahole.1
@@ -320,6 +320,12 @@ Supported non-standard features (not enabled for 'all')
 
 So for example, specifying \-\-btf_encode=var,enum64 will result in a BTF encoding that (as well as encoding basic BTF information) will contain variables and enum64 values.
 
+.fi
+
+If one wants to add an extra feature to the set of standard ones, the '+' prefix can be used, i.e.:
+\-\-btf_features=+reproducible_build will add all standard features plus the 'reproducible_build' extra
+feature.
+
 .TP
 .B \-\-btf_features_strict
 Identical to \-\-btf_features above, but pahole will exit if it encounters an unrecognized feature.
diff --git a/pahole.c b/pahole.c
index af94d2a45ee96cbe..42c5b03ee1d1a8f8 100644
--- a/pahole.c
+++ b/pahole.c
@@ -1364,6 +1364,12 @@ static void parse_btf_features(const char *features, bool strict)
 		return;
 	}
 
+	// Adding extra features to the set of standard features.
+	if (strstarts(features, "+")) {
+		btf_features__enable_for_all();
+		++features;
+	}
+
 	strncpy(f, features, BTF_MAX_FEATURE_STR)[BTF_MAX_FEATURE_STR] = '\0';
 	s = f;
 	while ((feature_name = strtok_r(s, ",", &saveptr)) != NULL) {
diff --git a/tests/reproducible_build.sh b/tests/reproducible_build.sh
index e2f836081b125119..1222cb42c6639235 100755
--- a/tests/reproducible_build.sh
+++ b/tests/reproducible_build.sh
@@ -29,7 +29,7 @@ nr_proc=$(getconf _NPROCESSORS_ONLN)
 
 for threads in $(seq $nr_proc) ; do
 	test -n "$VERBOSE" && echo $threads threads encoding
-	pahole -j$threads --btf_features=all,reproducible_build --btf_encode_detached=$outdir/vmlinux.btf.parallel.reproducible $vmlinux &
+	pahole -j$threads --btf_features=+reproducible_build --btf_encode_detached=$outdir/vmlinux.btf.parallel.reproducible $vmlinux &
 	pahole=$!
 	# HACK: Wait a bit for pahole to start its threads
 	sleep 0.3s
-- 
2.44.0

Re: [PATCH 2/2] pahole: Allow asking for extra features using the '+' prefix in --btf_features

[Andrii Nakryiko]

On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> From: Arnaldo Carvalho de Melo <acme@redhat.com>
>
> Instead of the somewhat confusing:
>
>   --btf_features=all,reproducible_build
>
> That means "'all' the standard BTF features plus the 'reproducible_build'
> extra BTF feature", use + directly, making it more compact:
>
>   --btf_features=+reproducible_build
>

for older paholes that don't yet know about + syntax, but support
--btf_features, will this effectively disable all features or how will
it work?

I'm thinking from the perspective of using +reproducible_build
unconditionally in kernel's build scripts, will it regress something
for current pahole versions?

> In the future we may want the '-' counterpart as a way to _remove_ some
> of the standard set of BTF features.
>
> Cc: Alan Maguire <alan.maguire@oracle.com>
> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
> Cc: Daniel Xu <dxu@dxuuu.xyz>
> Cc: Eduard Zingerman <eddyz87@gmail.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> ---
>  man-pages/pahole.1          | 6 ++++++
>  pahole.c                    | 6 ++++++
>  tests/reproducible_build.sh | 2 +-
>  3 files changed, 13 insertions(+), 1 deletion(-)
>

[...]

Re: [PATCH 2/2] pahole: Allow asking for extra features using the '+' prefix in --btf_features

[Arnaldo Carvalho de Melo]

On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote:
> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo
> <acme@kernel.org> wrote:
> >
> > From: Arnaldo Carvalho de Melo <acme@redhat.com>
> >
> > Instead of the somewhat confusing:
> >
> >   --btf_features=all,reproducible_build
> >
> > That means "'all' the standard BTF features plus the 'reproducible_build'
> > extra BTF feature", use + directly, making it more compact:
> >
> >   --btf_features=+reproducible_build
> >
> 
> for older paholes that don't yet know about + syntax, but support
> --btf_features, will this effectively disable all features or how will
> it work?
> 
> I'm thinking from the perspective of using +reproducible_build
> unconditionally in kernel's build scripts, will it regress something
> for current pahole versions?

Well, I think it will end up being discarded just like "all" or
"default", no? I.e. those were keywords not grokked by older pahole
versions, so ignored as we're not using --btf_features_strict, right?

Alan?

But then we're not yet using --btf_features in scripts/Makefile.btf,
right?

But as Daniel pointed out and Alan (I think) agreed, for things like
scripts we probably end up using the most verbose thing as:

	--btf_features=default,reproducible_build

to mean a set (the default set of BTF options) + an optional/extra
feature (reproducibe_build), that for people not used to the + syntax
may be more descriptive (I really think that both are confusing for
beginners knowing nothing about BTF and its evolution, etc).

Alan, also we released 1.26 with "all" meaning what we now call
"default", so we need to keep both meaning the same thing, right?

- Arnaldo
 
> > In the future we may want the '-' counterpart as a way to _remove_ some
> > of the standard set of BTF features.
> >
> > Cc: Alan Maguire <alan.maguire@oracle.com>
> > Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
> > Cc: Daniel Xu <dxu@dxuuu.xyz>
> > Cc: Eduard Zingerman <eddyz87@gmail.com>
> > Cc: Jiri Olsa <jolsa@kernel.org>
> > Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> > ---
> >  man-pages/pahole.1          | 6 ++++++
> >  pahole.c                    | 6 ++++++
> >  tests/reproducible_build.sh | 2 +-
> >  3 files changed, 13 insertions(+), 1 deletion(-)
> >
> 
> [...]

Re: [PATCH 2/2] pahole: Allow asking for extra features using the '+' prefix in --btf_features

[Alan Maguire]

On 26/04/2024 21:47, Arnaldo Carvalho de Melo wrote:
> On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote:
>> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo
>> <acme@kernel.org> wrote:
>>>
>>> From: Arnaldo Carvalho de Melo <acme@redhat.com>
>>>
>>> Instead of the somewhat confusing:
>>>
>>>   --btf_features=all,reproducible_build
>>>
>>> That means "'all' the standard BTF features plus the 'reproducible_build'
>>> extra BTF feature", use + directly, making it more compact:
>>>
>>>   --btf_features=+reproducible_build
>>>
>>
>> for older paholes that don't yet know about + syntax, but support
>> --btf_features, will this effectively disable all features or how will
>> it work?
>>
>> I'm thinking from the perspective of using +reproducible_build
>> unconditionally in kernel's build scripts, will it regress something
>> for current pahole versions?
> 
> Well, I think it will end up being discarded just like "all" or
> "default", no? I.e. those were keywords not grokked by older pahole
> versions, so ignored as we're not using --btf_features_strict, right?
> 
> Alan?
> 

Yep, it would just be ignored, so wouldn't have the desired behaviour
of enabling defaults + reproducible build option.

> But then we're not yet using --btf_features in scripts/Makefile.btf,
> right?
> 
> But as Daniel pointed out and Alan (I think) agreed, for things like
> scripts we probably end up using the most verbose thing as:
> 
> 	--btf_features=default,reproducible_build
> 
> to mean a set (the default set of BTF options) + an optional/extra
> feature (reproducibe_build), that for people not used to the + syntax
> may be more descriptive (I really think that both are confusing for
> beginners knowing nothing about BTF and its evolution, etc).
> 
> Alan, also we released 1.26 with "all" meaning what we now call
> "default", so we need to keep both meaning the same thing, right?
>

I might be missing something here, but I think we should always call out
explicitly the set of features we want in the kernel Makefile.btf
(something like [1]). The reason for this is that the concept of what is
"default" may evolve over time; for example it's going to include
Daniel's kfunc definitions for soon. That's a good thing, but it could
conceivably cause problems down the line. Consider a newer pahole - with
a newer set of defaults - running on an older kernel. In that case, we
could end up encoding BTF features we don't want.  By contrast, if we
always call out the full set of BTF features we want via
--btf_features=feature1,feature2 etc we'll always get the expected set.
Plus for folks consulting the code, it's much clearer which BTF features
are in use when they look at the Makefiles for a particular kernel.
So my sense of the value of "default" is as a shortcut for testing the
latest and greatest set of BTF feature encoding, but not for use in the
kernel tree Makefiles. Thanks!

Alan

[1]
https://lore.kernel.org/bpf/20240424154806.3417662-7-alan.maguire@oracle.com/

> - Arnaldo
>  
>>> In the future we may want the '-' counterpart as a way to _remove_ some
>>> of the standard set of BTF features.
>>>
>>> Cc: Alan Maguire <alan.maguire@oracle.com>
>>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
>>> Cc: Daniel Xu <dxu@dxuuu.xyz>
>>> Cc: Eduard Zingerman <eddyz87@gmail.com>
>>> Cc: Jiri Olsa <jolsa@kernel.org>
>>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
>>> ---
>>>  man-pages/pahole.1          | 6 ++++++
>>>  pahole.c                    | 6 ++++++
>>>  tests/reproducible_build.sh | 2 +-
>>>  3 files changed, 13 insertions(+), 1 deletion(-)
>>>
>>
>> [...]

Re: [PATCH 2/2] pahole: Allow asking for extra features using the '+' prefix in --btf_features

[Andrii Nakryiko]

On Mon, Apr 29, 2024 at 4:16 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> On 26/04/2024 21:47, Arnaldo Carvalho de Melo wrote:
> > On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote:
> >> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo
> >> <acme@kernel.org> wrote:
> >>>
> >>> From: Arnaldo Carvalho de Melo <acme@redhat.com>
> >>>
> >>> Instead of the somewhat confusing:
> >>>
> >>>   --btf_features=all,reproducible_build
> >>>
> >>> That means "'all' the standard BTF features plus the 'reproducible_build'
> >>> extra BTF feature", use + directly, making it more compact:
> >>>
> >>>   --btf_features=+reproducible_build
> >>>
> >>
> >> for older paholes that don't yet know about + syntax, but support
> >> --btf_features, will this effectively disable all features or how will
> >> it work?
> >>
> >> I'm thinking from the perspective of using +reproducible_build
> >> unconditionally in kernel's build scripts, will it regress something
> >> for current pahole versions?
> >
> > Well, I think it will end up being discarded just like "all" or
> > "default", no? I.e. those were keywords not grokked by older pahole
> > versions, so ignored as we're not using --btf_features_strict, right?
> >
> > Alan?
> >
>
> Yep, it would just be ignored, so wouldn't have the desired behaviour
> of enabling defaults + reproducible build option.
>
> > But then we're not yet using --btf_features in scripts/Makefile.btf,
> > right?
> >
> > But as Daniel pointed out and Alan (I think) agreed, for things like
> > scripts we probably end up using the most verbose thing as:
> >
> >       --btf_features=default,reproducible_build
> >
> > to mean a set (the default set of BTF options) + an optional/extra
> > feature (reproducibe_build), that for people not used to the + syntax
> > may be more descriptive (I really think that both are confusing for
> > beginners knowing nothing about BTF and its evolution, etc).
> >
> > Alan, also we released 1.26 with "all" meaning what we now call
> > "default", so we need to keep both meaning the same thing, right?
> >
>
> I might be missing something here, but I think we should always call out
> explicitly the set of features we want in the kernel Makefile.btf
> (something like [1]). The reason for this is that the concept of what is
> "default" may evolve over time; for example it's going to include
> Daniel's kfunc definitions for soon. That's a good thing, but it could
> conceivably cause problems down the line. Consider a newer pahole - with
> a newer set of defaults - running on an older kernel. In that case, we
> could end up encoding BTF features we don't want.  By contrast, if we
> always call out the full set of BTF features we want via
> --btf_features=feature1,feature2 etc we'll always get the expected set.
> Plus for folks consulting the code, it's much clearer which BTF features
> are in use when they look at the Makefiles for a particular kernel.
> So my sense of the value of "default" is as a shortcut for testing the
> latest and greatest set of BTF feature encoding, but not for use in the
> kernel tree Makefiles. Thanks!

Yep, I agree, the whole point was to not regress older kernel builds
with newer pahole, so we need to explicitly list used features.

>
> Alan
>
> [1]
> https://lore.kernel.org/bpf/20240424154806.3417662-7-alan.maguire@oracle.com/
>
> > - Arnaldo
> >
> >>> In the future we may want the '-' counterpart as a way to _remove_ some
> >>> of the standard set of BTF features.
> >>>
> >>> Cc: Alan Maguire <alan.maguire@oracle.com>
> >>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
> >>> Cc: Daniel Xu <dxu@dxuuu.xyz>
> >>> Cc: Eduard Zingerman <eddyz87@gmail.com>
> >>> Cc: Jiri Olsa <jolsa@kernel.org>
> >>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> >>> ---
> >>>  man-pages/pahole.1          | 6 ++++++
> >>>  pahole.c                    | 6 ++++++
> >>>  tests/reproducible_build.sh | 2 +-
> >>>  3 files changed, 13 insertions(+), 1 deletion(-)
> >>>
> >>
> >> [...]

Re: [PATCH 2/2] pahole: Allow asking for extra features using the '+' prefix in --btf_features

[Arnaldo Carvalho de Melo]

On Mon, Apr 29, 2024 at 12:16:18PM +0100, Alan Maguire wrote:
> On 26/04/2024 21:47, Arnaldo Carvalho de Melo wrote:
> > On Fri, Apr 26, 2024 at 01:26:40PM -0700, Andrii Nakryiko wrote:
> >> On Fri, Apr 19, 2024 at 1:58 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >> for older paholes that don't yet know about + syntax, but support
> >> --btf_features, will this effectively disable all features or how will
> >> it work?
> >>
> >> I'm thinking from the perspective of using +reproducible_build
> >> unconditionally in kernel's build scripts, will it regress something
> >> for current pahole versions?
> > 
> > Well, I think it will end up being discarded just like "all" or
> > "default", no? I.e. those were keywords not grokked by older pahole
> > versions, so ignored as we're not using --btf_features_strict, right?

> > Alan?
 
> Yep, it would just be ignored, so wouldn't have the desired behaviour
> of enabling defaults + reproducible build option.
 
> > But then we're not yet using --btf_features in scripts/Makefile.btf,
> > right?

> > But as Daniel pointed out and Alan (I think) agreed, for things like
> > scripts we probably end up using the most verbose thing as:

> > 	--btf_features=default,reproducible_build

> > to mean a set (the default set of BTF options) + an optional/extra
> > feature (reproducibe_build), that for people not used to the + syntax
> > may be more descriptive (I really think that both are confusing for
> > beginners knowing nothing about BTF and its evolution, etc).

> > Alan, also we released 1.26 with "all" meaning what we now call
> > "default", so we need to keep both meaning the same thing, right?

> I might be missing something here, but I think we should always call out

No you're not, I was just talking about what Daniel pointed out, that
when using a script using 'default,extra_feature' is more descriptive
than '+extra_feature', but you're right, for the kernel we go on adding
the features we need and older pahole versions will just ignore those.

The explanation you gave on this message is interesting to clarify when
'default' should be used and perhaps also use the kernel build process
needs/use of a list of features, etc. It would be good to have it in the
man page, can you provide a formal patch doing that?

Thanks,

- Arnaldo

> explicitly the set of features we want in the kernel Makefile.btf
> (something like [1]). The reason for this is that the concept of what is
> "default" may evolve over time; for example it's going to include
> Daniel's kfunc definitions for soon. That's a good thing, but it could
> conceivably cause problems down the line. Consider a newer pahole - with
> a newer set of defaults - running on an older kernel. In that case, we
> could end up encoding BTF features we don't want.  By contrast, if we
> always call out the full set of BTF features we want via
> --btf_features=feature1,feature2 etc we'll always get the expected set.
> Plus for folks consulting the code, it's much clearer which BTF features
> are in use when they look at the Makefiles for a particular kernel.
> So my sense of the value of "default" is as a shortcut for testing the
> latest and greatest set of BTF feature encoding, but not for use in the
> kernel tree Makefiles. Thanks!
> 
> Alan
> 
> [1]
> https://lore.kernel.org/bpf/20240424154806.3417662-7-alan.maguire@oracle.com/
> 
> > - Arnaldo
> >  
> >>> In the future we may want the '-' counterpart as a way to _remove_ some
> >>> of the standard set of BTF features.
> >>>
> >>> Cc: Alan Maguire <alan.maguire@oracle.com>
> >>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
> >>> Cc: Daniel Xu <dxu@dxuuu.xyz>
> >>> Cc: Eduard Zingerman <eddyz87@gmail.com>
> >>> Cc: Jiri Olsa <jolsa@kernel.org>
> >>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> >>> ---
> >>>  man-pages/pahole.1          | 6 ++++++
> >>>  pahole.c                    | 6 ++++++
> >>>  tests/reproducible_build.sh | 2 +-
> >>>  3 files changed, 13 insertions(+), 1 deletion(-)
> >>>
> >>
> >> [...]

Re: [PATCHES 0/2] Introduce --btf_features=+extra_features syntax

[Daniel Xu]

Hi Arnaldo,

On Fri, Apr 19, 2024 at 05:57:43PM -0300, Arnaldo Carvalho de Melo wrote:
> Hi,
> 
> 	Please take a look if you agree this is a more compact, less
> confusing way of asking for the set of standard BTF features + some
> extra features such as 'reproducible_build'.
> 
> 	We have this in perf, for things like:
> 
> ⬢[acme@toolbox pahole]$ perf report -h -F 
> 
>  Usage: perf report [<options>]
> 
>     -F, --fields <key[,keys...]>
>                           output field(s): overhead period sample  overhead overhead_sys
>                           overhead_us overhead_guest_sys overhead_guest_us overhead_children
>                           sample period weight1 weight2 weight3 ins_lat retire_lat
>                           p_stage_cyc pid comm dso symbol parent cpu socket
>                           srcline srcfile local_weight weight transaction trace
>                           symbol_size dso_size cgroup cgroup_id ipc_null time
>                           code_page_size local_ins_lat ins_lat local_p_stage_cyc
>                           p_stage_cyc addr local_retire_lat retire_lat simd
>                           type typeoff symoff dso_from dso_to symbol_from symbol_to
>                           mispredict abort in_tx cycles srcline_from srcline_to
>                           ipc_lbr addr_from addr_to symbol_daddr dso_daddr locked
>                           tlb mem snoop dcacheline symbol_iaddr phys_daddr data_page_size
>                           blocked
> 
> ⬢[acme@toolbox pahole]$
> 
> From the 'perf report' man page for '-F':
> 
>         If the keys starts with a prefix '+', then it will append the specified
>         field(s) to the default field order. For example: perf report -F +period,sample.

I think for perf it makes sense to have compact representation b/c
folks might be doing a lot of ad-hoc work. But encoding BTF seems more
like a write-once, read mostly. So having `+` notation doesn't feel like
it'd help that much.

As someone who's not seen that style of syntax before, it's not
immediately obvious what it does. But seeing `all`, I have a pretty
good idea.

[..]

Thanks,
Daniel

Re: [PATCHES 0/2] Introduce --btf_features=+extra_features syntax

[Alan Maguire]

On 23/04/2024 03:29, Daniel Xu wrote:
> Hi Arnaldo,
> 
> On Fri, Apr 19, 2024 at 05:57:43PM -0300, Arnaldo Carvalho de Melo wrote:
>> Hi,
>>
>> 	Please take a look if you agree this is a more compact, less
>> confusing way of asking for the set of standard BTF features + some
>> extra features such as 'reproducible_build'.
>>
>> 	We have this in perf, for things like:
>>
>> ⬢[acme@toolbox pahole]$ perf report -h -F 
>>
>>  Usage: perf report [<options>]
>>
>>     -F, --fields <key[,keys...]>
>>                           output field(s): overhead period sample  overhead overhead_sys
>>                           overhead_us overhead_guest_sys overhead_guest_us overhead_children
>>                           sample period weight1 weight2 weight3 ins_lat retire_lat
>>                           p_stage_cyc pid comm dso symbol parent cpu socket
>>                           srcline srcfile local_weight weight transaction trace
>>                           symbol_size dso_size cgroup cgroup_id ipc_null time
>>                           code_page_size local_ins_lat ins_lat local_p_stage_cyc
>>                           p_stage_cyc addr local_retire_lat retire_lat simd
>>                           type typeoff symoff dso_from dso_to symbol_from symbol_to
>>                           mispredict abort in_tx cycles srcline_from srcline_to
>>                           ipc_lbr addr_from addr_to symbol_daddr dso_daddr locked
>>                           tlb mem snoop dcacheline symbol_iaddr phys_daddr data_page_size
>>                           blocked
>>
>> ⬢[acme@toolbox pahole]$
>>
>> From the 'perf report' man page for '-F':
>>
>>         If the keys starts with a prefix '+', then it will append the specified
>>         field(s) to the default field order. For example: perf report -F +period,sample.
> 
> I think for perf it makes sense to have compact representation b/c
> folks might be doing a lot of ad-hoc work. But encoding BTF seems more
> like a write-once, read mostly. So having `+` notation doesn't feel like
> it'd help that much.
> 
> As someone who's not seen that style of syntax before, it's not
> immediately obvious what it does. But seeing `all`, I have a pretty
> good idea.
>

One thing we should probably bear in mind here is that for kernel builds
we will always explicitly call out the set of features we want rather
than use "all". So the "all" support is really more of a shortcut for
developers who run pahole standalone for testing BTF encoding. It is
still confusing though.

The +/- approach seems fine to me especially if there are precedents in
other tools; maybe we should also switch name to "default" instead of
"all" at the same time tho? The notion of default values internal to
pahole (when BTF features aren't explicitly set) isn't exposed to the
user, so I _think_ we can get away with using that term. We could
probably do a bit of internal renaming - set_btf_features_default() ->
set_btf_features_minimal() - to call these the minimal BTF features or
something similar..

> [..]
> 
> Thanks,
> Daniel

Re: [PATCHES 0/2] Introduce --btf_features=+extra_features syntax

[Arnaldo Carvalho de Melo]

On Tue, Apr 23, 2024 at 10:02:29AM +0100, Alan Maguire wrote:
> On 23/04/2024 03:29, Daniel Xu wrote:
> > On Fri, Apr 19, 2024 at 05:57:43PM -0300, Arnaldo Carvalho de Melo wrote:
> >> 	Please take a look if you agree this is a more compact, less
> >> confusing way of asking for the set of standard BTF features + some
> >> extra features such as 'reproducible_build'.
> >>
> >> 	We have this in perf, for things like:
> >>
> >> ⬢[acme@toolbox pahole]$ perf report -h -F 
> >>
> >>  Usage: perf report [<options>]
> >>
> >>     -F, --fields <key[,keys...]>
> >>                           output field(s): overhead period sample  overhead overhead_sys
> >>                           overhead_us overhead_guest_sys overhead_guest_us overhead_children
> >>                           sample period weight1 weight2 weight3 ins_lat retire_lat
> >>                           p_stage_cyc pid comm dso symbol parent cpu socket
> >>                           srcline srcfile local_weight weight transaction trace
> >>                           symbol_size dso_size cgroup cgroup_id ipc_null time
> >>                           code_page_size local_ins_lat ins_lat local_p_stage_cyc
> >>                           p_stage_cyc addr local_retire_lat retire_lat simd
> >>                           type typeoff symoff dso_from dso_to symbol_from symbol_to
> >>                           mispredict abort in_tx cycles srcline_from srcline_to
> >>                           ipc_lbr addr_from addr_to symbol_daddr dso_daddr locked
> >>                           tlb mem snoop dcacheline symbol_iaddr phys_daddr data_page_size
> >>                           blocked
> >>
> >> ⬢[acme@toolbox pahole]$
> >>
> >> From the 'perf report' man page for '-F':
> >>
> >>         If the keys starts with a prefix '+', then it will append the specified
> >>         field(s) to the default field order. For example: perf report -F +period,sample.

> > I think for perf it makes sense to have compact representation b/c
> > folks might be doing a lot of ad-hoc work. But encoding BTF seems more
> > like a write-once, read mostly. So having `+` notation doesn't feel like

In the case where documentation style is prefered, i.e. a write once
read mostly, as you said, then use the most descriptive form.

> > it'd help that much.

> > As someone who's not seen that style of syntax before, it's not
> > immediately obvious what it does. But seeing `all`, I have a pretty
> > good idea.
 
> One thing we should probably bear in mind here is that for kernel builds
> we will always explicitly call out the set of features we want rather
> than use "all". So the "all" support is really more of a shortcut for
> developers who run pahole standalone for testing BTF encoding. It is
> still confusing though.

Agreed, multiple people agreed 'all' is confusing as not _all_ BTF
features are selected by it.
 
> The +/- approach seems fine to me especially if there are precedents in

Yeah, so we'll have a very compact way of adding (and removing, if we
feel the need, by prefixing a undesired feature that is present in the
'default' set with -) features, in addition to a more detailed way,
i.e. these will be equivalent:

	--btf_features=default,reproducible_build

and:

	--btf_features=+reproducible_build

> other tools; maybe we should also switch name to "default" instead of
> "all" at the same time tho? The notion of default values internal to

I'm ok with this, so please send a patch renaming 'all' to 'default', on
top of what is now in the 'next' branch.

- Arnaldo

> pahole (when BTF features aren't explicitly set) isn't exposed to the
> user, so I _think_ we can get away with using that term. We could
> probably do a bit of internal renaming - set_btf_features_default() ->
> set_btf_features_minimal() - to call these the minimal BTF features or
> something similar..