[PATCH bpf-next 0/5] BTF: arbitrary __attribute__ encoding

[PATCH bpf-next 0/5] BTF: arbitrary __attribute__ encoding

[Ihor Solodrai]

This patch series extends BPF Type Format (BTF) to support arbitrary
__attribute__ encoding.

Setting the kind_flag to 1 in BTF type tags and decl tags now changes
the meaning for the encoded tag, in particular with respect to
btf_dump in libbpf.

If the kflag is set, then the string encoded by the tag represents the
full attribute-list of an attribute specifier [1].

This feature will allow extending tools such as pahole and bpftool to
capture and use more granular type information, and make it easier to
manage compatibility between clang and gcc BPF compilers.

[1] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Attribute-Syntax.html

Ihor Solodrai (5):
  libbpf: introduce kflag for type_tags and decl_tags in BTF
  libbpf: check the kflag of type tags in btf_dump
  selftests/bpf: add a btf_dump test for type_tags
  bpf: allow kind_flag for BTF type and decl tags
  selftests/bpf: add a BTF verification test for kflagged type_tag

 Documentation/bpf/btf.rst                     |  27 +++-
 kernel/bpf/btf.c                              |   7 +-
 tools/include/uapi/linux/btf.h                |   3 +-
 tools/lib/bpf/btf.c                           |  87 +++++++---
 tools/lib/bpf/btf.h                           |   3 +
 tools/lib/bpf/btf_dump.c                      |   5 +-
 tools/lib/bpf/libbpf.map                      |   2 +
 tools/testing/selftests/bpf/prog_tests/btf.c  |  23 ++-
 .../selftests/bpf/prog_tests/btf_dump.c       | 148 +++++++++++++-----
 tools/testing/selftests/bpf/test_btf.h        |   6 +
 10 files changed, 234 insertions(+), 77 deletions(-)

-- 
2.48.1

[PATCH bpf-next 1/5] libbpf: introduce kflag for type_tags and decl_tags in BTF

[Ihor Solodrai]

Add the following functions to libbpf API:
* btf__add_type_attr()
* btf__add_decl_attr()

These functions allow to add to BTF the type tags and decl tags with
info->kflag set to 1. The kflag indicates that the tag directly
encodes an __attribute__ and not a normal tag.

See Documentation/bpf/btf.rst changes for details on the semantics.

Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>
---
 Documentation/bpf/btf.rst      | 27 +++++++++--
 tools/include/uapi/linux/btf.h |  3 +-
 tools/lib/bpf/btf.c            | 87 +++++++++++++++++++++++++---------
 tools/lib/bpf/btf.h            |  3 ++
 tools/lib/bpf/libbpf.map       |  2 +
 5 files changed, 93 insertions(+), 29 deletions(-)

diff --git a/Documentation/bpf/btf.rst b/Documentation/bpf/btf.rst
index 2478cef758f8..615ded7b2442 100644
--- a/Documentation/bpf/btf.rst
+++ b/Documentation/bpf/btf.rst
@@ -102,8 +102,9 @@ Each type contains the following common data::
          * bits 24-28: kind (e.g. int, ptr, array...etc)
          * bits 29-30: unused
          * bit     31: kind_flag, currently used by
-         *             struct, union, fwd, enum and enum64.
-         */
+	 *             struct, union, enum, fwd, enum64,
+	 *             DECL_TAG and TYPE_TAG
+	 */
         __u32 info;
         /* "size" is used by INT, ENUM, STRUCT, UNION and ENUM64.
          * "size" tells the size of the type it is describing.
@@ -478,7 +479,7 @@ No additional type data follow ``btf_type``.
 
 ``struct btf_type`` encoding requirement:
  * ``name_off``: offset to a non-empty string
- * ``info.kind_flag``: 0
+ * ``info.kind_flag``: 0 or 1
  * ``info.kind``: BTF_KIND_DECL_TAG
  * ``info.vlen``: 0
  * ``type``: ``struct``, ``union``, ``func``, ``var`` or ``typedef``
@@ -489,7 +490,6 @@ No additional type data follow ``btf_type``.
         __u32   component_idx;
     };
 
-The ``name_off`` encodes btf_decl_tag attribute string.
 The ``type`` should be ``struct``, ``union``, ``func``, ``var`` or ``typedef``.
 For ``var`` or ``typedef`` type, ``btf_decl_tag.component_idx`` must be ``-1``.
 For the other three types, if the btf_decl_tag attribute is
@@ -499,12 +499,21 @@ the attribute is applied to a ``struct``/``union`` member or
 a ``func`` argument, and ``btf_decl_tag.component_idx`` should be a
 valid index (starting from 0) pointing to a member or an argument.
 
+If ``info.kind_flag`` is 0, then this is a normal decl tag, and the
+``name_off`` encodes btf_decl_tag attribute string.
+
+If ``info.kind_flag`` is 1, then the decl tag represents an arbitrary
+__attribute__. In this case, ``name_off`` encodes a string
+representing the attribute-list of the attribute specifier. For
+example, for an ``__attribute__((aligned(4)))`` the string's contents
+is ``aligned(4)``.
+
 2.2.18 BTF_KIND_TYPE_TAG
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
 ``struct btf_type`` encoding requirement:
  * ``name_off``: offset to a non-empty string
- * ``info.kind_flag``: 0
+ * ``info.kind_flag``: 0 or 1
  * ``info.kind``: BTF_KIND_TYPE_TAG
  * ``info.vlen``: 0
  * ``type``: the type with ``btf_type_tag`` attribute
@@ -522,6 +531,14 @@ type_tag, then zero or more const/volatile/restrict/typedef
 and finally the base type. The base type is one of
 int, ptr, array, struct, union, enum, func_proto and float types.
 
+Similarly to decl tags, if the ``info.kind_flag`` is 0, then this is a
+normal type tag, and the ``name_off`` encodes btf_type_tag attribute
+string.
+
+If ``info.kind_flag`` is 1, then the type tag represents an arbitrary
+__attribute__, and the ``name_off`` encodes a string representing the
+attribute-list of the attribute specifier.
+
 2.2.19 BTF_KIND_ENUM64
 ~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/tools/include/uapi/linux/btf.h b/tools/include/uapi/linux/btf.h
index ec1798b6d3ff..d602c26a0c2a 100644
--- a/tools/include/uapi/linux/btf.h
+++ b/tools/include/uapi/linux/btf.h
@@ -36,7 +36,8 @@ struct btf_type {
 	 * bits 24-28: kind (e.g. int, ptr, array...etc)
 	 * bits 29-30: unused
 	 * bit     31: kind_flag, currently used by
-	 *             struct, union, enum, fwd and enum64
+	 *             struct, union, enum, fwd, enum64,
+	 *             DECL_TAG and TYPE_TAG
 	 */
 	__u32 info;
 	/* "size" is used by INT, ENUM, STRUCT, UNION, DATASEC and ENUM64.
diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c
index 48c66f3a9200..df2808cee009 100644
--- a/tools/lib/bpf/btf.c
+++ b/tools/lib/bpf/btf.c
@@ -2090,7 +2090,7 @@ static int validate_type_id(int id)
 }
 
 /* generic append function for PTR, TYPEDEF, CONST/VOLATILE/RESTRICT */
-static int btf_add_ref_kind(struct btf *btf, int kind, const char *name, int ref_type_id)
+static int btf_add_ref_kind(struct btf *btf, int kind, const char *name, int ref_type_id, int kflag)
 {
 	struct btf_type *t;
 	int sz, name_off = 0;
@@ -2113,7 +2113,7 @@ static int btf_add_ref_kind(struct btf *btf, int kind, const char *name, int ref
 	}
 
 	t->name_off = name_off;
-	t->info = btf_type_info(kind, 0, 0);
+	t->info = btf_type_info(kind, 0, kflag);
 	t->type = ref_type_id;
 
 	return btf_commit_type(btf, sz);
@@ -2128,7 +2128,7 @@ static int btf_add_ref_kind(struct btf *btf, int kind, const char *name, int ref
  */
 int btf__add_ptr(struct btf *btf, int ref_type_id)
 {
-	return btf_add_ref_kind(btf, BTF_KIND_PTR, NULL, ref_type_id);
+	return btf_add_ref_kind(btf, BTF_KIND_PTR, NULL, ref_type_id, 0);
 }
 
 /*
@@ -2506,7 +2506,7 @@ int btf__add_fwd(struct btf *btf, const char *name, enum btf_fwd_kind fwd_kind)
 		struct btf_type *t;
 		int id;
 
-		id = btf_add_ref_kind(btf, BTF_KIND_FWD, name, 0);
+		id = btf_add_ref_kind(btf, BTF_KIND_FWD, name, 0, 0);
 		if (id <= 0)
 			return id;
 		t = btf_type_by_id(btf, id);
@@ -2536,7 +2536,7 @@ int btf__add_typedef(struct btf *btf, const char *name, int ref_type_id)
 	if (!name || !name[0])
 		return libbpf_err(-EINVAL);
 
-	return btf_add_ref_kind(btf, BTF_KIND_TYPEDEF, name, ref_type_id);
+	return btf_add_ref_kind(btf, BTF_KIND_TYPEDEF, name, ref_type_id, 0);
 }
 
 /*
@@ -2548,7 +2548,7 @@ int btf__add_typedef(struct btf *btf, const char *name, int ref_type_id)
  */
 int btf__add_volatile(struct btf *btf, int ref_type_id)
 {
-	return btf_add_ref_kind(btf, BTF_KIND_VOLATILE, NULL, ref_type_id);
+	return btf_add_ref_kind(btf, BTF_KIND_VOLATILE, NULL, ref_type_id, 0);
 }
 
 /*
@@ -2560,7 +2560,7 @@ int btf__add_volatile(struct btf *btf, int ref_type_id)
  */
 int btf__add_const(struct btf *btf, int ref_type_id)
 {
-	return btf_add_ref_kind(btf, BTF_KIND_CONST, NULL, ref_type_id);
+	return btf_add_ref_kind(btf, BTF_KIND_CONST, NULL, ref_type_id, 0);
 }
 
 /*
@@ -2572,7 +2572,7 @@ int btf__add_const(struct btf *btf, int ref_type_id)
  */
 int btf__add_restrict(struct btf *btf, int ref_type_id)
 {
-	return btf_add_ref_kind(btf, BTF_KIND_RESTRICT, NULL, ref_type_id);
+	return btf_add_ref_kind(btf, BTF_KIND_RESTRICT, NULL, ref_type_id, 0);
 }
 
 /*
@@ -2588,7 +2588,24 @@ int btf__add_type_tag(struct btf *btf, const char *value, int ref_type_id)
 	if (!value || !value[0])
 		return libbpf_err(-EINVAL);
 
-	return btf_add_ref_kind(btf, BTF_KIND_TYPE_TAG, value, ref_type_id);
+	return btf_add_ref_kind(btf, BTF_KIND_TYPE_TAG, value, ref_type_id, 0);
+}
+
+/*
+ * Append new BTF_KIND_TYPE_TAG type with:
+ *   - *value*, non-empty/non-NULL tag value;
+ *   - *ref_type_id* - referenced type ID, it might not exist yet;
+ * Set info->kflag to 1, indicating this tag is an __attribute__
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__add_type_attr(struct btf *btf, const char *value, int ref_type_id)
+{
+	if (!value || !value[0])
+		return libbpf_err(-EINVAL);
+
+	return btf_add_ref_kind(btf, BTF_KIND_TYPE_TAG, value, ref_type_id, 1);
 }
 
 /*
@@ -2610,7 +2627,7 @@ int btf__add_func(struct btf *btf, const char *name,
 	    linkage != BTF_FUNC_EXTERN)
 		return libbpf_err(-EINVAL);
 
-	id = btf_add_ref_kind(btf, BTF_KIND_FUNC, name, proto_type_id);
+	id = btf_add_ref_kind(btf, BTF_KIND_FUNC, name, proto_type_id, 0);
 	if (id > 0) {
 		struct btf_type *t = btf_type_by_id(btf, id);
 
@@ -2845,18 +2862,9 @@ int btf__add_datasec_var_info(struct btf *btf, int var_type_id, __u32 offset, __
 	return 0;
 }
 
-/*
- * Append new BTF_KIND_DECL_TAG type with:
- *   - *value* - non-empty/non-NULL string;
- *   - *ref_type_id* - referenced type ID, it might not exist yet;
- *   - *component_idx* - -1 for tagging reference type, otherwise struct/union
- *     member or function argument index;
- * Returns:
- *   - >0, type ID of newly added BTF type;
- *   - <0, on error.
- */
-int btf__add_decl_tag(struct btf *btf, const char *value, int ref_type_id,
-		 int component_idx)
+
+static int __btf__add_decl_tag(struct btf *btf, const char *value,
+		 int ref_type_id, int component_idx, int kflag)
 {
 	struct btf_type *t;
 	int sz, value_off;
@@ -2880,13 +2888,46 @@ int btf__add_decl_tag(struct btf *btf, const char *value, int ref_type_id,
 		return value_off;
 
 	t->name_off = value_off;
-	t->info = btf_type_info(BTF_KIND_DECL_TAG, 0, false);
+	t->info = btf_type_info(BTF_KIND_DECL_TAG, 0, kflag);
 	t->type = ref_type_id;
 	btf_decl_tag(t)->component_idx = component_idx;
 
 	return btf_commit_type(btf, sz);
 }
 
+/*
+ * Append new BTF_KIND_DECL_TAG type with:
+ *   - *value* - non-empty/non-NULL string;
+ *   - *ref_type_id* - referenced type ID, it might not exist yet;
+ *   - *component_idx* - -1 for tagging reference type, otherwise struct/union
+ *     member or function argument index;
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__add_decl_tag(struct btf *btf, const char *value, int ref_type_id,
+		 int component_idx)
+{
+	return __btf__add_decl_tag(btf, value, ref_type_id, component_idx, 0);
+}
+
+/*
+ * Append new BTF_KIND_DECL_TAG type with:
+ *   - *value* - non-empty/non-NULL string;
+ *   - *ref_type_id* - referenced type ID, it might not exist yet;
+ *   - *component_idx* - -1 for tagging reference type, otherwise struct/union
+ *     member or function argument index;
+ * Set info->kflag to 1, indicating this tag is an __attribute__
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__add_decl_attr(struct btf *btf, const char *value, int ref_type_id,
+		 int component_idx)
+{
+	return __btf__add_decl_tag(btf, value, ref_type_id, component_idx, 1);
+}
+
 struct btf_ext_sec_info_param {
 	__u32 off;
 	__u32 len;
diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h
index 47ee8f6ac489..1c969a530a3e 100644
--- a/tools/lib/bpf/btf.h
+++ b/tools/lib/bpf/btf.h
@@ -227,6 +227,7 @@ LIBBPF_API int btf__add_volatile(struct btf *btf, int ref_type_id);
 LIBBPF_API int btf__add_const(struct btf *btf, int ref_type_id);
 LIBBPF_API int btf__add_restrict(struct btf *btf, int ref_type_id);
 LIBBPF_API int btf__add_type_tag(struct btf *btf, const char *value, int ref_type_id);
+LIBBPF_API int btf__add_type_attr(struct btf *btf, const char *value, int ref_type_id);
 
 /* func and func_proto construction APIs */
 LIBBPF_API int btf__add_func(struct btf *btf, const char *name,
@@ -243,6 +244,8 @@ LIBBPF_API int btf__add_datasec_var_info(struct btf *btf, int var_type_id,
 /* tag construction API */
 LIBBPF_API int btf__add_decl_tag(struct btf *btf, const char *value, int ref_type_id,
 			    int component_idx);
+LIBBPF_API int btf__add_decl_attr(struct btf *btf, const char *value, int ref_type_id,
+			    int component_idx);
 
 struct btf_dedup_opts {
 	size_t sz;
diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
index a8b2936a1646..8616e10b3c1b 100644
--- a/tools/lib/bpf/libbpf.map
+++ b/tools/lib/bpf/libbpf.map
@@ -436,4 +436,6 @@ LIBBPF_1.6.0 {
 		bpf_linker__add_buf;
 		bpf_linker__add_fd;
 		bpf_linker__new_fd;
+                btf__add_decl_attr;
+                btf__add_type_attr;
 } LIBBPF_1.5.0;
-- 
2.48.1

Re: [PATCH bpf-next 1/5] libbpf: introduce kflag for type_tags and decl_tags in BTF

[Alan Maguire]

On 22/01/2025 02:53, Ihor Solodrai wrote:
> Add the following functions to libbpf API:
> * btf__add_type_attr()
> * btf__add_decl_attr()
> 
> These functions allow to add to BTF the type tags and decl tags with
> info->kflag set to 1. The kflag indicates that the tag directly
> encodes an __attribute__ and not a normal tag.
> 
> See Documentation/bpf/btf.rst changes for details on the semantics.
> 
> Suggested-by: Andrii Nakryiko <andrii@kernel.org>
> Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>

need to sync include/uapi/linux/btf.h with
tools/include/uapi/linux/btf.h, but other than that

Reviewed-by: Alan Maguire <alan.maguire@oracle.com>

> ---
>  Documentation/bpf/btf.rst      | 27 +++++++++--
>  tools/include/uapi/linux/btf.h |  3 +-

same change needed to include/uapi/linux/btf.h too, right?

>  tools/lib/bpf/btf.c            | 87 +++++++++++++++++++++++++---------
>  tools/lib/bpf/btf.h            |  3 ++
>  tools/lib/bpf/libbpf.map       |  2 +
>  5 files changed, 93 insertions(+), 29 deletions(-)
> 
> diff --git a/Documentation/bpf/btf.rst b/Documentation/bpf/btf.rst
> index 2478cef758f8..615ded7b2442 100644
> --- a/Documentation/bpf/btf.rst
> +++ b/Documentation/bpf/btf.rst
> @@ -102,8 +102,9 @@ Each type contains the following common data::
>           * bits 24-28: kind (e.g. int, ptr, array...etc)
>           * bits 29-30: unused
>           * bit     31: kind_flag, currently used by
> -         *             struct, union, fwd, enum and enum64.
> -         */
> +	 *             struct, union, enum, fwd, enum64,
> +	 *             DECL_TAG and TYPE_TAG
> +	 */
>          __u32 info;
>          /* "size" is used by INT, ENUM, STRUCT, UNION and ENUM64.
>           * "size" tells the size of the type it is describing.
> @@ -478,7 +479,7 @@ No additional type data follow ``btf_type``.
>  
>  ``struct btf_type`` encoding requirement:
>   * ``name_off``: offset to a non-empty string
> - * ``info.kind_flag``: 0
> + * ``info.kind_flag``: 0 or 1
>   * ``info.kind``: BTF_KIND_DECL_TAG
>   * ``info.vlen``: 0
>   * ``type``: ``struct``, ``union``, ``func``, ``var`` or ``typedef``
> @@ -489,7 +490,6 @@ No additional type data follow ``btf_type``.
>          __u32   component_idx;
>      };
>  
> -The ``name_off`` encodes btf_decl_tag attribute string.
>  The ``type`` should be ``struct``, ``union``, ``func``, ``var`` or ``typedef``.
>  For ``var`` or ``typedef`` type, ``btf_decl_tag.component_idx`` must be ``-1``.
>  For the other three types, if the btf_decl_tag attribute is
> @@ -499,12 +499,21 @@ the attribute is applied to a ``struct``/``union`` member or
>  a ``func`` argument, and ``btf_decl_tag.component_idx`` should be a
>  valid index (starting from 0) pointing to a member or an argument.
>  
> +If ``info.kind_flag`` is 0, then this is a normal decl tag, and the
> +``name_off`` encodes btf_decl_tag attribute string.
> +
> +If ``info.kind_flag`` is 1, then the decl tag represents an arbitrary
> +__attribute__. In this case, ``name_off`` encodes a string
> +representing the attribute-list of the attribute specifier. For
> +example, for an ``__attribute__((aligned(4)))`` the string's contents
> +is ``aligned(4)``.
> +
>  2.2.18 BTF_KIND_TYPE_TAG
>  ~~~~~~~~~~~~~~~~~~~~~~~~
>  
>  ``struct btf_type`` encoding requirement:
>   * ``name_off``: offset to a non-empty string
> - * ``info.kind_flag``: 0
> + * ``info.kind_flag``: 0 or 1
>   * ``info.kind``: BTF_KIND_TYPE_TAG
>   * ``info.vlen``: 0
>   * ``type``: the type with ``btf_type_tag`` attribute
> @@ -522,6 +531,14 @@ type_tag, then zero or more const/volatile/restrict/typedef
>  and finally the base type. The base type is one of
>  int, ptr, array, struct, union, enum, func_proto and float types.
>  
> +Similarly to decl tags, if the ``info.kind_flag`` is 0, then this is a
> +normal type tag, and the ``name_off`` encodes btf_type_tag attribute
> +string.
> +
> +If ``info.kind_flag`` is 1, then the type tag represents an arbitrary
> +__attribute__, and the ``name_off`` encodes a string representing the
> +attribute-list of the attribute specifier.
> +
>  2.2.19 BTF_KIND_ENUM64
>  ~~~~~~~~~~~~~~~~~~~~~~
>  
> diff --git a/tools/include/uapi/linux/btf.h b/tools/include/uapi/linux/btf.h
> index ec1798b6d3ff..d602c26a0c2a 100644
> --- a/tools/include/uapi/linux/btf.h
> +++ b/tools/include/uapi/linux/btf.h
> @@ -36,7 +36,8 @@ struct btf_type {
>  	 * bits 24-28: kind (e.g. int, ptr, array...etc)
>  	 * bits 29-30: unused
>  	 * bit     31: kind_flag, currently used by
> -	 *             struct, union, enum, fwd and enum64
> +	 *             struct, union, enum, fwd, enum64,
> +	 *             DECL_TAG and TYPE_TAG
>  	 */
>  	__u32 info;
>  	/* "size" is used by INT, ENUM, STRUCT, UNION, DATASEC and ENUM64.
> diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c
> index 48c66f3a9200..df2808cee009 100644
> --- a/tools/lib/bpf/btf.c
> +++ b/tools/lib/bpf/btf.c
> @@ -2090,7 +2090,7 @@ static int validate_type_id(int id)
>  }
>  
>  /* generic append function for PTR, TYPEDEF, CONST/VOLATILE/RESTRICT */
> -static int btf_add_ref_kind(struct btf *btf, int kind, const char *name, int ref_type_id)
> +static int btf_add_ref_kind(struct btf *btf, int kind, const char *name, int ref_type_id, int kflag)
>  {
>  	struct btf_type *t;
>  	int sz, name_off = 0;
> @@ -2113,7 +2113,7 @@ static int btf_add_ref_kind(struct btf *btf, int kind, const char *name, int ref
>  	}
>  
>  	t->name_off = name_off;
> -	t->info = btf_type_info(kind, 0, 0);
> +	t->info = btf_type_info(kind, 0, kflag);
>  	t->type = ref_type_id;
>  
>  	return btf_commit_type(btf, sz);
> @@ -2128,7 +2128,7 @@ static int btf_add_ref_kind(struct btf *btf, int kind, const char *name, int ref
>   */
>  int btf__add_ptr(struct btf *btf, int ref_type_id)
>  {
> -	return btf_add_ref_kind(btf, BTF_KIND_PTR, NULL, ref_type_id);
> +	return btf_add_ref_kind(btf, BTF_KIND_PTR, NULL, ref_type_id, 0);
>  }
>  
>  /*
> @@ -2506,7 +2506,7 @@ int btf__add_fwd(struct btf *btf, const char *name, enum btf_fwd_kind fwd_kind)
>  		struct btf_type *t;
>  		int id;
>  
> -		id = btf_add_ref_kind(btf, BTF_KIND_FWD, name, 0);
> +		id = btf_add_ref_kind(btf, BTF_KIND_FWD, name, 0, 0);
>  		if (id <= 0)
>  			return id;
>  		t = btf_type_by_id(btf, id);
> @@ -2536,7 +2536,7 @@ int btf__add_typedef(struct btf *btf, const char *name, int ref_type_id)
>  	if (!name || !name[0])
>  		return libbpf_err(-EINVAL);
>  
> -	return btf_add_ref_kind(btf, BTF_KIND_TYPEDEF, name, ref_type_id);
> +	return btf_add_ref_kind(btf, BTF_KIND_TYPEDEF, name, ref_type_id, 0);
>  }
>  
>  /*
> @@ -2548,7 +2548,7 @@ int btf__add_typedef(struct btf *btf, const char *name, int ref_type_id)
>   */
>  int btf__add_volatile(struct btf *btf, int ref_type_id)
>  {
> -	return btf_add_ref_kind(btf, BTF_KIND_VOLATILE, NULL, ref_type_id);
> +	return btf_add_ref_kind(btf, BTF_KIND_VOLATILE, NULL, ref_type_id, 0);
>  }
>  
>  /*
> @@ -2560,7 +2560,7 @@ int btf__add_volatile(struct btf *btf, int ref_type_id)
>   */
>  int btf__add_const(struct btf *btf, int ref_type_id)
>  {
> -	return btf_add_ref_kind(btf, BTF_KIND_CONST, NULL, ref_type_id);
> +	return btf_add_ref_kind(btf, BTF_KIND_CONST, NULL, ref_type_id, 0);
>  }
>  
>  /*
> @@ -2572,7 +2572,7 @@ int btf__add_const(struct btf *btf, int ref_type_id)
>   */
>  int btf__add_restrict(struct btf *btf, int ref_type_id)
>  {
> -	return btf_add_ref_kind(btf, BTF_KIND_RESTRICT, NULL, ref_type_id);
> +	return btf_add_ref_kind(btf, BTF_KIND_RESTRICT, NULL, ref_type_id, 0);
>  }
>  
>  /*
> @@ -2588,7 +2588,24 @@ int btf__add_type_tag(struct btf *btf, const char *value, int ref_type_id)
>  	if (!value || !value[0])
>  		return libbpf_err(-EINVAL);
>  
> -	return btf_add_ref_kind(btf, BTF_KIND_TYPE_TAG, value, ref_type_id);
> +	return btf_add_ref_kind(btf, BTF_KIND_TYPE_TAG, value, ref_type_id, 0);
> +}
> +
> +/*
> + * Append new BTF_KIND_TYPE_TAG type with:
> + *   - *value*, non-empty/non-NULL tag value;
> + *   - *ref_type_id* - referenced type ID, it might not exist yet;
> + * Set info->kflag to 1, indicating this tag is an __attribute__
> + * Returns:
> + *   - >0, type ID of newly added BTF type;
> + *   - <0, on error.
> + */
> +int btf__add_type_attr(struct btf *btf, const char *value, int ref_type_id)
> +{
> +	if (!value || !value[0])
> +		return libbpf_err(-EINVAL);
> +
> +	return btf_add_ref_kind(btf, BTF_KIND_TYPE_TAG, value, ref_type_id, 1);
>  }
>  
>  /*
> @@ -2610,7 +2627,7 @@ int btf__add_func(struct btf *btf, const char *name,
>  	    linkage != BTF_FUNC_EXTERN)
>  		return libbpf_err(-EINVAL);
>  
> -	id = btf_add_ref_kind(btf, BTF_KIND_FUNC, name, proto_type_id);
> +	id = btf_add_ref_kind(btf, BTF_KIND_FUNC, name, proto_type_id, 0);
>  	if (id > 0) {
>  		struct btf_type *t = btf_type_by_id(btf, id);
>  
> @@ -2845,18 +2862,9 @@ int btf__add_datasec_var_info(struct btf *btf, int var_type_id, __u32 offset, __
>  	return 0;
>  }
>  
> -/*
> - * Append new BTF_KIND_DECL_TAG type with:
> - *   - *value* - non-empty/non-NULL string;
> - *   - *ref_type_id* - referenced type ID, it might not exist yet;
> - *   - *component_idx* - -1 for tagging reference type, otherwise struct/union
> - *     member or function argument index;
> - * Returns:
> - *   - >0, type ID of newly added BTF type;
> - *   - <0, on error.
> - */
> -int btf__add_decl_tag(struct btf *btf, const char *value, int ref_type_id,
> -		 int component_idx)
> +
> +static int __btf__add_decl_tag(struct btf *btf, const char *value,
> +		 int ref_type_id, int component_idx, int kflag)
>  {
>  	struct btf_type *t;
>  	int sz, value_off;
> @@ -2880,13 +2888,46 @@ int btf__add_decl_tag(struct btf *btf, const char *value, int ref_type_id,
>  		return value_off;
>  
>  	t->name_off = value_off;
> -	t->info = btf_type_info(BTF_KIND_DECL_TAG, 0, false);
> +	t->info = btf_type_info(BTF_KIND_DECL_TAG, 0, kflag);
>  	t->type = ref_type_id;
>  	btf_decl_tag(t)->component_idx = component_idx;
>  
>  	return btf_commit_type(btf, sz);
>  }
>  
> +/*
> + * Append new BTF_KIND_DECL_TAG type with:
> + *   - *value* - non-empty/non-NULL string;
> + *   - *ref_type_id* - referenced type ID, it might not exist yet;
> + *   - *component_idx* - -1 for tagging reference type, otherwise struct/union
> + *     member or function argument index;
> + * Returns:
> + *   - >0, type ID of newly added BTF type;
> + *   - <0, on error.
> + */
> +int btf__add_decl_tag(struct btf *btf, const char *value, int ref_type_id,
> +		 int component_idx)
> +{
> +	return __btf__add_decl_tag(btf, value, ref_type_id, component_idx, 0);
> +}
> +
> +/*
> + * Append new BTF_KIND_DECL_TAG type with:
> + *   - *value* - non-empty/non-NULL string;
> + *   - *ref_type_id* - referenced type ID, it might not exist yet;
> + *   - *component_idx* - -1 for tagging reference type, otherwise struct/union
> + *     member or function argument index;
> + * Set info->kflag to 1, indicating this tag is an __attribute__
> + * Returns:
> + *   - >0, type ID of newly added BTF type;
> + *   - <0, on error.
> + */
> +int btf__add_decl_attr(struct btf *btf, const char *value, int ref_type_id,
> +		 int component_idx)
> +{
> +	return __btf__add_decl_tag(btf, value, ref_type_id, component_idx, 1);
> +}
> +
>  struct btf_ext_sec_info_param {
>  	__u32 off;
>  	__u32 len;
> diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h
> index 47ee8f6ac489..1c969a530a3e 100644
> --- a/tools/lib/bpf/btf.h
> +++ b/tools/lib/bpf/btf.h
> @@ -227,6 +227,7 @@ LIBBPF_API int btf__add_volatile(struct btf *btf, int ref_type_id);
>  LIBBPF_API int btf__add_const(struct btf *btf, int ref_type_id);
>  LIBBPF_API int btf__add_restrict(struct btf *btf, int ref_type_id);
>  LIBBPF_API int btf__add_type_tag(struct btf *btf, const char *value, int ref_type_id);
> +LIBBPF_API int btf__add_type_attr(struct btf *btf, const char *value, int ref_type_id);
>  
>  /* func and func_proto construction APIs */
>  LIBBPF_API int btf__add_func(struct btf *btf, const char *name,
> @@ -243,6 +244,8 @@ LIBBPF_API int btf__add_datasec_var_info(struct btf *btf, int var_type_id,
>  /* tag construction API */
>  LIBBPF_API int btf__add_decl_tag(struct btf *btf, const char *value, int ref_type_id,
>  			    int component_idx);
> +LIBBPF_API int btf__add_decl_attr(struct btf *btf, const char *value, int ref_type_id,
> +			    int component_idx);
>  
>  struct btf_dedup_opts {
>  	size_t sz;
> diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
> index a8b2936a1646..8616e10b3c1b 100644
> --- a/tools/lib/bpf/libbpf.map
> +++ b/tools/lib/bpf/libbpf.map
> @@ -436,4 +436,6 @@ LIBBPF_1.6.0 {
>  		bpf_linker__add_buf;
>  		bpf_linker__add_fd;
>  		bpf_linker__new_fd;
> +                btf__add_decl_attr;
> +                btf__add_type_attr;
>  } LIBBPF_1.5.0;

Re: [PATCH bpf-next 1/5] libbpf: introduce kflag for type_tags and decl_tags in BTF

[Ihor Solodrai]

On Wednesday, January 22nd, 2025 at 2:56 AM, Alan Maguire <alan.maguire@oracle.com> wrote:

> 
> 
> On 22/01/2025 02:53, Ihor Solodrai wrote:
> 
> > Add the following functions to libbpf API:
> > * btf__add_type_attr()
> > * btf__add_decl_attr()
> > 
> > These functions allow to add to BTF the type tags and decl tags with
> > info->kflag set to 1. The kflag indicates that the tag directly
> > encodes an attribute and not a normal tag.
> > 
> > See Documentation/bpf/btf.rst changes for details on the semantics.
> > 
> > Suggested-by: Andrii Nakryiko andrii@kernel.org
> > Signed-off-by: Ihor Solodrai ihor.solodrai@pm.me
> 
> 
> need to sync include/uapi/linux/btf.h with
> tools/include/uapi/linux/btf.h, but other than that
> 
> Reviewed-by: Alan Maguire alan.maguire@oracle.com

I haven't realized btf.h is copied in two places.
Will fix, thanks.

> 
> [...]

Re: [PATCH bpf-next 1/5] libbpf: introduce kflag for type_tags and decl_tags in BTF

[Andrii Nakryiko]

On Tue, Jan 21, 2025 at 6:53 PM Ihor Solodrai <ihor.solodrai@pm.me> wrote:
>
> Add the following functions to libbpf API:
> * btf__add_type_attr()
> * btf__add_decl_attr()
>
> These functions allow to add to BTF the type tags and decl tags with
> info->kflag set to 1. The kflag indicates that the tag directly
> encodes an __attribute__ and not a normal tag.
>
> See Documentation/bpf/btf.rst changes for details on the semantics.
>
> Suggested-by: Andrii Nakryiko <andrii@kernel.org>
> Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>
> ---
>  Documentation/bpf/btf.rst      | 27 +++++++++--
>  tools/include/uapi/linux/btf.h |  3 +-

kernel documentation and uapi changes should be in a separate patch
from libbpf-side changes

>  tools/lib/bpf/btf.c            | 87 +++++++++++++++++++++++++---------
>  tools/lib/bpf/btf.h            |  3 ++
>  tools/lib/bpf/libbpf.map       |  2 +
>  5 files changed, 93 insertions(+), 29 deletions(-)
>

Please double-check whitespacing, tabs vs space might have been messed
up in this patch.

pw-bot: cr

> diff --git a/Documentation/bpf/btf.rst b/Documentation/bpf/btf.rst
> index 2478cef758f8..615ded7b2442 100644
> --- a/Documentation/bpf/btf.rst
> +++ b/Documentation/bpf/btf.rst
> @@ -102,8 +102,9 @@ Each type contains the following common data::
>           * bits 24-28: kind (e.g. int, ptr, array...etc)
>           * bits 29-30: unused
>           * bit     31: kind_flag, currently used by
> -         *             struct, union, fwd, enum and enum64.
> -         */
> +        *             struct, union, enum, fwd, enum64,
> +        *             DECL_TAG and TYPE_TAG

keep it lower case

> +        */
>          __u32 info;
>          /* "size" is used by INT, ENUM, STRUCT, UNION and ENUM64.
>           * "size" tells the size of the type it is describing.

[...]

> -/*
> - * Append new BTF_KIND_DECL_TAG type with:
> - *   - *value* - non-empty/non-NULL string;
> - *   - *ref_type_id* - referenced type ID, it might not exist yet;
> - *   - *component_idx* - -1 for tagging reference type, otherwise struct/union
> - *     member or function argument index;
> - * Returns:
> - *   - >0, type ID of newly added BTF type;
> - *   - <0, on error.
> - */
> -int btf__add_decl_tag(struct btf *btf, const char *value, int ref_type_id,
> -                int component_idx)
> +
> +static int __btf__add_decl_tag(struct btf *btf, const char *value,
> +                int ref_type_id, int component_idx, int kflag)

nit: please call it "btf_add_decl_tag" (to distinguish it from public
API, but also not use double-underscored name unnecessarily)

>  {
>         struct btf_type *t;
>         int sz, value_off;

[..]

[PATCH bpf-next 2/5] libbpf: check the kflag of type tags in btf_dump

[Ihor Solodrai]

If the kflag is set for a BTF type tag, then the tag represents an
arbitrary __attribute__. Change btf_dump accordingly.

Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>
---
 tools/lib/bpf/btf_dump.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tools/lib/bpf/btf_dump.c b/tools/lib/bpf/btf_dump.c
index a3fc6908f6c9..460c3e57fadb 100644
--- a/tools/lib/bpf/btf_dump.c
+++ b/tools/lib/bpf/btf_dump.c
@@ -1494,7 +1494,10 @@ static void btf_dump_emit_type_chain(struct btf_dump *d,
 		case BTF_KIND_TYPE_TAG:
 			btf_dump_emit_mods(d, decls);
 			name = btf_name_of(d, t->name_off);
-			btf_dump_printf(d, " __attribute__((btf_type_tag(\"%s\")))", name);
+			if (btf_kflag(t))
+				btf_dump_printf(d, " __attribute__((%s))", name);
+			else
+				btf_dump_printf(d, " __attribute__((btf_type_tag(\"%s\")))", name);
 			break;
 		case BTF_KIND_ARRAY: {
 			const struct btf_array *a = btf_array(t);
-- 
2.48.1

Re: [PATCH bpf-next 2/5] libbpf: check the kflag of type tags in btf_dump

[Alan Maguire]

On 22/01/2025 02:53, Ihor Solodrai wrote:
> If the kflag is set for a BTF type tag, then the tag represents an
> arbitrary __attribute__. Change btf_dump accordingly.
> 
> Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>

Reviewed-by: Alan Maguire <alan.maguire@oracle.com>

> ---
>  tools/lib/bpf/btf_dump.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/lib/bpf/btf_dump.c b/tools/lib/bpf/btf_dump.c
> index a3fc6908f6c9..460c3e57fadb 100644
> --- a/tools/lib/bpf/btf_dump.c
> +++ b/tools/lib/bpf/btf_dump.c
> @@ -1494,7 +1494,10 @@ static void btf_dump_emit_type_chain(struct btf_dump *d,
>  		case BTF_KIND_TYPE_TAG:
>  			btf_dump_emit_mods(d, decls);
>  			name = btf_name_of(d, t->name_off);
> -			btf_dump_printf(d, " __attribute__((btf_type_tag(\"%s\")))", name);
> +			if (btf_kflag(t))
> +				btf_dump_printf(d, " __attribute__((%s))", name);
> +			else
> +				btf_dump_printf(d, " __attribute__((btf_type_tag(\"%s\")))", name);
>  			break;
>  		case BTF_KIND_ARRAY: {
>  			const struct btf_array *a = btf_array(t);

[PATCH bpf-next 3/5] selftests/bpf: add a btf_dump test for type_tags

[Ihor Solodrai]

Factor out common routines handling custom BTF from
test_btf_dump_incremental. Then use them in the
test_btf_dump_type_tags.

test_btf_dump_type_tags verifies that a type tag is dumped correctly
with respect to its kflag.

Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>
---
 .../selftests/bpf/prog_tests/btf_dump.c       | 148 +++++++++++++-----
 1 file changed, 111 insertions(+), 37 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/btf_dump.c b/tools/testing/selftests/bpf/prog_tests/btf_dump.c
index b293b8501fd6..690cf8cef7d2 100644
--- a/tools/testing/selftests/bpf/prog_tests/btf_dump.c
+++ b/tools/testing/selftests/bpf/prog_tests/btf_dump.c
@@ -126,26 +126,70 @@ static int test_btf_dump_case(int n, struct btf_dump_test_case *t)
 	return err;
 }
 
-static char *dump_buf;
-static size_t dump_buf_sz;
-static FILE *dump_buf_file;
+struct btf_dump__custom_btf_test {
+	struct btf *btf;
+	struct btf_dump *d;
+	char *dump_buf;
+	size_t dump_buf_sz;
+	FILE *dump_buf_file;
+};
 
-static void test_btf_dump_incremental(void)
+static void btf_dump__custom_btf_test__free(struct btf_dump__custom_btf_test *t)
 {
-	struct btf *btf = NULL;
-	struct btf_dump *d = NULL;
-	int id, err, i;
+	fclose(t->dump_buf_file);
+	free(t->dump_buf);
+	btf_dump__free(t->d);
+	btf__free(t->btf);
+}
 
-	dump_buf_file = open_memstream(&dump_buf, &dump_buf_sz);
-	if (!ASSERT_OK_PTR(dump_buf_file, "dump_memstream"))
-		return;
-	btf = btf__new_empty();
-	if (!ASSERT_OK_PTR(btf, "new_empty"))
+static int btf_dump__custom_btf_test__init(struct btf_dump__custom_btf_test *t)
+{
+	t->dump_buf_file = open_memstream(&t->dump_buf, &t->dump_buf_sz);
+	if (!ASSERT_OK_PTR(t->dump_buf_file, "dump_memstream"))
+		return -1;
+	t->btf = btf__new_empty();
+	if (!ASSERT_OK_PTR(t->btf, "new_empty"))
 		goto err_out;
-	d = btf_dump__new(btf, btf_dump_printf, dump_buf_file, NULL);
-	if (!ASSERT_OK(libbpf_get_error(d), "btf_dump__new"))
+	t->d = btf_dump__new(t->btf, btf_dump_printf, t->dump_buf_file, NULL);
+	if (!ASSERT_OK(libbpf_get_error(t->d), "btf_dump__new"))
 		goto err_out;
 
+	return 0;
+
+err_out:
+	btf_dump__custom_btf_test__free(t);
+	return -1;
+}
+
+static void btf_dump__custom_btf_test__dump_and_compare(
+		struct btf_dump__custom_btf_test *t,
+		const char *expected_output,
+		const char *message)
+{
+	int i, err;
+
+	for (i = 1; i < btf__type_cnt(t->btf); i++) {
+		err = btf_dump__dump_type(t->d, i);
+		ASSERT_OK(err, "dump_type_ok");
+	}
+
+	fflush(t->dump_buf_file);
+	t->dump_buf[t->dump_buf_sz] = 0; /* some libc implementations don't do this */
+
+	ASSERT_STREQ(t->dump_buf, expected_output, message);
+}
+
+static void test_btf_dump_incremental(void)
+{
+	struct btf_dump__custom_btf_test t;
+	struct btf *btf;
+	int id, err;
+
+	if (btf_dump__custom_btf_test__init(&t))
+		return;
+
+	btf = t.btf;
+
 	/* First, generate BTF corresponding to the following C code:
 	 *
 	 * enum x;
@@ -182,15 +226,7 @@ static void test_btf_dump_incremental(void)
 	err = btf__add_field(btf, "x", 4, 0, 0);
 	ASSERT_OK(err, "field_ok");
 
-	for (i = 1; i < btf__type_cnt(btf); i++) {
-		err = btf_dump__dump_type(d, i);
-		ASSERT_OK(err, "dump_type_ok");
-	}
-
-	fflush(dump_buf_file);
-	dump_buf[dump_buf_sz] = 0; /* some libc implementations don't do this */
-
-	ASSERT_STREQ(dump_buf,
+	btf_dump__custom_btf_test__dump_and_compare(&t,
 "enum x;\n"
 "\n"
 "enum x {\n"
@@ -221,7 +257,7 @@ static void test_btf_dump_incremental(void)
 	 * enum values don't conflict;
 	 *
 	 */
-	fseek(dump_buf_file, 0, SEEK_SET);
+	fseek(t.dump_buf_file, 0, SEEK_SET);
 
 	id = btf__add_struct(btf, "s", 4);
 	ASSERT_EQ(id, 7, "struct_id");
@@ -232,14 +268,7 @@ static void test_btf_dump_incremental(void)
 	err = btf__add_field(btf, "s", 6, 64, 0);
 	ASSERT_OK(err, "field_ok");
 
-	for (i = 1; i < btf__type_cnt(btf); i++) {
-		err = btf_dump__dump_type(d, i);
-		ASSERT_OK(err, "dump_type_ok");
-	}
-
-	fflush(dump_buf_file);
-	dump_buf[dump_buf_sz] = 0; /* some libc implementations don't do this */
-	ASSERT_STREQ(dump_buf,
+	btf_dump__custom_btf_test__dump_and_compare(&t,
 "struct s___2 {\n"
 "	enum x x;\n"
 "	enum {\n"
@@ -248,11 +277,53 @@ static void test_btf_dump_incremental(void)
 "	struct s s;\n"
 "};\n\n" , "c_dump1");
 
-err_out:
-	fclose(dump_buf_file);
-	free(dump_buf);
-	btf_dump__free(d);
-	btf__free(btf);
+	btf_dump__custom_btf_test__free(&t);
+}
+
+static void test_btf_dump_type_tags(void)
+{
+	struct btf_dump__custom_btf_test t;
+	struct btf *btf;
+	int id, err;
+
+	if (btf_dump__custom_btf_test__init(&t))
+		return;
+
+	btf = t.btf;
+
+	/* Generate BTF corresponding to the following C code:
+	 *
+	 * struct s {
+	 *   void __attribute__((btf_type_tag(\"void_tag\"))) *p1;
+	 *   void __attribute__((void_attr)) *p2;
+	 * };
+	 *
+	 */
+
+	id = btf__add_type_tag(btf, "void_tag", 0);
+	ASSERT_EQ(id, 1, "type_tag_id");
+	id = btf__add_ptr(btf, id);
+	ASSERT_EQ(id, 2, "void_ptr_id1");
+
+	id = btf__add_type_attr(btf, "void_attr", 0);
+	ASSERT_EQ(id, 3, "type_attr_id");
+	id = btf__add_ptr(btf, id);
+	ASSERT_EQ(id, 4, "void_ptr_id2");
+
+	id = btf__add_struct(btf, "s", 8);
+	ASSERT_EQ(id, 5, "struct_id");
+	err = btf__add_field(btf, "p1", 2, 0, 0);
+	ASSERT_OK(err, "field_ok1");
+	err = btf__add_field(btf, "p2", 4, 0, 0);
+	ASSERT_OK(err, "field_ok2");
+
+	btf_dump__custom_btf_test__dump_and_compare(&t,
+"struct s {\n"
+"	void __attribute__((btf_type_tag(\"void_tag\"))) *p1;\n"
+"	void __attribute__((void_attr)) *p2;\n"
+"};\n\n", "dump_and_compare");
+
+	btf_dump__custom_btf_test__free(&t);
 }
 
 #define STRSIZE				4096
@@ -874,6 +945,9 @@ void test_btf_dump() {
 	if (test__start_subtest("btf_dump: incremental"))
 		test_btf_dump_incremental();
 
+	if (test__start_subtest("btf_dump: type_tags"))
+		test_btf_dump_type_tags();
+
 	btf = libbpf_find_kernel_btf();
 	if (!ASSERT_OK_PTR(btf, "no kernel BTF found"))
 		return;
-- 
2.48.1

Re: [PATCH bpf-next 3/5] selftests/bpf: add a btf_dump test for type_tags

[Eduard Zingerman]

On Wed, 2025-01-22 at 02:53 +0000, Ihor Solodrai wrote:
> Factor out common routines handling custom BTF from
> test_btf_dump_incremental. Then use them in the
> test_btf_dump_type_tags.
> 
> test_btf_dump_type_tags verifies that a type tag is dumped correctly
> with respect to its kflag.
> 
> Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>
> ---

Acked-by: Eduard Zingerman <eddyz87@gmail.com>

>  .../selftests/bpf/prog_tests/btf_dump.c       | 148 +++++++++++++-----
>  1 file changed, 111 insertions(+), 37 deletions(-)
> 
> diff --git a/tools/testing/selftests/bpf/prog_tests/btf_dump.c b/tools/testing/selftests/bpf/prog_tests/btf_dump.c
> index b293b8501fd6..690cf8cef7d2 100644
> --- a/tools/testing/selftests/bpf/prog_tests/btf_dump.c
> +++ b/tools/testing/selftests/bpf/prog_tests/btf_dump.c
> @@ -126,26 +126,70 @@ static int test_btf_dump_case(int n, struct btf_dump_test_case *t)
>  	return err;
>  }
>  
> -static char *dump_buf;
> -static size_t dump_buf_sz;
> -static FILE *dump_buf_file;
> +struct btf_dump__custom_btf_test {

Nit: since there would be a v2, I'd give this thing a shorter name,
     e.g. test_ctx.

> +	struct btf *btf;
> +	struct btf_dump *d;
> +	char *dump_buf;
> +	size_t dump_buf_sz;
> +	FILE *dump_buf_file;
> +};

[...]

> +static void test_btf_dump_incremental(void)
> +{
> +	struct btf_dump__custom_btf_test t;

Nit: this should be 'struct btf_dump__custom_btf_test t = {};'
     otherwise btf_dump__custom_btf_test__init() would read
     garbage in the error cases.

> +	struct btf *btf;
> +	int id, err;
> +
> +	if (btf_dump__custom_btf_test__init(&t))
> +		return;
> +
> +	btf = t.btf;
> +
>  	/* First, generate BTF corresponding to the following C code:
>  	 *
>  	 * enum x;

[...]

[PATCH bpf-next 4/5] bpf: allow kind_flag for BTF type and decl tags

[Ihor Solodrai]

BTF type tags and decl tags now may have info->kflag set to 1,
changing the semantics of the tag.

Change BTF verification to permit BTF that makes use of this feature:
  * remove kflag check in btf_decl_tag_check_meta(), as both values
    are valid
  * allow kflag to be set for BTF_KIND_TYPE_TAG type in
    btf_ref_type_check_meta()

Modify a selftest checking for kflag in decl_tag accordingly.

Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>
---
 kernel/bpf/btf.c                             | 7 +------
 tools/testing/selftests/bpf/prog_tests/btf.c | 4 +---
 2 files changed, 2 insertions(+), 9 deletions(-)

diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 8396ce1d0fba..becdec583e00 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -2575,7 +2575,7 @@ static int btf_ref_type_check_meta(struct btf_verifier_env *env,
 		return -EINVAL;
 	}
 
-	if (btf_type_kflag(t)) {
+	if (btf_type_kflag(t) && BTF_INFO_KIND(t->info) != BTF_KIND_TYPE_TAG) {
 		btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
 		return -EINVAL;
 	}
@@ -4944,11 +4944,6 @@ static s32 btf_decl_tag_check_meta(struct btf_verifier_env *env,
 		return -EINVAL;
 	}
 
-	if (btf_type_kflag(t)) {
-		btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
-		return -EINVAL;
-	}
-
 	component_idx = btf_type_decl_tag(t)->component_idx;
 	if (component_idx < -1) {
 		btf_verifier_log_type(env, t, "Invalid component_idx");
diff --git a/tools/testing/selftests/bpf/prog_tests/btf.c b/tools/testing/selftests/bpf/prog_tests/btf.c
index e63d74ce046f..aab9ad88c845 100644
--- a/tools/testing/selftests/bpf/prog_tests/btf.c
+++ b/tools/testing/selftests/bpf/prog_tests/btf.c
@@ -3866,7 +3866,7 @@ static struct btf_raw_test raw_tests[] = {
 	.err_str = "vlen != 0",
 },
 {
-	.descr = "decl_tag test #8, invalid kflag",
+	.descr = "decl_tag test #8, tag with kflag",
 	.raw_types = {
 		BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4),	/* [1] */
 		BTF_VAR_ENC(NAME_TBD, 1, 0),			/* [2] */
@@ -3881,8 +3881,6 @@ static struct btf_raw_test raw_tests[] = {
 	.key_type_id = 1,
 	.value_type_id = 1,
 	.max_entries = 1,
-	.btf_load_err = true,
-	.err_str = "Invalid btf_info kind_flag",
 },
 {
 	.descr = "decl_tag test #9, var, invalid component_idx",
-- 
2.48.1

Re: [PATCH bpf-next 4/5] bpf: allow kind_flag for BTF type and decl tags

[Alan Maguire]

On 22/01/2025 02:53, Ihor Solodrai wrote:
> BTF type tags and decl tags now may have info->kflag set to 1,
> changing the semantics of the tag.
> 
> Change BTF verification to permit BTF that makes use of this feature:
>   * remove kflag check in btf_decl_tag_check_meta(), as both values
>     are valid
>   * allow kflag to be set for BTF_KIND_TYPE_TAG type in
>     btf_ref_type_check_meta()
> 
> Modify a selftest checking for kflag in decl_tag accordingly.
> 
> Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>

Looks good, but I have a few questions here. Today in btf_struct_walk()
we check pointer type tags like this:

			/* check type tag */
                        t = btf_type_by_id(btf, mtype->type);
                        if (btf_type_is_type_tag(t)) {
                                tag_value = __btf_name_by_offset(btf,
t->name_off);
                                /* check __user tag */
                                if (strcmp(tag_value, "user") == 0)
                                        tmp_flag = MEM_USER;
                                /* check __percpu tag */
                                if (strcmp(tag_value, "percpu") == 0)
                                        tmp_flag = MEM_PERCPU;
                                /* check __rcu tag */
                                if (strcmp(tag_value, "rcu") == 0)
                                        tmp_flag = MEM_RCU;
                        }


Do we need to add in a check for kind_flag == 0 here to ensure it's a
BTF type tag attribute? Similar question for type tag kptr checking in
btf_find_kptr() (we could perhaps add a btf_type_is_btf_type_tag() or
something to include the kind_flag == 0 check).

And I presume the btf_check_type_tags() logic still applies for generic
attributes - i.e. that they always precede the modifiers in the chain?


> ---
>  kernel/bpf/btf.c                             | 7 +------
>  tools/testing/selftests/bpf/prog_tests/btf.c | 4 +---
>  2 files changed, 2 insertions(+), 9 deletions(-)
> 
> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index 8396ce1d0fba..becdec583e00 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
> @@ -2575,7 +2575,7 @@ static int btf_ref_type_check_meta(struct btf_verifier_env *env,
>  		return -EINVAL;
>  	}
>  
> -	if (btf_type_kflag(t)) {
> +	if (btf_type_kflag(t) && BTF_INFO_KIND(t->info) != BTF_KIND_TYPE_TAG) {
>  		btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
>  		return -EINVAL;
>  	}
> @@ -4944,11 +4944,6 @@ static s32 btf_decl_tag_check_meta(struct btf_verifier_env *env,
>  		return -EINVAL;
>  	}
>  
> -	if (btf_type_kflag(t)) {
> -		btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
> -		return -EINVAL;
> -	}
> -
>  	component_idx = btf_type_decl_tag(t)->component_idx;
>  	if (component_idx < -1) {
>  		btf_verifier_log_type(env, t, "Invalid component_idx");
> diff --git a/tools/testing/selftests/bpf/prog_tests/btf.c b/tools/testing/selftests/bpf/prog_tests/btf.c
> index e63d74ce046f..aab9ad88c845 100644
> --- a/tools/testing/selftests/bpf/prog_tests/btf.c
> +++ b/tools/testing/selftests/bpf/prog_tests/btf.c
> @@ -3866,7 +3866,7 @@ static struct btf_raw_test raw_tests[] = {
>  	.err_str = "vlen != 0",
>  },
>  {
> -	.descr = "decl_tag test #8, invalid kflag",
> +	.descr = "decl_tag test #8, tag with kflag",
>  	.raw_types = {
>  		BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4),	/* [1] */
>  		BTF_VAR_ENC(NAME_TBD, 1, 0),			/* [2] */
> @@ -3881,8 +3881,6 @@ static struct btf_raw_test raw_tests[] = {
>  	.key_type_id = 1,
>  	.value_type_id = 1,
>  	.max_entries = 1,
> -	.btf_load_err = true,
> -	.err_str = "Invalid btf_info kind_flag",
>  },
>  {
>  	.descr = "decl_tag test #9, var, invalid component_idx",

Re: [PATCH bpf-next 4/5] bpf: allow kind_flag for BTF type and decl tags

[Ihor Solodrai]

On Wednesday, January 22nd, 2025 at 3:09 AM, Alan Maguire <alan.maguire@oracle.com> wrote:

> 
> 
> On 22/01/2025 02:53, Ihor Solodrai wrote:
> 
> > BTF type tags and decl tags now may have info->kflag set to 1,
> > changing the semantics of the tag.
> > 
> > Change BTF verification to permit BTF that makes use of this feature:
> > * remove kflag check in btf_decl_tag_check_meta(), as both values
> > are valid
> > * allow kflag to be set for BTF_KIND_TYPE_TAG type in
> > btf_ref_type_check_meta()
> > 
> > Modify a selftest checking for kflag in decl_tag accordingly.
> > 
> > Signed-off-by: Ihor Solodrai ihor.solodrai@pm.me
> 
> 
> Looks good, but I have a few questions here. Today in btf_struct_walk()
> we check pointer type tags like this:
> 
> /* check type tag */
> t = btf_type_by_id(btf, mtype->type);
> 
> if (btf_type_is_type_tag(t)) {
> tag_value = __btf_name_by_offset(btf,
> t->name_off);
> 
> /* check __user tag /
> if (strcmp(tag_value, "user") == 0)
> tmp_flag = MEM_USER;
> / check __percpu tag /
> if (strcmp(tag_value, "percpu") == 0)
> tmp_flag = MEM_PERCPU;
> / check __rcu tag */
> if (strcmp(tag_value, "rcu") == 0)
> tmp_flag = MEM_RCU;
> }
> 
> 
> Do we need to add in a check for kind_flag == 0 here to ensure it's a
> BTF type tag attribute? Similar question for type tag kptr checking in
> btf_find_kptr() (we could perhaps add a btf_type_is_btf_type_tag() or
> something to include the kind_flag == 0 check).

Yes, we should check for kflag here. Thanks for pointing this out.

    __attribute__((user))
vs
    __attribute__((bpf_type_tag("user")))

are very different things.

> 
> And I presume the btf_check_type_tags() logic still applies for generic
> attributes - i.e. that they always precede the modifiers in the chain?

Yes, that is my understanding too.

> 
> [...]

[PATCH bpf-next 5/5] selftests/bpf: add a BTF verification test for kflagged type_tag

[Ihor Solodrai]

Add a BTF verification test case for a type_tag with a kflag set.
Type tags with a kflag are now valid.

Add BTF_DECL_ATTR_ENC and BTF_TYPE_ATTR_ENC test helper macros,
corresponding to *_TAG_ENC.

Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>
---
 tools/testing/selftests/bpf/prog_tests/btf.c | 19 ++++++++++++++++++-
 tools/testing/selftests/bpf/test_btf.h       |  6 ++++++
 2 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/btf.c b/tools/testing/selftests/bpf/prog_tests/btf.c
index aab9ad88c845..8a9ba4292109 100644
--- a/tools/testing/selftests/bpf/prog_tests/btf.c
+++ b/tools/testing/selftests/bpf/prog_tests/btf.c
@@ -3870,7 +3870,7 @@ static struct btf_raw_test raw_tests[] = {
 	.raw_types = {
 		BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4),	/* [1] */
 		BTF_VAR_ENC(NAME_TBD, 1, 0),			/* [2] */
-		BTF_TYPE_ENC(NAME_TBD, BTF_INFO_ENC(BTF_KIND_DECL_TAG, 1, 0), 2), (-1),
+		BTF_DECL_ATTR_ENC(NAME_TBD, 2, -1),
 		BTF_END_RAW,
 	},
 	BTF_STR_SEC("\0local\0tag1"),
@@ -4204,6 +4204,23 @@ static struct btf_raw_test raw_tests[] = {
 	.btf_load_err = true,
 	.err_str = "Type tags don't precede modifiers",
 },
+{
+	.descr = "type_tag test #7, tag with kflag",
+	.raw_types = {
+		BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4),	/* [1] */
+		BTF_TYPE_ATTR_ENC(NAME_TBD, 1),			/* [2] */
+		BTF_PTR_ENC(2),					/* [3] */
+		BTF_END_RAW,
+	},
+	BTF_STR_SEC("\0tag"),
+	.map_type = BPF_MAP_TYPE_ARRAY,
+	.map_name = "tag_type_check_btf",
+	.key_size = sizeof(int),
+	.value_size = 4,
+	.key_type_id = 1,
+	.value_type_id = 1,
+	.max_entries = 1,
+},
 {
 	.descr = "enum64 test #1, unsigned, size 8",
 	.raw_types = {
diff --git a/tools/testing/selftests/bpf/test_btf.h b/tools/testing/selftests/bpf/test_btf.h
index fb4f4714eeb4..e65889ab4adf 100644
--- a/tools/testing/selftests/bpf/test_btf.h
+++ b/tools/testing/selftests/bpf/test_btf.h
@@ -72,9 +72,15 @@
 #define BTF_TYPE_FLOAT_ENC(name, sz) \
 	BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_FLOAT, 0, 0), sz)
 
+#define BTF_DECL_ATTR_ENC(value, type, component_idx)	\
+	BTF_TYPE_ENC(value, BTF_INFO_ENC(BTF_KIND_DECL_TAG, 1, 0), type), (component_idx)
+
 #define BTF_DECL_TAG_ENC(value, type, component_idx)	\
 	BTF_TYPE_ENC(value, BTF_INFO_ENC(BTF_KIND_DECL_TAG, 0, 0), type), (component_idx)
 
+#define BTF_TYPE_ATTR_ENC(value, type)	\
+	BTF_TYPE_ENC(value, BTF_INFO_ENC(BTF_KIND_TYPE_TAG, 1, 0), type)
+
 #define BTF_TYPE_TAG_ENC(value, type)	\
 	BTF_TYPE_ENC(value, BTF_INFO_ENC(BTF_KIND_TYPE_TAG, 0, 0), type)
 
-- 
2.48.1

Re: [PATCH bpf-next 0/5] BTF: arbitrary __attribute__ encoding

[Alan Maguire]

On 22/01/2025 02:53, Ihor Solodrai wrote:
> This patch series extends BPF Type Format (BTF) to support arbitrary
> __attribute__ encoding.
> 
> Setting the kind_flag to 1 in BTF type tags and decl tags now changes
> the meaning for the encoded tag, in particular with respect to
> btf_dump in libbpf.
> 
> If the kflag is set, then the string encoded by the tag represents the
> full attribute-list of an attribute specifier [1].
> 
> This feature will allow extending tools such as pahole and bpftool to
> capture and use more granular type information, and make it easier to
> manage compatibility between clang and gcc BPF compilers.
>

sounds good! So presumably pahole will then have a "full_attribute" or
similar BTF feature that will only do full attribute encoding for
kernels that expect the kind flag to be set? Otherwise we'll run the
risk of generating invalid BTF for older kernels with newer pahole
(since those older kernels will fail to verify tags with a kind flag set).

Thanks!

Alan

> [1] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Attribute-Syntax.html
> 
> Ihor Solodrai (5):
>   libbpf: introduce kflag for type_tags and decl_tags in BTF
>   libbpf: check the kflag of type tags in btf_dump
>   selftests/bpf: add a btf_dump test for type_tags
>   bpf: allow kind_flag for BTF type and decl tags
>   selftests/bpf: add a BTF verification test for kflagged type_tag
> 
>  Documentation/bpf/btf.rst                     |  27 +++-
>  kernel/bpf/btf.c                              |   7 +-
>  tools/include/uapi/linux/btf.h                |   3 +-
>  tools/lib/bpf/btf.c                           |  87 +++++++---
>  tools/lib/bpf/btf.h                           |   3 +
>  tools/lib/bpf/btf_dump.c                      |   5 +-
>  tools/lib/bpf/libbpf.map                      |   2 +
>  tools/testing/selftests/bpf/prog_tests/btf.c  |  23 ++-
>  .../selftests/bpf/prog_tests/btf_dump.c       | 148 +++++++++++++-----
>  tools/testing/selftests/bpf/test_btf.h        |   6 +
>  10 files changed, 234 insertions(+), 77 deletions(-)
> 

Re: [PATCH bpf-next 0/5] BTF: arbitrary __attribute__ encoding

[Ihor Solodrai]

On 1/22/25 2:13 AM, Alan Maguire wrote:
 > On 22/01/2025 02:53, Ihor Solodrai wrote:
 >> This patch series extends BPF Type Format (BTF) to support arbitrary
 >> __attribute__ encoding.
 >>
 >> Setting the kind_flag to 1 in BTF type tags and decl tags now changes
 >> the meaning for the encoded tag, in particular with respect to
 >> btf_dump in libbpf.
 >>
 >> If the kflag is set, then the string encoded by the tag represents the
 >> full attribute-list of an attribute specifier [1].
 >>
 >> This feature will allow extending tools such as pahole and bpftool to
 >> capture and use more granular type information, and make it easier to
 >> manage compatibility between clang and gcc BPF compilers.
 >>
 >
 > sounds good! So presumably pahole will then have a "full_attribute" or
 > similar BTF feature that will only do full attribute encoding for
 > kernels that expect the kind flag to be set? Otherwise we'll run the
 > risk of generating invalid BTF for older kernels with newer pahole
 > (since those older kernels will fail to verify tags with a kind flag 
set).

Hi Alan. Sorry I missed this message on the first pass.

Yes, this BTF extension enables pahole to encode any __attribute__ in
BTF. There is already a use case for that, which led me to this patch
series: expressing bpf_arena tags in vmlinux.h.

As to "full_attribute"-like feature in pahole, my intuition is that
it's a significant chunk of work, and it might not be necessary in the
near future. But with these changes in BTF it at least becomes a
possibility. And you're right, it will have to be an optional feature.

 >
 > Thanks!
 >
 > Alan
 >
 >> [1] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Attribute-Syntax.html
 >>
 >> Ihor Solodrai (5):
 >>   libbpf: introduce kflag for type_tags and decl_tags in BTF
 >>   libbpf: check the kflag of type tags in btf_dump
 >>   selftests/bpf: add a btf_dump test for type_tags
 >>   bpf: allow kind_flag for BTF type and decl tags
 >>   selftests/bpf: add a BTF verification test for kflagged type_tag
 >>
 >>  Documentation/bpf/btf.rst                     |  27 +++-
 >>  kernel/bpf/btf.c                              |   7 +-
 >>  tools/include/uapi/linux/btf.h                |   3 +-
 >>  tools/lib/bpf/btf.c                           |  87 +++++++---
 >>  tools/lib/bpf/btf.h                           |   3 +
 >>  tools/lib/bpf/btf_dump.c                      |   5 +-
 >>  tools/lib/bpf/libbpf.map                      |   2 +
 >>  tools/testing/selftests/bpf/prog_tests/btf.c  |  23 ++-
 >>  .../selftests/bpf/prog_tests/btf_dump.c       | 148 +++++++++++++-----
 >>  tools/testing/selftests/bpf/test_btf.h        |   6 +
 >>  10 files changed, 234 insertions(+), 77 deletions(-)
 >>
 >

Re: [PATCH bpf-next 0/5] BTF: arbitrary __attribute__ encoding

[Jose E. Marchesi]

> This patch series extends BPF Type Format (BTF) to support arbitrary
> __attribute__ encoding.
>
> Setting the kind_flag to 1 in BTF type tags and decl tags now changes
> the meaning for the encoded tag, in particular with respect to
> btf_dump in libbpf.
>
> If the kflag is set, then the string encoded by the tag represents the
> full attribute-list of an attribute specifier [1].

Why is extending BTF necessary for this?  Type and declaration tags
contain arbitrary strings, and AFAIK you can have more than one type tag
associated with a single type or declaration.  Why coupling the
interpretation of the contents of the string with the transport format?

Something like "cattribute:always_inline".

> This feature will allow extending tools such as pahole and bpftool to
> capture and use more granular type information, and make it easier to
> manage compatibility between clang and gcc BPF compilers.
>
> [1] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Attribute-Syntax.html
>
> Ihor Solodrai (5):
>   libbpf: introduce kflag for type_tags and decl_tags in BTF
>   libbpf: check the kflag of type tags in btf_dump
>   selftests/bpf: add a btf_dump test for type_tags
>   bpf: allow kind_flag for BTF type and decl tags
>   selftests/bpf: add a BTF verification test for kflagged type_tag
>
>  Documentation/bpf/btf.rst                     |  27 +++-
>  kernel/bpf/btf.c                              |   7 +-
>  tools/include/uapi/linux/btf.h                |   3 +-
>  tools/lib/bpf/btf.c                           |  87 +++++++---
>  tools/lib/bpf/btf.h                           |   3 +
>  tools/lib/bpf/btf_dump.c                      |   5 +-
>  tools/lib/bpf/libbpf.map                      |   2 +
>  tools/testing/selftests/bpf/prog_tests/btf.c  |  23 ++-
>  .../selftests/bpf/prog_tests/btf_dump.c       | 148 +++++++++++++-----
>  tools/testing/selftests/bpf/test_btf.h        |   6 +
>  10 files changed, 234 insertions(+), 77 deletions(-)

Re: [PATCH bpf-next 0/5] BTF: arbitrary __attribute__ encoding

[Ihor Solodrai]

On Wednesday, January 22nd, 2025 at 3:44 AM, Jose E. Marchesi <jose.marchesi@oracle.com> wrote:

> 
> 
> > This patch series extends BPF Type Format (BTF) to support arbitrary
> > attribute encoding.
> > 
> > Setting the kind_flag to 1 in BTF type tags and decl tags now changes
> > the meaning for the encoded tag, in particular with respect to
> > btf_dump in libbpf.
> > 
> > If the kflag is set, then the string encoded by the tag represents the
> > full attribute-list of an attribute specifier [1].
> 
> 
> Why is extending BTF necessary for this? Type and declaration tags
> contain arbitrary strings, and AFAIK you can have more than one type tag
> associated with a single type or declaration. Why coupling the
> interpretation of the contents of the string with the transport format?
> 
> Something like "cattribute:always_inline".

Hi Jose. Good questions.

You are correct that the tags can contain arbitrary strings already,
and that multiple tags can be associated with the same type or
declaration.

A specific problem I'm trying to solve is how to direct btf_dump in
interpreting tags as attributes, and do it in a generic way, as it's a
part of libbpf.

I discussed with Andrii, Eduard and Alexei a couple of approaches, and
tried some of them.

For example, a set of dump options could be introduced to handle
specific use-cases, similar to what you suggested in a
ATTR_PRESERVE_ACCESS_INDEX patch [1]. This is a valid approach,
however it is not very generic. An option will have to be introduced
and implemented for every new use-case.

A more generic approach is adding a set of callbacks to btf_dump. This
is a big design task, which I think should be avoided unless
absolutely necessary.

The benefit of this change - defining flagged tags as attributes - is
that it enables BTF to natively encode attributes as part of a type,
which is not possible currently. And it's a simple change.

Using the contents of the tag to indicate it's meaning (such as
"cattrubite:always_inline") will work too. However I don't think it's
desirable to have to parse the tag strings within libbpf, even more so
in BPF verifier.

In a discussion with Andrii we briefly entertained an idea of allowing
btf_dump to print the tag string directly (without requiring it to be
a tag or attribute), which would allow all kinds of hacks. Tempting,
but probably very bug-prone.

[1] https://lore.kernel.org/bpf/20240503111836.25275-1-jose.marchesi@oracle.com/

> 
> > This feature will allow extending tools such as pahole and bpftool to
> > capture and use more granular type information, and make it easier to
> > manage compatibility between clang and gcc BPF compilers.
> > 
> > [1] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Attribute-Syntax.html
> > 
> > Ihor Solodrai (5):
> > libbpf: introduce kflag for type_tags and decl_tags in BTF
> > libbpf: check the kflag of type tags in btf_dump
> > selftests/bpf: add a btf_dump test for type_tags
> > bpf: allow kind_flag for BTF type and decl tags
> > selftests/bpf: add a BTF verification test for kflagged type_tag
> > 
> > Documentation/bpf/btf.rst | 27 +++-
> > kernel/bpf/btf.c | 7 +-
> > tools/include/uapi/linux/btf.h | 3 +-
> > tools/lib/bpf/btf.c | 87 +++++++---
> > tools/lib/bpf/btf.h | 3 +
> > tools/lib/bpf/btf_dump.c | 5 +-
> > tools/lib/bpf/libbpf.map | 2 +
> > tools/testing/selftests/bpf/prog_tests/btf.c | 23 ++-
> > .../selftests/bpf/prog_tests/btf_dump.c | 148 +++++++++++++-----
> > tools/testing/selftests/bpf/test_btf.h | 6 +
> > 10 files changed, 234 insertions(+), 77 deletions(-)

Re: [PATCH bpf-next 0/5] BTF: arbitrary __attribute__ encoding

[Jose E. Marchesi]

> On Wednesday, January 22nd, 2025 at 3:44 AM, Jose E. Marchesi <jose.marchesi@oracle.com> wrote:
>
>> 
>> 
>> > This patch series extends BPF Type Format (BTF) to support arbitrary
>> > attribute encoding.
>> > 
>> > Setting the kind_flag to 1 in BTF type tags and decl tags now changes
>> > the meaning for the encoded tag, in particular with respect to
>> > btf_dump in libbpf.
>> > 
>> > If the kflag is set, then the string encoded by the tag represents the
>> > full attribute-list of an attribute specifier [1].
>> 
>> 
>> Why is extending BTF necessary for this? Type and declaration tags
>> contain arbitrary strings, and AFAIK you can have more than one type tag
>> associated with a single type or declaration. Why coupling the
>> interpretation of the contents of the string with the transport format?
>> 
>> Something like "cattribute:always_inline".
>
> Hi Jose. Good questions.
>
> You are correct that the tags can contain arbitrary strings already,
> and that multiple tags can be associated with the same type or
> declaration.
>
> A specific problem I'm trying to solve is how to direct btf_dump in
> interpreting tags as attributes, and do it in a generic way, as it's a
> part of libbpf.
>
> I discussed with Andrii, Eduard and Alexei a couple of approaches, and
> tried some of them.
>
> For example, a set of dump options could be introduced to handle
> specific use-cases, similar to what you suggested in a
> ATTR_PRESERVE_ACCESS_INDEX patch [1]. This is a valid approach,
> however it is not very generic. An option will have to be introduced
> and implemented for every new use-case.
>
> A more generic approach is adding a set of callbacks to btf_dump. This
> is a big design task, which I think should be avoided unless
> absolutely necessary.
>
> The benefit of this change - defining flagged tags as attributes - is
> that it enables BTF to natively encode attributes as part of a type,
> which is not possible currently. And it's a simple change.
>
> Using the contents of the tag to indicate it's meaning (such as
> "cattrubite:always_inline") will work too. However I don't think it's
> desirable to have to parse the tag strings within libbpf, even more so
> in BPF verifier.

I expect the verifier will in any case have to distinguish the different
strings it gets in the tags, for other purposes, right, so this wouldn't
be introducing anything different?

Also FWIW DWARF doesn't have a kind_flag.

>
> In a discussion with Andrii we briefly entertained an idea of allowing
> btf_dump to print the tag string directly (without requiring it to be
> a tag or attribute), which would allow all kinds of hacks. Tempting,
> but probably very bug-prone.
>
> [1] https://lore.kernel.org/bpf/20240503111836.25275-1-jose.marchesi@oracle.com/
>
>> 
>> > This feature will allow extending tools such as pahole and bpftool to
>> > capture and use more granular type information, and make it easier to
>> > manage compatibility between clang and gcc BPF compilers.
>> > 
>> > [1] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Attribute-Syntax.html
>> > 
>> > Ihor Solodrai (5):
>> > libbpf: introduce kflag for type_tags and decl_tags in BTF
>> > libbpf: check the kflag of type tags in btf_dump
>> > selftests/bpf: add a btf_dump test for type_tags
>> > bpf: allow kind_flag for BTF type and decl tags
>> > selftests/bpf: add a BTF verification test for kflagged type_tag
>> > 
>> > Documentation/bpf/btf.rst | 27 +++-
>> > kernel/bpf/btf.c | 7 +-
>> > tools/include/uapi/linux/btf.h | 3 +-
>> > tools/lib/bpf/btf.c | 87 +++++++---
>> > tools/lib/bpf/btf.h | 3 +
>> > tools/lib/bpf/btf_dump.c | 5 +-
>> > tools/lib/bpf/libbpf.map | 2 +
>> > tools/testing/selftests/bpf/prog_tests/btf.c | 23 ++-
>> > .../selftests/bpf/prog_tests/btf_dump.c | 148 +++++++++++++-----
>> > tools/testing/selftests/bpf/test_btf.h | 6 +
>> > 10 files changed, 234 insertions(+), 77 deletions(-)

Re: [PATCH bpf-next 0/5] BTF: arbitrary __attribute__ encoding

[Ihor Solodrai]

On Wednesday, January 22nd, 2025 at 11:47 AM, Jose E. Marchesi <jose.marchesi@oracle.com> wrote:

> [...]
> > 
> > Using the contents of the tag to indicate it's meaning (such as
> > "cattrubite:always_inline") will work too. However I don't think it's
> > desirable to have to parse the tag strings within libbpf, even more so
> > in BPF verifier.
> 
> 
> I expect the verifier will in any case have to distinguish the different
> strings it gets in the tags, for other purposes, right, so this wouldn't
> be introducing anything different?

I only see direct string comparisons in the BTF verification, for example:

    if (btf_type_is_type_tag(t)) {
    		tag_value = __btf_name_by_offset(btf, t->name_off);
    		if (strcmp(tag_value, "user") == 0)
    			info->reg_type |= MEM_USER;
    		if (strcmp(tag_value, "percpu") == 0)
    			info->reg_type |= MEM_PERCPU;
    	}

What's different is that this way a syntax is introduced, even if very
simple like "prefix:suffix". And so it potentially has to be parsed by
the tag reader, be it btf_dump or anything else. Testing a kflag is
just a much simpler operation. Maybe if we had N kinds of tags, and
not just two this would make sense?

Also. would this way of encoding be a part of the BTF spec then?
It can be done in principle, I just don't know if it's a good idea.

> 
> Also FWIW DWARF doesn't have a kind_flag.

Right, but BTF was designed with different goals, and one of them is to
be compact. kind_flag so far just hasn't been used by the tags, but it
is used in other BTF types.

> 
> [...]

Re: [PATCH bpf-next 0/5] BTF: arbitrary __attribute__ encoding

[Andrii Nakryiko]

On Wed, Jan 22, 2025 at 3:44 AM Jose E. Marchesi
<jose.marchesi@oracle.com> wrote:
>
>
> > This patch series extends BPF Type Format (BTF) to support arbitrary
> > __attribute__ encoding.
> >
> > Setting the kind_flag to 1 in BTF type tags and decl tags now changes
> > the meaning for the encoded tag, in particular with respect to
> > btf_dump in libbpf.
> >
> > If the kflag is set, then the string encoded by the tag represents the
> > full attribute-list of an attribute specifier [1].
>
> Why is extending BTF necessary for this?  Type and declaration tags
> contain arbitrary strings, and AFAIK you can have more than one type tag

Because currently TYPE_TAG(some_string) is
__attribute__((btf_type_tag("some_string"))).

That btf_type_tag() attribute name is hard-coded in the semantics of
current TYPE_TAG (and DECL_TAG as well). So here Ihor is generalizing
this to be __attribute__((some_string)).

> associated with a single type or declaration.  Why coupling the
> interpretation of the contents of the string with the transport format?
>
> Something like "cattribute:always_inline".

I think that ship has sailed. We didn't define any extra semantics for
any sort of "prefix:" part of TYPE_TAG's string, and I'm not sure we
want to retroactively define anything like that at this point.

What is exactly the problem with using kflag=1? Keep in mind, at least
initially, this is meant for tools like pahole and bpftool, not
GCC/Clang itself, to augment BTF with extra annotations (like
preserve_access_index attribute that is added when generating
vmlinux.h).

>
> > This feature will allow extending tools such as pahole and bpftool to
> > capture and use more granular type information, and make it easier to
> > manage compatibility between clang and gcc BPF compilers.
> >
> > [1] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Attribute-Syntax.html
> >
> > Ihor Solodrai (5):
> >   libbpf: introduce kflag for type_tags and decl_tags in BTF
> >   libbpf: check the kflag of type tags in btf_dump
> >   selftests/bpf: add a btf_dump test for type_tags
> >   bpf: allow kind_flag for BTF type and decl tags
> >   selftests/bpf: add a BTF verification test for kflagged type_tag
> >
> >  Documentation/bpf/btf.rst                     |  27 +++-
> >  kernel/bpf/btf.c                              |   7 +-
> >  tools/include/uapi/linux/btf.h                |   3 +-
> >  tools/lib/bpf/btf.c                           |  87 +++++++---
> >  tools/lib/bpf/btf.h                           |   3 +
> >  tools/lib/bpf/btf_dump.c                      |   5 +-
> >  tools/lib/bpf/libbpf.map                      |   2 +
> >  tools/testing/selftests/bpf/prog_tests/btf.c  |  23 ++-
> >  .../selftests/bpf/prog_tests/btf_dump.c       | 148 +++++++++++++-----
> >  tools/testing/selftests/bpf/test_btf.h        |   6 +
> >  10 files changed, 234 insertions(+), 77 deletions(-)

Re: [PATCH bpf-next 0/5] BTF: arbitrary __attribute__ encoding

[Jose E. Marchesi]

> On Wed, Jan 22, 2025 at 3:44 AM Jose E. Marchesi
> <jose.marchesi@oracle.com> wrote:
>>
>>
>> > This patch series extends BPF Type Format (BTF) to support arbitrary
>> > __attribute__ encoding.
>> >
>> > Setting the kind_flag to 1 in BTF type tags and decl tags now changes
>> > the meaning for the encoded tag, in particular with respect to
>> > btf_dump in libbpf.
>> >
>> > If the kflag is set, then the string encoded by the tag represents the
>> > full attribute-list of an attribute specifier [1].
>>
>> Why is extending BTF necessary for this?  Type and declaration tags
>> contain arbitrary strings, and AFAIK you can have more than one type tag
>
> Because currently TYPE_TAG(some_string) is
> __attribute__((btf_type_tag("some_string"))).
>
> That btf_type_tag() attribute name is hard-coded in the semantics of
> current TYPE_TAG (and DECL_TAG as well). So here Ihor is generalizing
> this to be __attribute__((some_string)).
>
>> associated with a single type or declaration.  Why coupling the
>> interpretation of the contents of the string with the transport format?
>>
>> Something like "cattribute:always_inline".
>
> I think that ship has sailed. We didn't define any extra semantics for
> any sort of "prefix:" part of TYPE_TAG's string, and I'm not sure we
> want to retroactively define anything like that at this point.
>
> What is exactly the problem with using kflag=1? Keep in mind, at least
> initially, this is meant for tools like pahole and bpftool, not
> GCC/Clang itself, to augment BTF with extra annotations (like
> preserve_access_index attribute that is added when generating
> vmlinux.h).

Ah ok, I misunderstood how this is intended to be used.

I thought it would be the BPF compiler that would be creating these
entries.  But it is these tools that will emit the attributes in the
headers, and then augment the BTF entries for these particular
attributes when loading the programs that include the headers?

>
>>
>> > This feature will allow extending tools such as pahole and bpftool to
>> > capture and use more granular type information, and make it easier to
>> > manage compatibility between clang and gcc BPF compilers.
>> >
>> > [1] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Attribute-Syntax.html
>> >
>> > Ihor Solodrai (5):
>> >   libbpf: introduce kflag for type_tags and decl_tags in BTF
>> >   libbpf: check the kflag of type tags in btf_dump
>> >   selftests/bpf: add a btf_dump test for type_tags
>> >   bpf: allow kind_flag for BTF type and decl tags
>> >   selftests/bpf: add a BTF verification test for kflagged type_tag
>> >
>> >  Documentation/bpf/btf.rst                     |  27 +++-
>> >  kernel/bpf/btf.c                              |   7 +-
>> >  tools/include/uapi/linux/btf.h                |   3 +-
>> >  tools/lib/bpf/btf.c                           |  87 +++++++---
>> >  tools/lib/bpf/btf.h                           |   3 +
>> >  tools/lib/bpf/btf_dump.c                      |   5 +-
>> >  tools/lib/bpf/libbpf.map                      |   2 +
>> >  tools/testing/selftests/bpf/prog_tests/btf.c  |  23 ++-
>> >  .../selftests/bpf/prog_tests/btf_dump.c       | 148 +++++++++++++-----
>> >  tools/testing/selftests/bpf/test_btf.h        |   6 +
>> >  10 files changed, 234 insertions(+), 77 deletions(-)