[PATCH bpf-next v4 0/4] bpftool: btf: Support dumping a single type from file

[PATCH bpf-next v4 0/4] bpftool: btf: Support dumping a single type from file

[Daniel Xu]

Some projects, for example xdp-tools [0], prefer to check in a minimized
vmlinux.h rather than the complete file which can get rather large.

However, when you try to add a minimized version of a complex struct (eg
struct xfrm_state), things can get quite complex if you're trying to
manually untangle and deduplicate the dependencies.

This commit teaches bpftool to do a minimized dump of a single type by
providing an optional root_id argument.

Example usage:

    $ ./bpftool btf dump file ~/dev/linux/vmlinux | rg "STRUCT 'xfrm_state'"
    [12643] STRUCT 'xfrm_state' size=912 vlen=58

    $ ./bpftool btf dump file ~/dev/linux/vmlinux root_id 12643 format c
    #ifndef __VMLINUX_H__
    #define __VMLINUX_H__

    [..]

    struct xfrm_type_offload;

    struct xfrm_sec_ctx;

    struct xfrm_state {
            possible_net_t xs_net;
            union {
                    struct hlist_node gclist;
                    struct hlist_node bydst;
            };
            union {
                    struct hlist_node dev_gclist;
                    struct hlist_node bysrc;
            };
            struct hlist_node byspi;
    [..]

[0]: https://github.com/xdp-project/xdp-tools/blob/master/headers/bpf/vmlinux.h

=== Changelog ===
Changes in v4:
* Support multiple instances of root_id

Changes in v3:
* Make `root_id` a top level btf-dump argument rather than attached to `file`
* Update bash completion script
* Refactor root_type_ids checking to after btf handle creation
* Update help messages and fix existing man page inconsistency

Changes in v2:
* Add early error check for invalid BTF ID

Daniel Xu (4):
  bpftool: man: Add missing format argument to command description
  bpftool: btf: Validate root_type_ids early
  bpftool: btf: Support dumping a specific types from file
  bpftool: bash: Add bash completion for root_id argument

 .../bpf/bpftool/Documentation/bpftool-btf.rst |  8 ++-
 tools/bpf/bpftool/bash-completion/bpftool     |  7 ++-
 tools/bpf/bpftool/btf.c                       | 51 ++++++++++++++++++-
 3 files changed, 60 insertions(+), 6 deletions(-)

-- 
2.46.0

[PATCH bpf-next v4 1/4] bpftool: man: Add missing format argument to command description

[Daniel Xu]

The command description was missing the optional argument. Add it there
for consistency with the rest of the commands.

Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
---
 tools/bpf/bpftool/Documentation/bpftool-btf.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/bpf/bpftool/Documentation/bpftool-btf.rst b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
index 3f6bca03ad2e..245569f43035 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-btf.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
@@ -43,7 +43,7 @@ bpftool btf { show | list } [id *BTF_ID*]
     that hold open file descriptors (FDs) against BTF objects. On such kernels
     bpftool will automatically emit this information as well.
 
-bpftool btf dump *BTF_SRC*
+bpftool btf dump *BTF_SRC* [format *FORMAT*]
     Dump BTF entries from a given *BTF_SRC*.
 
     When **id** is specified, BTF object with that ID will be loaded and all
-- 
2.46.0

[PATCH bpf-next v4 2/4] bpftool: btf: Validate root_type_ids early

[Daniel Xu]

Handle invalid root_type_ids early, as an invalid ID will cause dumpers
to half-emit valid boilerplate and then bail with an unclean exit. This
is ugly and possibly confusing for users, so preemptively handle the
common error case before any dumping begins.

Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
---
 tools/bpf/bpftool/btf.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/tools/bpf/bpftool/btf.c b/tools/bpf/bpftool/btf.c
index d005e4fd6128..3e995faf9efa 100644
--- a/tools/bpf/bpftool/btf.c
+++ b/tools/bpf/bpftool/btf.c
@@ -886,6 +886,7 @@ static int do_dump(int argc, char **argv)
 	const char *src;
 	int fd = -1;
 	int err = 0;
+	int i;
 
 	if (!REQ_ARGS(2)) {
 		usage();
@@ -1017,6 +1018,17 @@ static int do_dump(int argc, char **argv)
 		}
 	}
 
+	/* Invalid root IDs causes half emitted boilerplate and then unclean
+	 * exit. It's an ugly user experience, so handle common error here.
+	 */
+	for (i = 0; i < root_type_cnt; i++) {
+		if (root_type_ids[i] >= btf__type_cnt(btf)) {
+			err = -EINVAL;
+			p_err("invalid root ID: %u", root_type_ids[i]);
+			goto done;
+		}
+	}
+
 	if (dump_c) {
 		if (json_output) {
 			p_err("JSON output for C-syntax dump is not supported");
-- 
2.46.0

[PATCH bpf-next v4 3/4] bpftool: btf: Support dumping a specific types from file

[Daniel Xu]

Some projects, for example xdp-tools [0], prefer to check in a minimized
vmlinux.h rather than the complete file which can get rather large.

However, when you try to add a minimized version of a complex struct (eg
struct xfrm_state), things can get quite complex if you're trying to
manually untangle and deduplicate the dependencies.

This commit teaches bpftool to do a minimized dump of a specific types by
providing a optional root_id argument(s).

Example usage:

    $ ./bpftool btf dump file ~/dev/linux/vmlinux | rg "STRUCT 'xfrm_state'"
    [12643] STRUCT 'xfrm_state' size=912 vlen=58

    $ ./bpftool btf dump file ~/dev/linux/vmlinux root_id 12643 format c
    #ifndef __VMLINUX_H__
    #define __VMLINUX_H__

    [..]

    struct xfrm_type_offload;

    struct xfrm_sec_ctx;

    struct xfrm_state {
            possible_net_t xs_net;
            union {
                    struct hlist_node gclist;
                    struct hlist_node bydst;
            };
            union {
                    struct hlist_node dev_gclist;
                    struct hlist_node bysrc;
            };
            struct hlist_node byspi;
    [..]

[0]: https://github.com/xdp-project/xdp-tools/blob/master/headers/bpf/vmlinux.h

Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
---
 .../bpf/bpftool/Documentation/bpftool-btf.rst |  8 +++-
 tools/bpf/bpftool/btf.c                       | 39 ++++++++++++++++++-
 2 files changed, 43 insertions(+), 4 deletions(-)

diff --git a/tools/bpf/bpftool/Documentation/bpftool-btf.rst b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
index 245569f43035..dbe6d6d94e4c 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-btf.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
@@ -24,7 +24,7 @@ BTF COMMANDS
 =============
 
 | **bpftool** **btf** { **show** | **list** } [**id** *BTF_ID*]
-| **bpftool** **btf dump** *BTF_SRC* [**format** *FORMAT*]
+| **bpftool** **btf dump** *BTF_SRC* [**format** *FORMAT*] [**root_id** *ROOT_ID*]
 | **bpftool** **btf help**
 |
 | *BTF_SRC* := { **id** *BTF_ID* | **prog** *PROG* | **map** *MAP* [{**key** | **value** | **kv** | **all**}] | **file** *FILE* }
@@ -43,7 +43,7 @@ bpftool btf { show | list } [id *BTF_ID*]
     that hold open file descriptors (FDs) against BTF objects. On such kernels
     bpftool will automatically emit this information as well.
 
-bpftool btf dump *BTF_SRC* [format *FORMAT*]
+bpftool btf dump *BTF_SRC* [format *FORMAT*] [root_id *ROOT_ID*]
     Dump BTF entries from a given *BTF_SRC*.
 
     When **id** is specified, BTF object with that ID will be loaded and all
@@ -67,6 +67,10 @@ bpftool btf dump *BTF_SRC* [format *FORMAT*]
     formatting, the output is sorted by default. Use the **unsorted** option
     to avoid sorting the output.
 
+    **root_id** option can be used to filter a dump to a single type and all
+    its dependent types. It cannot be used with any other types of filtering.
+    It can be passed multiple times to dump multiple types.
+
 bpftool btf help
     Print short help message.
 
diff --git a/tools/bpf/bpftool/btf.c b/tools/bpf/bpftool/btf.c
index 3e995faf9efa..2636655ac180 100644
--- a/tools/bpf/bpftool/btf.c
+++ b/tools/bpf/bpftool/btf.c
@@ -27,6 +27,8 @@
 #define KFUNC_DECL_TAG		"bpf_kfunc"
 #define FASTCALL_DECL_TAG	"bpf_fastcall"
 
+#define MAX_ROOT_IDS		16
+
 static const char * const btf_kind_str[NR_BTF_KINDS] = {
 	[BTF_KIND_UNKN]		= "UNKNOWN",
 	[BTF_KIND_INT]		= "INT",
@@ -880,7 +882,8 @@ static int do_dump(int argc, char **argv)
 {
 	bool dump_c = false, sort_dump_c = true;
 	struct btf *btf = NULL, *base = NULL;
-	__u32 root_type_ids[2];
+	__u32 root_type_ids[MAX_ROOT_IDS];
+	bool have_id_filtering;
 	int root_type_cnt = 0;
 	__u32 btf_id = -1;
 	const char *src;
@@ -974,6 +977,8 @@ static int do_dump(int argc, char **argv)
 		goto done;
 	}
 
+	have_id_filtering = !!root_type_cnt;
+
 	while (argc) {
 		if (is_prefix(*argv, "format")) {
 			NEXT_ARG();
@@ -993,6 +998,36 @@ static int do_dump(int argc, char **argv)
 				goto done;
 			}
 			NEXT_ARG();
+		} else if (is_prefix(*argv, "root_id")) {
+			__u32 root_id;
+			char *end;
+
+			if (have_id_filtering) {
+				p_err("cannot use root_id with other type filtering");
+				err = -EINVAL;
+				goto done;
+			} else if (root_type_cnt == MAX_ROOT_IDS) {
+				p_err("only %d root_id are supported", MAX_ROOT_IDS);
+				err = -E2BIG;
+				goto done;
+			}
+
+			NEXT_ARG();
+			root_id = strtoul(*argv, &end, 0);
+			if (*end) {
+				err = -1;
+				p_err("can't parse %s as root ID", *argv);
+				goto done;
+			}
+			for (i = 0; i < root_type_cnt; i++) {
+				if (root_type_ids[i] == root_id) {
+					err = -EINVAL;
+					p_err("duplicate root_id %d supplied", root_id);
+					goto done;
+				}
+			}
+			root_type_ids[root_type_cnt++] = root_id;
+			NEXT_ARG();
 		} else if (is_prefix(*argv, "unsorted")) {
 			sort_dump_c = false;
 			NEXT_ARG();
@@ -1403,7 +1438,7 @@ static int do_help(int argc, char **argv)
 
 	fprintf(stderr,
 		"Usage: %1$s %2$s { show | list } [id BTF_ID]\n"
-		"       %1$s %2$s dump BTF_SRC [format FORMAT]\n"
+		"       %1$s %2$s dump BTF_SRC [format FORMAT] [root_id ROOT_ID]\n"
 		"       %1$s %2$s help\n"
 		"\n"
 		"       BTF_SRC := { id BTF_ID | prog PROG | map MAP [{key | value | kv | all}] | file FILE }\n"
-- 
2.46.0

[PATCH bpf-next v4 4/4] bpftool: bash: Add bash completion for root_id argument

[Daniel Xu]

This commit updates the bash completion script with the new root_id
argument.

Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
---
 tools/bpf/bpftool/bash-completion/bpftool | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/tools/bpf/bpftool/bash-completion/bpftool b/tools/bpf/bpftool/bash-completion/bpftool
index 0c541498c301..097d406ee21f 100644
--- a/tools/bpf/bpftool/bash-completion/bpftool
+++ b/tools/bpf/bpftool/bash-completion/bpftool
@@ -930,6 +930,9 @@ _bpftool()
                         format)
                             COMPREPLY=( $( compgen -W "c raw" -- "$cur" ) )
                             ;;
+                        root_id)
+                            return 0;
+                            ;;
                         c)
                             COMPREPLY=( $( compgen -W "unsorted" -- "$cur" ) )
                             ;;
@@ -937,13 +940,13 @@ _bpftool()
                             # emit extra options
                             case ${words[3]} in
                                 id|file)
-                                    _bpftool_once_attr 'format'
+                                    _bpftool_once_attr 'format root_id'
                                     ;;
                                 map|prog)
                                     if [[ ${words[3]} == "map" ]] && [[ $cword == 6 ]]; then
                                         COMPREPLY+=( $( compgen -W "key value kv all" -- "$cur" ) )
                                     fi
-                                    _bpftool_once_attr 'format'
+                                    _bpftool_once_attr 'format root_id'
                                     ;;
                                 *)
                                     ;;
-- 
2.46.0

Re: [PATCH bpf-next v4 1/4] bpftool: man: Add missing format argument to command description

[Quentin Monnet]

2024-12-12 18:24 UTC-0700 ~ Daniel Xu <dxu@dxuuu.xyz>
> The command description was missing the optional argument. Add it there
> for consistency with the rest of the commands.
> 
> Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>


Reviewed-by: Quentin Monnet <qmo@kernel.org>

Thanks!

Re: [PATCH bpf-next v4 2/4] bpftool: btf: Validate root_type_ids early

[Quentin Monnet]

2024-12-12 18:24 UTC-0700 ~ Daniel Xu <dxu@dxuuu.xyz>
> Handle invalid root_type_ids early, as an invalid ID will cause dumpers
> to half-emit valid boilerplate and then bail with an unclean exit. This
> is ugly and possibly confusing for users, so preemptively handle the
> common error case before any dumping begins.
> 
> Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>


Reviewed-by: Quentin Monnet <qmo@kernel.org>

Re: [PATCH bpf-next v4 3/4] bpftool: btf: Support dumping a specific types from file

[Quentin Monnet]

2024-12-12 18:24 UTC-0700 ~ Daniel Xu <dxu@dxuuu.xyz>
> Some projects, for example xdp-tools [0], prefer to check in a minimized
> vmlinux.h rather than the complete file which can get rather large.
> 
> However, when you try to add a minimized version of a complex struct (eg
> struct xfrm_state), things can get quite complex if you're trying to
> manually untangle and deduplicate the dependencies.
> 
> This commit teaches bpftool to do a minimized dump of a specific types by
> providing a optional root_id argument(s).
> 
> Example usage:
> 
>     $ ./bpftool btf dump file ~/dev/linux/vmlinux | rg "STRUCT 'xfrm_state'"
>     [12643] STRUCT 'xfrm_state' size=912 vlen=58
> 
>     $ ./bpftool btf dump file ~/dev/linux/vmlinux root_id 12643 format c
>     #ifndef __VMLINUX_H__
>     #define __VMLINUX_H__
> 
>     [..]
> 
>     struct xfrm_type_offload;
> 
>     struct xfrm_sec_ctx;
> 
>     struct xfrm_state {
>             possible_net_t xs_net;
>             union {
>                     struct hlist_node gclist;
>                     struct hlist_node bydst;
>             };
>             union {
>                     struct hlist_node dev_gclist;
>                     struct hlist_node bysrc;
>             };
>             struct hlist_node byspi;
>     [..]
> 
> [0]: https://github.com/xdp-project/xdp-tools/blob/master/headers/bpf/vmlinux.h
> 
> Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
> ---
>  .../bpf/bpftool/Documentation/bpftool-btf.rst |  8 +++-
>  tools/bpf/bpftool/btf.c                       | 39 ++++++++++++++++++-
>  2 files changed, 43 insertions(+), 4 deletions(-)
> 
> diff --git a/tools/bpf/bpftool/Documentation/bpftool-btf.rst b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
> index 245569f43035..dbe6d6d94e4c 100644
> --- a/tools/bpf/bpftool/Documentation/bpftool-btf.rst
> +++ b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
> @@ -24,7 +24,7 @@ BTF COMMANDS
>  =============
>  
>  | **bpftool** **btf** { **show** | **list** } [**id** *BTF_ID*]
> -| **bpftool** **btf dump** *BTF_SRC* [**format** *FORMAT*]
> +| **bpftool** **btf dump** *BTF_SRC* [**format** *FORMAT*] [**root_id** *ROOT_ID*]
>  | **bpftool** **btf help**
>  |
>  | *BTF_SRC* := { **id** *BTF_ID* | **prog** *PROG* | **map** *MAP* [{**key** | **value** | **kv** | **all**}] | **file** *FILE* }
> @@ -43,7 +43,7 @@ bpftool btf { show | list } [id *BTF_ID*]
>      that hold open file descriptors (FDs) against BTF objects. On such kernels
>      bpftool will automatically emit this information as well.
>  
> -bpftool btf dump *BTF_SRC* [format *FORMAT*]
> +bpftool btf dump *BTF_SRC* [format *FORMAT*] [root_id *ROOT_ID*]
>      Dump BTF entries from a given *BTF_SRC*.
>  
>      When **id** is specified, BTF object with that ID will be loaded and all
> @@ -67,6 +67,10 @@ bpftool btf dump *BTF_SRC* [format *FORMAT*]
>      formatting, the output is sorted by default. Use the **unsorted** option
>      to avoid sorting the output.
>  
> +    **root_id** option can be used to filter a dump to a single type and all
> +    its dependent types. It cannot be used with any other types of filtering.
> +    It can be passed multiple times to dump multiple types.
> +
>  bpftool btf help
>      Print short help message.
>  
> diff --git a/tools/bpf/bpftool/btf.c b/tools/bpf/bpftool/btf.c
> index 3e995faf9efa..2636655ac180 100644
> --- a/tools/bpf/bpftool/btf.c
> +++ b/tools/bpf/bpftool/btf.c
> @@ -27,6 +27,8 @@
>  #define KFUNC_DECL_TAG		"bpf_kfunc"
>  #define FASTCALL_DECL_TAG	"bpf_fastcall"
>  
> +#define MAX_ROOT_IDS		16
> +
>  static const char * const btf_kind_str[NR_BTF_KINDS] = {
>  	[BTF_KIND_UNKN]		= "UNKNOWN",
>  	[BTF_KIND_INT]		= "INT",
> @@ -880,7 +882,8 @@ static int do_dump(int argc, char **argv)
>  {
>  	bool dump_c = false, sort_dump_c = true;
>  	struct btf *btf = NULL, *base = NULL;
> -	__u32 root_type_ids[2];
> +	__u32 root_type_ids[MAX_ROOT_IDS];
> +	bool have_id_filtering;
>  	int root_type_cnt = 0;
>  	__u32 btf_id = -1;
>  	const char *src;
> @@ -974,6 +977,8 @@ static int do_dump(int argc, char **argv)
>  		goto done;
>  	}
>  
> +	have_id_filtering = !!root_type_cnt;
> +
>  	while (argc) {
>  		if (is_prefix(*argv, "format")) {
>  			NEXT_ARG();
> @@ -993,6 +998,36 @@ static int do_dump(int argc, char **argv)
>  				goto done;
>  			}
>  			NEXT_ARG();
> +		} else if (is_prefix(*argv, "root_id")) {
> +			__u32 root_id;
> +			char *end;
> +
> +			if (have_id_filtering) {
> +				p_err("cannot use root_id with other type filtering");
> +				err = -EINVAL;
> +				goto done;
> +			} else if (root_type_cnt == MAX_ROOT_IDS) {
> +				p_err("only %d root_id are supported", MAX_ROOT_IDS);


I doubt users will often reach this limit, but if they do, the message
can be confusing, because MAX_ROOT_IDS also accounts for root_type_ids[]
cells used when we pass map arguments ("key" or "value" or "kv"), so you
could pass 15 "root_id" on the command line and get a message telling
only 16 are supported.

Maybe add a counter to tell how many were defined from the rest of the
command line, and adjust the value in the error message?


> +				err = -E2BIG;
> +				goto done;
> +			}
> +
> +			NEXT_ARG();
> +			root_id = strtoul(*argv, &end, 0);
> +			if (*end) {
> +				err = -1;
> +				p_err("can't parse %s as root ID", *argv);
> +				goto done;
> +			}
> +			for (i = 0; i < root_type_cnt; i++) {
> +				if (root_type_ids[i] == root_id) {
> +					err = -EINVAL;
> +					p_err("duplicate root_id %d supplied", root_id);
> +					goto done;
> +				}
> +			}
> +			root_type_ids[root_type_cnt++] = root_id;
> +			NEXT_ARG();
>  		} else if (is_prefix(*argv, "unsorted")) {
>  			sort_dump_c = false;
>  			NEXT_ARG();
> @@ -1403,7 +1438,7 @@ static int do_help(int argc, char **argv)
>  
>  	fprintf(stderr,
>  		"Usage: %1$s %2$s { show | list } [id BTF_ID]\n"
> -		"       %1$s %2$s dump BTF_SRC [format FORMAT]\n"
> +		"       %1$s %2$s dump BTF_SRC [format FORMAT] [root_id ROOT_ID]\n"
>  		"       %1$s %2$s help\n"
>  		"\n"
>  		"       BTF_SRC := { id BTF_ID | prog PROG | map MAP [{key | value | kv | all}] | file FILE }\n"

Re: [PATCH bpf-next v4 4/4] bpftool: bash: Add bash completion for root_id argument

[Quentin Monnet]

2024-12-12 18:24 UTC-0700 ~ Daniel Xu <dxu@dxuuu.xyz>
> This commit updates the bash completion script with the new root_id
> argument.
> 
> Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
> ---
>  tools/bpf/bpftool/bash-completion/bpftool | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/bpf/bpftool/bash-completion/bpftool b/tools/bpf/bpftool/bash-completion/bpftool
> index 0c541498c301..097d406ee21f 100644
> --- a/tools/bpf/bpftool/bash-completion/bpftool
> +++ b/tools/bpf/bpftool/bash-completion/bpftool
> @@ -930,6 +930,9 @@ _bpftool()
>                          format)
>                              COMPREPLY=( $( compgen -W "c raw" -- "$cur" ) )
>                              ;;
> +                        root_id)
> +                            return 0;
> +                            ;;
>                          c)
>                              COMPREPLY=( $( compgen -W "unsorted" -- "$cur" ) )
>                              ;;
> @@ -937,13 +940,13 @@ _bpftool()
>                              # emit extra options
>                              case ${words[3]} in
>                                  id|file)
> -                                    _bpftool_once_attr 'format'
> +                                    _bpftool_once_attr 'format root_id'
>                                      ;;
>                                  map|prog)
>                                      if [[ ${words[3]} == "map" ]] && [[ $cword == 6 ]]; then
>                                          COMPREPLY+=( $( compgen -W "key value kv all" -- "$cur" ) )
>                                      fi
> -                                    _bpftool_once_attr 'format'
> +                                    _bpftool_once_attr 'format root_id'
>                                      ;;
>                                  *)
>                                      ;;


Thanks! If we support multiple occurrences for root_id, let's adjust the
completion (and also complete with root_id after "format c", by the way,
I missed it last time):

------
diff --git a/tools/bpf/bpftool/bash-completion/bpftool b/tools/bpf/bpftool/bash-completion/bpftool
index 0c541498c301..e5bf809656ed 100644
--- a/tools/bpf/bpftool/bash-completion/bpftool
+++ b/tools/bpf/bpftool/bash-completion/bpftool
@@ -930,19 +930,24 @@ _bpftool()
                         format)
                             COMPREPLY=( $( compgen -W "c raw" -- "$cur" ) )
                             ;;
+                        root_id)
+                            return 0
+                            ;;
                         c)
-                            COMPREPLY=( $( compgen -W "unsorted" -- "$cur" ) )
+                            COMPREPLY=( $( compgen -W "unsorted root_id" -- "$cur" ) )
                             ;;
                         *)
                             # emit extra options
                             case ${words[3]} in
                                 id|file)
+                                    COMPREPLY=( $( compgen -W "root_id" -- "$cur" ) )
                                     _bpftool_once_attr 'format'
                                     ;;
                                 map|prog)
                                     if [[ ${words[3]} == "map" ]] && [[ $cword == 6 ]]; then
                                         COMPREPLY+=( $( compgen -W "key value kv all" -- "$cur" ) )
                                     fi
+                                    COMPREPLY=( $( compgen -W "root_id" -- "$cur" ) )
                                     _bpftool_once_attr 'format'
                                     ;;
                                 *)
------

Re: [PATCH bpf-next v4 3/4] bpftool: btf: Support dumping a specific types from file

[Daniel Xu]

Hi Quentin,

On Fri, Dec 13, 2024 at 03:17:36PM GMT, Quentin Monnet wrote:
> 2024-12-12 18:24 UTC-0700 ~ Daniel Xu <dxu@dxuuu.xyz>
> > Some projects, for example xdp-tools [0], prefer to check in a minimized
> > vmlinux.h rather than the complete file which can get rather large.
> > 
> > However, when you try to add a minimized version of a complex struct (eg
> > struct xfrm_state), things can get quite complex if you're trying to
> > manually untangle and deduplicate the dependencies.
> > 
> > This commit teaches bpftool to do a minimized dump of a specific types by
> > providing a optional root_id argument(s).
> > 
> > Example usage:
> > 
> >     $ ./bpftool btf dump file ~/dev/linux/vmlinux | rg "STRUCT 'xfrm_state'"
> >     [12643] STRUCT 'xfrm_state' size=912 vlen=58
> > 
> >     $ ./bpftool btf dump file ~/dev/linux/vmlinux root_id 12643 format c
> >     #ifndef __VMLINUX_H__
> >     #define __VMLINUX_H__
> > 
> >     [..]
> > 
> >     struct xfrm_type_offload;
> > 
> >     struct xfrm_sec_ctx;
> > 
> >     struct xfrm_state {
> >             possible_net_t xs_net;
> >             union {
> >                     struct hlist_node gclist;
> >                     struct hlist_node bydst;
> >             };
> >             union {
> >                     struct hlist_node dev_gclist;
> >                     struct hlist_node bysrc;
> >             };
> >             struct hlist_node byspi;
> >     [..]
> > 
> > [0]: https://github.com/xdp-project/xdp-tools/blob/master/headers/bpf/vmlinux.h
> > 
> > Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
> > ---
> >  .../bpf/bpftool/Documentation/bpftool-btf.rst |  8 +++-
> >  tools/bpf/bpftool/btf.c                       | 39 ++++++++++++++++++-
> >  2 files changed, 43 insertions(+), 4 deletions(-)
> > 
> > diff --git a/tools/bpf/bpftool/Documentation/bpftool-btf.rst b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
> > index 245569f43035..dbe6d6d94e4c 100644
> > --- a/tools/bpf/bpftool/Documentation/bpftool-btf.rst
> > +++ b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
> > @@ -24,7 +24,7 @@ BTF COMMANDS
> >  =============
> >  
> >  | **bpftool** **btf** { **show** | **list** } [**id** *BTF_ID*]
> > -| **bpftool** **btf dump** *BTF_SRC* [**format** *FORMAT*]
> > +| **bpftool** **btf dump** *BTF_SRC* [**format** *FORMAT*] [**root_id** *ROOT_ID*]
> >  | **bpftool** **btf help**
> >  |
> >  | *BTF_SRC* := { **id** *BTF_ID* | **prog** *PROG* | **map** *MAP* [{**key** | **value** | **kv** | **all**}] | **file** *FILE* }
> > @@ -43,7 +43,7 @@ bpftool btf { show | list } [id *BTF_ID*]
> >      that hold open file descriptors (FDs) against BTF objects. On such kernels
> >      bpftool will automatically emit this information as well.
> >  
> > -bpftool btf dump *BTF_SRC* [format *FORMAT*]
> > +bpftool btf dump *BTF_SRC* [format *FORMAT*] [root_id *ROOT_ID*]
> >      Dump BTF entries from a given *BTF_SRC*.
> >  
> >      When **id** is specified, BTF object with that ID will be loaded and all
> > @@ -67,6 +67,10 @@ bpftool btf dump *BTF_SRC* [format *FORMAT*]
> >      formatting, the output is sorted by default. Use the **unsorted** option
> >      to avoid sorting the output.
> >  
> > +    **root_id** option can be used to filter a dump to a single type and all
> > +    its dependent types. It cannot be used with any other types of filtering.
> > +    It can be passed multiple times to dump multiple types.
> > +
> >  bpftool btf help
> >      Print short help message.
> >  
> > diff --git a/tools/bpf/bpftool/btf.c b/tools/bpf/bpftool/btf.c
> > index 3e995faf9efa..2636655ac180 100644
> > --- a/tools/bpf/bpftool/btf.c
> > +++ b/tools/bpf/bpftool/btf.c
> > @@ -27,6 +27,8 @@
> >  #define KFUNC_DECL_TAG		"bpf_kfunc"
> >  #define FASTCALL_DECL_TAG	"bpf_fastcall"
> >  
> > +#define MAX_ROOT_IDS		16
> > +
> >  static const char * const btf_kind_str[NR_BTF_KINDS] = {
> >  	[BTF_KIND_UNKN]		= "UNKNOWN",
> >  	[BTF_KIND_INT]		= "INT",
> > @@ -880,7 +882,8 @@ static int do_dump(int argc, char **argv)
> >  {
> >  	bool dump_c = false, sort_dump_c = true;
> >  	struct btf *btf = NULL, *base = NULL;
> > -	__u32 root_type_ids[2];
> > +	__u32 root_type_ids[MAX_ROOT_IDS];
> > +	bool have_id_filtering;
> >  	int root_type_cnt = 0;
> >  	__u32 btf_id = -1;
> >  	const char *src;
> > @@ -974,6 +977,8 @@ static int do_dump(int argc, char **argv)
> >  		goto done;
> >  	}
> >  
> > +	have_id_filtering = !!root_type_cnt;
> > +
> >  	while (argc) {
> >  		if (is_prefix(*argv, "format")) {
> >  			NEXT_ARG();
> > @@ -993,6 +998,36 @@ static int do_dump(int argc, char **argv)
> >  				goto done;
> >  			}
> >  			NEXT_ARG();
> > +		} else if (is_prefix(*argv, "root_id")) {
> > +			__u32 root_id;
> > +			char *end;
> > +
> > +			if (have_id_filtering) {
> > +				p_err("cannot use root_id with other type filtering");
> > +				err = -EINVAL;
> > +				goto done;
> > +			} else if (root_type_cnt == MAX_ROOT_IDS) {
> > +				p_err("only %d root_id are supported", MAX_ROOT_IDS);
> 
> 
> I doubt users will often reach this limit, but if they do, the message
> can be confusing, because MAX_ROOT_IDS also accounts for root_type_ids[]
> cells used when we pass map arguments ("key" or "value" or "kv"), so you
> could pass 15 "root_id" on the command line and get a message telling
> only 16 are supported.
> 
> Maybe add a counter to tell how many were defined from the rest of the
> command line, and adjust the value in the error message?

The above `if (have_id_filtering)` check prevents mixing key/value/kv
map args with root_id. That ought to prevent overcounting, right?

Thanks,
Daniel

Re: [PATCH bpf-next v4 3/4] bpftool: btf: Support dumping a specific types from file

[Quentin Monnet]

2024-12-13 09:45 UTC-0700 ~ Daniel Xu <dxu@dxuuu.xyz>
> Hi Quentin,
> 
> On Fri, Dec 13, 2024 at 03:17:36PM GMT, Quentin Monnet wrote:
>> 2024-12-12 18:24 UTC-0700 ~ Daniel Xu <dxu@dxuuu.xyz>
>>> Some projects, for example xdp-tools [0], prefer to check in a minimized
>>> vmlinux.h rather than the complete file which can get rather large.
>>>
>>> However, when you try to add a minimized version of a complex struct (eg
>>> struct xfrm_state), things can get quite complex if you're trying to
>>> manually untangle and deduplicate the dependencies.
>>>
>>> This commit teaches bpftool to do a minimized dump of a specific types by
>>> providing a optional root_id argument(s).
>>>
>>> Example usage:
>>>
>>>     $ ./bpftool btf dump file ~/dev/linux/vmlinux | rg "STRUCT 'xfrm_state'"
>>>     [12643] STRUCT 'xfrm_state' size=912 vlen=58
>>>
>>>     $ ./bpftool btf dump file ~/dev/linux/vmlinux root_id 12643 format c
>>>     #ifndef __VMLINUX_H__
>>>     #define __VMLINUX_H__
>>>
>>>     [..]
>>>
>>>     struct xfrm_type_offload;
>>>
>>>     struct xfrm_sec_ctx;
>>>
>>>     struct xfrm_state {
>>>             possible_net_t xs_net;
>>>             union {
>>>                     struct hlist_node gclist;
>>>                     struct hlist_node bydst;
>>>             };
>>>             union {
>>>                     struct hlist_node dev_gclist;
>>>                     struct hlist_node bysrc;
>>>             };
>>>             struct hlist_node byspi;
>>>     [..]
>>>
>>> [0]: https://github.com/xdp-project/xdp-tools/blob/master/headers/bpf/vmlinux.h
>>>
>>> Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
>>> ---
>>>  .../bpf/bpftool/Documentation/bpftool-btf.rst |  8 +++-
>>>  tools/bpf/bpftool/btf.c                       | 39 ++++++++++++++++++-
>>>  2 files changed, 43 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/tools/bpf/bpftool/Documentation/bpftool-btf.rst b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
>>> index 245569f43035..dbe6d6d94e4c 100644
>>> --- a/tools/bpf/bpftool/Documentation/bpftool-btf.rst
>>> +++ b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
>>> @@ -24,7 +24,7 @@ BTF COMMANDS
>>>  =============
>>>  
>>>  | **bpftool** **btf** { **show** | **list** } [**id** *BTF_ID*]
>>> -| **bpftool** **btf dump** *BTF_SRC* [**format** *FORMAT*]
>>> +| **bpftool** **btf dump** *BTF_SRC* [**format** *FORMAT*] [**root_id** *ROOT_ID*]
>>>  | **bpftool** **btf help**
>>>  |
>>>  | *BTF_SRC* := { **id** *BTF_ID* | **prog** *PROG* | **map** *MAP* [{**key** | **value** | **kv** | **all**}] | **file** *FILE* }
>>> @@ -43,7 +43,7 @@ bpftool btf { show | list } [id *BTF_ID*]
>>>      that hold open file descriptors (FDs) against BTF objects. On such kernels
>>>      bpftool will automatically emit this information as well.
>>>  
>>> -bpftool btf dump *BTF_SRC* [format *FORMAT*]
>>> +bpftool btf dump *BTF_SRC* [format *FORMAT*] [root_id *ROOT_ID*]
>>>      Dump BTF entries from a given *BTF_SRC*.
>>>  
>>>      When **id** is specified, BTF object with that ID will be loaded and all
>>> @@ -67,6 +67,10 @@ bpftool btf dump *BTF_SRC* [format *FORMAT*]
>>>      formatting, the output is sorted by default. Use the **unsorted** option
>>>      to avoid sorting the output.
>>>  
>>> +    **root_id** option can be used to filter a dump to a single type and all
>>> +    its dependent types. It cannot be used with any other types of filtering.
>>> +    It can be passed multiple times to dump multiple types.
>>> +
>>>  bpftool btf help
>>>      Print short help message.
>>>  
>>> diff --git a/tools/bpf/bpftool/btf.c b/tools/bpf/bpftool/btf.c
>>> index 3e995faf9efa..2636655ac180 100644
>>> --- a/tools/bpf/bpftool/btf.c
>>> +++ b/tools/bpf/bpftool/btf.c
>>> @@ -27,6 +27,8 @@
>>>  #define KFUNC_DECL_TAG		"bpf_kfunc"
>>>  #define FASTCALL_DECL_TAG	"bpf_fastcall"
>>>  
>>> +#define MAX_ROOT_IDS		16
>>> +
>>>  static const char * const btf_kind_str[NR_BTF_KINDS] = {
>>>  	[BTF_KIND_UNKN]		= "UNKNOWN",
>>>  	[BTF_KIND_INT]		= "INT",
>>> @@ -880,7 +882,8 @@ static int do_dump(int argc, char **argv)
>>>  {
>>>  	bool dump_c = false, sort_dump_c = true;
>>>  	struct btf *btf = NULL, *base = NULL;
>>> -	__u32 root_type_ids[2];
>>> +	__u32 root_type_ids[MAX_ROOT_IDS];
>>> +	bool have_id_filtering;
>>>  	int root_type_cnt = 0;
>>>  	__u32 btf_id = -1;
>>>  	const char *src;
>>> @@ -974,6 +977,8 @@ static int do_dump(int argc, char **argv)
>>>  		goto done;
>>>  	}
>>>  
>>> +	have_id_filtering = !!root_type_cnt;
>>> +
>>>  	while (argc) {
>>>  		if (is_prefix(*argv, "format")) {
>>>  			NEXT_ARG();
>>> @@ -993,6 +998,36 @@ static int do_dump(int argc, char **argv)
>>>  				goto done;
>>>  			}
>>>  			NEXT_ARG();
>>> +		} else if (is_prefix(*argv, "root_id")) {
>>> +			__u32 root_id;
>>> +			char *end;
>>> +
>>> +			if (have_id_filtering) {
>>> +				p_err("cannot use root_id with other type filtering");
>>> +				err = -EINVAL;
>>> +				goto done;
>>> +			} else if (root_type_cnt == MAX_ROOT_IDS) {
>>> +				p_err("only %d root_id are supported", MAX_ROOT_IDS);
>>
>>
>> I doubt users will often reach this limit, but if they do, the message
>> can be confusing, because MAX_ROOT_IDS also accounts for root_type_ids[]
>> cells used when we pass map arguments ("key" or "value" or "kv"), so you
>> could pass 15 "root_id" on the command line and get a message telling
>> only 16 are supported.
>>
>> Maybe add a counter to tell how many were defined from the rest of the
>> command line, and adjust the value in the error message?
> 
> The above `if (have_id_filtering)` check prevents mixing key/value/kv
> map args with root_id. That ought to prevent overcounting, right?

Ah, you're right, you even mentioned it in the docs, sorry. All good on
that side, then.

Regarding the restriction, would you mind making the mention from the
man page a bit more explicit, please? Something along:

    [...] It cannot be used with any other types of filtering, such as
    the "key", "value", or "kv" arguments when dumping BTF for a map.
    **root_id** can be passed multiple times...

Thanks,
Quentin

Re: [PATCH bpf-next v4 3/4] bpftool: btf: Support dumping a specific types from file

[Daniel Xu]

On Fri, Dec 13, 2024 at 04:55:34PM GMT, Quentin Monnet wrote:
> 2024-12-13 09:45 UTC-0700 ~ Daniel Xu <dxu@dxuuu.xyz>
> > Hi Quentin,
> > 
> > On Fri, Dec 13, 2024 at 03:17:36PM GMT, Quentin Monnet wrote:
> >> 2024-12-12 18:24 UTC-0700 ~ Daniel Xu <dxu@dxuuu.xyz>
> >>> Some projects, for example xdp-tools [0], prefer to check in a minimized
> >>> vmlinux.h rather than the complete file which can get rather large.
> >>>
> >>> However, when you try to add a minimized version of a complex struct (eg
> >>> struct xfrm_state), things can get quite complex if you're trying to
> >>> manually untangle and deduplicate the dependencies.
> >>>
> >>> This commit teaches bpftool to do a minimized dump of a specific types by
> >>> providing a optional root_id argument(s).
> >>>
> >>> Example usage:
> >>>
> >>>     $ ./bpftool btf dump file ~/dev/linux/vmlinux | rg "STRUCT 'xfrm_state'"
> >>>     [12643] STRUCT 'xfrm_state' size=912 vlen=58
> >>>
> >>>     $ ./bpftool btf dump file ~/dev/linux/vmlinux root_id 12643 format c
> >>>     #ifndef __VMLINUX_H__
> >>>     #define __VMLINUX_H__
> >>>
> >>>     [..]
> >>>
> >>>     struct xfrm_type_offload;
> >>>
> >>>     struct xfrm_sec_ctx;
> >>>
> >>>     struct xfrm_state {
> >>>             possible_net_t xs_net;
> >>>             union {
> >>>                     struct hlist_node gclist;
> >>>                     struct hlist_node bydst;
> >>>             };
> >>>             union {
> >>>                     struct hlist_node dev_gclist;
> >>>                     struct hlist_node bysrc;
> >>>             };
> >>>             struct hlist_node byspi;
> >>>     [..]
> >>>
> >>> [0]: https://github.com/xdp-project/xdp-tools/blob/master/headers/bpf/vmlinux.h
> >>>
> >>> Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
> >>> ---
> >>>  .../bpf/bpftool/Documentation/bpftool-btf.rst |  8 +++-
> >>>  tools/bpf/bpftool/btf.c                       | 39 ++++++++++++++++++-
> >>>  2 files changed, 43 insertions(+), 4 deletions(-)
> >>>
> >>> diff --git a/tools/bpf/bpftool/Documentation/bpftool-btf.rst b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
> >>> index 245569f43035..dbe6d6d94e4c 100644
> >>> --- a/tools/bpf/bpftool/Documentation/bpftool-btf.rst
> >>> +++ b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
> >>> @@ -24,7 +24,7 @@ BTF COMMANDS
> >>>  =============
> >>>  
> >>>  | **bpftool** **btf** { **show** | **list** } [**id** *BTF_ID*]
> >>> -| **bpftool** **btf dump** *BTF_SRC* [**format** *FORMAT*]
> >>> +| **bpftool** **btf dump** *BTF_SRC* [**format** *FORMAT*] [**root_id** *ROOT_ID*]
> >>>  | **bpftool** **btf help**
> >>>  |
> >>>  | *BTF_SRC* := { **id** *BTF_ID* | **prog** *PROG* | **map** *MAP* [{**key** | **value** | **kv** | **all**}] | **file** *FILE* }
> >>> @@ -43,7 +43,7 @@ bpftool btf { show | list } [id *BTF_ID*]
> >>>      that hold open file descriptors (FDs) against BTF objects. On such kernels
> >>>      bpftool will automatically emit this information as well.
> >>>  
> >>> -bpftool btf dump *BTF_SRC* [format *FORMAT*]
> >>> +bpftool btf dump *BTF_SRC* [format *FORMAT*] [root_id *ROOT_ID*]
> >>>      Dump BTF entries from a given *BTF_SRC*.
> >>>  
> >>>      When **id** is specified, BTF object with that ID will be loaded and all
> >>> @@ -67,6 +67,10 @@ bpftool btf dump *BTF_SRC* [format *FORMAT*]
> >>>      formatting, the output is sorted by default. Use the **unsorted** option
> >>>      to avoid sorting the output.
> >>>  
> >>> +    **root_id** option can be used to filter a dump to a single type and all
> >>> +    its dependent types. It cannot be used with any other types of filtering.
> >>> +    It can be passed multiple times to dump multiple types.
> >>> +
> >>>  bpftool btf help
> >>>      Print short help message.
> >>>  
> >>> diff --git a/tools/bpf/bpftool/btf.c b/tools/bpf/bpftool/btf.c
> >>> index 3e995faf9efa..2636655ac180 100644
> >>> --- a/tools/bpf/bpftool/btf.c
> >>> +++ b/tools/bpf/bpftool/btf.c
> >>> @@ -27,6 +27,8 @@
> >>>  #define KFUNC_DECL_TAG		"bpf_kfunc"
> >>>  #define FASTCALL_DECL_TAG	"bpf_fastcall"
> >>>  
> >>> +#define MAX_ROOT_IDS		16
> >>> +
> >>>  static const char * const btf_kind_str[NR_BTF_KINDS] = {
> >>>  	[BTF_KIND_UNKN]		= "UNKNOWN",
> >>>  	[BTF_KIND_INT]		= "INT",
> >>> @@ -880,7 +882,8 @@ static int do_dump(int argc, char **argv)
> >>>  {
> >>>  	bool dump_c = false, sort_dump_c = true;
> >>>  	struct btf *btf = NULL, *base = NULL;
> >>> -	__u32 root_type_ids[2];
> >>> +	__u32 root_type_ids[MAX_ROOT_IDS];
> >>> +	bool have_id_filtering;
> >>>  	int root_type_cnt = 0;
> >>>  	__u32 btf_id = -1;
> >>>  	const char *src;
> >>> @@ -974,6 +977,8 @@ static int do_dump(int argc, char **argv)
> >>>  		goto done;
> >>>  	}
> >>>  
> >>> +	have_id_filtering = !!root_type_cnt;
> >>> +
> >>>  	while (argc) {
> >>>  		if (is_prefix(*argv, "format")) {
> >>>  			NEXT_ARG();
> >>> @@ -993,6 +998,36 @@ static int do_dump(int argc, char **argv)
> >>>  				goto done;
> >>>  			}
> >>>  			NEXT_ARG();
> >>> +		} else if (is_prefix(*argv, "root_id")) {
> >>> +			__u32 root_id;
> >>> +			char *end;
> >>> +
> >>> +			if (have_id_filtering) {
> >>> +				p_err("cannot use root_id with other type filtering");
> >>> +				err = -EINVAL;
> >>> +				goto done;
> >>> +			} else if (root_type_cnt == MAX_ROOT_IDS) {
> >>> +				p_err("only %d root_id are supported", MAX_ROOT_IDS);
> >>
> >>
> >> I doubt users will often reach this limit, but if they do, the message
> >> can be confusing, because MAX_ROOT_IDS also accounts for root_type_ids[]
> >> cells used when we pass map arguments ("key" or "value" or "kv"), so you
> >> could pass 15 "root_id" on the command line and get a message telling
> >> only 16 are supported.
> >>
> >> Maybe add a counter to tell how many were defined from the rest of the
> >> command line, and adjust the value in the error message?
> > 
> > The above `if (have_id_filtering)` check prevents mixing key/value/kv
> > map args with root_id. That ought to prevent overcounting, right?
> 
> Ah, you're right, you even mentioned it in the docs, sorry. All good on
> that side, then.
> 
> Regarding the restriction, would you mind making the mention from the
> man page a bit more explicit, please? Something along:
> 
>     [...] It cannot be used with any other types of filtering, such as
>     the "key", "value", or "kv" arguments when dumping BTF for a map.
>     **root_id** can be passed multiple times...

Ack, will do.