Re: [PATCH dwarves v3] dwarf_loader: Fix skipped encoding of function BTF on 32-bit systems

[Alan Maguire <alan.maguire@oracle.com>]

On 22/05/2025 07:37, Tony Ambardar wrote:
> I encountered an issue building BTF kernels for 32-bit armhf, where many
> functions are missing in BTF data:
> 
>   LD      vmlinux
>   BTFIDS  vmlinux
> WARN: resolve_btfids: unresolved symbol vfs_truncate
> WARN: resolve_btfids: unresolved symbol vfs_fallocate
> WARN: resolve_btfids: unresolved symbol scx_bpf_select_cpu_dfl
> WARN: resolve_btfids: unresolved symbol scx_bpf_pick_idle_cpu_node
> WARN: resolve_btfids: unresolved symbol scx_bpf_pick_idle_cpu
> WARN: resolve_btfids: unresolved symbol scx_bpf_pick_any_cpu_node
> WARN: resolve_btfids: unresolved symbol scx_bpf_pick_any_cpu
> WARN: resolve_btfids: unresolved symbol scx_bpf_kick_cpu
> WARN: resolve_btfids: unresolved symbol scx_bpf_exit_bstr
> WARN: resolve_btfids: unresolved symbol scx_bpf_dsq_nr_queued
> WARN: resolve_btfids: unresolved symbol scx_bpf_dsq_move_vtime
> WARN: resolve_btfids: unresolved symbol scx_bpf_dsq_move_to_local
> WARN: resolve_btfids: unresolved symbol scx_bpf_dsq_move
> WARN: resolve_btfids: unresolved symbol scx_bpf_dsq_insert_vtime
> WARN: resolve_btfids: unresolved symbol scx_bpf_dsq_insert
> WARN: resolve_btfids: unresolved symbol scx_bpf_dispatch_vtime_from_dsq
> WARN: resolve_btfids: unresolved symbol scx_bpf_dispatch_vtime
> WARN: resolve_btfids: unresolved symbol scx_bpf_dispatch_from_dsq_set_vtime
> WARN: resolve_btfids: unresolved symbol scx_bpf_dispatch_from_dsq_set_slice
> WARN: resolve_btfids: unresolved symbol scx_bpf_dispatch_from_dsq
> WARN: resolve_btfids: unresolved symbol scx_bpf_dispatch
> WARN: resolve_btfids: unresolved symbol scx_bpf_destroy_dsq
> WARN: resolve_btfids: unresolved symbol scx_bpf_create_dsq
> WARN: resolve_btfids: unresolved symbol scx_bpf_consume
> WARN: resolve_btfids: unresolved symbol bpf_throw
> WARN: resolve_btfids: unresolved symbol bpf_sock_ops_enable_tx_tstamp
> WARN: resolve_btfids: unresolved symbol bpf_percpu_obj_new_impl
> WARN: resolve_btfids: unresolved symbol bpf_obj_new_impl
> WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> WARN: resolve_btfids: unresolved symbol bpf_iter_task_vma_new
> WARN: resolve_btfids: unresolved symbol bpf_iter_scx_dsq_new
> WARN: resolve_btfids: unresolved symbol bpf_get_kmem_cache
> WARN: resolve_btfids: unresolved symbol bpf_dynptr_from_xdp
> WARN: resolve_btfids: unresolved symbol bpf_dynptr_from_skb
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
>   NM      System.map
> 
> After further debugging this can be reproduced more simply:
> 
> $ pahole -J -j --btf_features=decl_tag,consistent_func,decl_tag_kfuncs .tmp_vmlinux_armhf
> btf_encoder__tag_kfunc: failed to find kfunc 'scx_bpf_select_cpu_dfl' in BTF
> btf_encoder__tag_kfuncs: failed to tag kfunc 'scx_bpf_select_cpu_dfl'
> 
> $ pfunct -Fbtf -E -f scx_bpf_select_cpu_dfl .tmp_vmlinux_armhf
> <nothing>
> 
> $ pfunct -Fdwarf -E -f scx_bpf_select_cpu_dfl .tmp_vmlinux_armhf
> s32 scx_bpf_select_cpu_dfl(struct task_struct * p, s32 prev_cpu, u64 wake_flags, bool * is_idle);
> 
> $ pahole -J -j --btf_features=decl_tag,decl_tag_kfuncs .tmp_vmlinux_armhf
> 
> $ pfunct -Fbtf -E -f scx_bpf_select_cpu_dfl .tmp_vmlinux_armhf
> bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct * p, s32 prev_cpu, u64 wake_flags, bool * is_idle);
> 
> The key things to note are the pahole 'consistent_func' feature and the u64
> 'wake_flags' parameter vs. arm 32-bit registers. These point to existing
> code handling arguments larger than register-size, allowing them to be
> BTF encoded but only if structs.
> 
> Generalize the code for any argument type larger than register size (i.e.
> size > cu->addr_size). This should work for integral or aggregate types,
> and also avoids a bug in the current code where a register-sized struct
> could be mistaken for larger. Note that zero-sized arguments will still
> be marked as inconsistent and not encoded.
> 
> Fixes: a53c58158b76 ("dwarf_loader: Mark functions that do not use expected registers for params")
> Tested-by: Alexis Lothoré <alexis.lothore@bootlin.com>
> Tested-by: Alan Maguire <alan.maguire@oracle.com>
> Signed-off-by: Tony Ambardar <tony.ambardar@gmail.com>

hi Tony,

I'm planning on landing this shortly unless anyone objects; and on that
topic if anyone has the cycles to test with this patch that would be
great! I ran it through the work-in-progress BTF comparison in github CI
and all looks good; see the "Compare functions generated" step in [1].

Thanks!

Alan

[1] https://github.com/alan-maguire/dwarves/actions/runs/15854137212

> ---
> v2 -> v3:
>  - Added Tested-by: from Alexis and Alan.
>  - Revert support for encoding 0-sized structs (as v1) after discussion:
>    https://lore.kernel.org/dwarves/9a41b21f-c0ae-4298-bf95-09d0cdc3f3ab@oracle.com/
>  - Inline param__is_wide() and clarify some naming/wording.
> 
> v1 -> v2:
>  - Update to preserve existing behaviour where zero-sized struct params
>    still permit the function to be encoded, as noted by Alan.
> 
> ---
>  dwarf_loader.c | 37 ++++++++++++-------------------------
>  1 file changed, 12 insertions(+), 25 deletions(-)
> 
> diff --git a/dwarf_loader.c b/dwarf_loader.c
> index e1ba7bc..134a76b 100644
> --- a/dwarf_loader.c
> +++ b/dwarf_loader.c
> @@ -2914,23 +2914,9 @@ out:
>  	return 0;
>  }
>  
> -static bool param__is_struct(struct cu *cu, struct tag *tag)
> +static inline bool param__is_wide(struct cu *cu, struct tag *tag)
>  {
> -	struct tag *type = cu__type(cu, tag->type);
> -
> -	if (!type)
> -		return false;
> -
> -	switch (type->tag) {
> -	case DW_TAG_structure_type:
> -		return true;
> -	case DW_TAG_const_type:
> -	case DW_TAG_typedef:
> -		/* handle "typedef struct", const parameter */
> -		return param__is_struct(cu, type);
> -	default:
> -		return false;
> -	}
> +	return tag__size(tag, cu) > cu->addr_size;
>  }
>  
>  static int cu__resolve_func_ret_types_optimized(struct cu *cu)
> @@ -2942,9 +2928,9 @@ static int cu__resolve_func_ret_types_optimized(struct cu *cu)
>  		struct tag *tag = pt->entries[i];
>  		struct parameter *pos;
>  		struct function *fn = tag__function(tag);
> -		bool has_unexpected_reg = false, has_struct_param = false;
> +		bool has_unexpected_reg = false, has_wide_param = false;
>  
> -		/* mark function as optimized if parameter is, or
> +		/* Mark function as optimized if parameter is, or
>  		 * if parameter does not have a location; at this
>  		 * point location presence has been marked in
>  		 * abstract origins for cases where a parameter
> @@ -2953,10 +2939,11 @@ static int cu__resolve_func_ret_types_optimized(struct cu *cu)
>  		 *
>  		 * Also mark functions which, due to optimization,
>  		 * use an unexpected register for a parameter.
> -		 * Exception is functions which have a struct
> -		 * as a parameter, as multiple registers may
> -		 * be used to represent it, throwing off register
> -		 * to parameter mapping.
> +		 * Exception is functions with a wide parameter,
> +		 * as single register won't be used to represent
> +		 * it, throwing off register to parameter mapping.
> +		 * Examples include large structs or 64-bit types
> +		 * on a 32-bit arch.
>  		 */
>  		ftype__for_each_parameter(&fn->proto, pos) {
>  			if (pos->optimized || !pos->has_loc)
> @@ -2967,11 +2954,11 @@ static int cu__resolve_func_ret_types_optimized(struct cu *cu)
>  		}
>  		if (has_unexpected_reg) {
>  			ftype__for_each_parameter(&fn->proto, pos) {
> -				has_struct_param = param__is_struct(cu, &pos->tag);
> -				if (has_struct_param)
> +				has_wide_param = param__is_wide(cu, &pos->tag);
> +				if (has_wide_param)
>  					break;
>  			}
> -			if (!has_struct_param)
> +			if (!has_wide_param)
>  				fn->proto.unexpected_reg = 1;
>  		}
>