Re: [PATCH dwarves v3 7/8] dwarf_loader: multithreading with a job/worker model

[Arnaldo Carvalho de Melo <acme@kernel.org>]

On Mon, Dec 30, 2024 at 01:07:36PM -0300, Arnaldo Carvalho de Melo wrote:
> On Sat, Dec 21, 2024 at 01:23:38AM +0000, Ihor Solodrai wrote:
> > Multithreading is now contained in dwarf_loader.c, and is implemented
> > using a jobs queue and a pool of worker threads. As a consequence,
> > multithreading-related code is removed from pahole.c.
> > 
> > A single-thread special case is removed: queueing setup works fine
> > with a single worker, which will switch between jobs as appropriate.
> > 
> > Code supporting previous version of the multithreading, such as
> > cu_state, thread_data and related functions, is also removed.
> > 
> > reproducible_build flag is now moot: the BTF encoding is always
> > reproducible with these changes.
> > 
> > The goal outlined in the RFC [1] - making parallel reproducible BTF
> > generation as fast as non-reproducible - is achieved by implementing
> > the requirement of ordered CU encoding (stealing) directly in the job
> > queue in dwarf_loader.c
> > 
> > The synchronization in the queue is implemented by a mutex (which
> > ensures consistency of queue state) and a job_added condition
> > variable. Motivation behind using condition variables is a classic
> > one: we want to avoid the threads checking the state of the queue in a
> > busy loop, competing for a single mutex.
> > 
> > In comparison to the previous version of this patch [2], job_taken
> > condition variable is removed. The number of decoded CUs in memory is
> > now limited by initial JOB_DECODE jobs. The enqueue/dequeue interface
> > is changed aiming to reduce locking. See relevant discussion [3].
> 
> So I'm running tests/tests after each of these patches and at this point
> I got:
> 
> root@number:/home/acme/git/pahole# tests/tests
>   1: Validation of BTF encoding of functions; this may take some time: grep: /tmp/btf_functions.sh.OgxoO4/dwarf.funcs: binary file matches
> ERROR: mismatch : BTF 'bool evtchn_fifo_is_pending(evtchn_port_t);' not found; DWARF ''
> Test data is in /tmp/btf_functions.sh.OgxoO4
>   2: Default BTF on a system without BTF: Ok
>   3: Flexible arrays accounting: Ok
>   4: Check that pfunct can print btf_decl_tags read from BTF: Ok
>   5: Pretty printing of files using DWARF type information: Ok
>   6: Parallel reproducible DWARF Loading/Serial BTF encoding: Ok
> /home/acme/git/pahole
> root@number:/home/acme/git/pahole#
> 
> root@number:/home/acme/git/pahole# grep evtchn_fifo_is_pending /tmp/btf_functions.sh.OgxoO4/dwarf.funcs 
> bool evtchn_fifo_is_pending(evtchn_port_t);
> root@number:/home/acme/git/pahole# grep evtchn_fifo_is_pending /tmp/btf_functions.sh.OgxoO4/btf.funcs 
> bool evtchn_fifo_is_pending(evtchn_port_t);
> root@number:/home/acme/git/pahole# 
> 
> Which seems like a test artifact:
> 
> root@number:/home/acme/git/pahole# pfunct -f evtchn_fifo_is_pending /tmp/btf_functions.sh.OgxoO4/vmlinux.btf 
> bool evtchn_fifo_is_pending(evtchn_port_t port);
> root@number:/home/acme/git/pahole# pfunct -F btf -f evtchn_fifo_is_pending /tmp/btf_functions.sh.OgxoO4/vmlinux.btf 
> bool evtchn_fifo_is_pending(evtchn_port_t port);
> root@number:/home/acme/git/pahole# 
> 
> root@number:/home/acme/git/pahole# pfunct -F btf -f evtchn_fifo_is_pending /lib/modules/6.13.0-rc2/build/vmlinux
> bool evtchn_fifo_is_pending(evtchn_port_t port);
> root@number:/home/acme/git/pahole# pfunct -F dwarf evtchn_fifo_is_pending /lib/modules/6.13.0-rc2/build/vmlinux
> bool evtchn_fifo_is_pending(evtchn_port_t port);
> bool evtchn_fifo_is_pending(evtchn_port_t port);
> root@number:/home/acme/git/pahole# 
> 
> Maybe because DWARF has two such functions? Alan?
> 
> But:
> 
> root@number:/home/acme/git/pahole# readelf -wi /lib/modules/6.13.0-rc2/build/vmlinux | grep -B2 -A12 evtchn_fifo_is_pending
>  <2><946502d>: Abbrev Number: 0
>  <1><946502e>: Abbrev Number: 128 (DW_TAG_subprogram)
>     <9465030>   DW_AT_name        : (indirect string, offset: 0x39779e): evtchn_fifo_is_pending
>     <9465034>   DW_AT_decl_file   : 1
>     <9465034>   DW_AT_decl_line   : 206
>     <9465035>   DW_AT_decl_column : 13
>     <9465036>   DW_AT_prototyped  : 1
>     <9465036>   DW_AT_type        : <0x9456abf>
>     <946503a>   DW_AT_low_pc      : 0xffffffff81b6d330
>     <9465042>   DW_AT_high_pc     : 0x65
>     <946504a>   DW_AT_frame_base  : 1 byte block: 9c 	(DW_OP_call_frame_cfa)
>     <946504c>   DW_AT_call_all_calls: 1
>     <946504c>   DW_AT_sibling     : <0x946517a>
>  <2><9465050>: Abbrev Number: 81 (DW_TAG_formal_parameter)
>     <9465051>   DW_AT_name        : (indirect string, offset: 0x3abde1): port
> root@number:/home/acme/git/pahole#
> 
> I.e. there is just one, but when pfunct prints it must be seeing some
> other usage that refers back to 946502e...
> 
> root@number:/home/acme/git/pahole# readelf -wi /lib/modules/6.13.0-rc2/build/vmlinux | grep -w -B2 -A12 0x946502e
>  <4><9464825>: Abbrev Number: 60 (DW_TAG_call_site)
>     <9464826>   DW_AT_call_return_pc: 0xffffffff81b6d4f5
>     <946482e>   DW_AT_call_origin : <0x946502e>
>     <9464832>   DW_AT_sibling     : <0x946483d>
>  <5><9464836>: Abbrev Number: 13 (DW_TAG_call_site_parameter)
>     <9464837>   DW_AT_location    : 1 byte block: 55 	(DW_OP_reg5 (rdi))
>     <9464839>   DW_AT_call_value  : 2 byte block: 7f 0 	(DW_OP_breg15 (r15): 0)
>  <5><946483c>: Abbrev Number: 0
>  <4><946483d>: Abbrev Number: 60 (DW_TAG_call_site)
>     <946483e>   DW_AT_call_return_pc: 0xffffffff81b6d590
>     <9464846>   DW_AT_call_origin : <0x9463caa>
>     <946484a>   DW_AT_sibling     : <0x946485d>
>  <5><946484e>: Abbrev Number: 13 (DW_TAG_call_site_parameter)
>     <946484f>   DW_AT_location    : 1 byte block: 55 	(DW_OP_reg5 (rdi))
>     <9464851>   DW_AT_call_value  : 2 byte block: 7f 0 	(DW_OP_breg15 (r15): 0)
> --
>     <9464f6b>   DW_AT_frame_base  : 1 byte block: 9c 	(DW_OP_call_frame_cfa)
>     <9464f6d>   DW_AT_call_all_calls: 1
>     <9464f6d>   DW_AT_sibling     : <0x946502e>
>  <2><9464f71>: Abbrev Number: 81 (DW_TAG_formal_parameter)
>     <9464f72>   DW_AT_name        : (indirect string, offset: 0x3abde1): port
>     <9464f76>   DW_AT_decl_file   : 1
>     <9464f76>   DW_AT_decl_line   : 212
>     <9464f77>   DW_AT_decl_column : 44
>     <9464f78>   DW_AT_type        : <0x9463263>
>     <9464f7c>   DW_AT_location    : 0x181ea81 (location list)
>     <9464f80>   DW_AT_GNU_locviews: 0x181ea79
>  <2><9464f84>: Abbrev Number: 64 (DW_TAG_variable)
>     <9464f85>   DW_AT_name        : (indirect string, offset: 0x9f2cd): word
>     <9464f89>   DW_AT_decl_file   : 1
>     <9464f89>   DW_AT_decl_line   : 214
> root@number:/home/acme/git/pahole#
> 
> Not really :-\
> 
> root@number:/home/acme/git/pahole# pfunct --decl_info -F dwarf evtchn_fifo_is_pending /lib/modules/6.13.0-rc2/build/vmlinux
> /* Used at: /home/acme/git/linux/drivers/xen/events/events_fifo.c */
> /* <946502e> /home/acme/git/linux/drivers/xen/events/events_fifo.c:206 */
> bool evtchn_fifo_is_pending(evtchn_port_t port);
> /* Used at: /home/acme/git/linux/drivers/xen/events/events_fifo.c */
> /* <946502e> /home/acme/git/linux/drivers/xen/events/events_fifo.c:206 */
> bool evtchn_fifo_is_pending(evtchn_port_t port);
> root@number:/home/acme/git/pahole#
> 
> So far I couldn't find an explanation for this oddity... Lets see if
> after applying all patches we get past this.

Its not related to this patch series, before we get two outputs for
these (and other functions in
/home/acme/git/linux/drivers/xen/events/events_fifo.c).

Still investigating.

- Arnaldo