Re: parallel pahole hangs while building modules from nvidia-open-kernel-dkms

["Ihor Solodrai" <ihor.solodrai@linux.dev>]

On 3/28/25 2:05 AM, Domenico Andreoli wrote:
> On Wed, Mar 26, 2025 at 08:48:51PM +0000, Ihor Solodrai wrote:
>> On 3/25/25 2:10 AM, Domenico Andreoli wrote:
>>> Hi,
>>>
>>>   This a forward of Debian bug report [0] where you can find more
>>> details. At [1] and [2] you can get the kernel and module to reproduce.
>>> I could reproduce on both amd64 and arm64 using pahole 1.29.
>>>
>>> This is marked as serious severity because it makes the autobuilder hang
>>> as well [3].
>>>
>>> Could you please help?
>>>
>>> Regards,
>>> Domenico
>>
>> Hi Domenico, thanks for the bug report.
>
> Hi Ihor,
>
>>
>> I debugged the hanging, and it appears that "abort" handling in case
>> of a BTF encoding error was overlooked in recent changes to speedup
>> parallel encoding.
>>
>> Could you please try the diff below, and check if it resolves the
>> hanging?
>>
>
> Yes, I tried it and the hanging is gone.
>
> Now both parallel and sequential invocations fail with this error:
>
>   dwarf_expr: unhandled 0x12 DW_OP_ operation
>   Unsupported DW_TAG_reference_type(0x10): type: 0x28172
>   Error while encoding BTF.
>   dwarf_expr: unhandled 0x12 DW_OP_ operation
>   dwarf_expr: unhandled 0x12 DW_OP_ operation
>   dwarf_expr: unhandled 0x12 DW_OP_ operation
>   libbpf: failed to find '.BTF' ELF section in nvidia-modeset.ko
>   pahole: nvidia-modeset.ko: Invalid argument
>
> I guess this is another story that was simply covered by the previous bug.

The "invalid argument" message and non-0 exit code are now appearing
because the encoding error is correctly propagated with the patch I
sent you.

The root cause of the problem you're seeing is unhandled DWARF
input. The resulting BTF is wrong in all cases: with or w/o the patch,
parallel or sequential.

The important part of the error message is:

    Unsupported DW_TAG_reference_type(0x10): type: 0x28172

This makes BTF encoder abort.

Could you please share the base vmlinux passed to the command that
produces this error? Maybe you could provide instructions on how to
build it?

>
>> diff --git a/dwarf_loader.c b/dwarf_loader.c
>> index 84122d0..e1ba7bc 100644
>> --- a/dwarf_loader.c
>> +++ b/dwarf_loader.c
>> @@ -3459,6 +3459,7 @@ static struct {
>>  	 */
>>  	uint32_t next_cu_id;
>>  	struct list_head jobs;
>> +	bool abort;
>>  } cus_processing_queue;
>>  
>>  enum job_type {
>> @@ -3479,6 +3480,7 @@ static void cus_queue__init(void)
>>  	pthread_cond_init(&cus_processing_queue.job_added, NULL);
>>  	INIT_LIST_HEAD(&cus_processing_queue.jobs);
>>  	cus_processing_queue.next_cu_id = 0;
>> +	cus_processing_queue.abort = false;
>>  }
>>  
>>  static void cus_queue__destroy(void)
>> @@ -3535,8 +3537,9 @@ static struct cu_processing_job *cus_queue__enqdeq_job(struct cu_processing_job
>>  		pthread_cond_signal(&cus_processing_queue.job_added);
>>  	}
>>  	for (;;) {
>> +		bool abort = __atomic_load_n(&cus_processing_queue.abort, __ATOMIC_SEQ_CST);
>>  		job = cus_queue__try_dequeue();
>> -		if (job)
>> +		if (job || abort)
>>  			break;
>>  		/* No jobs or only steals out of order */
>>  		pthread_cond_wait(&cus_processing_queue.job_added, &cus_processing_queue.mutex);
>> @@ -3653,6 +3656,9 @@ static void *dwarf_loader__worker_thread(void *arg)
>>  
>>  	while (!stop) {
>>  		job = cus_queue__enqdeq_job(job);
>> +		if (!job)
>> +			goto out_abort;
>> +
>>  		switch (job->type) {
>>  
>>  		case JOB_DECODE:
>> @@ -3688,6 +3694,8 @@ static void *dwarf_loader__worker_thread(void *arg)
>>  
>>  	return (void *)DWARF_CB_OK;
>>  out_abort:
>> +	__atomic_store_n(&cus_processing_queue.abort, true, __ATOMIC_SEQ_CST);
>> +	pthread_cond_signal(&cus_processing_queue.job_added);
>>  	return (void *)DWARF_CB_ABORT;
>>  }
>>  
>> @@ -4028,7 +4036,7 @@ static int cus__process_file(struct cus *cus, struct conf_load *conf, int fd,
>>  
>>  	/* Process the one or more modules gleaned from this file. */
>>  	int err = dwfl_getmodules(dwfl, cus__process_dwflmod, &parms, 0);
>> -	if (err < 0)
>> +	if (err)
>>  		return -1;
>>  
>>  	// We can't call dwfl_end(dwfl) here, as we keep pointers to strings
>>
>
> Is this patch already final or do you prefer I'd wait for review and marge first?

I'd prefer the patch to get reviewed and merged into pahole/next
first. I'll submit it separately.

But as I said, it fixes just the hanging part, not the root cause of
the error.

>
> I would apply it on top of Debian's 1.29 and release a new 1.29-3 package.
>
> Thank,
> dom
>