[REGRESSION] module BTF validation failure (Error -22) on next

[REGRESSION] module BTF validation failure (Error -22) on next

[Laura Nao]

Hello,

KernelCI has detected a module loading regression affecting all AMD and 
Intel Chromebooks in the Collabora LAVA lab, occurring between 
next-20241024 and next-20241025.

The logs indicate a failure in BTF module validation, preventing all 
modules from loading correctly (with CONFIG_MODULE_ALLOW_BTF_MISMATCH 
unset). The example below is from an AMD Chromebook (HP 14b na0052xx), 
with similar errors observed on other AMD and Intel devices:

[    5.284373] failed to validate module [cros_kbd_led_backlight] BTF: -22
[    5.291392] failed to validate module [i2c_hid] BTF: -22
[    5.293958] failed to validate module [chromeos_pstore] BTF: -22
[    5.302832] failed to validate module [coreboot_table] BTF: -22
[    5.309175] failed to validate module [raydium_i2c_ts] BTF: -22
[    5.309264] failed to validate module [i2c_cros_ec_tunnel] BTF: -22
[    5.322158] failed to validate module [typec] BTF: -22
[    5.327554] failed to validate module [snd_timer] BTF: -22
[    5.327573] failed to validate module [cros_usbpd_notify] BTF: -22
[    5.339272] failed to validate module [elan_i2c] BTF: -22
[    5.345821] failed to validate module [industrialio] BTF: -22
[    5.423113] failed to validate module [cfg80211] BTF: -22
[    5.443074] failed to validate module [cros_ec_dev] BTF: -22
[    5.448857] failed to validate module [snd_pci_acp3x] BTF: -22
[    5.454736] failed to validate module [cros_kbd_led_backlight] BTF: -22
[    5.461458] failed to validate module [regmap_i2c] BTF: -22
[    5.470228] failed to validate module [i2c_piix4] BTF: -22
[    5.491123] failed to validate module [i2c_hid] BTF: -22
[    5.491226] failed to validate module [chromeos_pstore] BTF: -22
[    5.496519] failed to validate module [coreboot_table] BTF: -22
[    5.502632] failed to validate module [snd_timer] BTF: -22
[    5.538916] failed to validate module [gsmi] BTF: -22
[    5.604971] failed to validate module [mii] BTF: -22
[    5.604971] failed to validate module [videobuf2_common] BTF: -22
[    5.604972] failed to validate module [sp5100_tco] BTF: -22
[    5.616068] failed to validate module [snd_soc_acpi] BTF: -22
[    5.680553] failed to validate module [bluetooth] BTF: -22
[    5.749320] failed to validate module [chromeos_pstore] BTF: -22
[    5.755440] failed to validate module [mii] BTF: -22
[    5.760522] failed to validate module [snd_timer] BTF: -22
[    5.783549] failed to validate module [bluetooth] BTF: -22
[    5.841561] failed to validate module [mii] BTF: -22
[    5.846699] failed to validate module [snd_timer] BTF: -22
[    5.892444] failed to validate module [mii] BTF: -22
[    5.897708] failed to validate module [snd_timer] BTF: -22
[    5.945507] failed to validate module [snd_timer] BTF: -22

The full kernel log is available on [1]. The config used is available on
[2] and the kernel/modules have been built using gcc-12.

The issue is still present on next-20241105.

I'm sending this report to track the regression while a fix is
identified. The culprit commit hasn't been pinpointed yet, I'll report
back once it's identified.

Any feedback or suggestion for additional debugging steps would be greatly 
appreciated.

Best,

Laura

[1] https://pastebin.com/raw/dtvzBkxh
[2] https://pastebin.com/raw/a1MGi3wH

#regzbot introduced: next-20241024..next-20241025

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Alan Maguire]

On 06/11/2024 16:08, Laura Nao wrote:
> Hello,
> 
> KernelCI has detected a module loading regression affecting all AMD and 
> Intel Chromebooks in the Collabora LAVA lab, occurring between 
> next-20241024 and next-20241025.
> 
> The logs indicate a failure in BTF module validation, preventing all 
> modules from loading correctly (with CONFIG_MODULE_ALLOW_BTF_MISMATCH 
> unset). The example below is from an AMD Chromebook (HP 14b na0052xx), 
> with similar errors observed on other AMD and Intel devices:
> 
> [    5.284373] failed to validate module [cros_kbd_led_backlight] BTF: -22
> [    5.291392] failed to validate module [i2c_hid] BTF: -22
> [    5.293958] failed to validate module [chromeos_pstore] BTF: -22
> [    5.302832] failed to validate module [coreboot_table] BTF: -22
> [    5.309175] failed to validate module [raydium_i2c_ts] BTF: -22
> [    5.309264] failed to validate module [i2c_cros_ec_tunnel] BTF: -22
> [    5.322158] failed to validate module [typec] BTF: -22
> [    5.327554] failed to validate module [snd_timer] BTF: -22
> [    5.327573] failed to validate module [cros_usbpd_notify] BTF: -22
> [    5.339272] failed to validate module [elan_i2c] BTF: -22
> [    5.345821] failed to validate module [industrialio] BTF: -22
> [    5.423113] failed to validate module [cfg80211] BTF: -22
> [    5.443074] failed to validate module [cros_ec_dev] BTF: -22
> [    5.448857] failed to validate module [snd_pci_acp3x] BTF: -22
> [    5.454736] failed to validate module [cros_kbd_led_backlight] BTF: -22
> [    5.461458] failed to validate module [regmap_i2c] BTF: -22
> [    5.470228] failed to validate module [i2c_piix4] BTF: -22
> [    5.491123] failed to validate module [i2c_hid] BTF: -22
> [    5.491226] failed to validate module [chromeos_pstore] BTF: -22
> [    5.496519] failed to validate module [coreboot_table] BTF: -22
> [    5.502632] failed to validate module [snd_timer] BTF: -22
> [    5.538916] failed to validate module [gsmi] BTF: -22
> [    5.604971] failed to validate module [mii] BTF: -22
> [    5.604971] failed to validate module [videobuf2_common] BTF: -22
> [    5.604972] failed to validate module [sp5100_tco] BTF: -22
> [    5.616068] failed to validate module [snd_soc_acpi] BTF: -22
> [    5.680553] failed to validate module [bluetooth] BTF: -22
> [    5.749320] failed to validate module [chromeos_pstore] BTF: -22
> [    5.755440] failed to validate module [mii] BTF: -22
> [    5.760522] failed to validate module [snd_timer] BTF: -22
> [    5.783549] failed to validate module [bluetooth] BTF: -22
> [    5.841561] failed to validate module [mii] BTF: -22
> [    5.846699] failed to validate module [snd_timer] BTF: -22
> [    5.892444] failed to validate module [mii] BTF: -22
> [    5.897708] failed to validate module [snd_timer] BTF: -22
> [    5.945507] failed to validate module [snd_timer] BTF: -22
> 
> The full kernel log is available on [1]. The config used is available on
> [2] and the kernel/modules have been built using gcc-12.
> 
> The issue is still present on next-20241105.
> 
> I'm sending this report to track the regression while a fix is
> identified. The culprit commit hasn't been pinpointed yet, I'll report
> back once it's identified.
> 
> Any feedback or suggestion for additional debugging steps would be greatly 
> appreciated.
> 
> Best,
>

Thanks for the report! Judging from the config, you're seeing this with
pahole v1.24. I have seen issues like this in the past where during a
kernel build, module BTF has been built against vmlinux BTF, and then
something later re-triggers vmlinux BTF generation. If that re-triggered
vmlinux BTF does not use the same type ids for types, this can result in
mismatch errors as above since modules are referring to out-of-date type
ids in vmlinux. That's just a preliminary guess though, we'll
need more info to help get to the bottom of this.

A few suggestions to help debug this:

- if you have build logs, check BTF generation of vmlinux. Did it in
fact happen twice perhaps? Even better if, if kernel CI saves logs, feel
free to send a pointer and I'll take a look.
- can you post the vmlinux (stripped of DWARF data if possible to limit
size) and one of the failing modules somewhere so we can analyze?
- Failing that,
bpftool btf dump file /path/2/vmlinux_from_build > vmlinux.raw
and upload of the vmlinux.raw and one of the failing module .kos would help.

I've tried to reproduce this; no luck so far at my end.

Alan

> Laura
> 
> [1] https://pastebin.com/raw/dtvzBkxh
> [2] https://pastebin.com/raw/a1MGi3wH
> 
> #regzbot introduced: next-20241024..next-20241025
> 
> 

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Laura Nao]

Hi Alan,

On 11/7/24 16:05, Alan Maguire wrote:
> Thanks for the report! Judging from the config, you're seeing this with
> pahole v1.24. I have seen issues like this in the past where during a
> kernel build, module BTF has been built against vmlinux BTF, and then
> something later re-triggers vmlinux BTF generation. If that re-triggered
> vmlinux BTF does not use the same type ids for types, this can result in
> mismatch errors as above since modules are referring to out-of-date type
> ids in vmlinux. That's just a preliminary guess though, we'll
> need more info to help get to the bottom of this.
> 
> A few suggestions to help debug this:
> 
> - if you have build logs, check BTF generation of vmlinux. Did it in
> fact happen twice perhaps? Even better if, if kernel CI saves logs, feel
> free to send a pointer and I'll take a look.

Thanks for the pointers!

From what I can tell in the logs, the BTF generation of vmlinux only 
occurred once. The automated build process in KernelCI generally involves 
building the kernel first, followed by the modules and other artifacts 
(such as the kselftest archive). 
The full build log can be downloaded by selecting 'build_log' from 
the dropdown menu at the top of this page:

https://kernelci-api.westus3.cloudapp.azure.com/viewer?node_id=6732f41d58937056c61734ab

I do see some warnings reported in the logs though:

WARN: resolve_btfids: unresolved symbol bpf_lsm_task_getsecid_obj
WARN: resolve_btfids: unresolved symbol bpf_lsm_current_getsecid_subj

> - can you post the vmlinux (stripped of DWARF data if possible to limit
> size) and one of the failing modules somewhere so we can analyze?
> - Failing that,
> bpftool btf dump file /path/2/vmlinux_from_build > vmlinux.raw
> and upload of the vmlinux.raw and one of the failing module .kos would help.
> 

Currently, KernelCI only retains the bzImage, not the vmlinux binary. The 
bzImage can be downloaded from the same link mentioned above by selecting 
'kernel' from the dropdown menu (modules can also be downloaded the same
way). I’ll try to replicate the build on my end and share the vmlinux 
with DWARF data stripped for convenience.

Thanks,

Laura

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Laura Nao]

On 11/13/24 10:37, Laura Nao wrote:
> 
> Currently, KernelCI only retains the bzImage, not the vmlinux binary. The
> bzImage can be downloaded from the same link mentioned above by selecting
> 'kernel' from the dropdown menu (modules can also be downloaded the same
> way). I’ll try to replicate the build on my end and share the vmlinux
> with DWARF data stripped for convenience.
> 

I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
(stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the 
modules[3] and its btf data[4] extracted with:

bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw

Looking again at the logs[5], I've noticed the following is reported:

[    0.415885] BPF: 	 type_id=115803 offset=177920 size=1152
[    0.416029] BPF:  
[    0.416083] BPF: Invalid offset
[    0.416165] BPF: 

There are two different definitions of rcu_data in '.data..percpu', one
is a struct and the other is an integer:

type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')

[115801] VAR 'rcu_data' type_id=115572, linkage=static
[115803] VAR 'rcu_data' type_id=1, linkage=static

[115572] STRUCT 'rcu_data' size=1152 vlen=69
[1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)

I assume that's not expected, correct?

I'll dig a bit deeper and report back if I can find anything else.

[1] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/vmlinux
[2] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/vmlinux.raw
[3] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/cros_kbd_led_backlight.ko
[4] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/cros_kbd_led_backlight.ko.raw
[5] https://pastebin.com/raw/FvvrPhAY

Best,

Laura

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Laura Nao]

On 11/15/24 18:17, Laura Nao wrote:
> I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the
> modules[3] and its btf data[4] extracted with:
> 
> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
> 
> Looking again at the logs[5], I've noticed the following is reported:
> 
> [    0.415885] BPF: 	 type_id=115803 offset=177920 size=1152
> [    0.416029] BPF:
> [    0.416083] BPF: Invalid offset
> [    0.416165] BPF:
> 
> There are two different definitions of rcu_data in '.data..percpu', one
> is a struct and the other is an integer:
> 
> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
> 
> [115801] VAR 'rcu_data' type_id=115572, linkage=static
> [115803] VAR 'rcu_data' type_id=1, linkage=static
> 
> [115572] STRUCT 'rcu_data' size=1152 vlen=69
> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> 
> I assume that's not expected, correct?
> 
> I'll dig a bit deeper and report back if I can find anything else.

I ran a bisection, and it appears the culprit commit is:
https://lore.kernel.org/all/20241021080856.48746-2-ubizjak@gmail.com/

Hi Uros, do you have any suggestions or insights on resolving this issue?

This problem is now impacting mainline as well. The full context can be 
found at the beginning of this thread[1].

Thanks,

Laura

[1] https://lore.kernel.org/all/20241106160820.259829-1-laura.nao@collabora.com/

#regzbot introduced: 001217defd

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Uros Bizjak]

On Wed, Dec 4, 2024 at 4:52 PM Laura Nao <laura.nao@collabora.com> wrote:
>
> On 11/15/24 18:17, Laura Nao wrote:
> > I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
> > (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the
> > modules[3] and its btf data[4] extracted with:
> >
> > bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
> >
> > Looking again at the logs[5], I've noticed the following is reported:
> >
> > [    0.415885] BPF:    type_id=115803 offset=177920 size=1152
> > [    0.416029] BPF:
> > [    0.416083] BPF: Invalid offset
> > [    0.416165] BPF:
> >
> > There are two different definitions of rcu_data in '.data..percpu', one
> > is a struct and the other is an integer:
> >
> > type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
> > type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
> >
> > [115801] VAR 'rcu_data' type_id=115572, linkage=static
> > [115803] VAR 'rcu_data' type_id=1, linkage=static
> >
> > [115572] STRUCT 'rcu_data' size=1152 vlen=69
> > [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> >
> > I assume that's not expected, correct?
> >
> > I'll dig a bit deeper and report back if I can find anything else.
>
> I ran a bisection, and it appears the culprit commit is:
> https://lore.kernel.org/all/20241021080856.48746-2-ubizjak@gmail.com/
>
> Hi Uros, do you have any suggestions or insights on resolving this issue?

There is a stray ";" at the end of the #define, perhaps this makes a difference:

+#define PERCPU_PTR(__p) \
+ (typeof(*(__p)) __force __kernel *)(__p);
+

and SHIFT_PERCPU_PTR macro now expands to:

RELOC_HIDE((typeof(*(p)) __force __kernel *)(p);, (offset))

A follow-up patch in the series changes PERCPU_PTR macro to:

#define PERCPU_PTR(__p) \
({ \
unsigned long __pcpu_ptr = (__force unsigned long)(__p); \
(typeof(*(__p)) __force __kernel *)(__pcpu_ptr); \
})

so this should again correctly cast the value.

Uros.

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Cong Wang]

On Thu, Dec 05, 2024 at 08:36:33AM +0100, Uros Bizjak wrote:
> On Wed, Dec 4, 2024 at 4:52 PM Laura Nao <laura.nao@collabora.com> wrote:
> >
> > On 11/15/24 18:17, Laura Nao wrote:
> > > I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
> > > (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the
> > > modules[3] and its btf data[4] extracted with:
> > >
> > > bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
> > >
> > > Looking again at the logs[5], I've noticed the following is reported:
> > >
> > > [    0.415885] BPF:    type_id=115803 offset=177920 size=1152
> > > [    0.416029] BPF:
> > > [    0.416083] BPF: Invalid offset
> > > [    0.416165] BPF:
> > >
> > > There are two different definitions of rcu_data in '.data..percpu', one
> > > is a struct and the other is an integer:
> > >
> > > type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
> > > type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
> > >
> > > [115801] VAR 'rcu_data' type_id=115572, linkage=static
> > > [115803] VAR 'rcu_data' type_id=1, linkage=static
> > >
> > > [115572] STRUCT 'rcu_data' size=1152 vlen=69
> > > [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> > >
> > > I assume that's not expected, correct?
> > >
> > > I'll dig a bit deeper and report back if I can find anything else.
> >
> > I ran a bisection, and it appears the culprit commit is:
> > https://lore.kernel.org/all/20241021080856.48746-2-ubizjak@gmail.com/
> >
> > Hi Uros, do you have any suggestions or insights on resolving this issue?
> 
> There is a stray ";" at the end of the #define, perhaps this makes a difference:
> 
> +#define PERCPU_PTR(__p) \
> + (typeof(*(__p)) __force __kernel *)(__p);
> +
> 
> and SHIFT_PERCPU_PTR macro now expands to:
> 
> RELOC_HIDE((typeof(*(p)) __force __kernel *)(p);, (offset))
> 
> A follow-up patch in the series changes PERCPU_PTR macro to:
> 
> #define PERCPU_PTR(__p) \
> ({ \
> unsigned long __pcpu_ptr = (__force unsigned long)(__p); \
> (typeof(*(__p)) __force __kernel *)(__pcpu_ptr); \
> })
> 
> so this should again correctly cast the value.

Hm, I saw a similar bug but with pahole 1.28. My kernel complains about
BTF invalid offset:

[    7.785788] BPF: 	 type_id=2394 offset=0 size=1
[    7.786411] BPF:
[    7.786703] BPF: Invalid offset
[    7.787119] BPF:

Dumping the vmlinux (there is no module invovled), I saw it is related to
percpu pointer too:

[2394] VAR '__pcpu_unique_cpu_hw_events' type_id=2, linkage=global
...
[163643] DATASEC '.data..percpu' size=2123280 vlen=808
        type_id=2393 offset=0 size=1 (VAR '__pcpu_scope_cpu_hw_events')
        type_id=2394 offset=0 size=1 (VAR '__pcpu_unique_cpu_hw_events')
...

I compiled and installed the latest pahole from its git repo:

$ pahole --version
v1.28

Thanks.

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Alan Maguire]

On 14/12/2024 04:41, Cong Wang wrote:
> On Thu, Dec 05, 2024 at 08:36:33AM +0100, Uros Bizjak wrote:
>> On Wed, Dec 4, 2024 at 4:52 PM Laura Nao <laura.nao@collabora.com> wrote:
>>>
>>> On 11/15/24 18:17, Laura Nao wrote:
>>>> I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
>>>> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the
>>>> modules[3] and its btf data[4] extracted with:
>>>>
>>>> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
>>>>
>>>> Looking again at the logs[5], I've noticed the following is reported:
>>>>
>>>> [    0.415885] BPF:    type_id=115803 offset=177920 size=1152
>>>> [    0.416029] BPF:
>>>> [    0.416083] BPF: Invalid offset
>>>> [    0.416165] BPF:
>>>>
>>>> There are two different definitions of rcu_data in '.data..percpu', one
>>>> is a struct and the other is an integer:
>>>>
>>>> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
>>>> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
>>>>
>>>> [115801] VAR 'rcu_data' type_id=115572, linkage=static
>>>> [115803] VAR 'rcu_data' type_id=1, linkage=static
>>>>
>>>> [115572] STRUCT 'rcu_data' size=1152 vlen=69
>>>> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
>>>>
>>>> I assume that's not expected, correct?
>>>>
>>>> I'll dig a bit deeper and report back if I can find anything else.
>>>
>>> I ran a bisection, and it appears the culprit commit is:
>>> https://lore.kernel.org/all/20241021080856.48746-2-ubizjak@gmail.com/
>>>
>>> Hi Uros, do you have any suggestions or insights on resolving this issue?
>>
>> There is a stray ";" at the end of the #define, perhaps this makes a difference:
>>
>> +#define PERCPU_PTR(__p) \
>> + (typeof(*(__p)) __force __kernel *)(__p);
>> +
>>
>> and SHIFT_PERCPU_PTR macro now expands to:
>>
>> RELOC_HIDE((typeof(*(p)) __force __kernel *)(p);, (offset))
>>
>> A follow-up patch in the series changes PERCPU_PTR macro to:
>>
>> #define PERCPU_PTR(__p) \
>> ({ \
>> unsigned long __pcpu_ptr = (__force unsigned long)(__p); \
>> (typeof(*(__p)) __force __kernel *)(__pcpu_ptr); \
>> })
>>
>> so this should again correctly cast the value.
> 
> Hm, I saw a similar bug but with pahole 1.28. My kernel complains about
> BTF invalid offset:
> 
> [    7.785788] BPF: 	 type_id=2394 offset=0 size=1
> [    7.786411] BPF:
> [    7.786703] BPF: Invalid offset
> [    7.787119] BPF:
> 
> Dumping the vmlinux (there is no module invovled), I saw it is related to
> percpu pointer too:
> 
> [2394] VAR '__pcpu_unique_cpu_hw_events' type_id=2, linkage=global
> ...
> [163643] DATASEC '.data..percpu' size=2123280 vlen=808
>         type_id=2393 offset=0 size=1 (VAR '__pcpu_scope_cpu_hw_events')
>         type_id=2394 offset=0 size=1 (VAR '__pcpu_unique_cpu_hw_events')
> ...
> 
> I compiled and installed the latest pahole from its git repo:
> 
> $ pahole --version
> v1.28
> 
> Thanks.

Thanks for the report! Looking at percpu-defs.h it looks like the
existence of such variables requires either

#if defined(ARCH_NEEDS_WEAK_PER_CPU) ||
defined(CONFIG_DEBUG_FORCE_WEAK_PER_CPU)

...

#define DEFINE_PER_CPU_SECTION(type, name, sec)                         \
        __PCPU_DUMMY_ATTRS char __pcpu_scope_##name;                    \
        extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;            \
        __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;                   \
        extern __PCPU_ATTRS(sec) __typeof__(type) name;                 \
        __PCPU_ATTRS(sec) __weak __typeof__(type) name


I'm guessing your .config has CONFIG_DEBUG_FORCE_WEAK_PER_CPU, or are
you building on s390/alpha?

I've reproduced this on bpf-next with CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y,
pahole v1.28 and gcc-12; I see ~900 __pcpu_ variables and get the same
BTF errors since multipe __pcpu_ vars share the offset 0.

A simple workaround in dwarves - and I verified this resolved the issue
for me - would be

diff --git a/btf_encoder.c b/btf_encoder.c
index 3754884..4a1799a 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -2174,7 +2174,8 @@ static bool filter_variable_name(const char *name)
                X("__UNIQUE_ID"),
                X("__tpstrtab_"),
                X("__exitcall_"),
-               X("__func_stack_frame_non_standard_")
+               X("__func_stack_frame_non_standard_"),
+               X("__pcpu_")
                #undef X
        };
        int i;

...but I'd like us to understand further why variables which were
supposed to be in a .discard section end up being encoded as there may
be other problems lurking here aside from this one. More soon hopefully...

Alan

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Alan Maguire]

On 14/12/2024 12:15, Alan Maguire wrote:
> On 14/12/2024 04:41, Cong Wang wrote:
>> On Thu, Dec 05, 2024 at 08:36:33AM +0100, Uros Bizjak wrote:
>>> On Wed, Dec 4, 2024 at 4:52 PM Laura Nao <laura.nao@collabora.com> wrote:
>>>>
>>>> On 11/15/24 18:17, Laura Nao wrote:
>>>>> I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
>>>>> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the
>>>>> modules[3] and its btf data[4] extracted with:
>>>>>
>>>>> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
>>>>>
>>>>> Looking again at the logs[5], I've noticed the following is reported:
>>>>>
>>>>> [    0.415885] BPF:    type_id=115803 offset=177920 size=1152
>>>>> [    0.416029] BPF:
>>>>> [    0.416083] BPF: Invalid offset
>>>>> [    0.416165] BPF:
>>>>>
>>>>> There are two different definitions of rcu_data in '.data..percpu', one
>>>>> is a struct and the other is an integer:
>>>>>
>>>>> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
>>>>> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
>>>>>
>>>>> [115801] VAR 'rcu_data' type_id=115572, linkage=static
>>>>> [115803] VAR 'rcu_data' type_id=1, linkage=static
>>>>>
>>>>> [115572] STRUCT 'rcu_data' size=1152 vlen=69
>>>>> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
>>>>>
>>>>> I assume that's not expected, correct?
>>>>>
>>>>> I'll dig a bit deeper and report back if I can find anything else.
>>>>
>>>> I ran a bisection, and it appears the culprit commit is:
>>>> https://lore.kernel.org/all/20241021080856.48746-2-ubizjak@gmail.com/
>>>>
>>>> Hi Uros, do you have any suggestions or insights on resolving this issue?
>>>
>>> There is a stray ";" at the end of the #define, perhaps this makes a difference:
>>>
>>> +#define PERCPU_PTR(__p) \
>>> + (typeof(*(__p)) __force __kernel *)(__p);
>>> +
>>>
>>> and SHIFT_PERCPU_PTR macro now expands to:
>>>
>>> RELOC_HIDE((typeof(*(p)) __force __kernel *)(p);, (offset))
>>>
>>> A follow-up patch in the series changes PERCPU_PTR macro to:
>>>
>>> #define PERCPU_PTR(__p) \
>>> ({ \
>>> unsigned long __pcpu_ptr = (__force unsigned long)(__p); \
>>> (typeof(*(__p)) __force __kernel *)(__pcpu_ptr); \
>>> })
>>>
>>> so this should again correctly cast the value.
>>
>> Hm, I saw a similar bug but with pahole 1.28. My kernel complains about
>> BTF invalid offset:
>>
>> [    7.785788] BPF: 	 type_id=2394 offset=0 size=1
>> [    7.786411] BPF:
>> [    7.786703] BPF: Invalid offset
>> [    7.787119] BPF:
>>
>> Dumping the vmlinux (there is no module invovled), I saw it is related to
>> percpu pointer too:
>>
>> [2394] VAR '__pcpu_unique_cpu_hw_events' type_id=2, linkage=global
>> ...
>> [163643] DATASEC '.data..percpu' size=2123280 vlen=808
>>         type_id=2393 offset=0 size=1 (VAR '__pcpu_scope_cpu_hw_events')
>>         type_id=2394 offset=0 size=1 (VAR '__pcpu_unique_cpu_hw_events')
>> ...
>>
>> I compiled and installed the latest pahole from its git repo:
>>
>> $ pahole --version
>> v1.28
>>
>> Thanks.
> 
> Thanks for the report! Looking at percpu-defs.h it looks like the
> existence of such variables requires either
> 
> #if defined(ARCH_NEEDS_WEAK_PER_CPU) ||
> defined(CONFIG_DEBUG_FORCE_WEAK_PER_CPU)
> 
> ...
> 
> #define DEFINE_PER_CPU_SECTION(type, name, sec)                         \
>         __PCPU_DUMMY_ATTRS char __pcpu_scope_##name;                    \
>         extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;            \
>         __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;                   \
>         extern __PCPU_ATTRS(sec) __typeof__(type) name;                 \
>         __PCPU_ATTRS(sec) __weak __typeof__(type) name
> 
> 
> I'm guessing your .config has CONFIG_DEBUG_FORCE_WEAK_PER_CPU, or are
> you building on s390/alpha?
> 
> I've reproduced this on bpf-next with CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y,
> pahole v1.28 and gcc-12; I see ~900 __pcpu_ variables and get the same
> BTF errors since multipe __pcpu_ vars share the offset 0.
> 
> A simple workaround in dwarves - and I verified this resolved the issue
> for me - would be
> 
> diff --git a/btf_encoder.c b/btf_encoder.c
> index 3754884..4a1799a 100644
> --- a/btf_encoder.c
> +++ b/btf_encoder.c
> @@ -2174,7 +2174,8 @@ static bool filter_variable_name(const char *name)
>                 X("__UNIQUE_ID"),
>                 X("__tpstrtab_"),
>                 X("__exitcall_"),
> -               X("__func_stack_frame_non_standard_")
> +               X("__func_stack_frame_non_standard_"),
> +               X("__pcpu_")
>                 #undef X
>         };
>         int i;
> 
> ...but I'd like us to understand further why variables which were
> supposed to be in a .discard section end up being encoded as there may
> be other problems lurking here aside from this one. More soon hopefully...
>


A bit more context here - variable encoding takes the address of the
variable from DWARF to locate the associated ELF section. Because we
insist on having a variable specification - with a location - this
usually works fine. However the problem is that because these dummy
__pcpu_ variables specify a .discard section, their addresses are 0, so
we get for example:

 <1><1e535>: Abbrev Number: 114 (DW_TAG_variable)
    <1e536>   DW_AT_name        : (indirect string, offset: 0x5e97):
__pcpu_unique_kstack_offset
    <1e53a>   DW_AT_decl_file   : 1
    <1e53b>   DW_AT_decl_line   : 823
    <1e53d>   DW_AT_decl_column : 1
    <1e53e>   DW_AT_type        : <0x57>
    <1e542>   DW_AT_external    : 1
    <1e542>   DW_AT_declaration : 1
 <1><1e542>: Abbrev Number: 156 (DW_TAG_variable)
    <1e544>   DW_AT_specification: <0x1e535>
    <1e548>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0
(DW_OP_addr: 0)


You can see the same thing for a simple program like this:

#include <stdio.h>

#define SEC(name) __attribute__((section(name)))

SEC("/DISCARD/") int d1;
extern int d1;
SEC("/DISCARD/") int d2;
extern int d2;

int main(int argc, char *argv[])
{
	return 0;
}


If you compile it with -g, the DWARF shows that d1 and d2 both have
address 0:

 <1><72>: Abbrev Number: 5 (DW_TAG_variable)
    <73>   DW_AT_name        : d1
    <76>   DW_AT_decl_file   : 1
    <77>   DW_AT_decl_line   : 5
    <78>   DW_AT_decl_column : 22
    <79>   DW_AT_type        : <0x57>
    <7d>   DW_AT_external    : 1
    <7d>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0
(DW_OP_addr: 0)
 <1><87>: Abbrev Number: 5 (DW_TAG_variable)
    <88>   DW_AT_name        : d2
    <8b>   DW_AT_decl_file   : 1
    <8c>   DW_AT_decl_line   : 7
    <8d>   DW_AT_decl_column : 22
    <8e>   DW_AT_type        : <0x57>
    <92>   DW_AT_external    : 1
    <92>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0
(DW_OP_addr: 0)


So the reason this happens for dwarves v1.28 in particular is - as I
understand it - we moved away from recording ELF section information for
each variable and matching that with DWARF info, instead relying on the
address to locate the associated ELF section. In cases like the above
the address information unfortunately leads us astray.

Seems like there's a few approaches we can take in fixing this:

1. designate "__pcpu_" prefix as a variable prefix to filter out. This
resolves the immediate problem but is too narrowly focused IMO and we
may end up playing whack-a-mole with other dummy variable prefixes.
2. resurrect ELF section variable information fully; i.e. record a list
of variables per ELF section (or at least per ELF section we care
about). If variable is not on the list for the ELF section, do not
encode it.
3. midway between the two; for the 0 address case specifically, verify
that the variable name really _is_ in the associated ELF section. No
need to create a local ELF table variable representation, we could just
walk the table in the case of the 0 addresses.

Diff for approach 3 is as follows

diff --git a/btf_encoder.c b/btf_encoder.c
index 3754884..21a0ab6 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -2189,6 +2189,26 @@ static bool filter_variable_name(const char *name)
        return false;
 }

+bool variable_in_sec(struct btf_encoder *encoder, const char *name,
size_t shndx)
+{
+       uint32_t sym_sec_idx;
+       uint32_t core_id;
+       GElf_Sym sym;
+
+       elf_symtab__for_each_symbol_index(encoder->symtab, core_id, sym,
sym_sec_idx) {
+               const char *sym_name;
+
+               if (sym_sec_idx != shndx || elf_sym__type(&sym) !=
STT_OBJECT)
+                       continue;
+               sym_name = elf_sym__name(&sym, encoder->symtab);
+               if (!sym_name)
+                       continue;
+               if (strcmp(name, sym_name) == 0)
+                       return true;
+       }
+       return false;
+}
+
 static int btf_encoder__encode_cu_variables(struct btf_encoder *encoder)
 {
        struct cu *cu = encoder->cu;
@@ -2258,6 +2278,11 @@ static int
btf_encoder__encode_cu_variables(struct btf_encoder *encoder)
                if (filter_variable_name(name))
                        continue;

+               /* A 0 address may be in a .discard section; ensure the
+                * variable really is in this section by checking ELF
symtab.
+                */
+               if (addr == 0 && !variable_in_sec(encoder, name, shndx))
+                       continue;
                /* Check for invalid BTF names */
                if (!btf_name_valid(name)) {
                        dump_invalid_symbol("Found invalid variable name
when encoding btf",


...so slightly more complex than option 1, but a bit more general in its
applicability to .discard section variables.

For the pahole folks, what do we think? Which option (or indeed other
ones I haven't thought of) makes sense for a fix for this? Thanks!

Alan

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Stephen Brennan]

Alan Maguire <alan.maguire@oracle.com> writes:
> On 14/12/2024 12:15, Alan Maguire wrote:
>> On 14/12/2024 04:41, Cong Wang wrote:
>>> On Thu, Dec 05, 2024 at 08:36:33AM +0100, Uros Bizjak wrote:
>>>> On Wed, Dec 4, 2024 at 4:52 PM Laura Nao <laura.nao@collabora.com> wrote:
>>>>>
>>>>> On 11/15/24 18:17, Laura Nao wrote:
>>>>>> I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
>>>>>> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the
>>>>>> modules[3] and its btf data[4] extracted with:
>>>>>>
>>>>>> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
>>>>>>
>>>>>> Looking again at the logs[5], I've noticed the following is reported:
>>>>>>
>>>>>> [    0.415885] BPF:    type_id=115803 offset=177920 size=1152
>>>>>> [    0.416029] BPF:
>>>>>> [    0.416083] BPF: Invalid offset
>>>>>> [    0.416165] BPF:
>>>>>>
>>>>>> There are two different definitions of rcu_data in '.data..percpu', one
>>>>>> is a struct and the other is an integer:
>>>>>>
>>>>>> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
>>>>>> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
>>>>>>
>>>>>> [115801] VAR 'rcu_data' type_id=115572, linkage=static
>>>>>> [115803] VAR 'rcu_data' type_id=1, linkage=static
>>>>>>
>>>>>> [115572] STRUCT 'rcu_data' size=1152 vlen=69
>>>>>> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
>>>>>>
>>>>>> I assume that's not expected, correct?
>>>>>>
>>>>>> I'll dig a bit deeper and report back if I can find anything else.
>>>>>
>>>>> I ran a bisection, and it appears the culprit commit is:
>>>>> https://lore.kernel.org/all/20241021080856.48746-2-ubizjak@gmail.com/
>>>>>
>>>>> Hi Uros, do you have any suggestions or insights on resolving this issue?
>>>>
>>>> There is a stray ";" at the end of the #define, perhaps this makes a difference:
>>>>
>>>> +#define PERCPU_PTR(__p) \
>>>> + (typeof(*(__p)) __force __kernel *)(__p);
>>>> +
>>>>
>>>> and SHIFT_PERCPU_PTR macro now expands to:
>>>>
>>>> RELOC_HIDE((typeof(*(p)) __force __kernel *)(p);, (offset))
>>>>
>>>> A follow-up patch in the series changes PERCPU_PTR macro to:
>>>>
>>>> #define PERCPU_PTR(__p) \
>>>> ({ \
>>>> unsigned long __pcpu_ptr = (__force unsigned long)(__p); \
>>>> (typeof(*(__p)) __force __kernel *)(__pcpu_ptr); \
>>>> })
>>>>
>>>> so this should again correctly cast the value.
>>>
>>> Hm, I saw a similar bug but with pahole 1.28. My kernel complains about
>>> BTF invalid offset:
>>>
>>> [    7.785788] BPF: 	 type_id=2394 offset=0 size=1
>>> [    7.786411] BPF:
>>> [    7.786703] BPF: Invalid offset
>>> [    7.787119] BPF:
>>>
>>> Dumping the vmlinux (there is no module invovled), I saw it is related to
>>> percpu pointer too:
>>>
>>> [2394] VAR '__pcpu_unique_cpu_hw_events' type_id=2, linkage=global
>>> ...
>>> [163643] DATASEC '.data..percpu' size=2123280 vlen=808
>>>         type_id=2393 offset=0 size=1 (VAR '__pcpu_scope_cpu_hw_events')
>>>         type_id=2394 offset=0 size=1 (VAR '__pcpu_unique_cpu_hw_events')
>>> ...
>>>
>>> I compiled and installed the latest pahole from its git repo:
>>>
>>> $ pahole --version
>>> v1.28
>>>
>>> Thanks.
>> 
>> Thanks for the report! Looking at percpu-defs.h it looks like the
>> existence of such variables requires either
>> 
>> #if defined(ARCH_NEEDS_WEAK_PER_CPU) ||
>> defined(CONFIG_DEBUG_FORCE_WEAK_PER_CPU)
>> 
>> ...
>> 
>> #define DEFINE_PER_CPU_SECTION(type, name, sec)                         \
>>         __PCPU_DUMMY_ATTRS char __pcpu_scope_##name;                    \
>>         extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;            \
>>         __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;                   \
>>         extern __PCPU_ATTRS(sec) __typeof__(type) name;                 \
>>         __PCPU_ATTRS(sec) __weak __typeof__(type) name
>> 
>> 
>> I'm guessing your .config has CONFIG_DEBUG_FORCE_WEAK_PER_CPU, or are
>> you building on s390/alpha?
>> 
>> I've reproduced this on bpf-next with CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y,
>> pahole v1.28 and gcc-12; I see ~900 __pcpu_ variables and get the same
>> BTF errors since multipe __pcpu_ vars share the offset 0.
>> 
>> A simple workaround in dwarves - and I verified this resolved the issue
>> for me - would be
>> 
>> diff --git a/btf_encoder.c b/btf_encoder.c
>> index 3754884..4a1799a 100644
>> --- a/btf_encoder.c
>> +++ b/btf_encoder.c
>> @@ -2174,7 +2174,8 @@ static bool filter_variable_name(const char *name)
>>                 X("__UNIQUE_ID"),
>>                 X("__tpstrtab_"),
>>                 X("__exitcall_"),
>> -               X("__func_stack_frame_non_standard_")
>> +               X("__func_stack_frame_non_standard_"),
>> +               X("__pcpu_")
>>                 #undef X
>>         };
>>         int i;
>> 
>> ...but I'd like us to understand further why variables which were
>> supposed to be in a .discard section end up being encoded as there may
>> be other problems lurking here aside from this one. More soon hopefully...
>>
>
>
> A bit more context here - variable encoding takes the address of the
> variable from DWARF to locate the associated ELF section. Because we
> insist on having a variable specification - with a location - this
> usually works fine. However the problem is that because these dummy
> __pcpu_ variables specify a .discard section, their addresses are 0, so
> we get for example:
>
>  <1><1e535>: Abbrev Number: 114 (DW_TAG_variable)
>     <1e536>   DW_AT_name        : (indirect string, offset: 0x5e97):
> __pcpu_unique_kstack_offset
>     <1e53a>   DW_AT_decl_file   : 1
>     <1e53b>   DW_AT_decl_line   : 823
>     <1e53d>   DW_AT_decl_column : 1
>     <1e53e>   DW_AT_type        : <0x57>
>     <1e542>   DW_AT_external    : 1
>     <1e542>   DW_AT_declaration : 1
>  <1><1e542>: Abbrev Number: 156 (DW_TAG_variable)
>     <1e544>   DW_AT_specification: <0x1e535>
>     <1e548>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0
> (DW_OP_addr: 0)
>
>
> You can see the same thing for a simple program like this:
>
> #include <stdio.h>
>
> #define SEC(name) __attribute__((section(name)))
>
> SEC("/DISCARD/") int d1;
> extern int d1;
> SEC("/DISCARD/") int d2;
> extern int d2;
>
> int main(int argc, char *argv[])
> {
> 	return 0;
> }
>
>
> If you compile it with -g, the DWARF shows that d1 and d2 both have
> address 0:
>
>  <1><72>: Abbrev Number: 5 (DW_TAG_variable)
>     <73>   DW_AT_name        : d1
>     <76>   DW_AT_decl_file   : 1
>     <77>   DW_AT_decl_line   : 5
>     <78>   DW_AT_decl_column : 22
>     <79>   DW_AT_type        : <0x57>
>     <7d>   DW_AT_external    : 1
>     <7d>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0
> (DW_OP_addr: 0)
>  <1><87>: Abbrev Number: 5 (DW_TAG_variable)
>     <88>   DW_AT_name        : d2
>     <8b>   DW_AT_decl_file   : 1
>     <8c>   DW_AT_decl_line   : 7
>     <8d>   DW_AT_decl_column : 22
>     <8e>   DW_AT_type        : <0x57>
>     <92>   DW_AT_external    : 1
>     <92>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0
> (DW_OP_addr: 0)
>
>
> So the reason this happens for dwarves v1.28 in particular is - as I
> understand it - we moved away from recording ELF section information for
> each variable and matching that with DWARF info, instead relying on the
> address to locate the associated ELF section. In cases like the above
> the address information unfortunately leads us astray.
>
> Seems like there's a few approaches we can take in fixing this:
>
> 1. designate "__pcpu_" prefix as a variable prefix to filter out. This
> resolves the immediate problem but is too narrowly focused IMO and we
> may end up playing whack-a-mole with other dummy variable prefixes.
> 2. resurrect ELF section variable information fully; i.e. record a list
> of variables per ELF section (or at least per ELF section we care
> about). If variable is not on the list for the ELF section, do not
> encode it.
> 3. midway between the two; for the 0 address case specifically, verify
> that the variable name really _is_ in the associated ELF section. No
> need to create a local ELF table variable representation, we could just
> walk the table in the case of the 0 addresses.
>
> Diff for approach 3 is as follows
>
> diff --git a/btf_encoder.c b/btf_encoder.c
> index 3754884..21a0ab6 100644
> --- a/btf_encoder.c
> +++ b/btf_encoder.c
> @@ -2189,6 +2189,26 @@ static bool filter_variable_name(const char *name)
>         return false;
>  }
>
> +bool variable_in_sec(struct btf_encoder *encoder, const char *name,
> size_t shndx)
> +{
> +       uint32_t sym_sec_idx;
> +       uint32_t core_id;
> +       GElf_Sym sym;
> +
> +       elf_symtab__for_each_symbol_index(encoder->symtab, core_id, sym,
> sym_sec_idx) {
> +               const char *sym_name;
> +
> +               if (sym_sec_idx != shndx || elf_sym__type(&sym) !=
> STT_OBJECT)
> +                       continue;
> +               sym_name = elf_sym__name(&sym, encoder->symtab);
> +               if (!sym_name)
> +                       continue;
> +               if (strcmp(name, sym_name) == 0)
> +                       return true;
> +       }
> +       return false;
> +}
> +
>  static int btf_encoder__encode_cu_variables(struct btf_encoder *encoder)
>  {
>         struct cu *cu = encoder->cu;
> @@ -2258,6 +2278,11 @@ static int
> btf_encoder__encode_cu_variables(struct btf_encoder *encoder)
>                 if (filter_variable_name(name))
>                         continue;
>
> +               /* A 0 address may be in a .discard section; ensure the
> +                * variable really is in this section by checking ELF
> symtab.
> +                */
> +               if (addr == 0 && !variable_in_sec(encoder, name, shndx))
> +                       continue;
>                 /* Check for invalid BTF names */
>                 if (!btf_name_valid(name)) {
>                         dump_invalid_symbol("Found invalid variable name
> when encoding btf",
>
>
> ...so slightly more complex than option 1, but a bit more general in its
> applicability to .discard section variables.
>
> For the pahole folks, what do we think? Which option (or indeed other
> ones I haven't thought of) makes sense for a fix for this? Thanks!

Hi Alan,

Thanks so much for the analysis. It strikes me that it would be very
nice if the compiler could somehow annotate DIEs that are discarded. But
maybe the discarding happens far enough along that the DWARF can't be
modified?

In any case, while we wish for better compilers, we need a solution now.
I think Option 3 is a really great compromise. Reading it in context
with the code, it makes intuitive sense and it works when we're
outputting just the percpu globals, or when we're outputting all
globals. And there's little risk of issues, given that up until 1.28,
we've been using the ELF name for all variables anyway, so we know that
there have always been ELF symbols for all the variables we care about.

Stephen

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Jiri Olsa]

On Mon, Dec 16, 2024 at 03:19:01PM +0000, Alan Maguire wrote:
> On 14/12/2024 12:15, Alan Maguire wrote:
> > On 14/12/2024 04:41, Cong Wang wrote:
> >> On Thu, Dec 05, 2024 at 08:36:33AM +0100, Uros Bizjak wrote:
> >>> On Wed, Dec 4, 2024 at 4:52 PM Laura Nao <laura.nao@collabora.com> wrote:
> >>>>
> >>>> On 11/15/24 18:17, Laura Nao wrote:
> >>>>> I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
> >>>>> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the
> >>>>> modules[3] and its btf data[4] extracted with:
> >>>>>
> >>>>> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
> >>>>>
> >>>>> Looking again at the logs[5], I've noticed the following is reported:
> >>>>>
> >>>>> [    0.415885] BPF:    type_id=115803 offset=177920 size=1152
> >>>>> [    0.416029] BPF:
> >>>>> [    0.416083] BPF: Invalid offset
> >>>>> [    0.416165] BPF:
> >>>>>
> >>>>> There are two different definitions of rcu_data in '.data..percpu', one
> >>>>> is a struct and the other is an integer:
> >>>>>
> >>>>> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
> >>>>> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
> >>>>>
> >>>>> [115801] VAR 'rcu_data' type_id=115572, linkage=static
> >>>>> [115803] VAR 'rcu_data' type_id=1, linkage=static
> >>>>>
> >>>>> [115572] STRUCT 'rcu_data' size=1152 vlen=69
> >>>>> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> >>>>>
> >>>>> I assume that's not expected, correct?
> >>>>>
> >>>>> I'll dig a bit deeper and report back if I can find anything else.
> >>>>
> >>>> I ran a bisection, and it appears the culprit commit is:
> >>>> https://lore.kernel.org/all/20241021080856.48746-2-ubizjak@gmail.com/
> >>>>
> >>>> Hi Uros, do you have any suggestions or insights on resolving this issue?
> >>>
> >>> There is a stray ";" at the end of the #define, perhaps this makes a difference:
> >>>
> >>> +#define PERCPU_PTR(__p) \
> >>> + (typeof(*(__p)) __force __kernel *)(__p);
> >>> +
> >>>
> >>> and SHIFT_PERCPU_PTR macro now expands to:
> >>>
> >>> RELOC_HIDE((typeof(*(p)) __force __kernel *)(p);, (offset))
> >>>
> >>> A follow-up patch in the series changes PERCPU_PTR macro to:
> >>>
> >>> #define PERCPU_PTR(__p) \
> >>> ({ \
> >>> unsigned long __pcpu_ptr = (__force unsigned long)(__p); \
> >>> (typeof(*(__p)) __force __kernel *)(__pcpu_ptr); \
> >>> })
> >>>
> >>> so this should again correctly cast the value.
> >>
> >> Hm, I saw a similar bug but with pahole 1.28. My kernel complains about
> >> BTF invalid offset:
> >>
> >> [    7.785788] BPF: 	 type_id=2394 offset=0 size=1
> >> [    7.786411] BPF:
> >> [    7.786703] BPF: Invalid offset
> >> [    7.787119] BPF:
> >>
> >> Dumping the vmlinux (there is no module invovled), I saw it is related to
> >> percpu pointer too:
> >>
> >> [2394] VAR '__pcpu_unique_cpu_hw_events' type_id=2, linkage=global
> >> ...
> >> [163643] DATASEC '.data..percpu' size=2123280 vlen=808
> >>         type_id=2393 offset=0 size=1 (VAR '__pcpu_scope_cpu_hw_events')
> >>         type_id=2394 offset=0 size=1 (VAR '__pcpu_unique_cpu_hw_events')
> >> ...
> >>
> >> I compiled and installed the latest pahole from its git repo:
> >>
> >> $ pahole --version
> >> v1.28
> >>
> >> Thanks.
> > 
> > Thanks for the report! Looking at percpu-defs.h it looks like the
> > existence of such variables requires either
> > 
> > #if defined(ARCH_NEEDS_WEAK_PER_CPU) ||
> > defined(CONFIG_DEBUG_FORCE_WEAK_PER_CPU)
> > 
> > ...
> > 
> > #define DEFINE_PER_CPU_SECTION(type, name, sec)                         \
> >         __PCPU_DUMMY_ATTRS char __pcpu_scope_##name;                    \
> >         extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;            \
> >         __PCPU_DUMMY_ATTRS char __pcpu_unique_##name;                   \
> >         extern __PCPU_ATTRS(sec) __typeof__(type) name;                 \
> >         __PCPU_ATTRS(sec) __weak __typeof__(type) name
> > 
> > 
> > I'm guessing your .config has CONFIG_DEBUG_FORCE_WEAK_PER_CPU, or are
> > you building on s390/alpha?
> > 
> > I've reproduced this on bpf-next with CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y,
> > pahole v1.28 and gcc-12; I see ~900 __pcpu_ variables and get the same
> > BTF errors since multipe __pcpu_ vars share the offset 0.
> > 
> > A simple workaround in dwarves - and I verified this resolved the issue
> > for me - would be
> > 
> > diff --git a/btf_encoder.c b/btf_encoder.c
> > index 3754884..4a1799a 100644
> > --- a/btf_encoder.c
> > +++ b/btf_encoder.c
> > @@ -2174,7 +2174,8 @@ static bool filter_variable_name(const char *name)
> >                 X("__UNIQUE_ID"),
> >                 X("__tpstrtab_"),
> >                 X("__exitcall_"),
> > -               X("__func_stack_frame_non_standard_")
> > +               X("__func_stack_frame_non_standard_"),
> > +               X("__pcpu_")
> >                 #undef X
> >         };
> >         int i;
> > 
> > ...but I'd like us to understand further why variables which were
> > supposed to be in a .discard section end up being encoded as there may
> > be other problems lurking here aside from this one. More soon hopefully...
> >
> 
> 
> A bit more context here - variable encoding takes the address of the
> variable from DWARF to locate the associated ELF section. Because we
> insist on having a variable specification - with a location - this
> usually works fine. However the problem is that because these dummy
> __pcpu_ variables specify a .discard section, their addresses are 0, so
> we get for example:
> 
>  <1><1e535>: Abbrev Number: 114 (DW_TAG_variable)
>     <1e536>   DW_AT_name        : (indirect string, offset: 0x5e97):
> __pcpu_unique_kstack_offset
>     <1e53a>   DW_AT_decl_file   : 1
>     <1e53b>   DW_AT_decl_line   : 823
>     <1e53d>   DW_AT_decl_column : 1
>     <1e53e>   DW_AT_type        : <0x57>
>     <1e542>   DW_AT_external    : 1
>     <1e542>   DW_AT_declaration : 1
>  <1><1e542>: Abbrev Number: 156 (DW_TAG_variable)
>     <1e544>   DW_AT_specification: <0x1e535>
>     <1e548>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0
> (DW_OP_addr: 0)
> 
> 
> You can see the same thing for a simple program like this:
> 
> #include <stdio.h>
> 
> #define SEC(name) __attribute__((section(name)))
> 
> SEC("/DISCARD/") int d1;
> extern int d1;
> SEC("/DISCARD/") int d2;
> extern int d2;
> 
> int main(int argc, char *argv[])
> {
> 	return 0;
> }
> 
> 
> If you compile it with -g, the DWARF shows that d1 and d2 both have
> address 0:
> 
>  <1><72>: Abbrev Number: 5 (DW_TAG_variable)
>     <73>   DW_AT_name        : d1
>     <76>   DW_AT_decl_file   : 1
>     <77>   DW_AT_decl_line   : 5
>     <78>   DW_AT_decl_column : 22
>     <79>   DW_AT_type        : <0x57>
>     <7d>   DW_AT_external    : 1
>     <7d>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0
> (DW_OP_addr: 0)
>  <1><87>: Abbrev Number: 5 (DW_TAG_variable)
>     <88>   DW_AT_name        : d2
>     <8b>   DW_AT_decl_file   : 1
>     <8c>   DW_AT_decl_line   : 7
>     <8d>   DW_AT_decl_column : 22
>     <8e>   DW_AT_type        : <0x57>
>     <92>   DW_AT_external    : 1
>     <92>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0
> (DW_OP_addr: 0)
> 
> 
> So the reason this happens for dwarves v1.28 in particular is - as I
> understand it - we moved away from recording ELF section information for
> each variable and matching that with DWARF info, instead relying on the
> address to locate the associated ELF section. In cases like the above
> the address information unfortunately leads us astray.
> 
> Seems like there's a few approaches we can take in fixing this:
> 
> 1. designate "__pcpu_" prefix as a variable prefix to filter out. This
> resolves the immediate problem but is too narrowly focused IMO and we
> may end up playing whack-a-mole with other dummy variable prefixes.
> 2. resurrect ELF section variable information fully; i.e. record a list
> of variables per ELF section (or at least per ELF section we care
> about). If variable is not on the list for the ELF section, do not
> encode it.
> 3. midway between the two; for the 0 address case specifically, verify
> that the variable name really _is_ in the associated ELF section. No
> need to create a local ELF table variable representation, we could just
> walk the table in the case of the 0 addresses.
> 
> Diff for approach 3 is as follows
> 
> diff --git a/btf_encoder.c b/btf_encoder.c
> index 3754884..21a0ab6 100644
> --- a/btf_encoder.c
> +++ b/btf_encoder.c
> @@ -2189,6 +2189,26 @@ static bool filter_variable_name(const char *name)
>         return false;
>  }
> 
> +bool variable_in_sec(struct btf_encoder *encoder, const char *name,
> size_t shndx)
> +{
> +       uint32_t sym_sec_idx;
> +       uint32_t core_id;
> +       GElf_Sym sym;
> +
> +       elf_symtab__for_each_symbol_index(encoder->symtab, core_id, sym,
> sym_sec_idx) {
> +               const char *sym_name;
> +
> +               if (sym_sec_idx != shndx || elf_sym__type(&sym) !=
> STT_OBJECT)
> +                       continue;
> +               sym_name = elf_sym__name(&sym, encoder->symtab);
> +               if (!sym_name)
> +                       continue;
> +               if (strcmp(name, sym_name) == 0)
> +                       return true;
> +       }
> +       return false;
> +}
> +
>  static int btf_encoder__encode_cu_variables(struct btf_encoder *encoder)
>  {
>         struct cu *cu = encoder->cu;
> @@ -2258,6 +2278,11 @@ static int
> btf_encoder__encode_cu_variables(struct btf_encoder *encoder)
>                 if (filter_variable_name(name))
>                         continue;
> 
> +               /* A 0 address may be in a .discard section; ensure the
> +                * variable really is in this section by checking ELF
> symtab.
> +                */
> +               if (addr == 0 && !variable_in_sec(encoder, name, shndx))
> +                       continue;
>                 /* Check for invalid BTF names */
>                 if (!btf_name_valid(name)) {
>                         dump_invalid_symbol("Found invalid variable name
> when encoding btf",
> 
> 
> ...so slightly more complex than option 1, but a bit more general in its
> applicability to .discard section variables.
> 
> For the pahole folks, what do we think? Which option (or indeed other
> ones I haven't thought of) makes sense for a fix for this? Thanks!

I can reproduce this with the CONFIG_DEBUG_FORCE_WEAK_PER_CPU enable,
the fix looks fine, could you send formal patch?

thanks,
jirka

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Cong Wang]

On Mon, Dec 16, 2024 at 03:19:01PM +0000, Alan Maguire wrote:
> Seems like there's a few approaches we can take in fixing this:
> 
> 1. designate "__pcpu_" prefix as a variable prefix to filter out. This
> resolves the immediate problem but is too narrowly focused IMO and we
> may end up playing whack-a-mole with other dummy variable prefixes.
> 2. resurrect ELF section variable information fully; i.e. record a list
> of variables per ELF section (or at least per ELF section we care
> about). If variable is not on the list for the ELF section, do not
> encode it.
> 3. midway between the two; for the 0 address case specifically, verify
> that the variable name really _is_ in the associated ELF section. No
> need to create a local ELF table variable representation, we could just
> walk the table in the case of the 0 addresses.
> 
> Diff for approach 3 is as follows

Hi Alan,

Thanks for your detailed analysis.

Is your patch formally submitted and merged? A quick code search with
variable_in_sec() tells me no, but I could very easily miss things here,
hence I am asking you. :)

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Alan Maguire]

On 15/01/2025 17:38, Cong Wang wrote:
> On Mon, Dec 16, 2024 at 03:19:01PM +0000, Alan Maguire wrote:
>> Seems like there's a few approaches we can take in fixing this:
>>
>> 1. designate "__pcpu_" prefix as a variable prefix to filter out. This
>> resolves the immediate problem but is too narrowly focused IMO and we
>> may end up playing whack-a-mole with other dummy variable prefixes.
>> 2. resurrect ELF section variable information fully; i.e. record a list
>> of variables per ELF section (or at least per ELF section we care
>> about). If variable is not on the list for the ELF section, do not
>> encode it.
>> 3. midway between the two; for the 0 address case specifically, verify
>> that the variable name really _is_ in the associated ELF section. No
>> need to create a local ELF table variable representation, we could just
>> walk the table in the case of the 0 addresses.
>>
>> Diff for approach 3 is as follows
> 
> Hi Alan,
> 
> Thanks for your detailed analysis.
> 
> Is your patch formally submitted and merged? A quick code search with
> variable_in_sec() tells me no, but I could very easily miss things here,
> hence I am asking you. :)

hi Cong, I submitted the patch here:

https://lore.kernel.org/dwarves/20241217103629.2383809-1-alan.maguire@oracle.com/

..but it looks like it hasn't landed officially yet, apologies about
that. Would you mind testing the master branch of dwarves with the patch
applied to confirm the problem is fixed? It worked at my end but would
be great to confirm it fixes the issue on your side too. Thanks!

Alan

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Jiri Olsa]

On Wed, Dec 04, 2024 at 04:53:05PM +0100, Laura Nao wrote:
> On 11/15/24 18:17, Laura Nao wrote:
> > I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
> > (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the
> > modules[3] and its btf data[4] extracted with:
> > 
> > bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
> > 
> > Looking again at the logs[5], I've noticed the following is reported:
> > 
> > [    0.415885] BPF: 	 type_id=115803 offset=177920 size=1152
> > [    0.416029] BPF:
> > [    0.416083] BPF: Invalid offset
> > [    0.416165] BPF:
> > 
> > There are two different definitions of rcu_data in '.data..percpu', one
> > is a struct and the other is an integer:
> > 
> > type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
> > type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
> > 
> > [115801] VAR 'rcu_data' type_id=115572, linkage=static
> > [115803] VAR 'rcu_data' type_id=1, linkage=static
> > 
> > [115572] STRUCT 'rcu_data' size=1152 vlen=69
> > [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> > 
> > I assume that's not expected, correct?
> > 
> > I'll dig a bit deeper and report back if I can find anything else.
> 
> I ran a bisection, and it appears the culprit commit is:
> https://lore.kernel.org/all/20241021080856.48746-2-ubizjak@gmail.com/

which tree are you using, I can't see this in linu-next ?

thanks,
jirka

> 
> Hi Uros, do you have any suggestions or insights on resolving this issue?
> 
> This problem is now impacting mainline as well. The full context can be 
> found at the beginning of this thread[1].
> 
> Thanks,
> 
> Laura
> 
> [1] https://lore.kernel.org/all/20241106160820.259829-1-laura.nao@collabora.com/
> 
> #regzbot introduced: 001217defd
> 
> 

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Uros Bizjak]

On Thu, Dec 5, 2024 at 11:33 AM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Wed, Dec 04, 2024 at 04:53:05PM +0100, Laura Nao wrote:
> > On 11/15/24 18:17, Laura Nao wrote:
> > > I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
> > > (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the
> > > modules[3] and its btf data[4] extracted with:
> > >
> > > bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
> > >
> > > Looking again at the logs[5], I've noticed the following is reported:
> > >
> > > [    0.415885] BPF:          type_id=115803 offset=177920 size=1152
> > > [    0.416029] BPF:
> > > [    0.416083] BPF: Invalid offset
> > > [    0.416165] BPF:
> > >
> > > There are two different definitions of rcu_data in '.data..percpu', one
> > > is a struct and the other is an integer:
> > >
> > > type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
> > > type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
> > >
> > > [115801] VAR 'rcu_data' type_id=115572, linkage=static
> > > [115803] VAR 'rcu_data' type_id=1, linkage=static
> > >
> > > [115572] STRUCT 'rcu_data' size=1152 vlen=69
> > > [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> > >
> > > I assume that's not expected, correct?
> > >
> > > I'll dig a bit deeper and report back if I can find anything else.
> >
> > I ran a bisection, and it appears the culprit commit is:
> > https://lore.kernel.org/all/20241021080856.48746-2-ubizjak@gmail.com/
>
> which tree are you using, I can't see this in linu-next ?

The offending commit is now part of linus tree:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=001217defda86d0d6a5a9e6cf77a6b813857e7e3

Uros.

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Jiri Olsa]

On Fri, Nov 15, 2024 at 06:17:12PM +0100, Laura Nao wrote:
> On 11/13/24 10:37, Laura Nao wrote:
> > 
> > Currently, KernelCI only retains the bzImage, not the vmlinux binary. The
> > bzImage can be downloaded from the same link mentioned above by selecting
> > 'kernel' from the dropdown menu (modules can also be downloaded the same
> > way). I’ll try to replicate the build on my end and share the vmlinux
> > with DWARF data stripped for convenience.
> > 
> 
> I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the 
> modules[3] and its btf data[4] extracted with:
> 
> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
> 
> Looking again at the logs[5], I've noticed the following is reported:
> 
> [    0.415885] BPF: 	 type_id=115803 offset=177920 size=1152
> [    0.416029] BPF:  
> [    0.416083] BPF: Invalid offset
> [    0.416165] BPF: 
> 
> There are two different definitions of rcu_data in '.data..percpu', one
> is a struct and the other is an integer:
> 
> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
> 
> [115801] VAR 'rcu_data' type_id=115572, linkage=static
> [115803] VAR 'rcu_data' type_id=1, linkage=static
> 
> [115572] STRUCT 'rcu_data' size=1152 vlen=69
> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> 
> I assume that's not expected, correct?

yes, that seems wrong.. but I can't reproduce with your config
together with pahole 1.24 .. could you try with latest one?

jirka

> 
> I'll dig a bit deeper and report back if I can find anything else.
> 
> [1] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/vmlinux
> [2] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/vmlinux.raw
> [3] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/cros_kbd_led_backlight.ko
> [4] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/cros_kbd_led_backlight.ko.raw
> [5] https://pastebin.com/raw/FvvrPhAY
> 
> Best,
> 
> Laura
> 

Re: [REGRESSION] module BTF validation failure (Error -22) on

[Laura Nao]

Hi Jiri,

Thanks for the feedback!

On 12/6/24 13:35, Jiri Olsa wrote:
> On Fri, Nov 15, 2024 at 06:17:12PM +0100, Laura Nao wrote:
>> On 11/13/24 10:37, Laura Nao wrote:
>>>
>>> Currently, KernelCI only retains the bzImage, not the vmlinux
>>> binary. The
>>> bzImage can be downloaded from the same link mentioned above by
>>> selecting
>>> 'kernel' from the dropdown menu (modules can also be downloaded the
>>> same
>>> way). I’ll try to replicate the build on my end and share the
>>> vmlinux
>>> with DWARF data stripped for convenience.
>>>
>>
>> I managed to reproduce the issue locally and I've uploaded the
>> vmlinux[1]
>> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of
>> the
>> modules[3] and its btf data[4] extracted with:
>>
>> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko >
>> cros_kbd_led_backlight.ko.raw
>>
>> Looking again at the logs[5], I've noticed the following is reported:
>>
>> [    0.415885] BPF: 	 type_id=115803 offset=177920 size=1152
>> [    0.416029] BPF:
>> [    0.416083] BPF: Invalid offset
>> [    0.416165] BPF:
>>
>> There are two different definitions of rcu_data in '.data..percpu',
>> one
>> is a struct and the other is an integer:
>>
>> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
>> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
>>
>> [115801] VAR 'rcu_data' type_id=115572, linkage=static
>> [115803] VAR 'rcu_data' type_id=1, linkage=static
>>
>> [115572] STRUCT 'rcu_data' size=1152 vlen=69
>> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64
>> encoding=(none)
>>
>> I assume that's not expected, correct?
> 
> yes, that seems wrong.. but I can't reproduce with your config
> together with pahole 1.24 .. could you try with latest one?

I just tested next-20241210 with the latest pahole version (1.28 from
the master branch[1]), and the issue does not occur with this version
(I can see only one instance of rcu_data in the BTF data, as expected).

I can confirm that the same kernel revision still exhibits the issue
with pahole 1.24.

If helpful, I can also test versions between 1.24 and 1.28 to identify
which ones work.

Thanks,

Laura

[1] https://git.kernel.org/pub/scm/devel/pahole/pahole.git

> 
> jirka
> 
>>
>> I'll dig a bit deeper and report back if I can find anything else.
>>
>> [1]
>> https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/vmlinux
>> [2]
>> https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/vmlinux.raw
>> [3]
>> https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/cros_kbd_led_backlight.ko
>> [4]
>> https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/cros_kbd_led_backlight.ko.raw
>> [5] https://pastebin.com/raw/FvvrPhAY
>>
>> Best,
>>
>> Laura
>>

Re: [REGRESSION] module BTF validation failure (Error -22) on

[Jiri Olsa]

On Tue, Dec 10, 2024 at 02:55:01PM +0100, Laura Nao wrote:
> Hi Jiri,
> 
> Thanks for the feedback!
> 
> On 12/6/24 13:35, Jiri Olsa wrote:
> > On Fri, Nov 15, 2024 at 06:17:12PM +0100, Laura Nao wrote:
> >> On 11/13/24 10:37, Laura Nao wrote:
> >>>
> >>> Currently, KernelCI only retains the bzImage, not the vmlinux
> >>> binary. The
> >>> bzImage can be downloaded from the same link mentioned above by
> >>> selecting
> >>> 'kernel' from the dropdown menu (modules can also be downloaded the
> >>> same
> >>> way). I’ll try to replicate the build on my end and share the
> >>> vmlinux
> >>> with DWARF data stripped for convenience.
> >>>
> >>
> >> I managed to reproduce the issue locally and I've uploaded the
> >> vmlinux[1]
> >> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of
> >> the
> >> modules[3] and its btf data[4] extracted with:
> >>
> >> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko >
> >> cros_kbd_led_backlight.ko.raw
> >>
> >> Looking again at the logs[5], I've noticed the following is reported:
> >>
> >> [    0.415885] BPF: 	 type_id=115803 offset=177920 size=1152
> >> [    0.416029] BPF:
> >> [    0.416083] BPF: Invalid offset
> >> [    0.416165] BPF:
> >>
> >> There are two different definitions of rcu_data in '.data..percpu',
> >> one
> >> is a struct and the other is an integer:
> >>
> >> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
> >> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
> >>
> >> [115801] VAR 'rcu_data' type_id=115572, linkage=static
> >> [115803] VAR 'rcu_data' type_id=1, linkage=static
> >>
> >> [115572] STRUCT 'rcu_data' size=1152 vlen=69
> >> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64
> >> encoding=(none)
> >>
> >> I assume that's not expected, correct?
> > 
> > yes, that seems wrong.. but I can't reproduce with your config
> > together with pahole 1.24 .. could you try with latest one?
> 
> I just tested next-20241210 with the latest pahole version (1.28 from
> the master branch[1]), and the issue does not occur with this version
> (I can see only one instance of rcu_data in the BTF data, as expected).
> 
> I can confirm that the same kernel revision still exhibits the issue
> with pahole 1.24.
> 
> If helpful, I can also test versions between 1.24 and 1.28 to identify
> which ones work.

I managed to reproduce finally with gcc-12, but had to use pahole 1.25,
1.24 failed with unknown attribute

	[95096] VAR 'rcu_data' type_id=94868, linkage=static
	[95098] VAR 'rcu_data' type_id=4, linkage=static
	type_id=95096 offset=177088 size=1152 (VAR 'rcu_data')
	type_id=95098 offset=177088 size=1152 (VAR 'rcu_data')

will try to check what's going on

jirka

Re: [REGRESSION] module BTF validation failure (Error -22) on

[Jiri Olsa]

On Wed, Dec 11, 2024 at 10:10:24PM +0100, Jiri Olsa wrote:
> On Tue, Dec 10, 2024 at 02:55:01PM +0100, Laura Nao wrote:
> > Hi Jiri,
> > 
> > Thanks for the feedback!
> > 
> > On 12/6/24 13:35, Jiri Olsa wrote:
> > > On Fri, Nov 15, 2024 at 06:17:12PM +0100, Laura Nao wrote:
> > >> On 11/13/24 10:37, Laura Nao wrote:
> > >>>
> > >>> Currently, KernelCI only retains the bzImage, not the vmlinux
> > >>> binary. The
> > >>> bzImage can be downloaded from the same link mentioned above by
> > >>> selecting
> > >>> 'kernel' from the dropdown menu (modules can also be downloaded the
> > >>> same
> > >>> way). I’ll try to replicate the build on my end and share the
> > >>> vmlinux
> > >>> with DWARF data stripped for convenience.
> > >>>
> > >>
> > >> I managed to reproduce the issue locally and I've uploaded the
> > >> vmlinux[1]
> > >> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of
> > >> the
> > >> modules[3] and its btf data[4] extracted with:
> > >>
> > >> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko >
> > >> cros_kbd_led_backlight.ko.raw
> > >>
> > >> Looking again at the logs[5], I've noticed the following is reported:
> > >>
> > >> [    0.415885] BPF: 	 type_id=115803 offset=177920 size=1152
> > >> [    0.416029] BPF:
> > >> [    0.416083] BPF: Invalid offset
> > >> [    0.416165] BPF:
> > >>
> > >> There are two different definitions of rcu_data in '.data..percpu',
> > >> one
> > >> is a struct and the other is an integer:
> > >>
> > >> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
> > >> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
> > >>
> > >> [115801] VAR 'rcu_data' type_id=115572, linkage=static
> > >> [115803] VAR 'rcu_data' type_id=1, linkage=static
> > >>
> > >> [115572] STRUCT 'rcu_data' size=1152 vlen=69
> > >> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64
> > >> encoding=(none)
> > >>
> > >> I assume that's not expected, correct?
> > > 
> > > yes, that seems wrong.. but I can't reproduce with your config
> > > together with pahole 1.24 .. could you try with latest one?
> > 
> > I just tested next-20241210 with the latest pahole version (1.28 from
> > the master branch[1]), and the issue does not occur with this version
> > (I can see only one instance of rcu_data in the BTF data, as expected).
> > 
> > I can confirm that the same kernel revision still exhibits the issue
> > with pahole 1.24.
> > 
> > If helpful, I can also test versions between 1.24 and 1.28 to identify
> > which ones work.
> 
> I managed to reproduce finally with gcc-12, but had to use pahole 1.25,
> 1.24 failed with unknown attribute
> 
> 	[95096] VAR 'rcu_data' type_id=94868, linkage=static
> 	[95098] VAR 'rcu_data' type_id=4, linkage=static
> 	type_id=95096 offset=177088 size=1152 (VAR 'rcu_data')
> 	type_id=95098 offset=177088 size=1152 (VAR 'rcu_data')

so for me the difference seems to be using gcc-12 and this commit in linux tree:
  dabddd687c9e percpu: cast percpu pointer in PERCPU_PTR() via unsigned long

which adds extra __pcpu_ptr variable into dwarf, and it has the same
address as the per cpu variable and that confuses pahole

it ends up with adding per cpu variable twice.. one with real type
(type_id=94868) and the other with unsigned long type (type_id=4)

however this got fixed in pahole 1.28 commit:
  47dcb534e253 btf_encoder: Stop indexing symbols for VARs

which filters out __pcpu_ptr variable completely, adding Stephen to the loop

with gcc-14 the __pcpu_ptr variable has VSCOPE_OPTIMIZED scope, so it won't
get into btf even without above pahole fix

I suggest gcc/pahole upgrade ;-)

thanks,
jirka

Re: [REGRESSION] module BTF validation failure (Error -22) on

[Stephen Brennan]

Jiri Olsa <olsajiri@gmail.com> writes:
> On Wed, Dec 11, 2024 at 10:10:24PM +0100, Jiri Olsa wrote:
>> On Tue, Dec 10, 2024 at 02:55:01PM +0100, Laura Nao wrote:
>> > Hi Jiri,
>> > 
>> > Thanks for the feedback!
>> > 
>> > On 12/6/24 13:35, Jiri Olsa wrote:
>> > > On Fri, Nov 15, 2024 at 06:17:12PM +0100, Laura Nao wrote:
>> > >> On 11/13/24 10:37, Laura Nao wrote:
>> > >>>
>> > >>> Currently, KernelCI only retains the bzImage, not the vmlinux
>> > >>> binary. The
>> > >>> bzImage can be downloaded from the same link mentioned above by
>> > >>> selecting
>> > >>> 'kernel' from the dropdown menu (modules can also be downloaded the
>> > >>> same
>> > >>> way). I’ll try to replicate the build on my end and share the
>> > >>> vmlinux
>> > >>> with DWARF data stripped for convenience.
>> > >>>
>> > >>
>> > >> I managed to reproduce the issue locally and I've uploaded the
>> > >> vmlinux[1]
>> > >> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of
>> > >> the
>> > >> modules[3] and its btf data[4] extracted with:
>> > >>
>> > >> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko >
>> > >> cros_kbd_led_backlight.ko.raw
>> > >>
>> > >> Looking again at the logs[5], I've noticed the following is reported:
>> > >>
>> > >> [    0.415885] BPF: 	 type_id=115803 offset=177920 size=1152
>> > >> [    0.416029] BPF:
>> > >> [    0.416083] BPF: Invalid offset
>> > >> [    0.416165] BPF:
>> > >>
>> > >> There are two different definitions of rcu_data in '.data..percpu',
>> > >> one
>> > >> is a struct and the other is an integer:
>> > >>
>> > >> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
>> > >> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
>> > >>
>> > >> [115801] VAR 'rcu_data' type_id=115572, linkage=static
>> > >> [115803] VAR 'rcu_data' type_id=1, linkage=static
>> > >>
>> > >> [115572] STRUCT 'rcu_data' size=1152 vlen=69
>> > >> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64
>> > >> encoding=(none)
>> > >>
>> > >> I assume that's not expected, correct?
>> > > 
>> > > yes, that seems wrong.. but I can't reproduce with your config
>> > > together with pahole 1.24 .. could you try with latest one?
>> > 
>> > I just tested next-20241210 with the latest pahole version (1.28 from
>> > the master branch[1]), and the issue does not occur with this version
>> > (I can see only one instance of rcu_data in the BTF data, as expected).
>> > 
>> > I can confirm that the same kernel revision still exhibits the issue
>> > with pahole 1.24.
>> > 
>> > If helpful, I can also test versions between 1.24 and 1.28 to identify
>> > which ones work.
>> 
>> I managed to reproduce finally with gcc-12, but had to use pahole 1.25,
>> 1.24 failed with unknown attribute
>> 
>> 	[95096] VAR 'rcu_data' type_id=94868, linkage=static
>> 	[95098] VAR 'rcu_data' type_id=4, linkage=static
>> 	type_id=95096 offset=177088 size=1152 (VAR 'rcu_data')
>> 	type_id=95098 offset=177088 size=1152 (VAR 'rcu_data')
>
> so for me the difference seems to be using gcc-12 and this commit in linux tree:
>   dabddd687c9e percpu: cast percpu pointer in PERCPU_PTR() via unsigned long
>
> which adds extra __pcpu_ptr variable into dwarf, and it has the same
> address as the per cpu variable and that confuses pahole
>
> it ends up with adding per cpu variable twice.. one with real type
> (type_id=94868) and the other with unsigned long type (type_id=4)
>
> however this got fixed in pahole 1.28 commit:
>   47dcb534e253 btf_encoder: Stop indexing symbols for VARs
>
> which filters out __pcpu_ptr variable completely, adding Stephen to the loop

Thanks for sharing this. Your analysis is spot-on, but I can fill in the
details a bit. I just grabbed 6.13-rc2 and built it with gcc 11 and
pahole 1.27, and observed the same issue:

  $ bpftool btf dump file vmlinux | grep "VAR 'rcu_data"
  [4045] VAR 'rcu_data' type_id=3962, linkage=static
  [4047] VAR 'rcu_data' type_id=1, linkage=static
          type_id=4045 offset=196608 size=520 (VAR 'rcu_data')
          type_id=4047 offset=196608 size=520 (VAR 'rcu_data')

In pahole 1.27, the (simplified) process for generating variables for
BTF is:

1. Look through the ELF symbol table, and find all symbols whose
addresses are within the percpu section, and add them to a list.

2. Look through the DWARF: for each tag of type DW_TAG_variable,
determine if the variable is "global". If so, and if the address matches
one of the symbols found in Step 1, continue.

3. Except for one special case, pahole doesn't check whether the DWARF
variable's name matches the symbol name. It simply emits a variable
using the name of the symbol from Step 1, and the type information from
Step 2.

The result of this process, in this case, is:

1. kernel/rcu/tree.c contains a declaration of "rcu_data". This results
in an ELF symbol in vmlinux of the same name. Great!

  $ eu-readelf -s vmlinux | grep '\brcu_data\b'
  12319: 0000000000030000    520 OBJECT  LOCAL  DEFAULT       21 rcu_data


2. A DWARF entry is emitted for "rcu_data" which has a matching location
(DW_AT_location has value DW_OP_addr 0x30000, matching the ELF symbol).
So far so good - pahole emits a BTF variable with the expected type.

  $ llvm-dwarfdump --name=rcu_data
  ...
  0x01af03f1: DW_TAG_variable
                DW_AT_name        ("rcu_data")
                DW_AT_decl_file   ("/home/stepbren/repos/linux-upstream/kernel/rcu/tree.c")
                DW_AT_decl_line   (80)
                DW_AT_decl_column (8)
                DW_AT_type        (0x01aefb38 "rcu_data")
                DW_AT_alignment   (0x40)
                DW_AT_location    (DW_OP_addr 0x30000)

3. In kernel/rcu/tree.c, we also have the following declaration at line
5227 which uses per_cpu_ptr() on &rcu_data:

5222 void rcutree_migrate_callbacks(int cpu)
5223 {
5224 	unsigned long flags;
5225 	struct rcu_data *my_rdp;
5226 	struct rcu_node *my_rnp;
5227 	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
                               ^^^^^^^^^^^

With the new changes in dabddd687c9e ("percpu: cast percpu pointer in
PERCPU_PTR() via unsigned long"), this expands to a lexical block which
contains a variable named "__pcpu_ptr", of type unsigned long. The
compiler emits the following DW_TAG_variable in the DWARF:

0x01b05d20:         DW_TAG_variable
                      DW_AT_name        ("__pcpu_ptr")
                      DW_AT_decl_file   ("/home/stepbren/repos/linux-upstream/kernel/rcu/tree.c")
                      DW_AT_decl_line   (5227)
                      DW_AT_decl_column (25)
                      DW_AT_type        (0x01adb52e "long unsigned int")
                      DW_AT_location    (DW_OP_addr 0x30000, DW_OP_stack_value)

Since the DW_AT_location has a DW_OP_addr - pahole understands this to
mean that the variable is located in global memory, and thus has
VSCOPE_GLOBAL. But of course, the actual "scope" of this variable is not
global, it is limited to the lexical block, which is completely hidden
away by the macro. But pahole 1.27 does not consider this, and since the
address matches the "rcu_data" symbol, it emits a variable of type "long
unsigned int" under the name "rcu_data" -- despite the fact that the
DWARF info has a name of "__pcpu_ptr".

The changes I made in 1.28 address this (unintentionally) by:

1. Requiring global variables be both "in the global scope" (i.e. in the
CU-level, rather than any function or other lexical block.
2. Requiring global variables have global memory (some of them could be
register variables, despite having global scope -- e.g.
current_stack_pointer).
3. No longer using the ELF symbol table, and instead using the DWARF
names for variables.

With #1, we would filter this variable. And with #3, even if the
variable were not filtered, we would output (a bunch of) variables with
the correct __pcpu_ptr variable name, which is unhelpful but at least
helps us understand where these things come from.

Rebuilding with GCC 14, we can see that the "__pcpu_ptr" variable no
longer has a DW_AT_location:

0x01afa82f:         DW_TAG_variable
                      DW_AT_name        ("__pcpu_ptr")
                      DW_AT_decl_file   ("/home/stepbren/repos/linux-upstream/kernel/rcu/tree.c")
                      DW_AT_decl_line   (5227)
                      DW_AT_decl_column (25)
                      DW_AT_type        (0x01ad0267 "long unsigned int")

This is the reason that pahole 1.27 now recognizes it as
VSCOPE_OPTIMIZED. Without a memory location pahole can't do anything to
match it against the "rcu_data" variable so nothing is emitted, and we
don't get the issue.

I'm not sure if this adds at all to the discussion, since the overall
answer is the same, an upgrade of pahole and/or gcc. (Pahole would be
recommended; GCC just changed the generated DWARF and I could imagine
other situations popping up elsewhere).


Thanks,
Stephen

> with gcc-14 the __pcpu_ptr variable has VSCOPE_OPTIMIZED scope, so it won't
> get into btf even without above pahole fix
>
> I suggest gcc/pahole upgrade ;-)
>
> thanks,
> jirka

Re: [REGRESSION] module BTF validation failure (Error -22) on

[Laura Nao]

On 12/12/24 22:49, Stephen Brennan wrote:
> Jiri Olsa <olsajiri@gmail.com> writes:
>> On Wed, Dec 11, 2024 at 10:10:24PM +0100, Jiri Olsa wrote:
>>> On Tue, Dec 10, 2024 at 02:55:01PM +0100, Laura Nao wrote:
>>>> Hi Jiri,
>>>>
>>>> Thanks for the feedback!
>>>>
>>>> On 12/6/24 13:35, Jiri Olsa wrote:
>>>>> On Fri, Nov 15, 2024 at 06:17:12PM +0100, Laura Nao wrote:
>>>>>> On 11/13/24 10:37, Laura Nao wrote:
>>>>>>>
>>>>>>> Currently, KernelCI only retains the bzImage, not the vmlinux
>>>>>>> binary. The
>>>>>>> bzImage can be downloaded from the same link mentioned above by
>>>>>>> selecting
>>>>>>> 'kernel' from the dropdown menu (modules can also be downloaded
>>>>>>> the
>>>>>>> same
>>>>>>> way). I’ll try to replicate the build on my end and share the
>>>>>>> vmlinux
>>>>>>> with DWARF data stripped for convenience.
>>>>>>>
>>>>>>
>>>>>> I managed to reproduce the issue locally and I've uploaded the
>>>>>> vmlinux[1]
>>>>>> (stripped of DWARF data) and vmlinux.raw[2] files, as well as one
>>>>>> of
>>>>>> the
>>>>>> modules[3] and its btf data[4] extracted with:
>>>>>>
>>>>>> bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko >
>>>>>> cros_kbd_led_backlight.ko.raw
>>>>>>
>>>>>> Looking again at the logs[5], I've noticed the following is
>>>>>> reported:
>>>>>>
>>>>>> [    0.415885] BPF: 	 type_id=115803 offset=177920 size=1152
>>>>>> [    0.416029] BPF:
>>>>>> [    0.416083] BPF: Invalid offset
>>>>>> [    0.416165] BPF:
>>>>>>
>>>>>> There are two different definitions of rcu_data in
>>>>>> '.data..percpu',
>>>>>> one
>>>>>> is a struct and the other is an integer:
>>>>>>
>>>>>> type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
>>>>>> type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
>>>>>>
>>>>>> [115801] VAR 'rcu_data' type_id=115572, linkage=static
>>>>>> [115803] VAR 'rcu_data' type_id=1, linkage=static
>>>>>>
>>>>>> [115572] STRUCT 'rcu_data' size=1152 vlen=69
>>>>>> [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64
>>>>>> encoding=(none)
>>>>>>
>>>>>> I assume that's not expected, correct?
>>>>>
>>>>> yes, that seems wrong.. but I can't reproduce with your config
>>>>> together with pahole 1.24 .. could you try with latest one?
>>>>
>>>> I just tested next-20241210 with the latest pahole version (1.28
>>>> from
>>>> the master branch[1]), and the issue does not occur with this
>>>> version
>>>> (I can see only one instance of rcu_data in the BTF data, as
>>>> expected).
>>>>
>>>> I can confirm that the same kernel revision still exhibits the
>>>> issue
>>>> with pahole 1.24.
>>>>
>>>> If helpful, I can also test versions between 1.24 and 1.28 to
>>>> identify
>>>> which ones work.
>>>
>>> I managed to reproduce finally with gcc-12, but had to use pahole
>>> 1.25,
>>> 1.24 failed with unknown attribute
>>>
>>> 	[95096] VAR 'rcu_data' type_id=94868, linkage=static
>>> 	[95098] VAR 'rcu_data' type_id=4, linkage=static
>>> 	type_id=95096 offset=177088 size=1152 (VAR 'rcu_data')
>>> 	type_id=95098 offset=177088 size=1152 (VAR 'rcu_data')
>>
>> so for me the difference seems to be using gcc-12 and this commit in
>> linux tree:
>>    dabddd687c9e percpu: cast percpu pointer in PERCPU_PTR() via
>>    unsigned long
>>
>> which adds extra __pcpu_ptr variable into dwarf, and it has the same
>> address as the per cpu variable and that confuses pahole
>>
>> it ends up with adding per cpu variable twice.. one with real type
>> (type_id=94868) and the other with unsigned long type (type_id=4)
>>
>> however this got fixed in pahole 1.28 commit:
>>    47dcb534e253 btf_encoder: Stop indexing symbols for VARs
>>
>> which filters out __pcpu_ptr variable completely, adding Stephen to
>> the loop
> 
> Thanks for sharing this. Your analysis is spot-on, but I can fill in
> the
> details a bit. I just grabbed 6.13-rc2 and built it with gcc 11 and
> pahole 1.27, and observed the same issue:
> 
>    $ bpftool btf dump file vmlinux | grep "VAR 'rcu_data"
>    [4045] VAR 'rcu_data' type_id=3962, linkage=static
>    [4047] VAR 'rcu_data' type_id=1, linkage=static
>            type_id=4045 offset=196608 size=520 (VAR 'rcu_data')
>            type_id=4047 offset=196608 size=520 (VAR 'rcu_data')
> 
> In pahole 1.27, the (simplified) process for generating variables for
> BTF is:
> 
> 1. Look through the ELF symbol table, and find all symbols whose
> addresses are within the percpu section, and add them to a list.
> 
> 2. Look through the DWARF: for each tag of type DW_TAG_variable,
> determine if the variable is "global". If so, and if the address
> matches
> one of the symbols found in Step 1, continue.
> 
> 3. Except for one special case, pahole doesn't check whether the DWARF
> variable's name matches the symbol name. It simply emits a variable
> using the name of the symbol from Step 1, and the type information
> from
> Step 2.
> 
> The result of this process, in this case, is:
> 
> 1. kernel/rcu/tree.c contains a declaration of "rcu_data". This
> results
> in an ELF symbol in vmlinux of the same name. Great!
> 
>    $ eu-readelf -s vmlinux | grep '\brcu_data\b'
>    12319: 0000000000030000    520 OBJECT  LOCAL  DEFAULT       21
>    rcu_data
> 
> 
> 2. A DWARF entry is emitted for "rcu_data" which has a matching
> location
> (DW_AT_location has value DW_OP_addr 0x30000, matching the ELF
> symbol).
> So far so good - pahole emits a BTF variable with the expected type.
> 
>    $ llvm-dwarfdump --name=rcu_data
>    ...
>    0x01af03f1: DW_TAG_variable
>                  DW_AT_name        ("rcu_data")
>                  DW_AT_decl_file
>                  ("/home/stepbren/repos/linux-upstream/kernel/rcu/tree.c")
>                  DW_AT_decl_line   (80)
>                  DW_AT_decl_column (8)
>                  DW_AT_type        (0x01aefb38 "rcu_data")
>                  DW_AT_alignment   (0x40)
>                  DW_AT_location    (DW_OP_addr 0x30000)
> 
> 3. In kernel/rcu/tree.c, we also have the following declaration at
> line
> 5227 which uses per_cpu_ptr() on &rcu_data:
> 
> 5222 void rcutree_migrate_callbacks(int cpu)
> 5223 {
> 5224 	unsigned long flags;
> 5225 	struct rcu_data *my_rdp;
> 5226 	struct rcu_node *my_rnp;
> 5227 	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
>                                 ^^^^^^^^^^^
> 
> With the new changes in dabddd687c9e ("percpu: cast percpu pointer in
> PERCPU_PTR() via unsigned long"), this expands to a lexical block
> which
> contains a variable named "__pcpu_ptr", of type unsigned long. The
> compiler emits the following DW_TAG_variable in the DWARF:
> 
> 0x01b05d20:         DW_TAG_variable
>                        DW_AT_name        ("__pcpu_ptr")
>                        DW_AT_decl_file
>                        ("/home/stepbren/repos/linux-upstream/kernel/rcu/tree.c")
>                        DW_AT_decl_line   (5227)
>                        DW_AT_decl_column (25)
>                        DW_AT_type        (0x01adb52e "long unsigned
>                        int")
>                        DW_AT_location    (DW_OP_addr 0x30000,
>                        DW_OP_stack_value)
> 
> Since the DW_AT_location has a DW_OP_addr - pahole understands this to
> mean that the variable is located in global memory, and thus has
> VSCOPE_GLOBAL. But of course, the actual "scope" of this variable is
> not
> global, it is limited to the lexical block, which is completely hidden
> away by the macro. But pahole 1.27 does not consider this, and since
> the
> address matches the "rcu_data" symbol, it emits a variable of type
> "long
> unsigned int" under the name "rcu_data" -- despite the fact that the
> DWARF info has a name of "__pcpu_ptr".
> 
> The changes I made in 1.28 address this (unintentionally) by:
> 
> 1. Requiring global variables be both "in the global scope" (i.e. in
> the
> CU-level, rather than any function or other lexical block.
> 2. Requiring global variables have global memory (some of them could
> be
> register variables, despite having global scope -- e.g.
> current_stack_pointer).
> 3. No longer using the ELF symbol table, and instead using the DWARF
> names for variables.
> 
> With #1, we would filter this variable. And with #3, even if the
> variable were not filtered, we would output (a bunch of) variables
> with
> the correct __pcpu_ptr variable name, which is unhelpful but at least
> helps us understand where these things come from.
> 
> Rebuilding with GCC 14, we can see that the "__pcpu_ptr" variable no
> longer has a DW_AT_location:
> 
> 0x01afa82f:         DW_TAG_variable
>                        DW_AT_name        ("__pcpu_ptr")
>                        DW_AT_decl_file
>                        ("/home/stepbren/repos/linux-upstream/kernel/rcu/tree.c")
>                        DW_AT_decl_line   (5227)
>                        DW_AT_decl_column (25)
>                        DW_AT_type        (0x01ad0267 "long unsigned
>                        int")
> 
> This is the reason that pahole 1.27 now recognizes it as
> VSCOPE_OPTIMIZED. Without a memory location pahole can't do anything
> to
> match it against the "rcu_data" variable so nothing is emitted, and we
> don't get the issue.
> 
> I'm not sure if this adds at all to the discussion, since the overall
> answer is the same, an upgrade of pahole and/or gcc. (Pahole would be
> recommended; GCC just changed the generated DWARF and I could imagine
> other situations popping up elsewhere).
>

Thank you for the help with debugging and for the detailed explanation!  
                                                                         
We'll proceed with updating pahole to v1.28 in the KernelCI build        
environment.                                                             
                                                                         
Best,                                                                    
                                                                         
Laura                                                                    
                                                                         
#regzbot resolve: fixed by changes in pahole 1.28
 
> 
> Thanks,
> Stephen
> 
>> with gcc-14 the __pcpu_ptr variable has VSCOPE_OPTIMIZED scope, so it
>> won't
>> get into btf even without above pahole fix
>>
>> I suggest gcc/pahole upgrade ;-)
>>
>> thanks,
>> jirka

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Kumar Kartikeya Dwivedi]

On Fri, 6 Dec 2024 at 13:37, Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Fri, Nov 15, 2024 at 06:17:12PM +0100, Laura Nao wrote:
> > On 11/13/24 10:37, Laura Nao wrote:
> > >
> > > Currently, KernelCI only retains the bzImage, not the vmlinux binary. The
> > > bzImage can be downloaded from the same link mentioned above by selecting
> > > 'kernel' from the dropdown menu (modules can also be downloaded the same
> > > way). I’ll try to replicate the build on my end and share the vmlinux
> > > with DWARF data stripped for convenience.
> > >
> >
> > I managed to reproduce the issue locally and I've uploaded the vmlinux[1]
> > (stripped of DWARF data) and vmlinux.raw[2] files, as well as one of the
> > modules[3] and its btf data[4] extracted with:
> >
> > bpftool -B vmlinux btf dump file cros_kbd_led_backlight.ko > cros_kbd_led_backlight.ko.raw
> >
> > Looking again at the logs[5], I've noticed the following is reported:
> >
> > [    0.415885] BPF:    type_id=115803 offset=177920 size=1152
> > [    0.416029] BPF:
> > [    0.416083] BPF: Invalid offset
> > [    0.416165] BPF:
> >
> > There are two different definitions of rcu_data in '.data..percpu', one
> > is a struct and the other is an integer:
> >
> > type_id=115801 offset=177920 size=1152 (VAR 'rcu_data')
> > type_id=115803 offset=177920 size=1152 (VAR 'rcu_data')
> >
> > [115801] VAR 'rcu_data' type_id=115572, linkage=static
> > [115803] VAR 'rcu_data' type_id=1, linkage=static
> >
> > [115572] STRUCT 'rcu_data' size=1152 vlen=69
> > [1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> >
> > I assume that's not expected, correct?
>
> yes, that seems wrong.. but I can't reproduce with your config
> together with pahole 1.24 .. could you try with latest one?
>

As a data point, I ran into this problem with pahole 1.27 on
bpf/master, I see the same rcu_data duplication.
For now I'm probably going to test patches with the latest pahole.

> jirka
>
> >
> > I'll dig a bit deeper and report back if I can find anything else.
> >
> > [1] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/vmlinux
> > [2] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/vmlinux.raw
> > [3] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/cros_kbd_led_backlight.ko
> > [4] https://people.collabora.com/~laura.nao/dbg-btf-mismatch-next-20241113/cros_kbd_led_backlight.ko.raw
> > [5] https://pastebin.com/raw/FvvrPhAY
> >
> > Best,
> >
> > Laura
> >
>

Re: [REGRESSION] module BTF validation failure (Error -22) on next

[Jiri Olsa]

On Wed, Nov 13, 2024 at 10:37:03AM +0100, Laura Nao wrote:
> Hi Alan,
> 
> On 11/7/24 16:05, Alan Maguire wrote:
> > Thanks for the report! Judging from the config, you're seeing this with
> > pahole v1.24. I have seen issues like this in the past where during a
> > kernel build, module BTF has been built against vmlinux BTF, and then
> > something later re-triggers vmlinux BTF generation. If that re-triggered
> > vmlinux BTF does not use the same type ids for types, this can result in
> > mismatch errors as above since modules are referring to out-of-date type
> > ids in vmlinux. That's just a preliminary guess though, we'll
> > need more info to help get to the bottom of this.
> > 
> > A few suggestions to help debug this:
> > 
> > - if you have build logs, check BTF generation of vmlinux. Did it in
> > fact happen twice perhaps? Even better if, if kernel CI saves logs, feel
> > free to send a pointer and I'll take a look.
> 
> Thanks for the pointers!
> 
> From what I can tell in the logs, the BTF generation of vmlinux only 
> occurred once. The automated build process in KernelCI generally involves 
> building the kernel first, followed by the modules and other artifacts 
> (such as the kselftest archive). 
> The full build log can be downloaded by selecting 'build_log' from 
> the dropdown menu at the top of this page:
> 
> https://kernelci-api.westus3.cloudapp.azure.com/viewer?node_id=6732f41d58937056c61734ab
> 
> I do see some warnings reported in the logs though:
> 
> WARN: resolve_btfids: unresolved symbol bpf_lsm_task_getsecid_obj
> WARN: resolve_btfids: unresolved symbol bpf_lsm_current_getsecid_subj

hi,
this is fixed in bpf/master already:
  8618f5ffba4d bpf, lsm: Remove getlsmprop hooks BTF IDs

I can't reproduce this as well, will check the logs you posted

jirka

> 
> > - can you post the vmlinux (stripped of DWARF data if possible to limit
> > size) and one of the failing modules somewhere so we can analyze?
> > - Failing that,
> > bpftool btf dump file /path/2/vmlinux_from_build > vmlinux.raw
> > and upload of the vmlinux.raw and one of the failing module .kos would help.
> > 
> 
> Currently, KernelCI only retains the bzImage, not the vmlinux binary. The 
> bzImage can be downloaded from the same link mentioned above by selecting 
> 'kernel' from the dropdown menu (modules can also be downloaded the same
> way). I’ll try to replicate the build on my end and share the vmlinux 
> with DWARF data stripped for convenience.
> 
> Thanks,
> 
> Laura
> 
>