s390x: selftests/bpf are failing on CI

s390x: selftests/bpf are failing on CI

[Ihor Solodrai]

Hi Ilya,

After recent merges from upstream, CI started failing both on bpf and
bpf-next trees. Yonghong Song and Song Liu submitted a couple of fixes
that are already applied to bpf tree, but there are still failures on
s390x.

https://github.com/kernel-patches/bpf/actions/runs/14163772245

Could you please investigate?

A snippet from the log:

    #11      atomics:OK
    trace_kprobe: Could not probe notrace function __s390x_sys_bpf
    trace_kprobe: Could not probe notrace function __s390x_sys_nanosleep
    trace_kprobe: Could not probe notrace function __s390x_sys_nanosleep
    trace_kprobe: Could not probe notrace function __s390x_sys_nanosleep
    trace_kprobe: Could not probe notrace function __s390x_sys_nanosleep
    trace_kprobe: Could not probe notrace function __s390x_sys_nanosleep
    test_attach_probe:PASS:skel_open 0 nsec
    test_attach_probe:PASS:skel_load 0 nsec
    test_attach_probe:PASS:check_bss 0 nsec
    test_attach_probe_manual:PASS:skel_kprobe_manual_open_and_load 0 nsec
    test_attach_probe_manual:PASS:uprobe_offset 0 nsec
    libbpf: prog 'handle_kprobe': failed to create kprobe '__s390x_sys_nanosleep+0x0' perf event: -EINVAL
    test_attach_probe_manual:FAIL:attach_kprobe unexpected error: -22
    #12/1    attach_probe/manual-default:FAIL
    test_attach_probe_manual:PASS:skel_kprobe_manual_open_and_load 0 nsec
    test_attach_probe_manual:PASS:uprobe_offset 0 nsec
    libbpf: failed to add legacy kprobe event for '__s390x_sys_nanosleep+0x0': -EINVAL
    libbpf: prog 'handle_kprobe': failed to create kprobe '__s390x_sys_nanosleep+0x0' perf event: -EINVAL
    test_attach_probe_manual:FAIL:attach_kprobe unexpected error: -22
    #12/2    attach_probe/manual-legacy:FAIL
    test_attach_probe_manual:PASS:skel_kprobe_manual_open_and_load 0 nsec
    test_attach_probe_manual:PASS:uprobe_offset 0 nsec
    libbpf: prog 'handle_kprobe': failed to create kprobe '__s390x_sys_nanosleep+0x0' perf event: -EINVAL
    test_attach_probe_manual:FAIL:attach_kprobe unexpected error: -22
    #12/3    attach_probe/manual-perf:FAIL
    test_attach_probe_manual:PASS:skel_kprobe_manual_open_and_load 0 nsec
    test_attach_probe_manual:PASS:uprobe_offset 0 nsec
    libbpf: prog 'handle_kprobe': failed to create kprobe '__s390x_sys_nanosleep+0x0' perf event: -EINVAL
    test_attach_probe_manual:FAIL:attach_kprobe unexpected error: -22
    #12/4    attach_probe/manual-link:FAIL
    libbpf: prog 'handle_kprobe_auto': failed to create kprobe '__se_sys_nanosleep+0x0' perf event: -ENOENT
    test_attach_probe_auto:FAIL:attach_kprobe_auto unexpected error: -2
    libbpf: prog 'handle_kretprobe_auto': failed to create kretprobe '__se_sys_nanosleep+0x0' perf event: -ENOENT
    test_attach_probe_auto:FAIL:attach_kretprobe_auto unexpected error: -2
    test_attach_probe_auto:PASS:auto-attach should fail for old-style name 0 nsec
    test_attach_probe_auto:PASS:attach_uretprobe_byname 0 nsec
    test_attach_probe_auto:FAIL:check_kprobe_auto_res unexpected check_kprobe_auto_res: actual 0 != expected 11
    test_attach_probe_auto:FAIL:check_kretprobe_auto_res unexpected check_kretprobe_auto_res: actual 0 != expected 22
    test_attach_probe_auto:PASS:check_uretprobe_byname_res 0 nsec
    #12/5    attach_probe/auto:FAIL

Re: s390x: selftests/bpf are failing on CI

[Ihor Solodrai]

On 3/31/25 3:25 PM, iii wrote:
> On 2025-03-31 20:25, Ihor Solodrai wrote:
>> Hi Ilya,
>>
>> After recent merges from upstream, CI started failing both on bpf and
>> bpf-next trees. Yonghong Song and Song Liu submitted a couple of fixes
>> that are already applied to bpf tree, but there are still failures on
>> s390x.
>>
>> https://github.com/kernel-patches/bpf/actions/runs/14163772245
>>
>> Could you please investigate?
>>
>> [...]
>
> Hi Ihor,
>
> Thanks for the heads up.
> I tried this manually with the kernel commit 07be1f644ff9 and the tests are passing:
>
> # ./test_progs -t attach_probe
>
> #12/1    attach_probe/manual-default:OK
> #12/2    attach_probe/manual-legacy:OK
> #12/3    attach_probe/manual-perf:OK
> #12/4    attach_probe/manual-link:OK
> #12/5    attach_probe/auto:OK
> #12/6    attach_probe/kprobe-sleepable:OK
> #12/7    attach_probe/uprobe-lib:OK
> #12/8    attach_probe/uprobe-sleepable:OK
> #12/9    attach_probe/uprobe-ref_ctr:OK
> #12      attach_probe:OK
> Summary: 1/9 PASSED, 0 SKIPPED, 0 FAILED
>
> So this must be a config issue.
> I'm not sure what is causing __s390x_sys_nanosleep to be notrace, but this doesn't look normal.

Hi, thank you for taking a look. Do they succeed if you run prior
tests too? I remember situations when a test would succeed when run
independently, but fail when running after other tests.

>
> I also see that the newer test runs are green:
>
> https://github.com/kernel-patches/bpf/actions/runs/14182247375
>
> Do you know if something changed in the meantime?

Yeah, I just disabled the tests failing on s390x for now to unblock
the CI. This is temporary.

A little off-topic: it looks like ebpf runners are offline again,
could be due to recent github runner version bump.

>
> Best regards,
> Ilya

Re: s390x: selftests/bpf are failing on CI

[Vasily Gorbik]

On Tue, Apr 01, 2025 at 05:16:28PM +0200, iii wrote:
> On 2025-04-01 10:06, iii wrote:
> > On 2025-04-01 00:45, Ihor Solodrai wrote:
> > > On 3/31/25 3:25 PM, iii wrote:
> > > > On 2025-03-31 20:25, Ihor Solodrai wrote:
> > > > > Hi Ilya,
> > > > > 
> > > > > After recent merges from upstream, CI started failing both
> > > > > on bpf and
> > > > > bpf-next trees. Yonghong Song and Song Liu submitted a
> > > > > couple of fixes
> > > > > that are already applied to bpf tree, but there are still
> > > > > failures on
> > > > > s390x.
> > > > > 
> > > > > https://github.com/kernel-patches/bpf/actions/runs/14163772245
> > > > > 
> > > > > Could you please investigate?
> > > > > 
> > > > > [...]
> 
> I could not reproduce this on a s390x machine, so I downloaded the CI
> artifacts and noticed that
> /sys/kernel/debug/tracing/available_filter_functions is empty.
> Turns out this is caused by cross-compilation.
> I have let our kernel team (on Cc:) know and they will investigate this
> further.

There must be an endianness issue in the sorttable tool. Reverting just
the commit fa1518875286 ("s390: Sort mcount locations at build time")
fixes the issue. I'll take a look.

Re: s390x: selftests/bpf are failing on CI

[Ihor Solodrai]

On 4/1/25 8:53 AM, Vasily Gorbik wrote:
> On Tue, Apr 01, 2025 at 05:16:28PM +0200, iii wrote:
>> On 2025-04-01 10:06, iii wrote:
>>> On 2025-04-01 00:45, Ihor Solodrai wrote:
>>>> On 3/31/25 3:25 PM, iii wrote:
>>>>> On 2025-03-31 20:25, Ihor Solodrai wrote:
>>>>>> Hi Ilya,
>>>>>>
>>>>>> After recent merges from upstream, CI started failing both
>>>>>> on bpf and
>>>>>> bpf-next trees. Yonghong Song and Song Liu submitted a
>>>>>> couple of fixes
>>>>>> that are already applied to bpf tree, but there are still
>>>>>> failures on
>>>>>> s390x.
>>>>>>
>>>>>> https://github.com/kernel-patches/bpf/actions/runs/14163772245
>>>>>>
>>>>>> Could you please investigate?
>>>>>>
>>>>>> [...]
>>
>> I could not reproduce this on a s390x machine, so I downloaded the CI
>> artifacts and noticed that
>> /sys/kernel/debug/tracing/available_filter_functions is empty.
>> Turns out this is caused by cross-compilation.
>> I have let our kernel team (on Cc:) know and they will investigate this
>> further.
>
> There must be an endianness issue in the sorttable tool. Reverting just
> the commit fa1518875286 ("s390: Sort mcount locations at build time")
> fixes the issue. I'll take a look.

Hi Vasily.

I can confirm that reverting fa1518875286 fixes the selftests
failures. I added a revert commit as a CI-specific patch for now.

Please keep me in cc.

Thank you for the help!

Re: s390x: selftests/bpf are failing on CI

[Ihor Solodrai]

On 4/1/25 1:06 AM, iii wrote:
> On 2025-04-01 00:45, Ihor Solodrai wrote:
>> [...]
>> A little off-topic: it looks like ebpf runners are offline again,
>> could be due to recent github runner version bump. 
>
> The new docker image looks malformed:
>
> $ docker run -it --rm ghcr.io/kernel-patches/runner:main-noble-s390x
> exec /entrypoint.sh: exec format error
>
> $ docker run -it --rm --entrypoint=/bin/sh ghcr.io/kernel-patches/runner:main-noble-s390x
> # cat /entrypoint.sh
> 404: Not Found

Please make sure you're using an up-to-date image.

Link: https://github.com/kernel-patches/runner/pkgs/container/runner/385557209?tag=main-noble-s390x
Digest: sha256:946298c96c51af666f8747be3afcf9a8e59483c40cf790cd94939abf6ec22969

A malformed image was pushed on 2025-03-20, then reverted and later
updated correctly.

If you're using the same ansible setup that we do, restarting the
systemd service controlling the runners should help.


>
>>
>>>
>>> Best regards,
>>> Ilya