Re: [PATCH bpf-next v3 0/8] Support bpf trampoline for s390x - CI issue

Joanne Koong
2023-02-02T18:31:54Z

On Tue, Jan 31, 2023 at 5:47 AM Ilya Leoshkevich <iii@linux.ibm.com> wrote:
>
> On Mon, 2023-01-30 at 19:13 -0800, Alexei Starovoitov wrote:
> > On Mon, Jan 30, 2023 at 10:56 AM Ilya Leoshkevich <iii@linux.ibm.com>
> > wrote:
> > >
> > > On Sun, 2023-01-29 at 19:28 -0800, Alexei Starovoitov wrote:
> > > > On Sun, Jan 29, 2023 at 11:05 AM Ilya Leoshkevich
> > > > <iii@linux.ibm.com>
> > > > wrote:
> > > > >
> > > > > v2:
> > > > > https://lore.kernel.org/bpf/20230128000650.1516334-1-iii@linux.ibm.com/#t
> > > > > v2 -> v3:
> > > > > - Make __arch_prepare_bpf_trampoline static.
> > > > > (Reported-by: kernel test robot <lkp@intel.com>)
> > > > > - Support both old- and new- style map definitions in
> > > > > sk_assign.
> > > > > (Alexei)
> > > > > - Trim DENYLIST.s390x. (Alexei)
> > > > > - Adjust s390x vmlinux path in vmtest.sh.
> > > > > - Drop merged fixes.
> > > >
> > > > It looks great. Applied.
> > > >
> > > > Sadly clang repo is unreliable today. I've kicked BPF CI multiple
> > > > times,
> > > > but it didn't manage to fetch the clang. Pushed anyway.
> > > > Pls watch for BPF CI failures in future runs.
> > >
> > > I think this is because llvm-toolchain-focal contains llvm 17 now.
> > > So we need to either use llvm-toolchain-focal-16, or set
> > > llvm_default_version=16 in libbpf/ci.
> >
> > Yep. That was fixed.
> > Looks like only one test is failing on s390:
> > test_synproxy:PASS:./xdp_synproxy --iface tmp1 --single 0 nsec
> > expect_str:FAIL:SYNACKs after connection unexpected SYNACKs after
> > connection: actual '' != expected 'Total SYNACKs generated: 1\x0A'
> >
> > #284/1 xdp_synproxy/xdp:FAIL
> > #284 xdp_synproxy:FAIL
> > Summary: 260/1530 PASSED, 31 SKIPPED, 1 FAILED
>
> Thanks! Where do you see the xdp_synproxy failure? I checked the jobs
> at [1] and rather see two migrate_reuseport failures ([2], [3]):

Hi Ilya,

I'm seeing these xdp_synproxy failures consistently in CI on
"test_progs/test_progs_no_alu32 on s390x with gcc" builds. These links
are to some of the latest ones:

https://github.com/kernel-patches/bpf/actions/runs/4074723783/jobs/7021760646
https://github.com/kernel-patches/bpf/actions/runs/4073866949/jobs/7019322847
https://github.com/kernel-patches/bpf/actions/runs/4073861356/jobs/7018721175

>
> count_requests:FAIL:count in BPF prog unexpected count in BPF prog:
> actual 10 != expected 25
> #127/7 migrate_reuseport/IPv6 TCP_NEW_SYN_RECV
> reqsk_timer_handler:FAIL
>
> count_requests:FAIL:count in BPF prog unexpected count in BPF prog:
> actual 14 != expected 25
> #127/4 migrate_reuseport/IPv4 TCP_NEW_SYN_RECV
> inet_csk_complete_hashdance:FAIL
>
> I tried running vmtest.sh in a loop, and could not reproduce neither
> the xdp_synproxy nor the migrate_reuseport failure.
>
> In migrate_reuseport, from the userspace perspective everything works,
> (count_requests:PASS:count in userspace 0 nsec). This means that we
> always get to the bpf_sk_select_reuseport() call and it succeeds.
> The eBPF program still records at least some migrations while the
> connection is in the TCP_NEW_SYN_RECV state, so I wonder if other
> migrations, for whatever reason, happen in a different state?
>
> [1] https://github.com/libbpf/libbpf/actions/workflows/test.yml
> [2]
> https://github.com/libbpf/libbpf/actions/runs/4049227053/jobs/6965361085#step:30:8908
> [3]
> https://github.com/libbpf/libbpf/actions/runs/4049783307/jobs/6966526594#step:30:8911