BUG? flaky "migrate_reuseport/IPv4 TCP_NEW_SYN_RECV reqsk_timer_handler" test

BUG? flaky "migrate_reuseport/IPv4 TCP_NEW_SYN_RECV reqsk_timer_handler" test

[Martin KaFai Lau]

Hi Kuniyuki,

The bpf CI has recently hit failure in the "migrate_reuseport/IPv4 
TCP_NEW_SYN_RECV" test. It does not always fail but becomes more flaky recently:
https://github.com/kernel-patches/bpf/actions/runs/8354884067/job/22869153115

It could be due to some slowness in the bpf CI environment but failing because 
of environment slowness is still not expected.

I took a very quick look. It seems like the test depends on the firing of the 
timer's handler "reqsk_timer_handler()", so there is a sleep(1) in the test.
May be the timer fired slower and the test failed? If that is the case, it may 
help to directly trace the reqsk_timer_handler() and wait for enough time to 
ensure it is called.

This test has been temporarily disabled for now 
(https://github.com/kernel-patches/vmtest/blob/master/ci/vmtest/configs/DENYLIST). 
Once you have a fix, we can re-enable it again to ensure this migrate feature 
will not regress.

Thanks,
Martin

Re: BUG? flaky "migrate_reuseport/IPv4 TCP_NEW_SYN_RECV reqsk_timer_handler" test

[Jakub Kicinski]

On Thu, 21 Mar 2024 18:39:53 -0700 Martin KaFai Lau wrote:
> The bpf CI has recently hit failure in the "migrate_reuseport/IPv4 
> TCP_NEW_SYN_RECV" test. It does not always fail but becomes more flaky recently:
> https://github.com/kernel-patches/bpf/actions/runs/8354884067/job/22869153115

FWIW we had a handful of IPv6 and TCP tests start failing after we
merged first half of the merge window from Linus as well.
Maybe something in the scheduler changed and exposed timing problems
in the tests :(

Re: BUG? flaky "migrate_reuseport/IPv4 TCP_NEW_SYN_RECV reqsk_timer_handler" test

[Kuniyuki Iwashima]

Hi Martin,

Thanks for the report.

From: Martin KaFai Lau <martin.lau@linux.dev>
Date: Thu, 21 Mar 2024 18:39:53 -0700
> Hi Kuniyuki,
> 
> The bpf CI has recently hit failure in the "migrate_reuseport/IPv4 
> TCP_NEW_SYN_RECV" test. It does not always fail but becomes more flaky recently:
> https://github.com/kernel-patches/bpf/actions/runs/8354884067/job/22869153115
> 
> It could be due to some slowness in the bpf CI environment but failing because 
> of environment slowness is still not expected.
> 
> I took a very quick look. It seems like the test depends on the firing of the 
> timer's handler "reqsk_timer_handler()", so there is a sleep(1) in the test.

Yes, it depends on the timing of the reqsk tiemr.


> May be the timer fired slower and the test failed? If that is the case, it may 
> help to directly trace the reqsk_timer_handler() and wait for enough time to 
> ensure it is called.

Sounds good.  I'll give it a try.

Thanks!

> 
> This test has been temporarily disabled for now 
> (https://github.com/kernel-patches/vmtest/blob/master/ci/vmtest/configs/DENYLIST). 
> Once you have a fix, we can re-enable it again to ensure this migrate feature 
> will not regress.
> 
> Thanks,
> Martin