[TEST] conntrack_reverse_clash.sh flakes

[TEST] conntrack_reverse_clash.sh flakes

[Jakub Kicinski]

Hi Florian!

We have a new faster NIPA setup, and now on non-debug builds we see 
a few (4 a week to be exact) flakes in conntrack_reverse_clash.sh

List of flakes from the last 100 runs:
https://netdev.bots.linux.dev/contest.html?pass=0&test=conntrack-reverse-clash-sh

Example:

# selftests: net/netfilter: conntrack_reverse_clash.sh
# Port number changed, wanted 56789 got 5950
# ERROR: SNAT performed without any matching snat rule
# kill: sending signal to 16051 failed: No such process
not ok 1 selftests: net/netfilter: conntrack_reverse_clash.sh # exit=1

Looks like the test also occasionally flaked on the old setup ("remote"
column with "metal" instead of "virt") which is now shut down:

# selftests: net/netfilter: conntrack_reverse_clash.sh
# Port number changed, wanted 56789 got 54630
# Port number changed, wanted 56790 got 25814
# ERROR: SNAT performed without any matching snat rule
not ok 1 selftests: net/netfilter: conntrack_reverse_clash.sh # exit=1

so this isn't new, just more likely now..

Could you TAL when you have spare cycles? (BTW the new setup is owned 
by netdev foundation so I can give you access if that helps).

Re: [TEST] conntrack_reverse_clash.sh flakes

[Jakub Kicinski]

On Sat, 6 Dec 2025 17:51:35 -0800 Jakub Kicinski wrote:
> Hi Florian!
> 
> We have a new faster NIPA setup, and now on non-debug builds we see 
> a few (4 a week to be exact) flakes in conntrack_reverse_clash.sh

Ah, one more, the non-reverse conntrack-clash is SKIPping, occasionally:
https://netdev.bots.linux.dev/contest.html?pass=0&test=conntrack-clash-sh

If the event it's testing is probabilistic could we make it return XFAIL
when it doesn't trigger? We try to reserve SKIP for tests skipped
because tool is missing in env, something isn't built into the kernel
etc.

Re: [TEST] conntrack_reverse_clash.sh flakes

[Florian Westphal]

Jakub Kicinski <kuba@kernel.org> wrote:
> Ah, one more, the non-reverse conntrack-clash is SKIPping, occasionally:
> https://netdev.bots.linux.dev/contest.html?pass=0&test=conntrack-clash-sh
> 
> If the event it's testing is probabilistic could we make it return XFAIL
> when it doesn't trigger? We try to reserve SKIP for tests skipped
> because tool is missing in env, something isn't built into the kernel
> etc.

Sure, I can switch this to xfail.  Its trying to trigger a conntrack
race but there is no way to force this and I did not want the script
to spin for long when trying to trigger the nat clash undo logic.

Re: [TEST] conntrack_reverse_clash.sh flakes

[Florian Westphal]

Jakub Kicinski <kuba@kernel.org> wrote:
> We have a new faster NIPA setup, and now on non-debug builds we see 
> a few (4 a week to be exact) flakes in conntrack_reverse_clash.sh
> 
> List of flakes from the last 100 runs:
> https://netdev.bots.linux.dev/contest.html?pass=0&test=conntrack-reverse-clash-sh
> 
> Example:
> 
> # selftests: net/netfilter: conntrack_reverse_clash.sh
> # Port number changed, wanted 56789 got 5950
> # ERROR: SNAT performed without any matching snat rule
> # kill: sending signal to 16051 failed: No such process
> not ok 1 selftests: net/netfilter: conntrack_reverse_clash.sh # exit=1
> 
> Looks like the test also occasionally flaked on the old setup ("remote"
> column with "metal" instead of "virt") which is now shut down:
> 
> # selftests: net/netfilter: conntrack_reverse_clash.sh
> # Port number changed, wanted 56789 got 54630
> # Port number changed, wanted 56790 got 25814
> # ERROR: SNAT performed without any matching snat rule
> not ok 1 selftests: net/netfilter: conntrack_reverse_clash.sh # exit=1
> 
> so this isn't new, just more likely now..
> 
> Could you TAL when you have spare cycles? (BTW the new setup is owned 
> by netdev foundation so I can give you access if that helps).

No need, I can reproduce this:
# selftests: conntrack_reverse_clash.sh
# Port number changed, wanted 56790 got 64562 from 127.0.0.12
# ERROR: SNAT performed without any matching snat rule
# udp      17 30 src=127.0.0.11 dst=127.0.0.12 sport=56789 dport=56790 [UNREPLIED] src=127.0.0.12 dst=127.0.0.11 sport=56790 dport=56789 mark=0 use=1
# conntrack v1.4.8 (conntrack-tools): 1 flow entries have been shown.
# cpu=0         found=0 invalid=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0 clash_resolve=0 chaintoolong=0
...

Looks like an actual bug to me, will need some time to investigate this.
If its too annoying consider disabling this particular test for now.

Thanks for reporting.