[TEST] nft-flowtable-sh flaking after pulling first chunk of the merge window

[TEST] nft-flowtable-sh flaking after pulling first chunk of the merge window

[Jakub Kicinski]

Hi!

Could be very bad luck but after we fast forwarded net-next yesterday
we have 3 failures in less than 24h in nft_flowtabl.sh:

https://netdev.bots.linux.dev/contest.html?test=nft-flowtable-sh

# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter  2113852 exceeds expected value 2097152, reply counter  60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh/stdout

# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter  3530493 exceeds expected value 3478585, reply counter  60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960022/10-nft-flowtable-sh/stdout

# FAIL: dscp counters do not match, expected dscp3 and dscp0 > 0 but got  1431 , 0 
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh-retry/stdout

Re: [TEST] nft-flowtable-sh flaking after pulling first chunk of the merge window

[Pablo Neira Ayuso]

Hi Jakub,

On Thu, Jan 23, 2025 at 08:04:44AM -0800, Jakub Kicinski wrote:
> Hi!
> 
> Could be very bad luck but after we fast forwarded net-next yesterday
> we have 3 failures in less than 24h in nft_flowtabl.sh:
> 
> https://netdev.bots.linux.dev/contest.html?test=nft-flowtable-sh
> 
> # FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter  2113852 exceeds expected value 2097152, reply counter  60
> https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh/stdout
> 
> # FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter  3530493 exceeds expected value 3478585, reply counter  60
> https://netdev-3.bots.linux.dev/vmksft-nf/results/960022/10-nft-flowtable-sh/stdout
> 
> # FAIL: dscp counters do not match, expected dscp3 and dscp0 > 0 but got  1431 , 0 
> https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh-retry/stdout

Thanks for your report, let me take a look.

Re: [TEST] nft-flowtable-sh flaking after pulling first chunk of the merge window

[Pablo Neira Ayuso]

Hi Jakub,

On Thu, Jan 23, 2025 at 08:04:44AM -0800, Jakub Kicinski wrote:
> Hi!
> 
> Could be very bad luck but after we fast forwarded net-next yesterday
> we have 3 failures in less than 24h in nft_flowtabl.sh:
> 
> https://netdev.bots.linux.dev/contest.html?test=nft-flowtable-sh
> 
> # FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter  2113852 exceeds expected value 2097152, reply counter  60
> https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh/stdout
> 
> # FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter  3530493 exceeds expected value 3478585, reply counter  60
> https://netdev-3.bots.linux.dev/vmksft-nf/results/960022/10-nft-flowtable-sh/stdout

this is reporting a flow in forward chain going over the size of the
file, this is a flow that is not follow flowtable path.

> # FAIL: dscp counters do not match, expected dscp3 and dscp0 > 0 but got  1431 , 0 
> https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh-retry/stdout

this is reporting that occasionally a flow does not follow flowtable
path, dscp3 gets bumped from the forward chain.

I can rarely see this last dscp tests FAIL when running this test in a
loop here.

Just a follow up, I am still diagnosing.

Re: [TEST] nft-flowtable-sh flaking after pulling first chunk of the merge window

[Jakub Kicinski]

On Wed, 29 Jan 2025 12:21:24 +0100 Pablo Neira Ayuso wrote:
> > Could be very bad luck but after we fast forwarded net-next yesterday
> > we have 3 failures in less than 24h in nft_flowtabl.sh:
> > 
> > https://netdev.bots.linux.dev/contest.html?test=nft-flowtable-sh
> > 
> > # FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter  2113852 exceeds expected value 2097152, reply counter  60
> > https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh/stdout
> > 
> > # FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter  3530493 exceeds expected value 3478585, reply counter  60
> > https://netdev-3.bots.linux.dev/vmksft-nf/results/960022/10-nft-flowtable-sh/stdout  
> 
> this is reporting a flow in forward chain going over the size of the
> file, this is a flow that is not follow flowtable path.
> 
> > # FAIL: dscp counters do not match, expected dscp3 and dscp0 > 0 but got  1431 , 0 
> > https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh-retry/stdout  
> 
> this is reporting that occasionally a flow does not follow flowtable
> path, dscp3 gets bumped from the forward chain.
> 
> I can rarely see this last dscp tests FAIL when running this test in a
> loop here.
> 
> Just a follow up, I am still diagnosing.

Thanks for the update!

FWIW we hit 4 more flakes since I reported it to you last week
(first link from previous message will take you to them).
All four in dscp_fwd

Re: [TEST] nft-flowtable-sh flaking after pulling first chunk of the merge window

[Pablo Neira Ayuso]

Hi Jakub,

On Wed, Jan 29, 2025 at 05:00:57PM -0800, Jakub Kicinski wrote:
> On Wed, 29 Jan 2025 12:21:24 +0100 Pablo Neira Ayuso wrote:
> > > Could be very bad luck but after we fast forwarded net-next yesterday
> > > we have 3 failures in less than 24h in nft_flowtabl.sh:
> > > 
> > > https://netdev.bots.linux.dev/contest.html?test=nft-flowtable-sh
> > > 
> > > # FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter  2113852 exceeds expected value 2097152, reply counter  60
> > > https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh/stdout
> > > 
> > > # FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter  3530493 exceeds expected value 3478585, reply counter  60
> > > https://netdev-3.bots.linux.dev/vmksft-nf/results/960022/10-nft-flowtable-sh/stdout  
> > 
> > this is reporting a flow in forward chain going over the size of the
> > file, this is a flow that is not follow flowtable path.
> > 
> > > # FAIL: dscp counters do not match, expected dscp3 and dscp0 > 0 but got  1431 , 0 
> > > https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh-retry/stdout  
> > 
> > this is reporting that occasionally a flow does not follow flowtable
> > path, dscp3 gets bumped from the forward chain.
> > 
> > I can rarely see this last dscp tests FAIL when running this test in a
> > loop here.
> > 
> > Just a follow up, I am still diagnosing.
> 
> Thanks for the update!
> 
> FWIW we hit 4 more flakes since I reported it to you last week
> (first link from previous message will take you to them).
> All four in dscp_fwd

Just another follow up on this. I am testing here a revert of:

  b8baac3b9c5c ("netfilter: flowtable: teardown flow if cached mtu is stale")

nft_flowtable.sh shows too frequent re-offloads (create/teardown
cycles) with fragments that can lead no packets following the
flowtable path as dscp_fwd reports.

Let me give it more testing then, if results are positive, I will
formally propose this revert.

Thanks.