Re: Flowtable race condition error

[Pablo Neira Ayuso <pablo@netfilter.org>]

Hi Sven,

On Tue, Mar 12, 2024 at 05:29:45PM +0100, Sven Auhagen wrote:
> Hi,
> 
> I have a race condition problem in the flowtable and could
> use some hint where to start debugging.
> 
> Every now and then a TCP FIN is closing the flowtable with a call
> to flow_offload_teardown.
> 
> Right after another packet from the reply direction is readding
> the connection to the flowtable just before the FIN is actually
> transitioning the state from ESTABLISHED to FIN WAIT.
> Now the FIN WAIT connection is OFFLOADED.

Are you restricting your ruleset to only offload new connections?

Or is it conntrack creating a fresh connection that being offloaded
for this terminating TCP traffic what you are observing?

> This by itself should work itself out at gc time but
> the state is now deleted right away.
>
> Any idea why the state is deleted right away?

It might be conntrack which is killing the connection, it would be
good to have a nf_ct_kill_reason(). Last time we talk, NAT can also
kill the conntrack in masquerade scenarios.

> Here is the output of the state messages:
> 
>     [NEW] tcp      6 120 SYN_SENT src=192.168.97.23 dst=192.168.107.52 sport=63482 dport=443 [UNREPLIED] src=192.168.107.52 dst=192.168.97.23 sport=443 dport=63482 mark=92274785
>  [UPDATE] tcp      6 60 SYN_RECV src=192.168.97.23 dst=192.168.107.52 sport=63482 dport=443 src=192.168.107.52 dst=192.168.97.23 sport=443 dport=63482 mark=92274785
>  [UPDATE] tcp      6 432000 ESTABLISHED src=192.168.97.23 dst=192.168.107.52 sport=63482 dport=443 src=192.168.107.52 dst=192.168.97.23 sport=443 dport=63482 [OFFLOAD] mark=92274785
>  [UPDATE] tcp      6 86400 FIN_WAIT src=192.168.97.23 dst=192.168.107.52 sport=63482 dport=443 src=192.168.107.52 dst=192.168.97.23 sport=443 dport=63482 [OFFLOAD] mark=92274785
> [DESTROY] tcp      6 FIN_WAIT src=192.168.97.23 dst=192.168.107.52 sport=63482 dport=443 packets=10 bytes=1415 src=192.168.107.52 dst=192.168.97.23 sport=443 dport=63482 packets=11 bytes=6343 [ASSURED] mark=92274785 delta-time=0

Is there a [NEW] event after this [DESTROY] in FIN_WAIT state to pick
the terminating connection from the middle?

b6f27d322a0a ("netfilter: nf_flow_table: tear down TCP flows if RST or
FIN was seen") to let conntrack close the connection gracefully,
otherwise flowtable becomes stateless and already finished connections
remain in place which affects features such as connlimit.

The intention in that patch is to remove the entry from the flowtable
then hand over back the conntrack to the connection tracking system
following slow path.