Re: [PATCH net] net: drop secpath extension before skb deferral free

[Jianbo Liu <jianbol@nvidia.com>]

On Mon, 2024-05-13 at 12:29 +0200, Eric Dumazet wrote:
> On Mon, May 13, 2024 at 12:04 PM Jianbo Liu <jianbol@nvidia.com> wrote:
> > 
> > In commit 68822bdf76f1 ("net: generalize skb freeing deferral to
> > per-cpu lists"), skb can be queued on remote cpu list for deferral
> > free.
> > 
> > The remote cpu is kicked if the queue reaches half capacity. As
> > mentioned in the patch, this seems very unlikely to trigger
> > NET_RX_SOFTIRQ on the remote CPU in this way. But that seems not
> > true,
> > we actually saw something that indicates this: skb is not freed
> > immediately, or even kept for a long time. And the possibility is
> > increased if there are more cpu cores.
> > 
> > As skb is not freed, its extension is not freed as well. An error
> > occurred while unloading the driver after running TCP traffic with
> > IPsec, where both crypto and packet were offloaded. However, in the
> > case of crypto offload, this failure was rare and significantly more
> > challenging to replicate.
> > 
> >  unregister_netdevice: waiting for eth2 to become free. Usage count =
> > 2
> >  ref_tracker: eth%d@000000007421424b has 1/1 users at
> >       xfrm_dev_state_add+0xe5/0x4d0
> >       xfrm_add_sa+0xc5c/0x11e0
> >       xfrm_user_rcv_msg+0xfa/0x240
> >       netlink_rcv_skb+0x54/0x100
> >       xfrm_netlink_rcv+0x31/0x40
> >       netlink_unicast+0x1fc/0x2c0
> >       netlink_sendmsg+0x232/0x4a0
> >       __sock_sendmsg+0x38/0x60
> >       ____sys_sendmsg+0x1e3/0x200
> >       ___sys_sendmsg+0x80/0xc0
> >       __sys_sendmsg+0x51/0x90
> >       do_syscall_64+0x40/0xe0
> >       entry_SYSCALL_64_after_hwframe+0x46/0x4e
> > 
> > The ref_tracker shows the netdev is hold when the offloading xfrm
> > state is first added to hardware. When receiving packet, the secpath
> > extension, which saves xfrm state, is added to skb by ipsec offload,
> > and the xfrm state is hence hold by the received skb. It can't be
> > flushed till skb is dequeued from the defer list, then skb and its
> > extension are really freed. Also, the netdev can't be unregistered
> > because it still referred by xfrm state.
> > 
> > To fix this issue, drop this extension before skb is queued to the
> > defer list, so xfrm state destruction is not blocked.
> > 
> > Fixes: 68822bdf76f1 ("net: generalize skb freeing deferral to per-cpu
> > lists")
> > Signed-off-by: Jianbo Liu <jianbol@nvidia.com>
> > Reviewed-by: Leon Romanovsky <leonro@nvidia.com>
> > ---
> 
> 
> This attribution and patch seem wrong. Also you should CC XFRM
> maintainers.
> 
> Before being freed from tcp_recvmsg() path, packets can sit in TCP
> receive queues for arbitrary amounts of time.
> 
> secpath_reset() should be called much earlier than in the code you
> tried to change.

Yes, this also fixed the issue if I moved secpatch_reset() before
tcp_v4_do_rcv().
 
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -2314,6 +2314,7 @@ int tcp_v4_rcv(struct sk_buff *skb)
 	tcp_v4_fill_cb(skb, iph, th);
 
 	skb->dev = NULL;
+	secpath_reset(skb);
 
 	if (sk->sk_state == TCP_LISTEN) {
 		ret = tcp_v4_do_rcv(sk, skb);

Do you want me to send v2, or push a new one if you agree with this
change?

Thanks!
Jianbo

> 
> If XFRM state can stick to packets stored in protocol queues, we have
> a much bigger problem.
> 
> I suspect all callers of nf_reset_ct() need a fix of some kind.