Re: [bisected] xfrm: TCP connection initiating PMTU discovery stalls on v3.

Wolfgang Walter
2014-12-09T20:36:23Z

Am Dienstag, 9. Dezember 2014, 06:26:49 schrieb Eric Dumazet:
> On Tue, 2014-12-09 at 09:54 +0100, Thomas Jarosch wrote:
> > On Monday, 8. December 2014 23:20:42 Wolfgang Walter wrote:
> > > Am Freitag, 5. Dezember 2014, 05:26:25 schrieb Eric Dumazet:
> > > > On Fri, 2014-12-05 at 13:09 +0100, Wolfgang Walter wrote:
> > > > > Hello,
> > > > >
> > > > > as reverting this patch fixes this rather annoying problem: is it
> > > > > dangerous to revert it as a workaround until the root cause is
> > > > > found?
> > > >
> > > > Unfortunately no, this patch fixes a serious issue.
> > > >
> > > > We need to find the root cause of your problem instead of trying to
> > > > work
> > > > around it.
> > >
> > > I only wanted to use it as local workaround here.
> > >
> > >
> > > I looked a bit at at code. I'm not familiar with the network code,
> > > though
> > >
> > > :-).
> >
> > If it helps, I'm running the reverted patch on five production boxes
> > hitherto without a hiccup. As far as I understood the original commit
> > message, some packet counters might me wrong without it.
> >
> > @Eric: What could possibly go wrong(tm)? :)
>
> Crashes in TCP stack, because of packet count mismatches.
>
> The sk_can_gso() status is already tested in tcp_sendmsg() as a hint,
> since path behavior can dynamically be changed on existing flow :
>
> <start a TCP flow>
> ethtool -K eth0 tso off gso off
>
> In this case, core networking stack detects this and segments the
> packets _after_ TCP or IP stack, before they reach eth0.
>
> TCP stack does not have to know that something is changed right before
> giving a GSO packet to core networking stack, this would be racy by
> nature, as TCP does not know or control full path. Hopefully we do not
> take RTNL for every packet we send in TCP !
>
> It seems XFRM triggers in a slow path something which is not correctly
> handled.
>
> It is not correct to add a racy kludge in TCP fast path for this very
> unlikely case.
>
> I would disable TSO/GSO on xfrm, and problem should disappear.

How would that be done? I found no way to disable it especially for xfrm. I
disabled gso for the interface which serves the ipsec traffic but this does
not help. tcp still uses gso for the esp tunnel.

I put a view printk's in net/xfrm/xfrm_output.c and net/ipv4/tcp_output.c. (I
try to understand where in the xfrm transformation gso is handeled).

What I can say yet is:

xfrm_output() is used with ipsec (esp) tunnel mode but at I never see gso
packets here. xfrm_output_gso() is never called.

Everytime tcp_set_skb_tso_segs() is called for a tcp connection over the esp-
tunnel and it is a gso case then the tcp connection hangs. Those packets
always have skb->len 1398 and mss_now is 1374. I see a call of xfrm_output()
afterwards but for a packet of skb->len 52 (maybe ACK from other direction?).

As long as the tcp-connection over the ipsec-tunnel works and if I send bulk
traffic xfrm_output() is called 3 times with packet skb->len 1426 and then one
time with 78 (maybe other direction?), don't know if that is of any interest.

With non-ipsec-traffic gso works fine: in this case the skb->len() varies a
lot and mss_now is always 1288.

Regards,
--
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts