2.4.16: TCP shutdown generating infinite ACK storm

2.4.16: TCP shutdown generating infinite ACK storm

[Jeremy Fitzhardinge]

Hi,

I'm looking at my box and a server at Tom's Hardware pounding each other
with packets.  It looks like Linux has got confused about sequence
numbers (or maybe the other end is confused?).

02:51:20.405848 ad.tomshardware.com.http > ixodes.goop.org.33708: . ack 1 win 17376 <nop,nop,timestamp 14994023 3826698> (DF)
02:51:20.405880 ixodes.goop.org.33708 > ad.tomshardware.com.http: . ack 4294920827 win 6432 <nop,nop,timestamp 3826706 14993505> (DF)
02:51:20.415964 ad.tomshardware.com.http > ixodes.goop.org.33708: . ack 1 win 17376 <nop,nop,timestamp 14994023 3826699> (DF)
02:51:20.415995 ixodes.goop.org.33708 > ad.tomshardware.com.http: . ack 4294920827 win 6432 <nop,nop,timestamp 3826707 14993505> (DF)
02:51:20.422607 ad.tomshardware.com.http > ixodes.goop.org.33708: . ack 1 win 17376 <nop,nop,timestamp 14994023 3826700> (DF)
02:51:20.422638 ixodes.goop.org.33708 > ad.tomshardware.com.http: . ack 4294920827 win 6432 <nop,nop,timestamp 3826707 14993505> (DF)
02:51:20.429027 ad.tomshardware.com.http > ixodes.goop.org.33708: . ack 1 win 17376 <nop,nop,timestamp 14994023 3826700> (DF)
02:51:20.429057 ixodes.goop.org.33708 > ad.tomshardware.com.http: . ack 4294920827 win 6432 <nop,nop,timestamp 3826708 14993505> (DF)
02:51:20.435654 ad.tomshardware.com.http > ixodes.goop.org.33708: . ack 1 win 17376 <nop,nop,timestamp 14994023 3826701> (DF)
02:51:20.435684 ixodes.goop.org.33708 > ad.tomshardware.com.http: . ack 4294920827 win 6432 <nop,nop,timestamp 3826709 14993505> (DF)
02:51:20.446010 ad.tomshardware.com.http > ixodes.goop.org.33708: . ack 1 win 17376 <nop,nop,timestamp 14994023 3826702> (DF)
02:51:20.446041 ixodes.goop.org.33708 > ad.tomshardware.com.http: . ack 4294920827 win 6432 <nop,nop,timestamp 3826710 14993505> (DF)
02:51:20.462525 ad.tomshardware.com.http > ixodes.goop.org.33708: . ack 1 win 17376 <nop,nop,timestamp 14994023 3826704> (DF)
02:51:20.462557 ixodes.goop.org.33708 > ad.tomshardware.com.http: . ack 4294920827 win 6432 <nop,nop,timestamp 3826711 14993505> (DF)
02:51:20.475581 ad.tomshardware.com.http > ixodes.goop.org.33708: . ack 1 win 17376 <nop,nop,timestamp 14994023 3826705> (DF)
02:51:20.475611 ixodes.goop.org.33708 > ad.tomshardware.com.http: . ack 4294920827 win 6432 <nop,nop,timestamp 3826713 14993505> (DF)
02:51:20.482239 ad.tomshardware.com.http > ixodes.goop.org.33708: . ack 1 win 17376 <nop,nop,timestamp 14994023 3826706> (DF)
02:51:20.482271 ixodes.goop.org.33708 > ad.tomshardware.com.http: . ack 4294920827 win 6432 <nop,nop,timestamp 3826713 14993505> (DF)
02:51:20.492100 ad.tomshardware.com.http > ixodes.goop.org.33708: . ack 1 win 17376 <nop,nop,timestamp 14994023 3826707> (DF)
[...]

Naturally, this looks bad.  After a while it seemed to stop of its own
accord, I presume when something timed out of LAST_ACK.  While it was
happening, there were two sockets ixising to ad.tomshardware.com:

tcp        1      1 ixodes.goop.org:33674   ad.tomshardware.co:http LAST_ACK    
tcp        1      1 ixodes.goop.org:33708   ad.tomshardware.co:http LAST_ACK    

I'm using 2.4.16 with no particularly special patches.  The gateway
machine is another 2.4.16 box doing NAT with ipchains emulation.

Any other info needed?

	J

Re: 2.4.16: TCP shutdown generating infinite ACK storm

[Mika Liljeberg]

Jeremy Fitzhardinge wrote:
> I'm looking at my box and a server at Tom's Hardware pounding each other
> with packets.  It looks like Linux has got confused about sequence
> numbers (or maybe the other end is confused?).

Yes, it does look like a sequence synchronization problem.

> Naturally, this looks bad.  After a while it seemed to stop of its own
> accord, I presume when something timed out of LAST_ACK.  While it was
> happening, there were two sockets ixising to ad.tomshardware.com:
> 
> tcp        1      1 ixodes.goop.org:33674   ad.tomshardware.co:http LAST_ACK
> tcp        1      1 ixodes.goop.org:33708   ad.tomshardware.co:http LAST_ACK
> 
> I'm using 2.4.16 with no particularly special patches.  The gateway
> machine is another 2.4.16 box doing NAT with ipchains emulation.
> 
> Any other info needed?

If you can repeat this, you please run tcpdump with the -vv option in
order to get it to display the sequence numbers on ack packets as well.
It would also be good to see the last few packets of the connection (at
least the FIN packets).

BR,

	MikaL

Re: 2.4.16: TCP shutdown generating infinite ACK storm

[Alan Cox]

> with packets.  It looks like Linux has got confused about sequence
> numbers (or maybe the other end is confused?).

Probably  it went through some crap load balancer on the way.

> I'm using 2.4.16 with no particularly special patches.  The gateway
> machine is another 2.4.16 box doing NAT with ipchains emulation.
> 
> Any other info needed?

You need to capture the entire misbehaving session at both ends to really
see what is going on and what is mangling it in the middle