Re: 2.6.23-rc8-mm2 - tcp_fastretrans_alert() WARNING

Cedric Le Goater
2007-10-01T09:27:46Z

Ilpo Järvinen wrote:
> On Sat, 29 Sep 2007, Cedric Le Goater wrote:
>
>> Ilpo Järvinen wrote:
>>> On Fri, 28 Sep 2007, Ilpo Järvinen wrote:
>>>> On Fri, 28 Sep 2007, Cedric Le Goater wrote:
>>>>
>>>>> I just found that warning in my logs. It seems that it's been
>>>>> happening since rc7-mm1 at least.
>>>>>
>>>>> WARNING: at /home/legoater/linux/2.6.23-rc8-mm2/net/ipv4/tcp_input.c:2314 tcp_fastretrans_alert()
>>>>>
>>>>> Call Trace:
>>>>> <IRQ> [<ffffffff8040fdc3>] tcp_ack+0xcd6/0x1894
>>>>> ...snip...
>>>> ...Thanks for the report, I'll have look what could still break
>>>> fackets_out...
>>> I think this one is now clear to me, tcp_fragment/collapse adjusts
>>> fackets_out (incorrectly) also for reno flow when there were some dupACKs
>>> that made sacked_out != 0. Could you please try if patch below proves all
>>> them to be of non-SACK origin... In case that's true, it's rather
>>> harmless, I'll send a fix on Monday or so (this would anyway be needed)...
>>> If you find out that them occur with SACK enabled flow, that would be
>>> more interesting and requires more digging...
>> I'm trying now to reproduce this WARNING.
>>
>> It seems that the n/w behaves differently during the week ends. Probably
>> taking a break.
>
> Thanks.
>
> Of course there are other means too to determine if TCP flows do negotiate
> SACK enabled or not. Depending on your test case (which is fully unknown
> to me) they may or may not be usable... At least the value of tcp_sack
> sysctl on both systems or tcpdump catching SYN packets should give that
> detail. ...If you know to which hosts TCP could be connected (and active)
> to, while the WARNING triggers, it's really easy to test what is being
> negotiated as it's unlikely to change at short notice and any TCP flow to
> that host will get us the same information though the WARNING would not be
> triggered with it at this time. Obviously if at least one of the remotes
> is not known or the set ends up being mixture of reno and SACK flows, then
> we'll just have to wait and see which fish we get...

got it !

r3-06.test.meiosys.com login: WARNING: at /home/legoater/linux/2.6.23-rc8-mm2/net/ipv4/tcp_input.c:2314 tcp_fastretrans_alert()

Call Trace:
<IRQ> [<ffffffff8040fdc3>] tcp_ack+0xcd6/0x18af
[<ffffffff80412b6f>] tcp_rcv_established+0x61f/0x6df
[<ffffffff80254146>] __lock_acquire+0x8a1/0xf1b
[<ffffffff80419d19>] tcp_v4_do_rcv+0x3e/0x394
[<ffffffff8041a68b>] tcp_v4_rcv+0x61c/0x9a9
[<ffffffff803ff1e3>] ip_local_deliver+0x1da/0x2a4
[<ffffffff803ffb4e>] ip_rcv+0x583/0x5c9
[<ffffffff8046d35b>] packet_rcv_spkt+0x19a/0x1a8
[<ffffffff803e081c>] netif_receive_skb+0x2cf/0x2f5
[<ffffffff88042505>] :tg3:tg3_poll+0x65d/0x8a4
[<ffffffff803e09e8>] net_rx_action+0xb8/0x191
[<ffffffff8023a927>] __do_softirq+0x5f/0xe0
[<ffffffff8020c98c>] call_softirq+0x1c/0x28
[<ffffffff8020e9c3>] do_softirq+0x3b/0xb8
[<ffffffff8023aa1e>] irq_exit+0x4e/0x50
[<ffffffff8020e7df>] do_IRQ+0xbd/0xd7
[<ffffffff80209cb9>] mwait_idle+0x0/0x4d
[<ffffffff8020bce6>] ret_from_intr+0x0/0xf
<EOI> [<ffffffff80209cfc>] mwait_idle+0x43/0x4d
[<ffffffff802099fb>] enter_idle+0x22/0x24
[<ffffffff80209c4f>] cpu_idle+0x9d/0xc0
[<ffffffff80476aa1>] rest_init+0x55/0x57
[<ffffffff80630815>] start_kernel+0x2d6/0x2e2
[<ffffffff80630134>] _sinittext+0x134/0x13b

TCP 0

I wasn't doing any particular test on n/w so it took me a while to figure
out how I was triggering the WARNING. Apparently, this is happening when I
run ketchup, but not always. This test machine is behind many firewall &
routers so it might be a reason.

tcpdump gave me this output for a wget on kernel.org :

10:51:14.835981 IP r3-06.test.meiosys.com.40322 > pub2.kernel.org.http: S 737836267:737836267(0) win 5840 <mss 1460,sackOK,timestamp 1309245 0,nop,wscale 7>
10:51:14.975153 IP pub2.kernel.org.http > r3-06.test.meiosys.com.40321: F 524:524(0) ack 166 win 5840
10:51:14.975177 IP r3-06.test.meiosys.com.40321 > pub2.kernel.org.http: . ack 525 win 7504

I'm trying to get the WARNING and the tcpdump output for it but for the
moment, it seems it's beyond my reach :/

Hope it helps !

C.