Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+

Ingo Molnar
2008-06-02T09:24:36Z

* Eric Dumazet <dada1@cosmosbay.com> wrote:

>>> 22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62
>>> 22003 listen(4, 10) = 0
>>> 22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0
>>>
>>> i'll queue up your reverts for testing in -tip.

I turned off localhost distcc two days ago and there has not been a
single hung socket since then, so we now know it for sure that without
localhost distcc connections, -tip's QA will not produce any hung
sockets in about 1000 random-kernel-build+boot iterations.

i've added those reverts this morning and added back the localhost
distcc rules - we'll see whether the hung sockets are back.

> I believe Ingo problems come on long lived sockets (were many bytes
> were exchanged between the peers), so I dont think DEFER_ACCEPT is the
> cullprit.
>
> I suggest to enable CONFIG_TIMER_STATS and to check timers, because
> /proc/net/tcp can display apparently large timer values when the timer
> is elapsed (jiffies > icsk->icsk_timeout) and
> jiffies_to_clock_t(timer_expires - jiffies) is then overflowing doing
> a multiply and a divide.

i'm wondering whether your suspicion on broken TCP timers is consistent
with the symptoms i've seen: the hung sockets clearly produced periodic
packet activity every 180 seconds, up to 8 hours, without ever changing
their receive of send queue. So at least a part of the TCP timer
mechanism for that specific stuck socket was working fine.

is there no sysctl or other debug mechanism to somehow get its full TCP
state and the reasons for why it is stuck? I'm wondering how you debug
broken TCP state machines without enabling testers to be able to dump
all state and passing it to developers.

I have a clearly reproducable testcase and i'd like to help out, but the
whole effort is stalled on 'not enough information' it appears. Doing
random reverts might help in truly helpless situations where a bug has
no debuggable state - but this situation seems really routine to me:
it's very difficult to trigger the bug but once it triggers the bug
scenario is stable and analyzable. I'd be glad to test any
instrumentation patch that makes similar scenarios more analyzable.

Ingo