Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+

Ilpo Järvinen
2008-05-26T13:28:36Z

On Mon, 26 May 2008, Ingo Molnar wrote:

> in an overnight -tip testruns that is based on recent -git i got two
> stuck TCP connections:
>
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
> tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED
>
> on a previously reliable machine. That connection has been stuck for 9
> hours so it does not time out, etc. - and the distcc run that goes over
> that connection is stuck as well.
>
> kernel config is attached.
>
> in terms of debugging there's not much i can do i'm afraid. It's not
> possible to get a tcpdump of this incident, given the extreme amount of
> load these testboxes handle.

...but you can still tcpdump that particular flow once the situation is
discovered to see if TCP still tries to do something, no? One needs to
tcpdump couple of minutes at minimum. Also please get /proc/net/tcp for
that flow around the same time.

> This problem started sometime around rc3
> and it occured on two boxes (on a laptop and on a desktop), both are SMP
> Core2Duo based systems. I never saw this problem before on thousands of
> similar bootups, so i'm 99.9% sure the bug is either new or became
> easier to trigger.
>
> It's not possible to bisect it as it needs up to 12 hours of heavy
> workload to trigger. The incident happened about 5 times since the first
> incident a couple of days ago - 4 times on one box and once on another
> box. The first failing head i became aware of was 78b58e549a3098. (-tip
> has other changes beyond -git but changes nothing in networking.)
>
> One clue (which might or might not matter) is that distcc is one of the
> very few applications that makes use of sendfile().

Can you please try with /proc/sys/net/ipv4/tcp_frto set to zero though
recv-q symptom seems weird would it be related to that (but there were
some recent fixes to FRTO and retrans_stamp change could have some
significance here)?

Other than that, nothing since -rc1 seems suspicious to me (though
I hardly understand every part of networking).

--
i.