From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755029AbYE3QX5 (ORCPT ); Fri, 30 May 2008 12:23:57 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752574AbYE3QXr (ORCPT ); Fri, 30 May 2008 12:23:47 -0400 Received: from rv-out-0506.google.com ([209.85.198.230]:47259 "EHLO rv-out-0506.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752498AbYE3QXp (ORCPT ); Fri, 30 May 2008 12:23:45 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=MiOisHJuMOPnHCPhNf5RM4QHYW1rLl+3JJ1QY8i9Nvh0uvR0qtb2voRfJhnnT9AU+aN+GPZe69fIPWhMWaJE67xXUiOuPov+7hsXyI5VUW29lemickz9DTihZ3wSukkPmYFEjKbRIAQIyxZ14UMtBDN8w/qSb9Yh+2dwDJdPkzE= Message-ID: <2c0942db0805300923u10bb591am27c73ab4d48466cf@mail.gmail.com> Date: Fri, 30 May 2008 09:23:45 -0700 From: "Ray Lee" To: "Ingo Molnar" Subject: Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+ Cc: "=?UTF-8?Q?Ilpo_J=C3=A4rvinen?=" , LKML , Netdev , "David S. Miller" , "Rafael J. Wysocki" , "Andrew Morton" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline References: <20080526115628.GA31316@elte.hu> X-Google-Sender-Auth: 222fb289b3091f90 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by alpha.home.local id m4UGO5Hl001849 On Mon, May 26, 2008 at 6:28 AM, Ilpo Järvinen wrote:> On Mon, 26 May 2008, Ingo Molnar wrote:>>> in an overnight -tip testruns that is based on recent -git i got two>> stuck TCP connections:>>>> Active Internet connections (w/o servers)>> Proto Recv-Q Send-Q Local Address Foreign Address State>> tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED>> tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED>>>> on a previously reliable machine. That connection has been stuck for 9>> hours so it does not time out, etc. - and the distcc run that goes over>> that connection is stuck as well.>>>> kernel config is attached.>>>> in terms of debugging there's not much i can do i'm afraid. It's not>> possible to get a tcpdump of this incident, given the extreme amount of>> load these testboxes handle.>> ...but you can still tcpdump that particular flow once the situation is> discovered to see if TCP still tries to do something, no? One needs to> tcpdump couple of minutes at minimum. Also please get /proc/net/tcp for> that flow around the same time.>>> This problem started sometime around rc3>> and it occured on two boxes (on a laptop and on a desktop), both are SMP>> Core2Duo based systems. I never saw this problem before on thousands of>> similar bootups, so i'm 99.9% sure the bug is either new or became>> easier to trigger.>>>> It's not possible to bisect it as it needs up to 12 hours of heavy>> workload to trigger. The incident happened about 5 times since the first>> incident a couple of days ago - 4 times on one box and once on another>> box. The first failing head i became aware of was 78b58e549a3098. (-tip>> has other changes beyond -git but changes nothing in networking.) Okay, but in some sense you've already bisected this somewhat. I'massuming that your testing uses the latest tip and is refreshed daily. If that's the case, then I would (possibly naively) expect the culpritto show up in a: git log -p v2.6.26-rc1..78b58e549a3098net/{compat.c,core,ipv4,netfilter,packet,sched,socket.c} There are only a few commits in there that appear to touch network behavior: 79d44516b4b178ffb6e2159c75584cfcfc097914 a1c1f281b84a751fdb5ff919da3b09df7297619f 62ab22278308a40bcb7f4079e9719ab8b7fe11b5 Reverting just those three and running overnight might provide a clue.OTOH, I'm in no way a net/ expert, so if you are already working adebugging strategy then feel free to ignore this. I'm only piping upas it appears that the troubleshooting has stalled. > (but there were> some recent fixes to FRTO and retrans_stamp change could have some> significance here)?>> Other than that, nothing since -rc1 seems suspicious to me (though> I hardly understand every part of networking).{.n++%ݶw{.n+{G{ayʇڙ,jfhz_(階ݢj"mG?&~iOzv^m ?I From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Ray Lee" Subject: Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+ Date: Fri, 30 May 2008 09:23:45 -0700 Message-ID: <2c0942db0805300923u10bb591am27c73ab4d48466cf@mail.gmail.com> References: <20080526115628.GA31316@elte.hu> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Cc: "=?UTF-8?Q?Ilpo_J=C3=A4rvinen?=" , LKML , Netdev , "David S. Miller" , "Rafael J. Wysocki" , "Andrew Morton" To: "Ingo Molnar" Return-path: Received: from rv-out-0506.google.com ([209.85.198.228]:47413 "EHLO rv-out-0506.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752314AbYE3QXq (ORCPT ); Fri, 30 May 2008 12:23:46 -0400 Received: by rv-out-0506.google.com with SMTP id l9so4689481rvb.1 for ; Fri, 30 May 2008 09:23:45 -0700 (PDT) In-Reply-To: Content-Disposition: inline Sender: netdev-owner@vger.kernel.org List-ID: T24gTW9uLCBNYXkgMjYsIDIwMDggYXQgNjoyOCBBTSwgSWxwbyBKw6RydmluZW4KPGlscG8uamFy dmluZW5AaGVsc2lua2kuZmk+IHdyb3RlOgo+IE9uIE1vbiwgMjYgTWF5IDIwMDgsIEluZ28gTW9s bmFyIHdyb3RlOgo+Cj4+IGluIGFuIG92ZXJuaWdodCAtdGlwIHRlc3RydW5zIHRoYXQgaXMgYmFz ZWQgb24gcmVjZW50IC1naXQgaSBnb3QgdHdvCj4+IHN0dWNrIFRDUCBjb25uZWN0aW9uczoKPj4K Pj4gQWN0aXZlIEludGVybmV0IGNvbm5lY3Rpb25zICh3L28gc2VydmVycykKPj4gUHJvdG8gUmVj di1RIFNlbmQtUSBMb2NhbCBBZGRyZXNzICAgICAgICAgICAgICAgRm9yZWlnbiBBZGRyZXNzICAg ICAgICAgICAgIFN0YXRlCj4+IHRjcCAgICAgICAgMCAxNzQ1OTIgMTAuMC4xLjE0OjU4MDE1ICAg ICAgICAgICAgIDEwLjAuMS4xNDozNjMyICAgICAgICAgICAgICBFU1RBQkxJU0hFRAo+PiB0Y3Ag ICAgNzIxMzQgICAgICAwIDEwLjAuMS4xNDozNjMyICAgICAgICAgICAgICAxMC4wLjEuMTQ6NTgw MTUgICAgICAgICAgICAgRVNUQUJMSVNIRUQKPj4KPj4gb24gYSBwcmV2aW91c2x5IHJlbGlhYmxl IG1hY2hpbmUuIFRoYXQgY29ubmVjdGlvbiBoYXMgYmVlbiBzdHVjayBmb3IgOQo+PiBob3VycyBz byBpdCBkb2VzIG5vdCB0aW1lIG91dCwgZXRjLiAtIGFuZCB0aGUgZGlzdGNjIHJ1biB0aGF0IGdv ZXMgb3Zlcgo+PiB0aGF0IGNvbm5lY3Rpb24gaXMgc3R1Y2sgYXMgd2VsbC4KPj4KPj4ga2VybmVs IGNvbmZpZyBpcyBhdHRhY2hlZC4KPj4KPj4gaW4gdGVybXMgb2YgZGVidWdnaW5nIHRoZXJlJ3Mg bm90IG11Y2ggaSBjYW4gZG8gaSdtIGFmcmFpZC4gSXQncyBub3QKPj4gcG9zc2libGUgdG8gZ2V0 IGEgdGNwZHVtcCBvZiB0aGlzIGluY2lkZW50LCBnaXZlbiB0aGUgZXh0cmVtZSBhbW91bnQgb2YK Pj4gbG9hZCB0aGVzZSB0ZXN0Ym94ZXMgaGFuZGxlLgo+Cj4gLi4uYnV0IHlvdSBjYW4gc3RpbGwg dGNwZHVtcCB0aGF0IHBhcnRpY3VsYXIgZmxvdyBvbmNlIHRoZSBzaXR1YXRpb24gaXMKPiBkaXNj b3ZlcmVkIHRvIHNlZSBpZiBUQ1Agc3RpbGwgdHJpZXMgdG8gZG8gc29tZXRoaW5nLCBubz8gT25l IG5lZWRzIHRvCj4gdGNwZHVtcCBjb3VwbGUgb2YgbWludXRlcyBhdCBtaW5pbXVtLiBBbHNvIHBs ZWFzZSBnZXQgL3Byb2MvbmV0L3RjcCBmb3IKPiB0aGF0IGZsb3cgYXJvdW5kIHRoZSBzYW1lIHRp bWUuCj4KPj4gVGhpcyBwcm9ibGVtIHN0YXJ0ZWQgc29tZXRpbWUgYXJvdW5kIHJjMwo+PiBhbmQg aXQgb2NjdXJlZCBvbiB0d28gYm94ZXMgKG9uIGEgbGFwdG9wIGFuZCBvbiBhIGRlc2t0b3ApLCBi b3RoIGFyZSBTTVAKPj4gQ29yZTJEdW8gYmFzZWQgc3lzdGVtcy4gSSBuZXZlciBzYXcgdGhpcyBw cm9ibGVtIGJlZm9yZSBvbiB0aG91c2FuZHMgb2YKPj4gc2ltaWxhciBib290dXBzLCBzbyBpJ20g OTkuOSUgc3VyZSB0aGUgYnVnIGlzIGVpdGhlciBuZXcgb3IgYmVjYW1lCj4+IGVhc2llciB0byB0 cmlnZ2VyLgo+Pgo+PiBJdCdzIG5vdCBwb3NzaWJsZSB0byBiaXNlY3QgaXQgYXMgaXQgbmVlZHMg dXAgdG8gMTIgaG91cnMgb2YgaGVhdnkKPj4gd29ya2xvYWQgdG8gdHJpZ2dlci4gVGhlIGluY2lk ZW50IGhhcHBlbmVkIGFib3V0IDUgdGltZXMgc2luY2UgdGhlIGZpcnN0Cj4+IGluY2lkZW50IGEg Y291cGxlIG9mIGRheXMgYWdvIC0gNCB0aW1lcyBvbiBvbmUgYm94IGFuZCBvbmNlIG9uIGFub3Ro ZXIKPj4gYm94LiBUaGUgZmlyc3QgZmFpbGluZyBoZWFkIGkgYmVjYW1lIGF3YXJlIG9mIHdhcyA3 OGI1OGU1NDlhMzA5OC4gKC10aXAKPj4gaGFzIG90aGVyIGNoYW5nZXMgYmV5b25kIC1naXQgYnV0 IGNoYW5nZXMgbm90aGluZyBpbiBuZXR3b3JraW5nLikKCk9rYXksIGJ1dCBpbiBzb21lIHNlbnNl IHlvdSd2ZSBhbHJlYWR5IGJpc2VjdGVkIHRoaXMgc29tZXdoYXQuIEknbQphc3N1bWluZyB0aGF0 IHlvdXIgdGVzdGluZyB1c2VzIHRoZSBsYXRlc3QgdGlwIGFuZCBpcyByZWZyZXNoZWQgZGFpbHku CgpJZiB0aGF0J3MgdGhlIGNhc2UsIHRoZW4gSSB3b3VsZCAocG9zc2libHkgbmFpdmVseSkgZXhw ZWN0IHRoZSBjdWxwcml0CnRvIHNob3cgdXAgaW4gYToKICBnaXQgbG9nIC1wIHYyLjYuMjYtcmMx Li43OGI1OGU1NDlhMzA5OApuZXQve2NvbXBhdC5jLGNvcmUsaXB2NCxuZXRmaWx0ZXIscGFja2V0 LHNjaGVkLHNvY2tldC5jfQoKVGhlcmUgYXJlIG9ubHkgYSBmZXcgY29tbWl0cyBpbiB0aGVyZSB0 aGF0IGFwcGVhciB0byB0b3VjaCBuZXR3b3JrIGJlaGF2aW9yOgoKICA3OWQ0NDUxNmI0YjE3OGZm YjZlMjE1OWM3NTU4NGNmY2ZjMDk3OTE0CiAgYTFjMWYyODFiODRhNzUxZmRiNWZmOTE5ZGEzYjA5 ZGY3Mjk3NjE5ZgogIDYyYWIyMjI3ODMwOGE0MGJjYjdmNDA3OWU5NzE5YWI4YjdmZTExYjUKClJl dmVydGluZyBqdXN0IHRob3NlIHRocmVlIGFuZCBydW5uaW5nIG92ZXJuaWdodCBtaWdodCBwcm92 aWRlIGEgY2x1ZS4KT1RPSCwgSSdtIGluIG5vIHdheSBhIG5ldC8gZXhwZXJ0LCBzbyBpZiB5b3Ug YXJlIGFscmVhZHkgd29ya2luZyBhCmRlYnVnZ2luZyBzdHJhdGVneSB0aGVuIGZlZWwgZnJlZSB0 byBpZ25vcmUgdGhpcy4gSSdtIG9ubHkgcGlwaW5nIHVwCmFzIGl0IGFwcGVhcnMgdGhhdCB0aGUg dHJvdWJsZXNob290aW5nIGhhcyBzdGFsbGVkLgoKPiAoYnV0IHRoZXJlIHdlcmUKPiBzb21lIHJl Y2VudCBmaXhlcyB0byBGUlRPIGFuZCByZXRyYW5zX3N0YW1wIGNoYW5nZSBjb3VsZCBoYXZlIHNv bWUKPiBzaWduaWZpY2FuY2UgaGVyZSk/Cj4KPiBPdGhlciB0aGFuIHRoYXQsIG5vdGhpbmcgc2lu Y2UgLXJjMSBzZWVtcyBzdXNwaWNpb3VzIHRvIG1lICh0aG91Z2gKPiBJIGhhcmRseSB1bmRlcnN0 YW5kIGV2ZXJ5IHBhcnQgb2YgbmV0d29ya2luZykuCg==