From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754643AbYEaO0A (ORCPT ); Sat, 31 May 2008 10:26:00 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752641AbYEaOZu (ORCPT ); Sat, 31 May 2008 10:25:50 -0400 Received: from rv-out-0506.google.com ([209.85.198.238]:12249 "EHLO rv-out-0506.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752586AbYEaOZt (ORCPT ); Sat, 31 May 2008 10:25:49 -0400 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=MJbuYCBAv+9BpFUs51O9Kynald/Rf+7DCFLsXG30aC7LCP9yIjv1hR3k1iq3hq4ML/Wsp0czgjAHHDBajcy4wkacKunGxuPNZC8+Jtqjo09ZyR+9Nk5KzzxEbrco7nsEgEWzrvE06DI91irFnyjoEFDMTsiKW1UzVrIUJ60NyDI= Message-ID: Date: Sat, 31 May 2008 16:25:47 +0200 From: "=?UTF-8?Q?H=C3=A5kon_L=C3=B8vdal?=" To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Subject: Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+ Cc: "Ingo Molnar" , "David S. Miller" , "Rafael J. Wysocki" , "Andrew Morton" , "=?UTF-8?Q?Ilpo_J=C3=A4rvinen?=" In-Reply-To: <1211966834.12349.62.camel@twins> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline References: <20080526115628.GA31316@elte.hu> <1211966834.12349.62.camel@twins> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by alpha.home.local id m4VEQ6Hl024685 2008/5/28 Peter Zijlstra :> Just a quick note to say, me too!!>> same scenario: distcc on localhost. Me too, however with a completely different scenario; my hung connectionsare not related to distcc at all. The output from /proc/net/tcp that Ingoposted a few days ago are somewhat different from mine, however I believethis is the same problem or at least related. Just as Ingo experienced,netstat -p only shows PID/program as '-' for the hung connections whilefor other connections it shows the expected results. I have recently bought a new PC and have started the process of copyingstuff from my old PC to the new PC. During this I have experienced thishang several times. I started copying by using tar on both ends over a sshpipe but in order to eliminate possible ssh problems I also have tried tarover a ttcp connection which also fails. There is no obvious pattern ofwhen this happens, I have experienced failures after transferring 1.15GB,51.4GB and 23.6GB. Here is the output from netstat -n -o filtered for port 22 and slightlyedited. All the lines started with Proto == tcp and Recv-Q == 0. Send-Q Local Addr Foreign Addr State Timer 0 old_pc:22 new_pc:52667 ESTABLISHED keepalive (3513.93/0/0) 0 old_pc:22 new_pc:43825 ESTABLISHED keepalive (5467.38/0/0) 2896 old_pc:22 new_pc:58601 ESTABLISHED on (21020884.65/0/0) 4344 old_pc:22 new_pc:54105 ESTABLISHED on (21017016.33/0/0) 2896 old_pc:22 new_pc:34149 ESTABLISHED on (20986889.24/0/0) The first two connections are ongoing, working, interactive sshconnections. The other three connections died days ago on my new PC. One thing that caught my eyes was these very high timer values.Checking the netstat source reveals that the value printed is "(double)time_len / HZ" and that time_len is extracted from /proc/net/tcp. Whilemy CONFIG_HZ is 1000, I assume netstat has picked up HZ as 100 from/usr/include/asm/param.h, and then things really seems to imply thatthere is some integer overflow since 2^31 = 2147483648. Looking into get_tcp4_sock in net/ipv4/tcp_ipv4.c I see that timer_expiresis initialized with icsk->icsk_timeout for the troublesome cases. Buthere my competence to trace this further stops, so I have no idea ofhow icsk->icsk_timeout gets such high values. My old PC is currently still running with these stalled connectionspresent so let me know if there is something I should try to investigatefurther. I can post output from /proc/net/tcp and my .config if you wantto have a look. My old PC is 32 bit/Celeron single core, kernel 2.6.24,while my new is 64 bit/Q9300 quad core, kernel 2.6.25.3. The ethernetcards are the following: 02:0d.0 Ethernet controller: Realtek Semiconductor Co., Ltd.RTL-8139/8139C/8139C+ (rev 10)02:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056PCI-E Gigabit Ethernet Controller (rev 12) BR Håkon Løvdal{.n++%ݶw{.n+{G{ayʇڙ,jfhz_(階ݢj"mG?&~iOzv^m ?I From mboxrd@z Thu Jan 1 00:00:00 1970 From: "=?UTF-8?Q?H=C3=A5kon_L=C3=B8vdal?=" Subject: Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+ Date: Sat, 31 May 2008 16:25:47 +0200 Message-ID: References: <20080526115628.GA31316@elte.hu> <1211966834.12349.62.camel@twins> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Cc: "Ingo Molnar" , "David S. Miller" , "Rafael J. Wysocki" , "Andrew Morton" , "=?UTF-8?Q?Ilpo_J=C3=A4rvinen?=" To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Return-path: Received: from rv-out-0506.google.com ([209.85.198.239]:12298 "EHLO rv-out-0506.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752629AbYEaOZt (ORCPT ); Sat, 31 May 2008 10:25:49 -0400 Received: by rv-out-0506.google.com with SMTP id l9so304659rvb.1 for ; Sat, 31 May 2008 07:25:47 -0700 (PDT) In-Reply-To: <1211966834.12349.62.camel@twins> Content-Disposition: inline Sender: netdev-owner@vger.kernel.org List-ID: MjAwOC81LzI4IFBldGVyIFppamxzdHJhIDxwZXRlcnpAaW5mcmFkZWFkLm9yZz46Cj4gSnVzdCBh IHF1aWNrIG5vdGUgdG8gc2F5LCBtZSB0b28hIQo+Cj4gc2FtZSBzY2VuYXJpbzogZGlzdGNjIG9u IGxvY2FsaG9zdC4KCk1lIHRvbywgaG93ZXZlciB3aXRoIGEgY29tcGxldGVseSBkaWZmZXJlbnQg c2NlbmFyaW87IG15IGh1bmcgY29ubmVjdGlvbnMKYXJlIG5vdCByZWxhdGVkIHRvIGRpc3RjYyBh dCBhbGwuIFRoZSBvdXRwdXQgZnJvbSAvcHJvYy9uZXQvdGNwIHRoYXQgSW5nbwpwb3N0ZWQgYSBm ZXcgZGF5cyBhZ28gYXJlIHNvbWV3aGF0IGRpZmZlcmVudCBmcm9tIG1pbmUsIGhvd2V2ZXIgSSBi ZWxpZXZlCnRoaXMgaXMgdGhlIHNhbWUgcHJvYmxlbSBvciBhdCBsZWFzdCByZWxhdGVkLiBKdXN0 IGFzIEluZ28gZXhwZXJpZW5jZWQsCm5ldHN0YXQgLXAgb25seSBzaG93cyBQSUQvcHJvZ3JhbSBh cyAnLScgZm9yIHRoZSBodW5nIGNvbm5lY3Rpb25zIHdoaWxlCmZvciBvdGhlciBjb25uZWN0aW9u cyBpdCBzaG93cyB0aGUgZXhwZWN0ZWQgcmVzdWx0cy4KCkkgaGF2ZSByZWNlbnRseSBib3VnaHQg YSBuZXcgUEMgYW5kIGhhdmUgc3RhcnRlZCB0aGUgcHJvY2VzcyBvZiBjb3B5aW5nCnN0dWZmIGZy b20gbXkgb2xkIFBDIHRvIHRoZSBuZXcgUEMuIER1cmluZyB0aGlzIEkgaGF2ZSBleHBlcmllbmNl ZCB0aGlzCmhhbmcgc2V2ZXJhbCB0aW1lcy4gSSBzdGFydGVkIGNvcHlpbmcgYnkgdXNpbmcgdGFy IG9uIGJvdGggZW5kcyBvdmVyIGEgc3NoCnBpcGUgYnV0IGluIG9yZGVyIHRvIGVsaW1pbmF0ZSBw b3NzaWJsZSBzc2ggcHJvYmxlbXMgSSBhbHNvIGhhdmUgdHJpZWQgdGFyCm92ZXIgYSB0dGNwIGNv bm5lY3Rpb24gd2hpY2ggYWxzbyBmYWlscy4gVGhlcmUgaXMgbm8gb2J2aW91cyBwYXR0ZXJuIG9m CndoZW4gdGhpcyBoYXBwZW5zLCBJIGhhdmUgZXhwZXJpZW5jZWQgZmFpbHVyZXMgYWZ0ZXIgdHJh bnNmZXJyaW5nIDEuMTVHQiwKNTEuNEdCIGFuZCAyMy42R0IuCgpIZXJlIGlzIHRoZSBvdXRwdXQg ZnJvbSBuZXRzdGF0IC1uIC1vIGZpbHRlcmVkIGZvciBwb3J0IDIyIGFuZCBzbGlnaHRseQplZGl0 ZWQuIEFsbCB0aGUgbGluZXMgc3RhcnRlZCB3aXRoIFByb3RvID09IHRjcCBhbmQgUmVjdi1RID09 IDAuCgpTZW5kLVEgTG9jYWwgQWRkciBGb3JlaWduIEFkZHIgIFN0YXRlICAgICAgIFRpbWVyCiAg ICAgMCBvbGRfcGM6MjIgIG5ld19wYzo1MjY2NyAgRVNUQUJMSVNIRUQga2VlcGFsaXZlICgzNTEz LjkzLzAvMCkKICAgICAwIG9sZF9wYzoyMiAgbmV3X3BjOjQzODI1ICBFU1RBQkxJU0hFRCBrZWVw YWxpdmUgKDU0NjcuMzgvMC8wKQogIDI4OTYgb2xkX3BjOjIyICBuZXdfcGM6NTg2MDEgIEVTVEFC TElTSEVEIG9uICgyMTAyMDg4NC42NS8wLzApCiAgNDM0NCBvbGRfcGM6MjIgIG5ld19wYzo1NDEw NSAgRVNUQUJMSVNIRUQgb24gKDIxMDE3MDE2LjMzLzAvMCkKICAyODk2IG9sZF9wYzoyMiAgbmV3 X3BjOjM0MTQ5ICBFU1RBQkxJU0hFRCBvbiAoMjA5ODY4ODkuMjQvMC8wKQoKVGhlIGZpcnN0IHR3 byBjb25uZWN0aW9ucyBhcmUgb25nb2luZywgd29ya2luZywgaW50ZXJhY3RpdmUgc3NoCmNvbm5l Y3Rpb25zLiBUaGUgb3RoZXIgdGhyZWUgY29ubmVjdGlvbnMgZGllZCBkYXlzIGFnbyBvbiBteSBu ZXcgUEMuCgpPbmUgdGhpbmcgdGhhdCBjYXVnaHQgbXkgZXllcyB3YXMgdGhlc2UgdmVyeSBoaWdo IHRpbWVyIHZhbHVlcy4KQ2hlY2tpbmcgdGhlIG5ldHN0YXQgc291cmNlIHJldmVhbHMgdGhhdCB0 aGUgdmFsdWUgcHJpbnRlZCBpcyAiKGRvdWJsZSkKdGltZV9sZW4gLyBIWiIgYW5kIHRoYXQgdGlt ZV9sZW4gaXMgZXh0cmFjdGVkIGZyb20gL3Byb2MvbmV0L3RjcC4gV2hpbGUKbXkgQ09ORklHX0ha IGlzIDEwMDAsIEkgYXNzdW1lIG5ldHN0YXQgaGFzIHBpY2tlZCB1cCBIWiBhcyAxMDAgZnJvbQov dXNyL2luY2x1ZGUvYXNtL3BhcmFtLmgsIGFuZCB0aGVuIHRoaW5ncyByZWFsbHkgc2VlbXMgdG8g aW1wbHkgdGhhdAp0aGVyZSBpcyBzb21lIGludGVnZXIgb3ZlcmZsb3cgc2luY2UgMl4zMSA9IDIx NDc0ODM2NDguCgpMb29raW5nIGludG8gZ2V0X3RjcDRfc29jayBpbiBuZXQvaXB2NC90Y3BfaXB2 NC5jIEkgc2VlIHRoYXQgdGltZXJfZXhwaXJlcwppcyBpbml0aWFsaXplZCB3aXRoIGljc2stPmlj c2tfdGltZW91dCBmb3IgdGhlIHRyb3VibGVzb21lIGNhc2VzLiBCdXQKaGVyZSBteSBjb21wZXRl bmNlIHRvIHRyYWNlIHRoaXMgZnVydGhlciBzdG9wcywgc28gSSBoYXZlIG5vIGlkZWEgb2YKaG93 IGljc2stPmljc2tfdGltZW91dCBnZXRzIHN1Y2ggaGlnaCB2YWx1ZXMuCgpNeSBvbGQgUEMgaXMg Y3VycmVudGx5IHN0aWxsIHJ1bm5pbmcgd2l0aCB0aGVzZSBzdGFsbGVkIGNvbm5lY3Rpb25zCnBy ZXNlbnQgc28gbGV0IG1lIGtub3cgaWYgdGhlcmUgaXMgc29tZXRoaW5nIEkgc2hvdWxkIHRyeSB0 byBpbnZlc3RpZ2F0ZQpmdXJ0aGVyLiBJIGNhbiBwb3N0IG91dHB1dCBmcm9tIC9wcm9jL25ldC90 Y3AgYW5kIG15IC5jb25maWcgaWYgeW91IHdhbnQKdG8gaGF2ZSBhIGxvb2suIE15IG9sZCBQQyBp cyAzMiBiaXQvQ2VsZXJvbiBzaW5nbGUgY29yZSwga2VybmVsIDIuNi4yNCwKd2hpbGUgbXkgbmV3 IGlzIDY0IGJpdC9ROTMwMCBxdWFkIGNvcmUsIGtlcm5lbCAyLjYuMjUuMy4gVGhlIGV0aGVybmV0 CmNhcmRzIGFyZSB0aGUgZm9sbG93aW5nOgoKMDI6MGQuMCBFdGhlcm5ldCBjb250cm9sbGVyOiBS ZWFsdGVrIFNlbWljb25kdWN0b3IgQ28uLCBMdGQuClJUTC04MTM5LzgxMzlDLzgxMzlDKyAocmV2 IDEwKQowMjowMC4wIEV0aGVybmV0IGNvbnRyb2xsZXI6IE1hcnZlbGwgVGVjaG5vbG9neSBHcm91 cCBMdGQuIDg4RTgwNTYKUENJLUUgR2lnYWJpdCBFdGhlcm5ldCBDb250cm9sbGVyIChyZXYgMTIp CgpCUiBIw6Vrb24gTMO4dmRhbAo=