From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail143.messagelabs.com (mail143.messagelabs.com [216.82.254.35]) by kanga.kvack.org (Postfix) with ESMTP id 843D26B01AC for ; Fri, 2 Jul 2010 19:05:08 -0400 (EDT) Date: Fri, 2 Jul 2010 16:05:01 -0700 From: Andrew Morton Subject: Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O Message-Id: <20100702160501.45861821.akpm@linux-foundation.org> In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: linux-mm@kvack.org, Jens Axboe List-ID: On Thu, 1 Jul 2010 11:57:37 GMT bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=16321 > > Summary: os unresponsive during buffered I/O > Product: IO/Storage > Version: 2.5 > Kernel Version: 2.6.34 > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: Block Layer > AssignedTo: axboe@kernel.dk > ReportedBy: rrs@researchut.com > Regression: No > > > I have been running these tests on my laptop running the 2.6.34 Debian kernel. > When doing buffered I/O, the OS completely stalls to any interactivity. I > cannot switch console tabs in my Desktop Environment and the mouse pointer does > not move. > > Eventually, I/O completes and every thing resumes to normal. There is no OOM > seen during the I/O operation. > If doing direct I/O, interactivity does not get penalized. > 1... 2... 3... FUCK!!! We've been trying to fix this stuff for ten years. Apparently, without success. Do we suck, or what? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail143.messagelabs.com (mail143.messagelabs.com [216.82.254.35]) by kanga.kvack.org (Postfix) with SMTP id 6B18C6B01AC for ; Sat, 3 Jul 2010 07:56:17 -0400 (EDT) Message-ID: <4C2F255D.6000908@kernel.dk> Date: Sat, 03 Jul 2010 13:56:13 +0200 From: Jens Axboe MIME-Version: 1.0 Subject: Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O References: <20100702160501.45861821.akpm@linux-foundation.org> In-Reply-To: <20100702160501.45861821.akpm@linux-foundation.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: Andrew Morton Cc: linux-mm@kvack.org List-ID: On 03/07/10 01.05, Andrew Morton wrote: > On Thu, 1 Jul 2010 11:57:37 GMT > bugzilla-daemon@bugzilla.kernel.org wrote: > >> https://bugzilla.kernel.org/show_bug.cgi?id=16321 >> >> Summary: os unresponsive during buffered I/O >> Product: IO/Storage >> Version: 2.5 >> Kernel Version: 2.6.34 >> Platform: All >> OS/Version: Linux >> Tree: Mainline >> Status: NEW >> Severity: normal >> Priority: P1 >> Component: Block Layer >> AssignedTo: axboe@kernel.dk >> ReportedBy: rrs@researchut.com >> Regression: No >> >> >> I have been running these tests on my laptop running the 2.6.34 Debian kernel. >> When doing buffered I/O, the OS completely stalls to any interactivity. I >> cannot switch console tabs in my Desktop Environment and the mouse pointer does >> not move. >> >> Eventually, I/O completes and every thing resumes to normal. There is no OOM >> seen during the I/O operation. >> If doing direct I/O, interactivity does not get penalized. >> > > 1... > > 2... > > 3... > > FUCK!!! > > We've been trying to fix this stuff for ten years. Apparently, without > success. Do we suck, or what? We suck. This is assigned to the block layer, but it must be something a lot more fundemental than this. Either the vm is shitting itself, or something else is sucking up the juice completely. Or the CPU scheduler is going to pieces, I dunno. The report says that he cannot even move the mouse or switch console tabs, clearly we are not (only) dealing with an issue at the IO side. What else is interesting is that only a few people seem to see this. What is different about their setup or their hardware?! I cannot reproduce reports like this, and if it happened for everybody for a simple 4 process random write like this, then everything would grind to a halt. I propose we fly Andrew out to investigate in person :-) -- Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail137.messagelabs.com (mail137.messagelabs.com [216.82.249.19]) by kanga.kvack.org (Postfix) with ESMTP id 761806B01AC for ; Sat, 3 Jul 2010 11:16:18 -0400 (EDT) Date: Sat, 3 Jul 2010 08:16:13 -0700 From: Andrew Morton Subject: Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O Message-Id: <20100703081613.36e1cba8.akpm@linux-foundation.org> In-Reply-To: <4C2F255D.6000908@kernel.dk> References: <20100702160501.45861821.akpm@linux-foundation.org> <4C2F255D.6000908@kernel.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: Jens Axboe Cc: linux-mm@kvack.org List-ID: On Sat, 03 Jul 2010 13:56:13 +0200 Jens Axboe wrote: > On 03/07/10 01.05, Andrew Morton wrote: > > On Thu, 1 Jul 2010 11:57:37 GMT > > bugzilla-daemon@bugzilla.kernel.org wrote: > > > >> https://bugzilla.kernel.org/show_bug.cgi?id=16321 > >> > >> Summary: os unresponsive during buffered I/O > >> Product: IO/Storage > >> Version: 2.5 > >> Kernel Version: 2.6.34 > >> Platform: All > >> OS/Version: Linux > >> Tree: Mainline > >> Status: NEW > >> Severity: normal > >> Priority: P1 > >> Component: Block Layer > >> AssignedTo: axboe@kernel.dk > >> ReportedBy: rrs@researchut.com > >> Regression: No > >> > >> > >> I have been running these tests on my laptop running the 2.6.34 Debian kernel. > >> When doing buffered I/O, the OS completely stalls to any interactivity. I > >> cannot switch console tabs in my Desktop Environment and the mouse pointer does > >> not move. > >> > >> Eventually, I/O completes and every thing resumes to normal. There is no OOM > >> seen during the I/O operation. > >> If doing direct I/O, interactivity does not get penalized. > >> > > > > 1... > > > > 2... > > > > 3... > > > > FUCK!!! > > > > We've been trying to fix this stuff for ten years. Apparently, without > > success. Do we suck, or what? > > We suck. This is assigned to the block layer, but it must be something > a lot more fundemental than this. Either the vm is shitting itself, or > something else is sucking up the juice completely. Or the CPU scheduler > is going to pieces, I dunno. The report says that he cannot even move the > mouse or switch console tabs, clearly we are not (only) dealing with > an issue at the IO side. > > What else is interesting is that only a few people seem to see this. > What is different about their setup or their hardware?! I cannot > reproduce reports like this, and if it happened for everybody for > a simple 4 process random write like this, then everything would > grind to a halt. > > I propose we fly Andrew out to investigate in person :-) Hey, I suck more than anyone! My laptop goes absolutely utterly mouse-wont-move comatose for tens of minutes when it fetchmails 100 emails and 100 spamassassins go berzerk. It could be either a CPU scheduler thing, or an IO thing, or an evil combination of both. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail190.messagelabs.com (mail190.messagelabs.com [216.82.249.51]) by kanga.kvack.org (Postfix) with SMTP id 73B186B01AC for ; Sat, 3 Jul 2010 15:56:12 -0400 (EDT) Message-ID: <4C2F95CF.4000308@redhat.com> Date: Sat, 03 Jul 2010 22:55:59 +0300 From: Avi Kivity MIME-Version: 1.0 Subject: Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O References: <20100702160501.45861821.akpm@linux-foundation.org> <4C2F255D.6000908@kernel.dk> <20100703081613.36e1cba8.akpm@linux-foundation.org> In-Reply-To: <20100703081613.36e1cba8.akpm@linux-foundation.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: Andrew Morton Cc: Jens Axboe , linux-mm@kvack.org List-ID: On 07/03/2010 06:16 PM, Andrew Morton wrote: > > My laptop goes absolutely utterly mouse-wont-move comatose for tens of > minutes when it fetchmails 100 emails and 100 spamassassins go berzerk. > It could be either a CPU scheduler thing, or an IO thing, or an evil > combination of both. > Try putting your spamassasins in a cgroup and see. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail191.messagelabs.com (mail191.messagelabs.com [216.82.242.19]) by kanga.kvack.org (Postfix) with ESMTP id B02E460080B for ; Tue, 20 Jul 2010 00:11:11 -0400 (EDT) Date: Mon, 19 Jul 2010 21:10:36 -0700 From: Andrew Morton Subject: Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O Message-Id: <20100719211036.3cfa1727.akpm@linux-foundation.org> In-Reply-To: <20100702160501.45861821.akpm@linux-foundation.org> References: <20100702160501.45861821.akpm@linux-foundation.org> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: linux-mm@kvack.org, Jens Axboe List-ID: On Fri, 2 Jul 2010 16:05:01 -0700 Andrew Morton wrote: > > https://bugzilla.kernel.org/show_bug.cgi?id=16321 The reporter has updated the report, says "This bug is consistently reproducible.". But no signs of interest from kernel developers yet. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail172.messagelabs.com (mail172.messagelabs.com [216.82.254.3]) by kanga.kvack.org (Postfix) with ESMTP id EC2F56B024D for ; Tue, 20 Jul 2010 11:09:00 -0400 (EDT) Date: Tue, 20 Jul 2010 11:08:55 -0400 From: Christoph Hellwig Subject: Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O Message-ID: <20100720150855.GA17832@infradead.org> References: <20100702160501.45861821.akpm@linux-foundation.org> <20100719211036.3cfa1727.akpm@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100719211036.3cfa1727.akpm@linux-foundation.org> Sender: owner-linux-mm@kvack.org To: Andrew Morton Cc: linux-mm@kvack.org, Jens Axboe List-ID: On Mon, Jul 19, 2010 at 09:10:36PM -0700, Andrew Morton wrote: > The reporter has updated the report, says "This bug is consistently > reproducible.". But no signs of interest from kernel developers yet. If only the reported can reproduce it that doesn't really help anyone. And neither does hiding bugreports in bugzilla help when this could be discussed much more usefully on the list. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail191.messagelabs.com (mail191.messagelabs.com [216.82.242.19]) by kanga.kvack.org (Postfix) with ESMTP id 3ECEC6B024D for ; Tue, 20 Jul 2010 13:51:31 -0400 (EDT) Date: Tue, 20 Jul 2010 10:50:52 -0700 From: Andrew Morton Subject: Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O Message-Id: <20100720105052.e077be1c.akpm@linux-foundation.org> In-Reply-To: <20100720150855.GA17832@infradead.org> References: <20100702160501.45861821.akpm@linux-foundation.org> <20100719211036.3cfa1727.akpm@linux-foundation.org> <20100720150855.GA17832@infradead.org> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: Christoph Hellwig Cc: linux-mm@kvack.org, Jens Axboe List-ID: On Tue, 20 Jul 2010 11:08:55 -0400 Christoph Hellwig wrote: > On Mon, Jul 19, 2010 at 09:10:36PM -0700, Andrew Morton wrote: > > The reporter has updated the report, says "This bug is consistently > > reproducible.". But no signs of interest from kernel developers yet. > > If only the reported can reproduce it that doesn't really help anyone. Garbage. We have a reporter who is prepared to work with us. Has anyone tried talking him through using the relevant tracepoints? Sending patches to add new ones if needed? > And neither does hiding bugreports in bugzilla help when this could be > discussed much more usefully on the list. The guy's email address is right there. And there are over forty more people at https://bugzilla.kernel.org/show_bug.cgi?id=7372 seeing similar (or other!) problems. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org