Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O

Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O

[Andrew Morton]

On Thu, 1 Jul 2010 11:57:37 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=16321
> 
>            Summary: os unresponsive during buffered I/O
>            Product: IO/Storage
>            Version: 2.5
>     Kernel Version: 2.6.34
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Block Layer
>         AssignedTo: axboe@kernel.dk
>         ReportedBy: rrs@researchut.com
>         Regression: No
> 
> 
> I have been running these tests on my laptop running the 2.6.34 Debian kernel.
> When doing buffered I/O, the OS completely stalls to any interactivity. I
> cannot switch console tabs in my Desktop Environment and the mouse pointer does
> not move.
> 
> Eventually, I/O completes and every thing resumes to normal. There is no OOM
> seen during the I/O operation.
> If doing direct I/O, interactivity does not get penalized.
> 

1...

2...

3...

FUCK!!!

We've been trying to fix this stuff for ten years.  Apparently, without
success.  Do we suck, or what?


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O

[Jens Axboe]

On 03/07/10 01.05, Andrew Morton wrote:
> On Thu, 1 Jul 2010 11:57:37 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> 
>> https://bugzilla.kernel.org/show_bug.cgi?id=16321
>>
>>            Summary: os unresponsive during buffered I/O
>>            Product: IO/Storage
>>            Version: 2.5
>>     Kernel Version: 2.6.34
>>           Platform: All
>>         OS/Version: Linux
>>               Tree: Mainline
>>             Status: NEW
>>           Severity: normal
>>           Priority: P1
>>          Component: Block Layer
>>         AssignedTo: axboe@kernel.dk
>>         ReportedBy: rrs@researchut.com
>>         Regression: No
>>
>>
>> I have been running these tests on my laptop running the 2.6.34 Debian kernel.
>> When doing buffered I/O, the OS completely stalls to any interactivity. I
>> cannot switch console tabs in my Desktop Environment and the mouse pointer does
>> not move.
>>
>> Eventually, I/O completes and every thing resumes to normal. There is no OOM
>> seen during the I/O operation.
>> If doing direct I/O, interactivity does not get penalized.
>>
> 
> 1...
> 
> 2...
> 
> 3...
> 
> FUCK!!!
> 
> We've been trying to fix this stuff for ten years.  Apparently, without
> success.  Do we suck, or what?

We suck. This is assigned to the block layer, but it must be something
a lot more fundemental than this. Either the vm is shitting itself, or
something else is sucking up the juice completely. Or the CPU scheduler
is going to pieces, I dunno. The report says that he cannot even move the
mouse or switch console tabs, clearly we are not (only) dealing with
an issue at the IO side.

What else is interesting is that only a few people seem to see this.
What is different about their setup or their hardware?! I cannot
reproduce reports like this, and if it happened for everybody for
a simple 4 process random write like this, then everything would
grind to a halt.

I propose we fly Andrew out to investigate in person :-)

-- 
Jens Axboe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O

[Andrew Morton]

On Sat, 03 Jul 2010 13:56:13 +0200 Jens Axboe <axboe@kernel.dk> wrote:

> On 03/07/10 01.05, Andrew Morton wrote:
> > On Thu, 1 Jul 2010 11:57:37 GMT
> > bugzilla-daemon@bugzilla.kernel.org wrote:
> > 
> >> https://bugzilla.kernel.org/show_bug.cgi?id=16321
> >>
> >>            Summary: os unresponsive during buffered I/O
> >>            Product: IO/Storage
> >>            Version: 2.5
> >>     Kernel Version: 2.6.34
> >>           Platform: All
> >>         OS/Version: Linux
> >>               Tree: Mainline
> >>             Status: NEW
> >>           Severity: normal
> >>           Priority: P1
> >>          Component: Block Layer
> >>         AssignedTo: axboe@kernel.dk
> >>         ReportedBy: rrs@researchut.com
> >>         Regression: No
> >>
> >>
> >> I have been running these tests on my laptop running the 2.6.34 Debian kernel.
> >> When doing buffered I/O, the OS completely stalls to any interactivity. I
> >> cannot switch console tabs in my Desktop Environment and the mouse pointer does
> >> not move.
> >>
> >> Eventually, I/O completes and every thing resumes to normal. There is no OOM
> >> seen during the I/O operation.
> >> If doing direct I/O, interactivity does not get penalized.
> >>
> > 
> > 1...
> > 
> > 2...
> > 
> > 3...
> > 
> > FUCK!!!
> > 
> > We've been trying to fix this stuff for ten years.  Apparently, without
> > success.  Do we suck, or what?
> 
> We suck. This is assigned to the block layer, but it must be something
> a lot more fundemental than this. Either the vm is shitting itself, or
> something else is sucking up the juice completely. Or the CPU scheduler
> is going to pieces, I dunno. The report says that he cannot even move the
> mouse or switch console tabs, clearly we are not (only) dealing with
> an issue at the IO side.
> 
> What else is interesting is that only a few people seem to see this.
> What is different about their setup or their hardware?! I cannot
> reproduce reports like this, and if it happened for everybody for
> a simple 4 process random write like this, then everything would
> grind to a halt.
> 
> I propose we fly Andrew out to investigate in person :-)

Hey, I suck more than anyone!

My laptop goes absolutely utterly mouse-wont-move comatose for tens of
minutes when it fetchmails 100 emails and 100 spamassassins go berzerk.
It could be either a CPU scheduler thing, or an IO thing, or an evil
combination of both.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O

[Avi Kivity]

On 07/03/2010 06:16 PM, Andrew Morton wrote:
>
> My laptop goes absolutely utterly mouse-wont-move comatose for tens of
> minutes when it fetchmails 100 emails and 100 spamassassins go berzerk.
> It could be either a CPU scheduler thing, or an IO thing, or an evil
> combination of both.
>    

Try putting your spamassasins in a cgroup and see.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O

[Andrew Morton]

On Fri, 2 Jul 2010 16:05:01 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:

> > https://bugzilla.kernel.org/show_bug.cgi?id=16321

The reporter has updated the report, says "This bug is consistently
reproducible.".  But no signs of interest from kernel developers yet.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O

[Christoph Hellwig]

On Mon, Jul 19, 2010 at 09:10:36PM -0700, Andrew Morton wrote:
> The reporter has updated the report, says "This bug is consistently
> reproducible.".  But no signs of interest from kernel developers yet.

If only the reported can reproduce it that doesn't really help anyone.

And neither does hiding bugreports in bugzilla help when this could be
discussed much more usefully on the list.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 16321] New: os unresponsive during buffered I/O

[Andrew Morton]

On Tue, 20 Jul 2010 11:08:55 -0400 Christoph Hellwig <hch@infradead.org> wrote:

> On Mon, Jul 19, 2010 at 09:10:36PM -0700, Andrew Morton wrote:
> > The reporter has updated the report, says "This bug is consistently
> > reproducible.".  But no signs of interest from kernel developers yet.
> 
> If only the reported can reproduce it that doesn't really help anyone.

Garbage.  We have a reporter who is prepared to work with us.  Has
anyone tried talking him through using the relevant tracepoints? 
Sending patches to add new ones if needed?

> And neither does hiding bugreports in bugzilla help when this could be
> discussed much more usefully on the list.

The guy's email address is right there.  And there are over forty more
people at https://bugzilla.kernel.org/show_bug.cgi?id=7372 seeing
similar (or other!) problems.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>