Re: Kernel 2.6.8.1: swap storm of death - CFQ scheduler=culprit

Jens Axboe
2004-08-24T10:32:06Z

On Tue, Aug 24 2004, Karl Vogel wrote:
> > > > Original post with testcase + stats:
> > > > http://article.gmane.org/gmane.linux.kernel/228156
> > >
> > > 2.6.8.1-mm4 clean does not reproduce the problem. Marcelo, your
> > > 2.6.8-rc4 report is not valid due to the fixed problem
> > related to that
> > > in CFQ already. I'd still like for you to retest with 2.6.8.1.
> > >
>
> Did some extra testing yesterday. When not running X or anything
> substantial, I'm able to trigger it after running the expunge 2 or
> 3 times in a row.
> If I increase the calloc size, it triggers faster (tried with 1Gb
> calloc on a 512Mb box with 1Gb swap partition).

I'll try increasing the size.

> The first expunge run, completes fine. The ones after that, get
> OOM killed and I get a printk about page allocation order 0 failure.
>
> The 2.6.8.1-mm4 was a clean version, but I will double check this,
> this evening.
>
> I also tried with deadline, but was unable to trigger it.

I'm adding preempt to the mix, maybe that'll help provoke it.

> > Oh, and please do also do a sysrq-t from a hung box and save
> > the output.
>
> Note: the box doesn't hang completely. Just some processes get stuck
> in 'D' and the machine swaps heavily.

That's fine, I'd like a sysrq-t of that.

> The tests of yesterday evening, did recover. So I'm guessing if I had
> waited long enough the box would have recovered on the previous
> tests. Looking at the vmstat from my previous tests, shows that the
> box was low on memory (free/buff/cache are all very low):
>
> http://users.telenet.be/kvogel/vmstat-after-kill.txt
>
> That was probably why it was swapping like mad.

Ok, so now I'm confused - tests on what kernel recovered?

> Will provide you with that sysrq-t this evening.

Great.

--
Jens Axboe