Stress testing 2.4.14-pre6

Stress testing 2.4.14-pre6

[Bob Matthews]

Hi Linus,

We've been doing some stress-testing on 2.4.14-pre6 and have encountered
a couple of problems.  The platform is an 8xPIII with 8G RAM and 32G
swap.  After running Cerberus for about 3 hours, the machine hung
completely.  I was not able to switch VC's.

Unfortunately, this is as detailed a bug report as I can submit.  It
looks like Magic SysRq is broken in this kernel.  <alt><SysRq>T will
print the column headings but nothing else.
-- 
Bob Matthews
Red Hat, Inc.

Re: Stress testing 2.4.14-pre6

[Alan Cox]

> Unfortunately, this is as detailed a bug report as I can submit.  It
> looks like Magic SysRq is broken in this kernel.  <alt><SysRq>T will
> print the column headings but nothing else.

Thats consistent with memory corruption trashig the task list

Re: Stress testing 2.4.14-pre6

[Linus Torvalds]

In article <3BE073B6.BDCB3D56@redhat.com>,
Bob Matthews  <bmatthews@redhat.com> wrote:
>Hi Linus,
>
>We've been doing some stress-testing on 2.4.14-pre6 and have encountered
>a couple of problems.  The platform is an 8xPIII with 8G RAM and 32G
>swap.  After running Cerberus for about 3 hours, the machine hung
>completely.  I was not able to switch VC's.

There is some race somewhere - I've found one interrupt race (that
actually seems to exist in the 2.2.x VM too, but is probably _much_
harder to trigger where an interrupt at _just_ the right time will
corrupt the per-process local page list.  That looks so unlikely that I
doubt that is it, but I'm looking for others (the irq one wasn't even a
SMP race - it's on UP too, surprise surprise). 

Working on it, in other words.

		Linus

Re: Stress testing 2.4.14-pre6

[Marcelo Tosatti]

On Thu, 1 Nov 2001, Linus Torvalds wrote:

> In article <3BE073B6.BDCB3D56@redhat.com>,
> Bob Matthews  <bmatthews@redhat.com> wrote:
> >Hi Linus,
> >
> >We've been doing some stress-testing on 2.4.14-pre6 and have encountered
> >a couple of problems.  The platform is an 8xPIII with 8G RAM and 32G
> >swap.  After running Cerberus for about 3 hours, the machine hung
> >completely.  I was not able to switch VC's.
> 
> There is some race somewhere - I've found one interrupt race (that
> actually seems to exist in the 2.2.x VM too, but is probably _much_
> harder to trigger where an interrupt at _just_ the right time will
> corrupt the per-process local page list.  That looks so unlikely that I
> doubt that is it, but I'm looking for others (the irq one wasn't even a
> SMP race - it's on UP too, surprise surprise). 
> 
> Working on it, in other words.

Would you mind to describe this race? 

Thanks

Re: Stress testing 2.4.14-pre6

[Linus Torvalds]

On Thu, 1 Nov 2001, Marcelo Tosatti wrote:
> >
> > There is some race somewhere - I've found one interrupt race (that
> > actually seems to exist in the 2.2.x VM too, but is probably _much_
> > harder to trigger where an interrupt at _just_ the right time will
> > corrupt the per-process local page list.  That looks so unlikely that I
> > doubt that is it, but I'm looking for others (the irq one wasn't even a
> > SMP race - it's on UP too, surprise surprise).
> >
> > Working on it, in other words.
>
> Would you mind to describe this race?

Both 2.2.x and the new VM (which, through Andrea, has a lot of the same
things) have this notion of a per-process "free pages list" that it
replenished by any freeing that the process does itself when it gets into
the "try_to_free_memory()" path.

The trigger for refilling this list is "current->flags & PF_FREE_PAGES".

The bug is that ytou can be in the middle of adding such a recently free'd
page to the per-process list of free pages, and an interrupt comes in.

The interrupt (or bottom half), in turn, might do something like

	page = get_free_page(GFP_ATOMIC);
	...
	free_page(page);

and now the free_page() inside the interrupt context will _also_ trigger
the PF_FREE_PAGES test, and _also_ add the page to the list. Except, of
course, the list is totally unprotected by any locks, so it may not be
valid at this point.

Fix is to only care about the PF_FREE_PAGES bit when not in an interrupt
context.

Anyway, I seriously doubt this explains any real-world bad behaviour: the
window for the interrupt hitting a half-way updated list is something like
two instructions long out of the whole memory freeing path. AND most
interrupts don't actually do any allocation.

		Linus

Re: Stress testing 2.4.14-pre6

[Jeff Garzik]

Linus Torvalds wrote:
> Anyway, I seriously doubt this explains any real-world bad behaviour: the
> window for the interrupt hitting a half-way updated list is something like
> two instructions long out of the whole memory freeing path. AND most
> interrupts don't actually do any allocation.

Network Rx interrupts do....  definitely not as frequent as IDE
interrupts, but not infrequent.

-- 
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno

Re: Stress testing 2.4.14-pre6

[Arjan van de Ven]

Jeff Garzik wrote:
> 
> Linus Torvalds wrote:
> > Anyway, I seriously doubt this explains any real-world bad behaviour: the
> > window for the interrupt hitting a half-way updated list is something like
> > two instructions long out of the whole memory freeing path. AND most
> > interrupts don't actually do any allocation.
> 
> Network Rx interrupts do....  definitely not as frequent as IDE
> interrupts, but not infrequent.

Cerberus doesn't use networking in the tested setup iirc

Re: Stress testing 2.4.14-pre6

[Jens Axboe]

On Thu, Nov 01 2001, Jeff Garzik wrote:
> Linus Torvalds wrote:
> > Anyway, I seriously doubt this explains any real-world bad behaviour: the
> > window for the interrupt hitting a half-way updated list is something like
> > two instructions long out of the whole memory freeing path. AND most
> > interrupts don't actually do any allocation.
> 
> Network Rx interrupts do....  definitely not as frequent as IDE
> interrupts, but not infrequent.

Which IDE interrupts allocate memory?!

-- 
Jens Axboe

Re: Stress testing 2.4.14-pre6

[Jeff Garzik]

Jens Axboe wrote:
> 
> On Thu, Nov 01 2001, Jeff Garzik wrote:
> > Linus Torvalds wrote:
> > > Anyway, I seriously doubt this explains any real-world bad behaviour: the
> > > window for the interrupt hitting a half-way updated list is something like
> > > two instructions long out of the whole memory freeing path. AND most
> > > interrupts don't actually do any allocation.
> >
> > Network Rx interrupts do....  definitely not as frequent as IDE
> > interrupts, but not infrequent.
> 
> Which IDE interrupts allocate memory?!

Sorry, I meant as in, IDE interrupts occur more frequently than Rx
interrupts.

English is my first language... really.

-- 
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno