fallout of 16K stacks

fallout of 16K stacks

[Andi Kleen]

Since the 16K stack change I noticed a number of problems with
my usual stress tests. They have a tendency to bomb out
because something cannot fork.

- AIM7 on a dual socket socket system now cannot reliably run 
>1000 parallel jobs.
- LTP stress + memhog stress in parallel to something else
usually doesn't survive the night.

Do we need to strengthen the memory allocator to try
harder for 16K?

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

fallout of 16K stacks

[Andi Kleen]

Since the 16K stack change I noticed a number of problems with
my usual stress tests. They have a tendency to bomb out
because something cannot fork.

- AIM7 on a dual socket socket system now cannot reliably run 
>1000 parallel jobs.
- LTP stress + memhog stress in parallel to something else
usually doesn't survive the night.

Do we need to strengthen the memory allocator to try
harder for 16K?

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: fallout of 16K stacks

[H. Peter Anvin]

On 07/07/2014 03:30 PM, Andi Kleen wrote:
> 
> Since the 16K stack change I noticed a number of problems with
> my usual stress tests. They have a tendency to bomb out
> because something cannot fork.

As in ENOMEM or does something worse happen?

> - AIM7 on a dual socket socket system now cannot reliably run 
>> 1000 parallel jobs.

... with how much RAM?

> - LTP stress + memhog stress in parallel to something else
> usually doesn't survive the night.
> 
> Do we need to strengthen the memory allocator to try
> harder for 16K?

Can we even?  The probability of success goes down exponentially in the
order requested.  Movable pages can help, of course, but still, there is
a very real cost to this :(

	-hpa

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: fallout of 16K stacks

[H. Peter Anvin]

On 07/07/2014 03:30 PM, Andi Kleen wrote:
> 
> Since the 16K stack change I noticed a number of problems with
> my usual stress tests. They have a tendency to bomb out
> because something cannot fork.

As in ENOMEM or does something worse happen?

> - AIM7 on a dual socket socket system now cannot reliably run 
>> 1000 parallel jobs.

... with how much RAM?

> - LTP stress + memhog stress in parallel to something else
> usually doesn't survive the night.
> 
> Do we need to strengthen the memory allocator to try
> harder for 16K?

Can we even?  The probability of success goes down exponentially in the
order requested.  Movable pages can help, of course, but still, there is
a very real cost to this :(

	-hpa

Re: fallout of 16K stacks

[Andi Kleen]

On Mon, Jul 07, 2014 at 03:49:48PM -0700, H. Peter Anvin wrote:
> On 07/07/2014 03:30 PM, Andi Kleen wrote:
> > 
> > Since the 16K stack change I noticed a number of problems with
> > my usual stress tests. They have a tendency to bomb out
> > because something cannot fork.
> 
> As in ENOMEM or does something worse happen?

EAGAIN, then the workload stops. For an overnight stress
test that's pretty catastrophic. It may have killed some stuff
with the OOM killer too.

> > - AIM7 on a dual socket socket system now cannot reliably run 
> >> 1000 parallel jobs.
> 
> ... with how much RAM?

This system has 32G

> > - LTP stress + memhog stress in parallel to something else
> > usually doesn't survive the night.
> > 
> > Do we need to strengthen the memory allocator to try
> > harder for 16K?
> 
> Can we even?  The probability of success goes down exponentially in the
> order requested.  Movable pages can help, of course, but still, there is
> a very real cost to this :(

I hope so. In the worst case just try longer.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: fallout of 16K stacks

[Andi Kleen]

On Mon, Jul 07, 2014 at 03:49:48PM -0700, H. Peter Anvin wrote:
> On 07/07/2014 03:30 PM, Andi Kleen wrote:
> > 
> > Since the 16K stack change I noticed a number of problems with
> > my usual stress tests. They have a tendency to bomb out
> > because something cannot fork.
> 
> As in ENOMEM or does something worse happen?

EAGAIN, then the workload stops. For an overnight stress
test that's pretty catastrophic. It may have killed some stuff
with the OOM killer too.

> > - AIM7 on a dual socket socket system now cannot reliably run 
> >> 1000 parallel jobs.
> 
> ... with how much RAM?

This system has 32G

> > - LTP stress + memhog stress in parallel to something else
> > usually doesn't survive the night.
> > 
> > Do we need to strengthen the memory allocator to try
> > harder for 16K?
> 
> Can we even?  The probability of success goes down exponentially in the
> order requested.  Movable pages can help, of course, but still, there is
> a very real cost to this :(

I hope so. In the worst case just try longer.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: fallout of 16K stacks

[Linus Torvalds]

On Mon, Jul 7, 2014 at 4:04 PM, Andi Kleen <andi@firstfloor.org> wrote:
>>
>> As in ENOMEM or does something worse happen?
>
> EAGAIN, then the workload stops. For an overnight stress
> test that's pretty catastrophic. It may have killed some stuff
> with the OOM killer too.

I don't think it's OOM.

We have long had the rule that order <= PAGE_ALLOC_COSTLY_ORDER (which
is 3) allocations imply __GFP_RETRY unless you explicitly ask it not
to.

And THREAD_SIZE_ORDER is still smaller than that.

Sure, if the system makes no progress at all, it will still oom for
allocations like that, but that's *not* going to happen for something
like a 32GB machine afaik.

And if it was the actual dup_task_struct() that failed (due to
alloc_thread_info_node() now failing), it should have returned ENOMEM
anyway.

So EAGAIN is due to something else.

The only cases for fork() returning EAGAIN I can find are the
RLIMIT_NPROC and max_threads checks.

And the thing is, the default value for RLIMIT_NPROC is actually
initialized based on THREAD_SIZE (which doubled), so maybe it's really
just that rlimit check that now triggers.

Hmm?

          Linus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: fallout of 16K stacks

[Linus Torvalds]

On Mon, Jul 7, 2014 at 4:04 PM, Andi Kleen <andi@firstfloor.org> wrote:
>>
>> As in ENOMEM or does something worse happen?
>
> EAGAIN, then the workload stops. For an overnight stress
> test that's pretty catastrophic. It may have killed some stuff
> with the OOM killer too.

I don't think it's OOM.

We have long had the rule that order <= PAGE_ALLOC_COSTLY_ORDER (which
is 3) allocations imply __GFP_RETRY unless you explicitly ask it not
to.

And THREAD_SIZE_ORDER is still smaller than that.

Sure, if the system makes no progress at all, it will still oom for
allocations like that, but that's *not* going to happen for something
like a 32GB machine afaik.

And if it was the actual dup_task_struct() that failed (due to
alloc_thread_info_node() now failing), it should have returned ENOMEM
anyway.

So EAGAIN is due to something else.

The only cases for fork() returning EAGAIN I can find are the
RLIMIT_NPROC and max_threads checks.

And the thing is, the default value for RLIMIT_NPROC is actually
initialized based on THREAD_SIZE (which doubled), so maybe it's really
just that rlimit check that now triggers.

Hmm?

          Linus