From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-we0-f177.google.com (mail-we0-f177.google.com [74.125.82.177]) by kanga.kvack.org (Postfix) with ESMTP id BEFB66B0044 for ; Mon, 7 Jul 2014 18:30:03 -0400 (EDT) Received: by mail-we0-f177.google.com with SMTP id u56so5074083wes.22 for ; Mon, 07 Jul 2014 15:30:03 -0700 (PDT) Received: from one.firstfloor.org (one.firstfloor.org. [193.170.194.197]) by mx.google.com with ESMTPS id n7si51407568wja.159.2014.07.07.15.30.02 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 07 Jul 2014 15:30:02 -0700 (PDT) Date: Tue, 8 Jul 2014 00:30:01 +0200 From: Andi Kleen Subject: fallout of 16K stacks Message-ID: <20140707223001.GD18735@two.firstfloor.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-linux-mm@kvack.org List-ID: To: torvalds@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org Since the 16K stack change I noticed a number of problems with my usual stress tests. They have a tendency to bomb out because something cannot fork. - AIM7 on a dual socket socket system now cannot reliably run >1000 parallel jobs. - LTP stress + memhog stress in parallel to something else usually doesn't survive the night. Do we need to strengthen the memory allocator to try harder for 16K? -Andi -- ak@linux.intel.com -- Speaking for myself only. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yk0-f172.google.com (mail-yk0-f172.google.com [209.85.160.172]) by kanga.kvack.org (Postfix) with ESMTP id E906B6B0036 for ; Mon, 7 Jul 2014 18:50:18 -0400 (EDT) Received: by mail-yk0-f172.google.com with SMTP id 142so1977092ykq.17 for ; Mon, 07 Jul 2014 15:50:18 -0700 (PDT) Received: from mail.zytor.com (terminus.zytor.com. [2001:1868:205::10]) by mx.google.com with ESMTPS id n12si48600322yhh.165.2014.07.07.15.50.17 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Jul 2014 15:50:17 -0700 (PDT) Message-ID: <53BB240C.30400@zytor.com> Date: Mon, 07 Jul 2014 15:49:48 -0700 From: "H. Peter Anvin" MIME-Version: 1.0 Subject: Re: fallout of 16K stacks References: <20140707223001.GD18735@two.firstfloor.org> In-Reply-To: <20140707223001.GD18735@two.firstfloor.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Andi Kleen , torvalds@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org On 07/07/2014 03:30 PM, Andi Kleen wrote: > > Since the 16K stack change I noticed a number of problems with > my usual stress tests. They have a tendency to bomb out > because something cannot fork. As in ENOMEM or does something worse happen? > - AIM7 on a dual socket socket system now cannot reliably run >> 1000 parallel jobs. ... with how much RAM? > - LTP stress + memhog stress in parallel to something else > usually doesn't survive the night. > > Do we need to strengthen the memory allocator to try > harder for 16K? Can we even? The probability of success goes down exponentially in the order requested. Movable pages can help, of course, but still, there is a very real cost to this :( -hpa -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wg0-f49.google.com (mail-wg0-f49.google.com [74.125.82.49]) by kanga.kvack.org (Postfix) with ESMTP id 760FA6B0038 for ; Mon, 7 Jul 2014 19:05:01 -0400 (EDT) Received: by mail-wg0-f49.google.com with SMTP id a1so1746697wgh.32 for ; Mon, 07 Jul 2014 16:05:00 -0700 (PDT) Received: from one.firstfloor.org (one.firstfloor.org. [193.170.194.197]) by mx.google.com with ESMTPS id n9si43195429wiz.23.2014.07.07.16.05.00 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 07 Jul 2014 16:05:00 -0700 (PDT) Date: Tue, 8 Jul 2014 01:04:59 +0200 From: Andi Kleen Subject: Re: fallout of 16K stacks Message-ID: <20140707230459.GF18735@two.firstfloor.org> References: <20140707223001.GD18735@two.firstfloor.org> <53BB240C.30400@zytor.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <53BB240C.30400@zytor.com> Sender: owner-linux-mm@kvack.org List-ID: To: "H. Peter Anvin" Cc: Andi Kleen , torvalds@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org On Mon, Jul 07, 2014 at 03:49:48PM -0700, H. Peter Anvin wrote: > On 07/07/2014 03:30 PM, Andi Kleen wrote: > > > > Since the 16K stack change I noticed a number of problems with > > my usual stress tests. They have a tendency to bomb out > > because something cannot fork. > > As in ENOMEM or does something worse happen? EAGAIN, then the workload stops. For an overnight stress test that's pretty catastrophic. It may have killed some stuff with the OOM killer too. > > - AIM7 on a dual socket socket system now cannot reliably run > >> 1000 parallel jobs. > > ... with how much RAM? This system has 32G > > - LTP stress + memhog stress in parallel to something else > > usually doesn't survive the night. > > > > Do we need to strengthen the memory allocator to try > > harder for 16K? > > Can we even? The probability of success goes down exponentially in the > order requested. Movable pages can help, of course, but still, there is > a very real cost to this :( I hope so. In the worst case just try longer. -Andi -- ak@linux.intel.com -- Speaking for myself only. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ve0-f174.google.com (mail-ve0-f174.google.com [209.85.128.174]) by kanga.kvack.org (Postfix) with ESMTP id 76B716B0039 for ; Mon, 7 Jul 2014 19:52:37 -0400 (EDT) Received: by mail-ve0-f174.google.com with SMTP id jx11so4872686veb.19 for ; Mon, 07 Jul 2014 16:52:37 -0700 (PDT) Received: from mail-vc0-x234.google.com (mail-vc0-x234.google.com [2607:f8b0:400c:c03::234]) by mx.google.com with ESMTPS id yr5si19393172vdb.14.2014.07.07.16.52.36 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 07 Jul 2014 16:52:36 -0700 (PDT) Received: by mail-vc0-f180.google.com with SMTP id im17so4763056vcb.11 for ; Mon, 07 Jul 2014 16:52:35 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20140707230459.GF18735@two.firstfloor.org> References: <20140707223001.GD18735@two.firstfloor.org> <53BB240C.30400@zytor.com> <20140707230459.GF18735@two.firstfloor.org> Date: Mon, 7 Jul 2014 16:52:35 -0700 Message-ID: Subject: Re: fallout of 16K stacks From: Linus Torvalds Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Andi Kleen Cc: "H. Peter Anvin" , linux-mm , Linux Kernel Mailing List On Mon, Jul 7, 2014 at 4:04 PM, Andi Kleen wrote: >> >> As in ENOMEM or does something worse happen? > > EAGAIN, then the workload stops. For an overnight stress > test that's pretty catastrophic. It may have killed some stuff > with the OOM killer too. I don't think it's OOM. We have long had the rule that order <= PAGE_ALLOC_COSTLY_ORDER (which is 3) allocations imply __GFP_RETRY unless you explicitly ask it not to. And THREAD_SIZE_ORDER is still smaller than that. Sure, if the system makes no progress at all, it will still oom for allocations like that, but that's *not* going to happen for something like a 32GB machine afaik. And if it was the actual dup_task_struct() that failed (due to alloc_thread_info_node() now failing), it should have returned ENOMEM anyway. So EAGAIN is due to something else. The only cases for fork() returning EAGAIN I can find are the RLIMIT_NPROC and max_threads checks. And the thing is, the default value for RLIMIT_NPROC is actually initialized based on THREAD_SIZE (which doubled), so maybe it's really just that rlimit check that now triggers. Hmm? Linus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org