From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx145.postini.com [74.125.245.145]) by kanga.kvack.org (Postfix) with SMTP id EB7B36B0124 for ; Thu, 4 Oct 2012 12:50:41 -0400 (EDT) Date: Thu, 4 Oct 2012 18:50:08 +0200 From: Andrea Arcangeli Message-ID: <20121004165008.GF25675@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-linux-mm@kvack.org List-ID: To: Christoph Lameter Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds , Andrew Morton , Peter Zijlstra , Ingo Molnar , Mel Gorman , Hugh Dickins , Rik van Riel , Johannes Weiner , Hillf Danton , Andrew Jones , Dan Smith , Thomas Gleixner , Paul Turner , Suresh Siddha , Mike Galbraith , "Paul E. McKenney" Subject: Re: [PATCH 29/33] autonuma: page_autonuma Reply-To: In-Reply-To: <0000013a2c223da2-632aa43e-21f8-4abd-a0ba-2e1b49881e3a-000000@email.amazonses.com> Hi Christoph, On Thu, Oct 04, 2012 at 02:16:14PM +0000, Christoph Lameter wrote: > On Thu, 4 Oct 2012, Andrea Arcangeli wrote: > > > Move the autonuma_last_nid from the "struct page" to a separate > > page_autonuma data structure allocated in the memsection (with > > sparsemem) or in the pgdat (with flatmem). > > Note that there is a available word in struct page before the autonuma > patches on x86_64 with CONFIG_HAVE_ALIGNED_STRUCT_PAGE. > > In fact the page_autonuma fills up the structure to nicely fit in one 64 > byte cacheline. Good point indeed. So we could drop page_autonuma by creating a CONFIG_SLUB=y dependency (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it also wouldn't be available on 32bit archs but the latter isn't a problem). I think it's a reasonable alternative to page_autonuma. Certainly it looks more appealing than taking over 16 precious bits from page->flags. There are still pros and cons. I'm neutral on it so more comments would be welcome ;). Andrea PS. randomly moved some in Cc over to Bcc as I overflowed the max header allowed on linux-kernel oops! -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx122.postini.com [74.125.245.122]) by kanga.kvack.org (Postfix) with SMTP id 9D5F46B0130 for ; Thu, 4 Oct 2012 14:17:39 -0400 (EDT) Date: Thu, 4 Oct 2012 18:17:37 +0000 From: Christoph Lameter Subject: Re: your mail In-Reply-To: <20121004165008.GF25675@redhat.com> Message-ID: <0000013a2cff3c3d-76e00716-2869-4dc8-8717-82f0136018d0-000000@email.amazonses.com> References: <20121004165008.GF25675@redhat.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Andrea Arcangeli Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds , Andrew Morton , Peter Zijlstra , Ingo Molnar , Mel Gorman , Hugh Dickins , Rik van Riel , Johannes Weiner , Hillf Danton , Andrew Jones , Dan Smith , Thomas Gleixner , Paul Turner , Suresh Siddha , Mike Galbraith , "Paul E. McKenney" On Thu, 4 Oct 2012, Andrea Arcangeli wrote: > So we could drop page_autonuma by creating a CONFIG_SLUB=y dependency > (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it > also wouldn't be available on 32bit archs but the latter isn't a > problem). Nope it should depend on page struct alignment. Other kernel subsystems may be depeding on page struct alignment in the future (and some other arches may already have that requirement) -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx157.postini.com [74.125.245.157]) by kanga.kvack.org (Postfix) with SMTP id 9B9796B0134 for ; Thu, 4 Oct 2012 14:38:38 -0400 (EDT) Date: Thu, 4 Oct 2012 20:38:19 +0200 From: Andrea Arcangeli Subject: Re: [PATCH 29/33] autonuma: page_autonuma Message-ID: <20121004183819.GM25675@redhat.com> References: <20121004165008.GF25675@redhat.com> <0000013a2cff3c3d-76e00716-2869-4dc8-8717-82f0136018d0-000000@email.amazonses.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0000013a2cff3c3d-76e00716-2869-4dc8-8717-82f0136018d0-000000@email.amazonses.com> Sender: owner-linux-mm@kvack.org List-ID: To: Christoph Lameter Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds , Andrew Morton , Peter Zijlstra , Ingo Molnar , Mel Gorman , Hugh Dickins , Rik van Riel , Johannes Weiner , Hillf Danton , Andrew Jones , Dan Smith , Thomas Gleixner , Paul Turner , Suresh Siddha , Mike Galbraith , "Paul E. McKenney" Hi Christoph, On Thu, Oct 04, 2012 at 06:17:37PM +0000, Christoph Lameter wrote: > On Thu, 4 Oct 2012, Andrea Arcangeli wrote: > > > So we could drop page_autonuma by creating a CONFIG_SLUB=y dependency > > (AUTONUMA wouldn't be available in the kernel config if SLAB=y, and it > > also wouldn't be available on 32bit archs but the latter isn't a > > problem). > > Nope it should depend on page struct alignment. Other kernel subsystems > may be depeding on page struct alignment in the future (and some other > arches may already have that requirement) But currently only SLUB x86 64bit selects CONFIG_HAVE_ALIGNED_STRUCT_PAGE: arch/Kconfig:config HAVE_ALIGNED_STRUCT_PAGE arch/x86/Kconfig: select HAVE_ALIGNED_STRUCT_PAGE if SLUB && !M386 include/linux/mm_types.h: defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) include/linux/mm_types.h:#ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE mm/slub.c: defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) mm/slub.c: defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) mm/slub.c: defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) So in practice a dependency on CONFIG_HAVE_ALIGNED_STRUCT_PAGE would still mean the same: only available when SLUB enables it, and only on x86 64bit (ppc64?). If you mean CONFIG_AUTONUMA=y should select (not depend) on CONFIG_HAVE_ALIGNED_STRUCT_PAGE, that would allow to enable it in all .configs but it would have a worse cons: losing 8bytes per page unconditionally (even when booting on non-NUMA hardware). The current page_autonuma solution is substantially memory-cheaper than selecting CONFIG_HAVE_ALIGNED_STRUCT_PAGE: it allocates 2bytes per page at boot time but only if booting on real NUMA hardware (without altering the page structure). So to me it looks still quite a decent tradeoff. Thanks, Andrea -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx112.postini.com [74.125.245.112]) by kanga.kvack.org (Postfix) with SMTP id 2582C6B013A for ; Thu, 4 Oct 2012 15:11:53 -0400 (EDT) Date: Thu, 4 Oct 2012 19:11:51 +0000 From: Christoph Lameter Subject: Re: [PATCH 29/33] autonuma: page_autonuma In-Reply-To: <20121004183819.GM25675@redhat.com> Message-ID: <0000013a2d30ebf2-1a2bb821-92a0-464b-9db0-f960b2fd074d-000000@email.amazonses.com> References: <20121004165008.GF25675@redhat.com> <0000013a2cff3c3d-76e00716-2869-4dc8-8717-82f0136018d0-000000@email.amazonses.com> <20121004183819.GM25675@redhat.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Andrea Arcangeli Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds , Andrew Morton , Peter Zijlstra , Ingo Molnar , Mel Gorman , Hugh Dickins , Rik van Riel , Johannes Weiner , Hillf Danton , Andrew Jones , Dan Smith , Thomas Gleixner , Paul Turner , Suresh Siddha , Mike Galbraith , "Paul E. McKenney" On Thu, 4 Oct 2012, Andrea Arcangeli wrote: > If you mean CONFIG_AUTONUMA=y should select (not depend) on > CONFIG_HAVE_ALIGNED_STRUCT_PAGE, that would allow to enable it in all > .configs but it would have a worse cons: losing 8bytes per page > unconditionally (even when booting on non-NUMA hardware). I did not say anything like that. Still not convinced that autonuma is worth doing and that it is beneficial given the complexity it adds to the kernel. Just wanted to point out that there is a case to be made for adding another word to the page struct. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx188.postini.com [74.125.245.188]) by kanga.kvack.org (Postfix) with SMTP id BE8B66B0044 for ; Fri, 5 Oct 2012 07:12:01 -0400 (EDT) Date: Fri, 5 Oct 2012 13:11:40 +0200 From: Andrea Arcangeli Subject: Re: [PATCH 29/33] autonuma: page_autonuma Message-ID: <20121005111140.GE6793@redhat.com> References: <20121004165008.GF25675@redhat.com> <0000013a2cff3c3d-76e00716-2869-4dc8-8717-82f0136018d0-000000@email.amazonses.com> <20121004183819.GM25675@redhat.com> <0000013a2d30ebf2-1a2bb821-92a0-464b-9db0-f960b2fd074d-000000@email.amazonses.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0000013a2d30ebf2-1a2bb821-92a0-464b-9db0-f960b2fd074d-000000@email.amazonses.com> Sender: owner-linux-mm@kvack.org List-ID: To: Christoph Lameter Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds , Andrew Morton , Peter Zijlstra , Ingo Molnar , Mel Gorman , Hugh Dickins , Rik van Riel , Johannes Weiner , Hillf Danton , Andrew Jones , Dan Smith , Thomas Gleixner , Paul Turner , Suresh Siddha , Mike Galbraith , "Paul E. McKenney" Hi Christoph, On Thu, Oct 04, 2012 at 07:11:51PM +0000, Christoph Lameter wrote: > I did not say anything like that. Still not convinced that autonuma is > worth doing and that it is beneficial given the complexity it adds to the > kernel. Just wanted to point out that there is a case to be made for > adding another word to the page struct. You've seen the benchmarks, no other solution that exists today solves all those cases and never showed a regression compared to upstream. Running that much faster is very beneficial in my view. Expecting the admin of a 2 socket system to use hard bindings manually is unrealistic, even for a 4 socket is unrealistic. If you've 512 node system well then you can afford to setup everything manually and boot with noautonuma, no argument about that. About the complexity, well there's no simple solution to an hard problem. The proof comes from the schednuma crowd that is currently copying the AutoNUMA scheduler cpu-follow-memory design at full force as we speak. Thanks, Andrea -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org