From mboxrd@z Thu Jan 1 00:00:00 1970 From: Glauber Costa Subject: Re: [PATCH v6 00/15] memory-hotplug: hot-remove physical memory Date: Thu, 10 Jan 2013 11:14:16 +0400 Message-ID: <50EE6A48.7060307@parallels.com> References: <1357723959-5416-1-git-send-email-tangchen@cn.fujitsu.com> <20130109142314.1ce04a96.akpm@linux-foundation.org> <50EE24A4.8020601@cn.fujitsu.com> Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <50EE24A4.8020601@cn.fujitsu.com> Sender: owner-linux-mm@kvack.org To: Tang Chen Cc: Andrew Morton , rientjes@google.com, len.brown@intel.com, benh@kernel.crashing.org, paulus@samba.org, cl@linux.com, minchan.kim@gmail.com, kosaki.motohiro@jp.fujitsu.com, isimatu.yasuaki@jp.fujitsu.com, wujianguo@huawei.com, wency@cn.fujitsu.com, hpa@zytor.com, linfeng@cn.fujitsu.com, laijs@cn.fujitsu.com, mgorman@suse.de, yinghai@kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-acpi@vger.kernel.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, linux-ia64@vger.kernel.org, cmetcalf@tilera.com, sparclinux@vger.kernel.org List-Id: linux-acpi@vger.kernel.org On 01/10/2013 06:17 AM, Tang Chen wrote: >>> Note: if the memory provided by the memory device is used by the >>> kernel, it >>> can't be offlined. It is not a bug. >> >> Right. But how often does this happen in testing? In other words, >> please provide an overall description of how well memory hot-remove is >> presently operating. Is it reliable? What is the success rate in >> real-world situations? > > We test the hot-remove functionality mostly with movable_online used. > And the memory used by kernel is not allowed to be removed. Can you try doing this using cpusets configured to hardwall ? It is my understanding that the object allocators will try hard not to allocate anything outside the walls defined by cpuset. Which means that if you have one process per node, and they are hardwalled, your kernel memory will be spread evenly among the machine. With a big enough load, they should eventually be present in all blocks. Another question I have for you: Have you considering calling shrink_slab to try to deplete the caches and therefore free at least slab memory in the nodes that can't be offlined? Is it relevant? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Glauber Costa Date: Thu, 10 Jan 2013 07:14:16 +0000 Subject: Re: [PATCH v6 00/15] memory-hotplug: hot-remove physical memory Message-Id: <50EE6A48.7060307@parallels.com> List-Id: References: <1357723959-5416-1-git-send-email-tangchen@cn.fujitsu.com> <20130109142314.1ce04a96.akpm@linux-foundation.org> <50EE24A4.8020601@cn.fujitsu.com> In-Reply-To: <50EE24A4.8020601@cn.fujitsu.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Tang Chen Cc: Andrew Morton , rientjes@google.com, len.brown@intel.com, benh@kernel.crashing.org, paulus@samba.org, cl@linux.com, minchan.kim@gmail.com, kosaki.motohiro@jp.fujitsu.com, isimatu.yasuaki@jp.fujitsu.com, wujianguo@huawei.com, wency@cn.fujitsu.com, hpa@zytor.com, linfeng@cn.fujitsu.com, laijs@cn.fujitsu.com, mgorman@suse.de, yinghai@kernel.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-acpi@vger.kernel.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, linux-ia64@vger.kernel.org, cmetcalf@tilera.com, sparclinux@vger.kernel.org On 01/10/2013 06:17 AM, Tang Chen wrote: >>> Note: if the memory provided by the memory device is used by the >>> kernel, it >>> can't be offlined. It is not a bug. >> >> Right. But how often does this happen in testing? In other words, >> please provide an overall description of how well memory hot-remove is >> presently operating. Is it reliable? What is the success rate in >> real-world situations? > > We test the hot-remove functionality mostly with movable_online used. > And the memory used by kernel is not allowed to be removed. Can you try doing this using cpusets configured to hardwall ? It is my understanding that the object allocators will try hard not to allocate anything outside the walls defined by cpuset. Which means that if you have one process per node, and they are hardwalled, your kernel memory will be spread evenly among the machine. With a big enough load, they should eventually be present in all blocks. Another question I have for you: Have you considering calling shrink_slab to try to deplete the caches and therefore free at least slab memory in the nodes that can't be offlined? Is it relevant? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx2.parallels.com (mx2.parallels.com [64.131.90.16]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 804FA2C0328 for ; Thu, 10 Jan 2013 18:14:24 +1100 (EST) Message-ID: <50EE6A48.7060307@parallels.com> Date: Thu, 10 Jan 2013 11:14:16 +0400 From: Glauber Costa MIME-Version: 1.0 To: Tang Chen Subject: Re: [PATCH v6 00/15] memory-hotplug: hot-remove physical memory References: <1357723959-5416-1-git-send-email-tangchen@cn.fujitsu.com> <20130109142314.1ce04a96.akpm@linux-foundation.org> <50EE24A4.8020601@cn.fujitsu.com> In-Reply-To: <50EE24A4.8020601@cn.fujitsu.com> Content-Type: text/plain; charset="ISO-8859-1" Cc: linux-ia64@vger.kernel.org, linux-sh@vger.kernel.org, linux-mm@kvack.org, paulus@samba.org, hpa@zytor.com, sparclinux@vger.kernel.org, cl@linux.com, linux-s390@vger.kernel.org, x86@kernel.org, linux-acpi@vger.kernel.org, isimatu.yasuaki@jp.fujitsu.com, linfeng@cn.fujitsu.com, mgorman@suse.de, kosaki.motohiro@jp.fujitsu.com, rientjes@google.com, len.brown@intel.com, wency@cn.fujitsu.com, cmetcalf@tilera.com, wujianguo@huawei.com, yinghai@kernel.org, laijs@cn.fujitsu.com, linux-kernel@vger.kernel.org, minchan.kim@gmail.com, Andrew Morton , linuxppc-dev@lists.ozlabs.org List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 01/10/2013 06:17 AM, Tang Chen wrote: >>> Note: if the memory provided by the memory device is used by the >>> kernel, it >>> can't be offlined. It is not a bug. >> >> Right. But how often does this happen in testing? In other words, >> please provide an overall description of how well memory hot-remove is >> presently operating. Is it reliable? What is the success rate in >> real-world situations? > > We test the hot-remove functionality mostly with movable_online used. > And the memory used by kernel is not allowed to be removed. Can you try doing this using cpusets configured to hardwall ? It is my understanding that the object allocators will try hard not to allocate anything outside the walls defined by cpuset. Which means that if you have one process per node, and they are hardwalled, your kernel memory will be spread evenly among the machine. With a big enough load, they should eventually be present in all blocks. Another question I have for you: Have you considering calling shrink_slab to try to deplete the caches and therefore free at least slab memory in the nodes that can't be offlined? Is it relevant? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932447Ab3AJHOU (ORCPT ); Thu, 10 Jan 2013 02:14:20 -0500 Received: from mx2.parallels.com ([64.131.90.16]:40086 "EHLO mx2.parallels.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752171Ab3AJHOR (ORCPT ); Thu, 10 Jan 2013 02:14:17 -0500 Message-ID: <50EE6A48.7060307@parallels.com> Date: Thu, 10 Jan 2013 11:14:16 +0400 From: Glauber Costa User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Tang Chen CC: Andrew Morton , , , , , , , , , , , , , , , , , , , , , , , , , Subject: Re: [PATCH v6 00/15] memory-hotplug: hot-remove physical memory References: <1357723959-5416-1-git-send-email-tangchen@cn.fujitsu.com> <20130109142314.1ce04a96.akpm@linux-foundation.org> <50EE24A4.8020601@cn.fujitsu.com> In-Reply-To: <50EE24A4.8020601@cn.fujitsu.com> Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 01/10/2013 06:17 AM, Tang Chen wrote: >>> Note: if the memory provided by the memory device is used by the >>> kernel, it >>> can't be offlined. It is not a bug. >> >> Right. But how often does this happen in testing? In other words, >> please provide an overall description of how well memory hot-remove is >> presently operating. Is it reliable? What is the success rate in >> real-world situations? > > We test the hot-remove functionality mostly with movable_online used. > And the memory used by kernel is not allowed to be removed. Can you try doing this using cpusets configured to hardwall ? It is my understanding that the object allocators will try hard not to allocate anything outside the walls defined by cpuset. Which means that if you have one process per node, and they are hardwalled, your kernel memory will be spread evenly among the machine. With a big enough load, they should eventually be present in all blocks. Another question I have for you: Have you considering calling shrink_slab to try to deplete the caches and therefore free at least slab memory in the nodes that can't be offlined? Is it relevant?