[RFC PATCH 0/8] memory-hotplug : hot-remove physical memory(clear page table)

[RFC PATCH 0/8] memory-hotplug : hot-remove physical memory(clear page table)

[Wen Congyang]

This patch series aims to support physical memory hot-remove(clear page table).

This patch series base on ishimatsu's patch series. You can get it here:
http://www.spinics.net/lists/linux-acpi/msg36804.html

The patches can remove following things:
  - page table of removed memory

If you find lack of function for physical memory hot-remove, please let me
know.

Note:
* The patch "remove memory info from list before freeing it" is being disccussed
  in other thread. But for testing the patch series, the patch is needed.
  So I added the patch as [PATCH 0/8].
* You need to apply ishimatsu's patch series first before applying this patch
  series.

Wen Congyang (8):
  memory-hotplug: store the node id in acpi_memory_device
  memory-hotplug: offline memory only when it is onlined
  memory-hotplug: call remove_memory() to cleanup when removing memory
    device
  memory-hotplug: export the function acpi_bus_remove()
  memory-hotplug: call acpi_bus_remove() to remove memory device
  memory-hotplug: introduce new function arch_remove_memory()
  x86: make __split_large_page() generally avialable
  memory-hotplug: implement arch_remove_memory()

 arch/ia64/mm/init.c                  |   16 ++++
 arch/powerpc/mm/mem.c                |   14 +++
 arch/s390/mm/init.c                  |    8 ++
 arch/sh/mm/init.c                    |   15 +++
 arch/tile/mm/init.c                  |    8 ++
 arch/x86/include/asm/pgtable_types.h |    1 +
 arch/x86/mm/init_32.c                |   10 ++
 arch/x86/mm/init_64.c                |  160 ++++++++++++++++++++++++++++++++++
 arch/x86/mm/pageattr.c               |   47 +++++-----
 drivers/acpi/acpi_memhotplug.c       |   24 ++++--
 drivers/acpi/scan.c                  |    3 +-
 include/acpi/acpi_bus.h              |    1 +
 include/linux/memory_hotplug.h       |    1 +
 mm/memory_hotplug.c                  |    2 +-
 14 files changed, 280 insertions(+), 30 deletions(-)

[PATCH 0/8] remove memory info from list before freeing it

[Wen Congyang]

We free info, but we forget to remove it from the list. It will cause
unexpected problem when we access the list next time.

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 drivers/acpi/acpi_memhotplug.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
index 8fe0e02..5cafd6b 100644
--- a/drivers/acpi/acpi_memhotplug.c
+++ b/drivers/acpi/acpi_memhotplug.c
@@ -323,6 +323,7 @@ static int acpi_memory_disable_device(struct acpi_memory_device *mem_device)
 			if (result)
 				return result;
 		}
+		list_del(&info->list);
 		kfree(info);
 	}
 
-- 
1.7.1

[RFC PATCH 1/8] memory-hotplug: store the node id in acpi_memory_device

[Wen Congyang]

The memory device has only one node id. Store the node id when
enabling the memory device, and we can reuse it when removing the
memory device.

CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <liuj97@gmail.com>
CC: Len Brown <len.brown@intel.com>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <paulus@samba.org>
CC: Christoph Lameter <cl@linux.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 drivers/acpi/acpi_memhotplug.c |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
index 5cafd6b..db8de39 100644
--- a/drivers/acpi/acpi_memhotplug.c
+++ b/drivers/acpi/acpi_memhotplug.c
@@ -84,6 +84,7 @@ struct acpi_memory_info {
 struct acpi_memory_device {
 	struct acpi_device * device;
 	unsigned int state;	/* State of the memory device */
+	int nid;
 	struct list_head res_list;
 };
 
@@ -257,6 +258,9 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)
 		info->enabled = 1;
 		num_enabled++;
 	}
+
+	mem_device->nid = node;
+
 	if (!num_enabled) {
 		printk(KERN_ERR PREFIX "add_memory failed\n");
 		mem_device->state = MEMORY_INVALID_STATE;
@@ -463,7 +467,7 @@ static int acpi_memory_device_remove(struct acpi_device *device, int type)
 
 	mem_device = acpi_driver_data(device);
 
-	node = acpi_get_node(mem_device->device->handle);
+	node = mem_device->nid;
 	list_for_each_entry_safe(info, tmp, &mem_device->res_list, list) {
 		if (!info->enabled)
 			continue;
@@ -473,8 +477,6 @@ static int acpi_memory_device_remove(struct acpi_device *device, int type)
 			if (result)
 				return result;
 		}
-		if (node < 0)
-			node = memory_add_physaddr_to_nid(info->start_addr);
 
 		result = remove_memory(node, info->start_addr, info->length);
 		if (result)
-- 
1.7.1

Re: [RFC PATCH 1/8] memory-hotplug: store the node id in acpi_memory_device

[Yasuaki Ishimatsu]

Hi Wen,

2012/07/20 16:09, Wen Congyang wrote:
> The memory device has only one node id. Store the node id when
> enabling the memory device, and we can reuse it when removing the
> memory device.
>
> CC: David Rientjes <rientjes@google.com>
> CC: Jiang Liu <liuj97@gmail.com>
> CC: Len Brown <len.brown@intel.com>
> CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> CC: Paul Mackerras <paulus@samba.org>
> CC: Christoph Lameter <cl@linux.com>
> Cc: Minchan Kim <minchan.kim@gmail.com>
> CC: Andrew Morton <akpm@linux-foundation.org>
> CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
> CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> ---

It looks to me.
Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>

Thanks,
Yasuaki Ishimatsu

>   drivers/acpi/acpi_memhotplug.c |    8 +++++---
>   1 files changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
> index 5cafd6b..db8de39 100644
> --- a/drivers/acpi/acpi_memhotplug.c
> +++ b/drivers/acpi/acpi_memhotplug.c
> @@ -84,6 +84,7 @@ struct acpi_memory_info {
>   struct acpi_memory_device {
>   	struct acpi_device * device;
>   	unsigned int state;	/* State of the memory device */
> +	int nid;
>   	struct list_head res_list;
>   };
>
> @@ -257,6 +258,9 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)
>   		info->enabled = 1;
>   		num_enabled++;
>   	}
> +
> +	mem_device->nid = node;
> +
>   	if (!num_enabled) {
>   		printk(KERN_ERR PREFIX "add_memory failed\n");
>   		mem_device->state = MEMORY_INVALID_STATE;
> @@ -463,7 +467,7 @@ static int acpi_memory_device_remove(struct acpi_device *device, int type)
>
>   	mem_device = acpi_driver_data(device);
>
> -	node = acpi_get_node(mem_device->device->handle);
> +	node = mem_device->nid;
>   	list_for_each_entry_safe(info, tmp, &mem_device->res_list, list) {
>   		if (!info->enabled)
>   			continue;
> @@ -473,8 +477,6 @@ static int acpi_memory_device_remove(struct acpi_device *device, int type)
>   			if (result)
>   				return result;
>   		}
> -		if (node < 0)
> -			node = memory_add_physaddr_to_nid(info->start_addr);
>
>   		result = remove_memory(node, info->start_addr, info->length);
>   		if (result)
>

[RFC PATCH 2/8] memory-hotplug: offline memory only when it is onlined

[Wen Congyang]

offline_memory() will fail if the memory is not onlined. So check
whether the memory is onlined before calling offline_memory().

CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <liuj97@gmail.com>
CC: Len Brown <len.brown@intel.com>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <paulus@samba.org>
CC: Christoph Lameter <cl@linux.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 drivers/acpi/acpi_memhotplug.c |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
index db8de39..712e767 100644
--- a/drivers/acpi/acpi_memhotplug.c
+++ b/drivers/acpi/acpi_memhotplug.c
@@ -323,9 +323,13 @@ static int acpi_memory_disable_device(struct acpi_memory_device *mem_device)
 	 */
 	list_for_each_entry_safe(info, n, &mem_device->res_list, list) {
 		if (info->enabled) {
-			result = offline_memory(info->start_addr, info->length);
-			if (result)
-				return result;
+			if (!is_memblk_offline(info->start_addr,
+					       info->length)) {
+				result = offline_memory(info->start_addr,
+							info->length);
+				if (result)
+					return result;
+			}
 		}
 		list_del(&info->list);
 		kfree(info);
-- 
1.7.1

Re: [RFC PATCH 2/8] memory-hotplug: offline memory only when it is onlined

[Yasuaki Ishimatsu]

Hi Wen,

2012/07/20 16:10, Wen Congyang wrote:
> offline_memory() will fail if the memory is not onlined. So check
> whether the memory is onlined before calling offline_memory().
>
> CC: David Rientjes <rientjes@google.com>
> CC: Jiang Liu <liuj97@gmail.com>
> CC: Len Brown <len.brown@intel.com>
> CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> CC: Paul Mackerras <paulus@samba.org>
> CC: Christoph Lameter <cl@linux.com>
> Cc: Minchan Kim <minchan.kim@gmail.com>
> CC: Andrew Morton <akpm@linux-foundation.org>
> CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
> CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> ---

I have no comment.
Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>

Thanks,
Yasuaki Ishimatsu

>   drivers/acpi/acpi_memhotplug.c |   10 +++++++---
>   1 files changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
> index db8de39..712e767 100644
> --- a/drivers/acpi/acpi_memhotplug.c
> +++ b/drivers/acpi/acpi_memhotplug.c
> @@ -323,9 +323,13 @@ static int acpi_memory_disable_device(struct acpi_memory_device *mem_device)
>   	 */
>   	list_for_each_entry_safe(info, n, &mem_device->res_list, list) {
>   		if (info->enabled) {
> -			result = offline_memory(info->start_addr, info->length);
> -			if (result)
> -				return result;
> +			if (!is_memblk_offline(info->start_addr,
> +					       info->length)) {
> +				result = offline_memory(info->start_addr,
> +							info->length);
> +				if (result)
> +					return result;
> +			}
>   		}
>   		list_del(&info->list);
>   		kfree(info);
>

[RFC PATCH 3/8] memory-hotplug: call remove_memory() to cleanup when removing memory device

[Wen Congyang]

We should remove the following things when removing the memory device:
1. memmap and related sysfs files
2. iomem_resource
3. mem_section and related sysfs files
4. node and related sysfs files

The function remove_memory() can do this. So call it after the memory device
is offlined.

CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <liuj97@gmail.com>
CC: Len Brown <len.brown@intel.com>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <paulus@samba.org>
CC: Christoph Lameter <cl@linux.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 drivers/acpi/acpi_memhotplug.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
index 712e767..58e4e63 100644
--- a/drivers/acpi/acpi_memhotplug.c
+++ b/drivers/acpi/acpi_memhotplug.c
@@ -315,7 +315,7 @@ static int acpi_memory_disable_device(struct acpi_memory_device *mem_device)
 {
 	int result;
 	struct acpi_memory_info *info, *n;
-
+	int node = mem_device->nid;
 
 	/*
 	 * Ask the VM to offline this memory range.
@@ -330,6 +330,11 @@ static int acpi_memory_disable_device(struct acpi_memory_device *mem_device)
 				if (result)
 					return result;
 			}
+
+			result = remove_memory(node, info->start_addr,
+					       info->length);
+			if (result)
+				return result;
 		}
 		list_del(&info->list);
 		kfree(info);
-- 
1.7.1

Re: [RFC PATCH 3/8] memory-hotplug: call remove_memory() to cleanup when removing memory device

[Yasuaki Ishimatsu]

Hi Wen,

2012/07/20 16:10, Wen Congyang wrote:
> We should remove the following things when removing the memory device:
> 1. memmap and related sysfs files
> 2. iomem_resource
> 3. mem_section and related sysfs files
> 4. node and related sysfs files
>
> The function remove_memory() can do this. So call it after the memory device
> is offlined.
>
> CC: David Rientjes <rientjes@google.com>
> CC: Jiang Liu <liuj97@gmail.com>
> CC: Len Brown <len.brown@intel.com>
> CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> CC: Paul Mackerras <paulus@samba.org>
> CC: Christoph Lameter <cl@linux.com>
> Cc: Minchan Kim <minchan.kim@gmail.com>
> CC: Andrew Morton <akpm@linux-foundation.org>
> CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
> CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> ---

I have no comment.
Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>

Thanks,
Yasuaki Ishimatsu


>   drivers/acpi/acpi_memhotplug.c |    7 ++++++-
>   1 files changed, 6 insertions(+), 1 deletions(-)
>
> diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
> index 712e767..58e4e63 100644
> --- a/drivers/acpi/acpi_memhotplug.c
> +++ b/drivers/acpi/acpi_memhotplug.c
> @@ -315,7 +315,7 @@ static int acpi_memory_disable_device(struct acpi_memory_device *mem_device)
>   {
>   	int result;
>   	struct acpi_memory_info *info, *n;
> -
> +	int node = mem_device->nid;
>
>   	/*
>   	 * Ask the VM to offline this memory range.
> @@ -330,6 +330,11 @@ static int acpi_memory_disable_device(struct acpi_memory_device *mem_device)
>   				if (result)
>   					return result;
>   			}
> +
> +			result = remove_memory(node, info->start_addr,
> +					       info->length);
> +			if (result)
> +				return result;
>   		}
>   		list_del(&info->list);
>   		kfree(info);
>

[RFC PATCH 4/8] memory-hotplug: export the function acpi_bus_remove()

[Wen Congyang]

The function acpi_bus_remove() can remove a acpi device from acpi device.
When a acpi device is removed, we need to call this function to remove
the acpi device from acpi bus. So export this function.

CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <liuj97@gmail.com>
CC: Len Brown <len.brown@intel.com>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <paulus@samba.org>
CC: Christoph Lameter <cl@linux.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 drivers/acpi/scan.c     |    3 ++-
 include/acpi/acpi_bus.h |    1 +
 2 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index c8a1f3b..d5e8d74 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1206,7 +1206,7 @@ static int acpi_device_set_context(struct acpi_device *device)
 	return -ENODEV;
 }
 
-static int acpi_bus_remove(struct acpi_device *dev, int rmdevice)
+int acpi_bus_remove(struct acpi_device *dev, int rmdevice)
 {
 	if (!dev)
 		return -EINVAL;
@@ -1228,6 +1228,7 @@ static int acpi_bus_remove(struct acpi_device *dev, int rmdevice)
 
 	return 0;
 }
+EXPORT_SYMBOL(acpi_bus_remove);
 
 static int acpi_add_single_object(struct acpi_device **child,
 				  acpi_handle handle, int type,
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index 9e6e1c6..3eb0ba7 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -356,6 +356,7 @@ bool acpi_bus_power_manageable(acpi_handle handle);
 bool acpi_bus_can_wakeup(acpi_handle handle);
 int acpi_power_resource_register_device(struct device *dev, acpi_handle handle);
 void acpi_power_resource_unregister_device(struct device *dev, acpi_handle handle);
+int acpi_bus_remove(struct acpi_device *dev, int rmdevice);
 #ifdef CONFIG_ACPI_PROC_EVENT
 int acpi_bus_generate_proc_event(struct acpi_device *device, u8 type, int data);
 int acpi_bus_generate_proc_event4(const char *class, const char *bid, u8 type, int data);
-- 
1.7.1

[RFC PATCH 5/8] memory-hotplug: call acpi_bus_remove() to remove memory device

[Wen Congyang]

The memory device has been ejected and powoffed, so we can call
acpi_bus_remove() to remove the memory device from acpi bus.

CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <liuj97@gmail.com>
CC: Len Brown <len.brown@intel.com>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <paulus@samba.org>
CC: Christoph Lameter <cl@linux.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 drivers/acpi/acpi_memhotplug.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/drivers/acpi/acpi_memhotplug.c b/drivers/acpi/acpi_memhotplug.c
index 58e4e63..431a17c 100644
--- a/drivers/acpi/acpi_memhotplug.c
+++ b/drivers/acpi/acpi_memhotplug.c
@@ -402,8 +402,9 @@ static void acpi_memory_device_notify(acpi_handle handle, u32 event, void *data)
 			printk(KERN_ERR PREFIX
 				    "Disable memory device\n");
 		/*
-		 * TBD: Invoke acpi_bus_remove to cleanup data structures
+		 * Invoke acpi_bus_remove() to remove memory device
 		 */
+		acpi_bus_remove(device, 1);
 		break;
 	default:
 		ACPI_DEBUG_PRINT((ACPI_DB_INFO,
-- 
1.7.1

[RFC PATCH 6/8] memory-hotplug: introduce new function arch_remove_memory()

[Wen Congyang]

We don't call __add_pages() directly in the function add_memory()
because some other architecture related thins needs to be done
before or after calling __add_pages(). So we should not call
__remove_pages() directly in the function remove_memory.
Introduce new function arch_remove_memory() to revert the things done
in arch_add_memory().

Note: the function for x86_64 will be implemented later. And I don't
know how to implement it for s390.

CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <liuj97@gmail.com>
CC: Len Brown <len.brown@intel.com>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <paulus@samba.org>
CC: Christoph Lameter <cl@linux.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 arch/ia64/mm/init.c            |   16 ++++++++++++++++
 arch/powerpc/mm/mem.c          |   14 ++++++++++++++
 arch/s390/mm/init.c            |    8 ++++++++
 arch/sh/mm/init.c              |   15 +++++++++++++++
 arch/tile/mm/init.c            |    8 ++++++++
 arch/x86/mm/init_32.c          |   10 ++++++++++
 arch/x86/mm/init_64.c          |    7 +++++++
 include/linux/memory_hotplug.h |    1 +
 mm/memory_hotplug.c            |    2 +-
 9 files changed, 80 insertions(+), 1 deletions(-)

diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
index 0eab454..1e345ed 100644
--- a/arch/ia64/mm/init.c
+++ b/arch/ia64/mm/init.c
@@ -688,6 +688,22 @@ int arch_add_memory(int nid, u64 start, u64 size)
 
 	return ret;
 }
+
+#ifdef CONFIG_MEMORY_HOTREMOVE
+int arch_remove_memory(u64 start, u64 size)
+{
+	unsigned long start_pfn = start >> PAGE_SHIFT;
+	unsigned long nr_pages = size >> PAGE_SHIFT;
+	int ret;
+
+	ret = __remove_pages(start_pfn, nr_pages);
+	if (ret)
+		pr_warn("%s: Problem encountered in __remove_pages() as"
+			" ret=%d\n", __func__,  ret);
+
+	return ret;
+}
+#endif
 #endif
 
 /*
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index baaafde..249cef4 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -133,6 +133,20 @@ int arch_add_memory(int nid, u64 start, u64 size)
 
 	return __add_pages(nid, zone, start_pfn, nr_pages);
 }
+
+#ifdef CONFIG_MEMORY_HOTREMOVE
+int arch_remove_memory(u64 start, u64 size)
+{
+	unsigned long start_pfn = start >> PAGE_SHIFT;
+	unsigned long nr_pages = size >> PAGE_SHIFT;
+
+	start = (unsigned long)__va(start);
+	if (remove_section_mapping(start, start + size))
+		return -EINVAL;
+
+	return __remove_pages(start_pfn, nr_pages);
+}
+#endif
 #endif /* CONFIG_MEMORY_HOTPLUG */
 
 /*
diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
index 2bea060..3de0d5b 100644
--- a/arch/s390/mm/init.c
+++ b/arch/s390/mm/init.c
@@ -259,4 +259,12 @@ int arch_add_memory(int nid, u64 start, u64 size)
 		vmem_remove_mapping(start, size);
 	return rc;
 }
+
+#ifdef CONFIG_MEMORY_HOTREMOVE
+int arch_remove_memory(u64 start, u64 size)
+{
+	/* TODO */
+	return -EBUSY;
+}
+#endif
 #endif /* CONFIG_MEMORY_HOTPLUG */
diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index 82cc576..fc84491 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -558,4 +558,19 @@ int memory_add_physaddr_to_nid(u64 addr)
 EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
 #endif
 
+#ifdef CONFIG_MEMORY_HOTREMOVE
+int arch_remove_memory(u64 start, u64 size)
+{
+	unsigned long start_pfn = start >> PAGE_SHIFT;
+	unsigned long nr_pages = size >> PAGE_SHIFT;
+	int ret;
+
+	ret = __remove_pages(start_pfn, nr_pages);
+	if (unlikely(ret))
+		pr_warn("%s: Failed, __remove_pages() == %d\n", __func__,
+			ret);
+
+	return ret;
+}
+#endif
 #endif /* CONFIG_MEMORY_HOTPLUG */
diff --git a/arch/tile/mm/init.c b/arch/tile/mm/init.c
index 630dd2c..bdd8a99 100644
--- a/arch/tile/mm/init.c
+++ b/arch/tile/mm/init.c
@@ -947,6 +947,14 @@ int remove_memory(u64 start, u64 size)
 {
 	return -EINVAL;
 }
+
+#ifdef CONFIG_MEMORY_HOTREMOVE
+int arch_remove_memory(u64 start, u64 size)
+{
+	/* TODO */
+	return -EBUSY;
+}
+#endif
 #endif
 
 struct kmem_cache *pgd_cache;
diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index 575d86f..a690153 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -842,6 +842,16 @@ int arch_add_memory(int nid, u64 start, u64 size)
 
 	return __add_pages(nid, zone, start_pfn, nr_pages);
 }
+
+#ifdef CONFIG_MEMORY_HOTREMOVE
+int arch_remove_memory(unsigned long start, unsigned long size)
+{
+	unsigned long start_pfn = start >> PAGE_SHIFT;
+	unsigned long nr_pages = size >> PAGE_SHIFT;
+
+	return __remove_pages(start_pfn, nr_pages);
+}
+#endif
 #endif
 
 /*
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 9e635b3..78b94bc 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -675,6 +675,13 @@ int arch_add_memory(int nid, u64 start, u64 size)
 }
 EXPORT_SYMBOL_GPL(arch_add_memory);
 
+#ifdef CONFIG_MEMORY_HOTREMOVE
+int arch_remove_memory(unsigned long start, unsigned long size)
+{
+	/* TODO */
+	return -EBUSY;
+}
+#endif
 #endif /* CONFIG_MEMORY_HOTPLUG */
 
 static struct kcore_list kcore_vsyscall;
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 2ba0a1a..8639799 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -84,6 +84,7 @@ extern void __online_page_free(struct page *page);
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
 extern bool is_pageblock_removable_nolock(struct page *page);
+extern int arch_remove_memory(unsigned long start, unsigned long size);
 #endif /* CONFIG_MEMORY_HOTREMOVE */
 
 /* reasonably generic interface to expand the physical pages in a zone  */
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index dccdf71..cc2c8b9 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1055,7 +1055,7 @@ int __ref remove_memory(int nid, u64 start, u64 size)
 		unregister_one_node(nid);
 	}
 
-	__remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT);
+	arch_remove_memory(start, size);
 out:
 	unlock_memory_hotplug();
 	return ret;
-- 
1.7.1

Re: [RFC PATCH 6/8] memory-hotplug: introduce new function arch_remove_memory()

[Yasuaki Ishimatsu]

2012/07/20 16:12, Wen Congyang wrote:
> We don't call __add_pages() directly in the function add_memory()
> because some other architecture related thins needs to be done
> before or after calling __add_pages(). So we should not call
> __remove_pages() directly in the function remove_memory.
> Introduce new function arch_remove_memory() to revert the things done
> in arch_add_memory().
>
> Note: the function for x86_64 will be implemented later. And I don't
> know how to implement it for s390.

I think you need cc to other arch ML for reviewing the patch.


> CC: David Rientjes <rientjes@google.com>
> CC: Jiang Liu <liuj97@gmail.com>
> CC: Len Brown <len.brown@intel.com>
> CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> CC: Paul Mackerras <paulus@samba.org>
> CC: Christoph Lameter <cl@linux.com>
> Cc: Minchan Kim <minchan.kim@gmail.com>
> CC: Andrew Morton <akpm@linux-foundation.org>
> CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
> CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> ---
>   arch/ia64/mm/init.c            |   16 ++++++++++++++++
>   arch/powerpc/mm/mem.c          |   14 ++++++++++++++
>   arch/s390/mm/init.c            |    8 ++++++++
>   arch/sh/mm/init.c              |   15 +++++++++++++++
>   arch/tile/mm/init.c            |    8 ++++++++
>   arch/x86/mm/init_32.c          |   10 ++++++++++
>   arch/x86/mm/init_64.c          |    7 +++++++
>   include/linux/memory_hotplug.h |    1 +
>   mm/memory_hotplug.c            |    2 +-
>   9 files changed, 80 insertions(+), 1 deletions(-)
>
> diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
> index 0eab454..1e345ed 100644
> --- a/arch/ia64/mm/init.c
> +++ b/arch/ia64/mm/init.c
> @@ -688,6 +688,22 @@ int arch_add_memory(int nid, u64 start, u64 size)
>
>   	return ret;
>   }
> +
> +#ifdef CONFIG_MEMORY_HOTREMOVE
> +int arch_remove_memory(u64 start, u64 size)
> +{
> +	unsigned long start_pfn = start >> PAGE_SHIFT;
> +	unsigned long nr_pages = size >> PAGE_SHIFT;
> +	int ret;
> +
> +	ret = __remove_pages(start_pfn, nr_pages);
> +	if (ret)
> +		pr_warn("%s: Problem encountered in __remove_pages() as"
> +			" ret=%d\n", __func__,  ret);
> +
> +	return ret;
> +}
> +#endif
>   #endif
>
>   /*
> diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
> index baaafde..249cef4 100644
> --- a/arch/powerpc/mm/mem.c
> +++ b/arch/powerpc/mm/mem.c
> @@ -133,6 +133,20 @@ int arch_add_memory(int nid, u64 start, u64 size)
>
>   	return __add_pages(nid, zone, start_pfn, nr_pages);
>   }
> +
> +#ifdef CONFIG_MEMORY_HOTREMOVE
> +int arch_remove_memory(u64 start, u64 size)
> +{
> +	unsigned long start_pfn = start >> PAGE_SHIFT;
> +	unsigned long nr_pages = size >> PAGE_SHIFT;
> +
> +	start = (unsigned long)__va(start);
> +	if (remove_section_mapping(start, start + size))
> +		return -EINVAL;
> +
> +	return __remove_pages(start_pfn, nr_pages);
> +}
> +#endif
>   #endif /* CONFIG_MEMORY_HOTPLUG */
>
>   /*
> diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
> index 2bea060..3de0d5b 100644
> --- a/arch/s390/mm/init.c
> +++ b/arch/s390/mm/init.c
> @@ -259,4 +259,12 @@ int arch_add_memory(int nid, u64 start, u64 size)
>   		vmem_remove_mapping(start, size);
>   	return rc;
>   }
> +
> +#ifdef CONFIG_MEMORY_HOTREMOVE
> +int arch_remove_memory(u64 start, u64 size)
> +{
> +	/* TODO */
> +	return -EBUSY;
> +}
> +#endif
>   #endif /* CONFIG_MEMORY_HOTPLUG */
> diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
> index 82cc576..fc84491 100644
> --- a/arch/sh/mm/init.c
> +++ b/arch/sh/mm/init.c
> @@ -558,4 +558,19 @@ int memory_add_physaddr_to_nid(u64 addr)
>   EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
>   #endif
>
> +#ifdef CONFIG_MEMORY_HOTREMOVE
> +int arch_remove_memory(u64 start, u64 size)
> +{
> +	unsigned long start_pfn = start >> PAGE_SHIFT;
> +	unsigned long nr_pages = size >> PAGE_SHIFT;
> +	int ret;
> +
> +	ret = __remove_pages(start_pfn, nr_pages);
> +	if (unlikely(ret))
> +		pr_warn("%s: Failed, __remove_pages() == %d\n", __func__,
> +			ret);
> +
> +	return ret;
> +}
> +#endif
>   #endif /* CONFIG_MEMORY_HOTPLUG */
> diff --git a/arch/tile/mm/init.c b/arch/tile/mm/init.c
> index 630dd2c..bdd8a99 100644
> --- a/arch/tile/mm/init.c
> +++ b/arch/tile/mm/init.c
> @@ -947,6 +947,14 @@ int remove_memory(u64 start, u64 size)
>   {
>   	return -EINVAL;
>   }
> +
> +#ifdef CONFIG_MEMORY_HOTREMOVE
> +int arch_remove_memory(u64 start, u64 size)
> +{
> +	/* TODO */
> +	return -EBUSY;
> +}
> +#endif
>   #endif
>
>   struct kmem_cache *pgd_cache;
> diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
> index 575d86f..a690153 100644
> --- a/arch/x86/mm/init_32.c
> +++ b/arch/x86/mm/init_32.c
> @@ -842,6 +842,16 @@ int arch_add_memory(int nid, u64 start, u64 size)
>
>   	return __add_pages(nid, zone, start_pfn, nr_pages);
>   }
> +
> +#ifdef CONFIG_MEMORY_HOTREMOVE
> +int arch_remove_memory(unsigned long start, unsigned long size)
> +{
> +	unsigned long start_pfn = start >> PAGE_SHIFT;
> +	unsigned long nr_pages = size >> PAGE_SHIFT;
> +
> +	return __remove_pages(start_pfn, nr_pages);
> +}
> +#endif
>   #endif
>
>   /*
> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
> index 9e635b3..78b94bc 100644
> --- a/arch/x86/mm/init_64.c
> +++ b/arch/x86/mm/init_64.c
> @@ -675,6 +675,13 @@ int arch_add_memory(int nid, u64 start, u64 size)
>   }
>   EXPORT_SYMBOL_GPL(arch_add_memory);
>
> +#ifdef CONFIG_MEMORY_HOTREMOVE
> +int arch_remove_memory(unsigned long start, unsigned long size)
> +{
> +	/* TODO */
> +	return -EBUSY;
> +}
> +#endif

Why does not the function call __remove_pages()?

Thanks,
Yasuaki ishimatsu

>   #endif /* CONFIG_MEMORY_HOTPLUG */
>
>   static struct kcore_list kcore_vsyscall;
> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
> index 2ba0a1a..8639799 100644
> --- a/include/linux/memory_hotplug.h
> +++ b/include/linux/memory_hotplug.h
> @@ -84,6 +84,7 @@ extern void __online_page_free(struct page *page);
>
>   #ifdef CONFIG_MEMORY_HOTREMOVE
>   extern bool is_pageblock_removable_nolock(struct page *page);
> +extern int arch_remove_memory(unsigned long start, unsigned long size);
>   #endif /* CONFIG_MEMORY_HOTREMOVE */
>
>   /* reasonably generic interface to expand the physical pages in a zone  */
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index dccdf71..cc2c8b9 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1055,7 +1055,7 @@ int __ref remove_memory(int nid, u64 start, u64 size)
>   		unregister_one_node(nid);
>   	}
>
> -	__remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT);
> +	arch_remove_memory(start, size);
>   out:
>   	unlock_memory_hotplug();
>   	return ret;
>

Re: [RFC PATCH 6/8] memory-hotplug: introduce new function arch_remove_memory()

[Wen Congyang]

At 07/20/2012 04:21 PM, Yasuaki Ishimatsu Wrote:
> 2012/07/20 16:12, Wen Congyang wrote:
>> We don't call __add_pages() directly in the function add_memory()
>> because some other architecture related thins needs to be done
>> before or after calling __add_pages(). So we should not call
>> __remove_pages() directly in the function remove_memory.
>> Introduce new function arch_remove_memory() to revert the things done
>> in arch_add_memory().
>>
>> Note: the function for x86_64 will be implemented later. And I don't
>> know how to implement it for s390.
> 
> I think you need cc to other arch ML for reviewing the patch.

I forgot to do it.

> 
> 
>> CC: David Rientjes <rientjes@google.com>
>> CC: Jiang Liu <liuj97@gmail.com>
>> CC: Len Brown <len.brown@intel.com>
>> CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
>> CC: Paul Mackerras <paulus@samba.org>
>> CC: Christoph Lameter <cl@linux.com>
>> Cc: Minchan Kim <minchan.kim@gmail.com>
>> CC: Andrew Morton <akpm@linux-foundation.org>
>> CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
>> CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> ---
>>   arch/ia64/mm/init.c            |   16 ++++++++++++++++
>>   arch/powerpc/mm/mem.c          |   14 ++++++++++++++
>>   arch/s390/mm/init.c            |    8 ++++++++
>>   arch/sh/mm/init.c              |   15 +++++++++++++++
>>   arch/tile/mm/init.c            |    8 ++++++++
>>   arch/x86/mm/init_32.c          |   10 ++++++++++
>>   arch/x86/mm/init_64.c          |    7 +++++++
>>   include/linux/memory_hotplug.h |    1 +
>>   mm/memory_hotplug.c            |    2 +-
>>   9 files changed, 80 insertions(+), 1 deletions(-)
>>
>> diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c
>> index 0eab454..1e345ed 100644
>> --- a/arch/ia64/mm/init.c
>> +++ b/arch/ia64/mm/init.c
>> @@ -688,6 +688,22 @@ int arch_add_memory(int nid, u64 start, u64 size)
>>
>>       return ret;
>>   }
>> +
>> +#ifdef CONFIG_MEMORY_HOTREMOVE
>> +int arch_remove_memory(u64 start, u64 size)
>> +{
>> +    unsigned long start_pfn = start >> PAGE_SHIFT;
>> +    unsigned long nr_pages = size >> PAGE_SHIFT;
>> +    int ret;
>> +
>> +    ret = __remove_pages(start_pfn, nr_pages);
>> +    if (ret)
>> +        pr_warn("%s: Problem encountered in __remove_pages() as"
>> +            " ret=%d\n", __func__,  ret);
>> +
>> +    return ret;
>> +}
>> +#endif
>>   #endif
>>
>>   /*
>> diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
>> index baaafde..249cef4 100644
>> --- a/arch/powerpc/mm/mem.c
>> +++ b/arch/powerpc/mm/mem.c
>> @@ -133,6 +133,20 @@ int arch_add_memory(int nid, u64 start, u64 size)
>>
>>       return __add_pages(nid, zone, start_pfn, nr_pages);
>>   }
>> +
>> +#ifdef CONFIG_MEMORY_HOTREMOVE
>> +int arch_remove_memory(u64 start, u64 size)
>> +{
>> +    unsigned long start_pfn = start >> PAGE_SHIFT;
>> +    unsigned long nr_pages = size >> PAGE_SHIFT;
>> +
>> +    start = (unsigned long)__va(start);
>> +    if (remove_section_mapping(start, start + size))
>> +        return -EINVAL;
>> +
>> +    return __remove_pages(start_pfn, nr_pages);
>> +}
>> +#endif
>>   #endif /* CONFIG_MEMORY_HOTPLUG */
>>
>>   /*
>> diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c
>> index 2bea060..3de0d5b 100644
>> --- a/arch/s390/mm/init.c
>> +++ b/arch/s390/mm/init.c
>> @@ -259,4 +259,12 @@ int arch_add_memory(int nid, u64 start, u64 size)
>>           vmem_remove_mapping(start, size);
>>       return rc;
>>   }
>> +
>> +#ifdef CONFIG_MEMORY_HOTREMOVE
>> +int arch_remove_memory(u64 start, u64 size)
>> +{
>> +    /* TODO */
>> +    return -EBUSY;
>> +}
>> +#endif
>>   #endif /* CONFIG_MEMORY_HOTPLUG */
>> diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
>> index 82cc576..fc84491 100644
>> --- a/arch/sh/mm/init.c
>> +++ b/arch/sh/mm/init.c
>> @@ -558,4 +558,19 @@ int memory_add_physaddr_to_nid(u64 addr)
>>   EXPORT_SYMBOL_GPL(memory_add_physaddr_to_nid);
>>   #endif
>>
>> +#ifdef CONFIG_MEMORY_HOTREMOVE
>> +int arch_remove_memory(u64 start, u64 size)
>> +{
>> +    unsigned long start_pfn = start >> PAGE_SHIFT;
>> +    unsigned long nr_pages = size >> PAGE_SHIFT;
>> +    int ret;
>> +
>> +    ret = __remove_pages(start_pfn, nr_pages);
>> +    if (unlikely(ret))
>> +        pr_warn("%s: Failed, __remove_pages() == %d\n", __func__,
>> +            ret);
>> +
>> +    return ret;
>> +}
>> +#endif
>>   #endif /* CONFIG_MEMORY_HOTPLUG */
>> diff --git a/arch/tile/mm/init.c b/arch/tile/mm/init.c
>> index 630dd2c..bdd8a99 100644
>> --- a/arch/tile/mm/init.c
>> +++ b/arch/tile/mm/init.c
>> @@ -947,6 +947,14 @@ int remove_memory(u64 start, u64 size)
>>   {
>>       return -EINVAL;
>>   }
>> +
>> +#ifdef CONFIG_MEMORY_HOTREMOVE
>> +int arch_remove_memory(u64 start, u64 size)
>> +{
>> +    /* TODO */
>> +    return -EBUSY;
>> +}
>> +#endif
>>   #endif
>>
>>   struct kmem_cache *pgd_cache;
>> diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
>> index 575d86f..a690153 100644
>> --- a/arch/x86/mm/init_32.c
>> +++ b/arch/x86/mm/init_32.c
>> @@ -842,6 +842,16 @@ int arch_add_memory(int nid, u64 start, u64 size)
>>
>>       return __add_pages(nid, zone, start_pfn, nr_pages);
>>   }
>> +
>> +#ifdef CONFIG_MEMORY_HOTREMOVE
>> +int arch_remove_memory(unsigned long start, unsigned long size)
>> +{
>> +    unsigned long start_pfn = start >> PAGE_SHIFT;
>> +    unsigned long nr_pages = size >> PAGE_SHIFT;
>> +
>> +    return __remove_pages(start_pfn, nr_pages);
>> +}
>> +#endif
>>   #endif
>>
>>   /*
>> diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
>> index 9e635b3..78b94bc 100644
>> --- a/arch/x86/mm/init_64.c
>> +++ b/arch/x86/mm/init_64.c
>> @@ -675,6 +675,13 @@ int arch_add_memory(int nid, u64 start, u64 size)
>>   }
>>   EXPORT_SYMBOL_GPL(arch_add_memory);
>>
>> +#ifdef CONFIG_MEMORY_HOTREMOVE
>> +int arch_remove_memory(unsigned long start, unsigned long size)
>> +{
>> +    /* TODO */
>> +    return -EBUSY;
>> +}
>> +#endif
> 
> Why does not the function call __remove_pages()?

arch_remove_memory() should clear the page table after remove pages here.
I implement this function in patch 8.

Thanks
Wen Congyang

> 
> Thanks,
> Yasuaki ishimatsu
> 
>>   #endif /* CONFIG_MEMORY_HOTPLUG */
>>
>>   static struct kcore_list kcore_vsyscall;
>> diff --git a/include/linux/memory_hotplug.h
>> b/include/linux/memory_hotplug.h
>> index 2ba0a1a..8639799 100644
>> --- a/include/linux/memory_hotplug.h
>> +++ b/include/linux/memory_hotplug.h
>> @@ -84,6 +84,7 @@ extern void __online_page_free(struct page *page);
>>
>>   #ifdef CONFIG_MEMORY_HOTREMOVE
>>   extern bool is_pageblock_removable_nolock(struct page *page);
>> +extern int arch_remove_memory(unsigned long start, unsigned long size);
>>   #endif /* CONFIG_MEMORY_HOTREMOVE */
>>
>>   /* reasonably generic interface to expand the physical pages in a
>> zone  */
>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>> index dccdf71..cc2c8b9 100644
>> --- a/mm/memory_hotplug.c
>> +++ b/mm/memory_hotplug.c
>> @@ -1055,7 +1055,7 @@ int __ref remove_memory(int nid, u64 start, u64
>> size)
>>           unregister_one_node(nid);
>>       }
>>
>> -    __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT);
>> +    arch_remove_memory(start, size);
>>   out:
>>       unlock_memory_hotplug();
>>       return ret;
>>
> 
> 
> 

[RFC PATCH 7/8] x86: make __split_large_page() generally avialable

[Wen Congyang]

We need to clear page table after calling __remove_pages(). We may
need to clear part of pmd or pud's entry when clearing the page table.
So we need to split it to small page. Make __split_large_page()
generally avialable, and we call call this function to split large page.

CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <liuj97@gmail.com>
CC: Len Brown <len.brown@intel.com>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <paulus@samba.org>
CC: Christoph Lameter <cl@linux.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 arch/x86/include/asm/pgtable_types.h |    1 +
 arch/x86/mm/pageattr.c               |   47 ++++++++++++++++++----------------
 2 files changed, 26 insertions(+), 22 deletions(-)

diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h
index 013286a..b725af2 100644
--- a/arch/x86/include/asm/pgtable_types.h
+++ b/arch/x86/include/asm/pgtable_types.h
@@ -334,6 +334,7 @@ static inline void update_page_count(int level, unsigned long pages) { }
  * as a pte too.
  */
 extern pte_t *lookup_address(unsigned long address, unsigned int *level);
+extern int __split_large_page(pte_t *kpte, unsigned long address, pte_t *pbase);
 
 #endif	/* !__ASSEMBLY__ */
 
diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c
index a718e0d..7dcb6f9 100644
--- a/arch/x86/mm/pageattr.c
+++ b/arch/x86/mm/pageattr.c
@@ -501,21 +501,13 @@ out_unlock:
 	return do_split;
 }
 
-static int split_large_page(pte_t *kpte, unsigned long address)
+int __split_large_page(pte_t *kpte, unsigned long address, pte_t *pbase)
 {
 	unsigned long pfn, pfninc = 1;
 	unsigned int i, level;
-	pte_t *pbase, *tmp;
+	pte_t *tmp;
 	pgprot_t ref_prot;
-	struct page *base;
-
-	if (!debug_pagealloc)
-		spin_unlock(&cpa_lock);
-	base = alloc_pages(GFP_KERNEL | __GFP_NOTRACK, 0);
-	if (!debug_pagealloc)
-		spin_lock(&cpa_lock);
-	if (!base)
-		return -ENOMEM;
+	struct page *base = virt_to_page(pbase);
 
 	spin_lock(&pgd_lock);
 	/*
@@ -523,10 +515,11 @@ static int split_large_page(pte_t *kpte, unsigned long address)
 	 * up for us already:
 	 */
 	tmp = lookup_address(address, &level);
-	if (tmp != kpte)
-		goto out_unlock;
+	if (tmp != kpte) {
+		spin_unlock(&pgd_lock);
+		return 1;
+	}
 
-	pbase = (pte_t *)page_address(base);
 	paravirt_alloc_pte(&init_mm, page_to_pfn(base));
 	ref_prot = pte_pgprot(pte_clrhuge(*kpte));
 	/*
@@ -579,17 +572,27 @@ static int split_large_page(pte_t *kpte, unsigned long address)
 	 * going on.
 	 */
 	__flush_tlb_all();
+	spin_unlock(&pgd_lock);
 
-	base = NULL;
+	return 0;
+}
 
-out_unlock:
-	/*
-	 * If we dropped out via the lookup_address check under
-	 * pgd_lock then stick the page back into the pool:
-	 */
-	if (base)
+static int split_large_page(pte_t *kpte, unsigned long address)
+{
+	pte_t *pbase;
+	struct page *base;
+
+	if (!debug_pagealloc)
+		spin_unlock(&cpa_lock);
+	base = alloc_pages(GFP_KERNEL | __GFP_NOTRACK, 0);
+	if (!debug_pagealloc)
+		spin_lock(&cpa_lock);
+	if (!base)
+		return -ENOMEM;
+
+	pbase = (pte_t *)page_address(base);
+	if (__split_large_page(kpte, address, pbase))
 		__free_page(base);
-	spin_unlock(&pgd_lock);
 
 	return 0;
 }
-- 
1.7.1

[RFC PATCH 8/8] memory-hotplug: implement arch_remove_memory()

[Wen Congyang]

Set the entry for the removed memory to 0. If the entry related meory
is not whole removed, split it to smaller page, and clear it.

CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <liuj97@gmail.com>
CC: Len Brown <len.brown@intel.com>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <paulus@samba.org>
CC: Christoph Lameter <cl@linux.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
CC: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 arch/x86/mm/init_64.c |  159 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 156 insertions(+), 3 deletions(-)

diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 78b94bc..d78f352 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -675,11 +675,164 @@ int arch_add_memory(int nid, u64 start, u64 size)
 }
 EXPORT_SYMBOL_GPL(arch_add_memory);
 
+static void __meminit
+phys_pte_remove(pte_t *pte_page, unsigned long addr, unsigned long end)
+{
+	unsigned pages = 0;
+	int i = pte_index(addr);
+
+	pte_t *pte = pte_page + pte_index(addr);
+
+	for (; i < PTRS_PER_PTE; i++, addr += PAGE_SIZE, pte++) {
+
+		if (addr >= end)
+			break;
+
+		if (!pte_present(*pte))
+			continue;
+
+		pages++;
+		set_pte(pte, __pte(0));
+	}
+
+	update_page_count(PG_LEVEL_4K, -pages);
+}
+
+static void __meminit
+phys_pmd_remove(pmd_t *pmd_page, unsigned long addr, unsigned long end)
+{
+	unsigned long pages = 0, next;
+	int i = pmd_index(addr);
+
+	for (; i < PTRS_PER_PMD; i++, addr = next) {
+		unsigned long pte_phys;
+		pmd_t *pmd = pmd_page + pmd_index(addr);
+		pte_t *pte;
+
+		if (addr >= end)
+			break;
+
+		next = (addr & PMD_MASK) + PMD_SIZE;
+
+		if (!pmd_present(*pmd))
+			continue;
+
+		if (pmd_large(*pmd)) {
+			if ((addr & ~PMD_MASK) == 0 && next <= end) {
+				set_pmd(pmd, __pmd(0));
+				pages++;
+				continue;
+			}
+
+			/*
+			 * We use 2M page, but we need to remove part of them,
+			 * so split 2M page to 4K page.
+			 */
+			pte = alloc_low_page(&pte_phys);
+			__split_large_page((pte_t *)pmd, addr, pte);
+
+			spin_lock(&init_mm.page_table_lock);
+			pmd_populate_kernel(&init_mm, pmd, __va(pte_phys));
+			spin_unlock(&init_mm.page_table_lock);
+		}
+
+		spin_lock(&init_mm.page_table_lock);
+		pte = map_low_page((pte_t *)pmd_page_vaddr(*pmd));
+		phys_pte_remove(pte, addr, end);
+		unmap_low_page(pte);
+		spin_unlock(&init_mm.page_table_lock);
+	}
+	update_page_count(PG_LEVEL_2M, -pages);
+}
+
+static void __meminit
+phys_pud_remove(pud_t *pud_page, unsigned long addr, unsigned long end)
+{
+	unsigned long pages = 0, next;
+	int i = pud_index(addr);
+
+	for (; i < PTRS_PER_PUD; i++, addr = next) {
+		unsigned long pmd_phys;
+		pud_t *pud = pud_page + pud_index(addr);
+		pmd_t *pmd;
+
+		if (addr >= end)
+			break;
+
+		next = (addr & PUD_MASK) + PUD_SIZE;
+
+		if (!pud_present(*pud))
+			continue;
+
+		if (pud_large(*pud)) {
+			if ((addr & ~PUD_MASK) == 0 && next <= end) {
+				set_pud(pud, __pud(0));
+				pages++;
+				continue;
+			}
+
+			/*
+			 * We use 1G page, but we need to remove part of them,
+			 * so split 1G page to 2M page.
+			 */
+			pmd = alloc_low_page(&pmd_phys);
+			__split_large_page((pte_t *)pud, addr, (pte_t *)pmd);
+
+			spin_lock(&init_mm.page_table_lock);
+			pud_populate(&init_mm, pud, __va(pmd_phys));
+			spin_unlock(&init_mm.page_table_lock);
+		}
+
+		pmd = map_low_page(pmd_offset(pud, 0));
+		phys_pmd_remove(pmd, addr, end);
+		unmap_low_page(pmd);
+		__flush_tlb_all();
+	}
+	__flush_tlb_all();
+
+	update_page_count(PG_LEVEL_1G, -pages);
+}
+
+void __meminit
+kernel_physical_mapping_remove(unsigned long start, unsigned long end)
+{
+	unsigned long next;
+
+	start = (unsigned long)__va(start);
+	end = (unsigned long)__va(end);
+
+	for (; start < end; start = next) {
+		pgd_t *pgd = pgd_offset_k(start);
+		pud_t *pud;
+
+		next = (start + PGDIR_SIZE) & PGDIR_MASK;
+		if (next > end)
+			next = end;
+
+		if (!pgd_present(*pgd))
+			continue;
+
+		pud = map_low_page((pud_t *)pgd_page_vaddr(*pgd));
+		phys_pud_remove(pud, __pa(start), __pa(end));
+		unmap_low_page(pud);
+	}
+
+	__flush_tlb_all();
+}
+
 #ifdef CONFIG_MEMORY_HOTREMOVE
-int arch_remove_memory(unsigned long start, unsigned long size)
+int __ref arch_remove_memory(unsigned long start, unsigned long size)
 {
-	/* TODO */
-	return -EBUSY;
+	unsigned long start_pfn = start >> PAGE_SHIFT;
+	unsigned long nr_pages = size >> PAGE_SHIFT;
+	int ret;
+
+	ret = __remove_pages(start_pfn, nr_pages);
+	WARN_ON_ONCE(ret);
+
+	kernel_physical_mapping_remove(start, start + size);
+
+	return ret;
 }
 #endif
 #endif /* CONFIG_MEMORY_HOTPLUG */
-- 
1.7.1

Re: [RFC PATCH 0/8] memory-hotplug : hot-remove physical memory(clear page table)

[Yasuaki Ishimatsu]

[Hi Wen,

Good news!! I was waiting for this patch to come.
Applying the patches, can we hot-remove physical memory completely?

Thanks,
Yasuaki Ishimatsu

2012/07/20 16:06, Wen Congyang wrote:
> This patch series aims to support physical memory hot-remove(clear page table).
>
> This patch series base on ishimatsu's patch series. You can get it here:
> http://www.spinics.net/lists/linux-acpi/msg36804.html
>
> The patches can remove following things:
>    - page table of removed memory
>
> If you find lack of function for physical memory hot-remove, please let me
> know.
>
> Note:
> * The patch "remove memory info from list before freeing it" is being disccussed
>    in other thread. But for testing the patch series, the patch is needed.
>    So I added the patch as [PATCH 0/8].
> * You need to apply ishimatsu's patch series first before applying this patch
>    series.
>
> Wen Congyang (8):
>    memory-hotplug: store the node id in acpi_memory_device
>    memory-hotplug: offline memory only when it is onlined
>    memory-hotplug: call remove_memory() to cleanup when removing memory
>      device
>    memory-hotplug: export the function acpi_bus_remove()
>    memory-hotplug: call acpi_bus_remove() to remove memory device
>    memory-hotplug: introduce new function arch_remove_memory()
>    x86: make __split_large_page() generally avialable
>    memory-hotplug: implement arch_remove_memory()
>
>   arch/ia64/mm/init.c                  |   16 ++++
>   arch/powerpc/mm/mem.c                |   14 +++
>   arch/s390/mm/init.c                  |    8 ++
>   arch/sh/mm/init.c                    |   15 +++
>   arch/tile/mm/init.c                  |    8 ++
>   arch/x86/include/asm/pgtable_types.h |    1 +
>   arch/x86/mm/init_32.c                |   10 ++
>   arch/x86/mm/init_64.c                |  160 ++++++++++++++++++++++++++++++++++
>   arch/x86/mm/pageattr.c               |   47 +++++-----
>   drivers/acpi/acpi_memhotplug.c       |   24 ++++--
>   drivers/acpi/scan.c                  |    3 +-
>   include/acpi/acpi_bus.h              |    1 +
>   include/linux/memory_hotplug.h       |    1 +
>   mm/memory_hotplug.c                  |    2 +-
>   14 files changed, 280 insertions(+), 30 deletions(-)
>

Re: [RFC PATCH 0/8] memory-hotplug : hot-remove physical memory(clear page table)

[Wen Congyang]

At 07/20/2012 03:31 PM, Yasuaki Ishimatsu Wrote:
> [Hi Wen,
> 
> Good news!! I was waiting for this patch to come.
> Applying the patches, can we hot-remove physical memory completely?

If all functions success, I guess so.

Thanks
Wen Congyang

> 
> Thanks,
> Yasuaki Ishimatsu
> 
> 2012/07/20 16:06, Wen Congyang wrote:
>> This patch series aims to support physical memory hot-remove(clear
>> page table).
>>
>> This patch series base on ishimatsu's patch series. You can get it here:
>> http://www.spinics.net/lists/linux-acpi/msg36804.html
>>
>> The patches can remove following things:
>>    - page table of removed memory
>>
>> If you find lack of function for physical memory hot-remove, please
>> let me
>> know.
>>
>> Note:
>> * The patch "remove memory info from list before freeing it" is being
>> disccussed
>>    in other thread. But for testing the patch series, the patch is
>> needed.
>>    So I added the patch as [PATCH 0/8].
>> * You need to apply ishimatsu's patch series first before applying
>> this patch
>>    series.
>>
>> Wen Congyang (8):
>>    memory-hotplug: store the node id in acpi_memory_device
>>    memory-hotplug: offline memory only when it is onlined
>>    memory-hotplug: call remove_memory() to cleanup when removing memory
>>      device
>>    memory-hotplug: export the function acpi_bus_remove()
>>    memory-hotplug: call acpi_bus_remove() to remove memory device
>>    memory-hotplug: introduce new function arch_remove_memory()
>>    x86: make __split_large_page() generally avialable
>>    memory-hotplug: implement arch_remove_memory()
>>
>>   arch/ia64/mm/init.c                  |   16 ++++
>>   arch/powerpc/mm/mem.c                |   14 +++
>>   arch/s390/mm/init.c                  |    8 ++
>>   arch/sh/mm/init.c                    |   15 +++
>>   arch/tile/mm/init.c                  |    8 ++
>>   arch/x86/include/asm/pgtable_types.h |    1 +
>>   arch/x86/mm/init_32.c                |   10 ++
>>   arch/x86/mm/init_64.c                |  160
>> ++++++++++++++++++++++++++++++++++
>>   arch/x86/mm/pageattr.c               |   47 +++++-----
>>   drivers/acpi/acpi_memhotplug.c       |   24 ++++--
>>   drivers/acpi/scan.c                  |    3 +-
>>   include/acpi/acpi_bus.h              |    1 +
>>   include/linux/memory_hotplug.h       |    1 +
>>   mm/memory_hotplug.c                  |    2 +-
>>   14 files changed, 280 insertions(+), 30 deletions(-)
>>
> 
> 
>