* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Tony Luck @ 2014-07-21 17:41 UTC (permalink / raw)
To: Nishanth Aravamudan
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
linux-mm@kvack.org, linux-hotplug, Linux Kernel Mailing List
In-Reply-To: <20140721172331.GB4156@linux.vnet.ibm.com>
On Mon, Jul 21, 2014 at 10:23 AM, Nishanth Aravamudan
<nacc@linux.vnet.ibm.com> wrote:
> It seems like the issue is the order of onlining of resources on a
> specific x86 platform?
Yes. When we online a node the BIOS hits us with some ACPI hotplug events:
First: Here are some new cpus
Next: Here is some new memory
Last; Here are some new I/O things (PCIe root ports, PCIe devices,
IOAPICs, IOMMUs, ...)
So there is a period where the node is memoryless - although that will generally
be resolved when the memory hot plug event arrives ... that isn't guaranteed to
occur (there might not be any memory on the node, or what memory there is
may have failed self-test and been disabled).
-Tony
^ permalink raw reply
* Re: [RFC Patch V1 15/30] mm, igb: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Nishanth Aravamudan @ 2014-07-21 17:42 UTC (permalink / raw)
To: Jiang Liu
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Jeff Kirsher,
Jesse Brandeburg, Bruce Allan, Carolyn Wyborny, Don Skidmore,
Greg Rose, Alex Duyck, John Ronciak, Mitch Williams, Linux NICS,
Tony Luck, linux-mm, linux-hotplug, linux-kernel, e1000-devel,
netdev
In-Reply-To: <1405064267-11678-16-git-send-email-jiang.liu@linux.intel.com>
On 11.07.2014 [15:37:32 +0800], Jiang Liu wrote:
> When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
> may return a node without memory, and later cause system failure/panic
> when calling kmalloc_node() and friends with returned node id.
> So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
> memory for the/current cpu.
>
> If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
> is the same as cpu_to_node()/numa_node_id().
>
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> ---
> drivers/net/ethernet/intel/igb/igb_main.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
> index f145adbb55ac..2b74bffa5648 100644
> --- a/drivers/net/ethernet/intel/igb/igb_main.c
> +++ b/drivers/net/ethernet/intel/igb/igb_main.c
> @@ -6518,7 +6518,7 @@ static bool igb_can_reuse_rx_page(struct igb_rx_buffer *rx_buffer,
> unsigned int truesize)
> {
> /* avoid re-using remote pages */
> - if (unlikely(page_to_nid(page) != numa_node_id()))
> + if (unlikely(page_to_nid(page) != numa_mem_id()))
> return false;
>
> #if (PAGE_SIZE < 8192)
> @@ -6588,7 +6588,7 @@ static bool igb_add_rx_frag(struct igb_ring *rx_ring,
> memcpy(__skb_put(skb, size), va, ALIGN(size, sizeof(long)));
>
> /* we can reuse buffer as-is, just make sure it is local */
> - if (likely(page_to_nid(page) = numa_node_id()))
> + if (likely(page_to_nid(page) = numa_mem_id()))
> return true;
>
> /* this page cannot be reused so discard it */
This doesn't seem to have anything to do with crashes or errors?
The original code is checking if the NUMA node of a page is remote to
the NUMA node current is running on. Your change makes it check if the
NUMA node of a page is not equal to the nearest NUMA node with memory.
That's not necessarily local, though, which seems like that is the whole
point. In this case, perhaps the driver author doesn't want to reuse the
memory at all for performance reasons? In any case, I don't think this
patch has appropriate justification.
Thanks,
Nish
^ permalink raw reply
* Re: [RFC Patch V1 28/30] mm: Update _mem_id_[] for every possible CPU when memory configuration chan
From: Nishanth Aravamudan @ 2014-07-21 17:47 UTC (permalink / raw)
To: Jiang Liu
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Rik van Riel, Johannes Weiner,
Srivatsa S. Bhat, Dave Hansen, Tony Luck, linux-mm, linux-hotplug,
linux-kernel
In-Reply-To: <1405064267-11678-29-git-send-email-jiang.liu@linux.intel.com>
On 11.07.2014 [15:37:45 +0800], Jiang Liu wrote:
> Current kernel only updates _mem_id_[cpu] for onlined CPUs when memory
> configuration changes. So kernel may allocate memory from remote node
> for a CPU if the CPU is still in absent or offline state even if the
> node associated with the CPU has already been onlined.
This just sounds like the topology information is being updated at the
wrong place/time? That is, the memory is online, the CPU is being
brought online, but isn't associated with any node?
> This patch tries to improve performance by updating _mem_id_[cpu] for
> each possible CPU when memory configuration changes, thus kernel could
> always allocate from local node once the node is onlined.
Ok, what is the impact? Do you actually see better performance?
> We check node_online(cpu_to_node(cpu)) because:
> 1) local_memory_node(nid) needs to access NODE_DATA(nid)
> 2) try_offline_node(nid) just zeroes out NODE_DATA(nid) instead of free it
>
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> ---
> mm/page_alloc.c | 10 +++++-----
> 1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 0ea758b898fd..de86e941ed57 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3844,13 +3844,13 @@ static int __build_all_zonelists(void *data)
> /*
> * We now know the "local memory node" for each node--
> * i.e., the node of the first zone in the generic zonelist.
> - * Set up numa_mem percpu variable for on-line cpus. During
> - * boot, only the boot cpu should be on-line; we'll init the
> - * secondary cpus' numa_mem as they come on-line. During
> - * node/memory hotplug, we'll fixup all on-line cpus.
> + * Set up numa_mem percpu variable for all possible cpus
> + * if associated node has been onlined.
> */
> - if (cpu_online(cpu))
> + if (node_online(cpu_to_node(cpu)))
> set_cpu_numa_mem(cpu, local_memory_node(cpu_to_node(cpu)));
> + else
> + set_cpu_numa_mem(cpu, NUMA_NO_NODE);
> #endif
^ permalink raw reply
* Re: [RFC Patch V1 22/30] mm, of: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Nishanth Aravamudan @ 2014-07-21 17:52 UTC (permalink / raw)
To: Jiang Liu
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Grant Likely, Rob Herring,
Tony Luck, linux-mm, linux-hotplug, linux-kernel, devicetree
In-Reply-To: <1405064267-11678-23-git-send-email-jiang.liu@linux.intel.com>
On 11.07.2014 [15:37:39 +0800], Jiang Liu wrote:
> When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
> may return a node without memory, and later cause system failure/panic
> when calling kmalloc_node() and friends with returned node id.
> So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
> memory for the/current cpu.
>
> If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
> is the same as cpu_to_node()/numa_node_id().
>
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> ---
> drivers/of/base.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/of/base.c b/drivers/of/base.c
> index b9864806e9b8..40d4772973ad 100644
> --- a/drivers/of/base.c
> +++ b/drivers/of/base.c
> @@ -85,7 +85,7 @@ EXPORT_SYMBOL(of_n_size_cells);
> #ifdef CONFIG_NUMA
> int __weak of_node_to_nid(struct device_node *np)
> {
> - return numa_node_id();
> + return numa_mem_id();
> }
> #endif
Um, NAK. of_node_to_nid() returns the NUMA node ID for a given device
tree node. The default should be the physically local NUMA node, not the
nearest memory-containing node.
I think the general direction of this patchset is good -- what NUMA
information do we actually are about at each callsite. But the execution
is blind and doesn't consider at all what the code is actually doing.
The changelogs are all identical and don't actually provide any
information about what errors this (or any) specific patch are
resolving.
Thanks,
Nish
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Nishanth Aravamudan @ 2014-07-21 17:57 UTC (permalink / raw)
To: Tony Luck
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
linux-mm@kvack.org, linux-hotplug, Linux Kernel Mailing List
In-Reply-To: <CA+8MBbK+ZdisT_yXh_jkWSd4hWEMisG614s4s0EyNV3j-7YOow@mail.gmail.com>
On 21.07.2014 [10:41:59 -0700], Tony Luck wrote:
> On Mon, Jul 21, 2014 at 10:23 AM, Nishanth Aravamudan
> <nacc@linux.vnet.ibm.com> wrote:
> > It seems like the issue is the order of onlining of resources on a
> > specific x86 platform?
>
> Yes. When we online a node the BIOS hits us with some ACPI hotplug events:
>
> First: Here are some new cpus
Ok, so during this period, you might get some remote allocations. Do you
know the topology of these CPUs? That is they belong to a
(soon-to-exist) NUMA node? Can you online that currently offline NUMA
node at this point (so that NODE_DATA()) resolves, etc.)?
> Next: Here is some new memory
And then update the NUMA topology at this point? That is,
set_cpu_numa_node/mem as appropriate so the underlying allocators do the
right thing?
> Last; Here are some new I/O things (PCIe root ports, PCIe devices,
> IOAPICs, IOMMUs, ...)
>
> So there is a period where the node is memoryless - although that will
> generally be resolved when the memory hot plug event arrives ... that
> isn't guaranteed to occur (there might not be any memory on the node,
> or what memory there is may have failed self-test and been disabled).
Right, but the allocator(s) generally does the right thing already in
the face of memoryless nodes -- they fallback to the nearest node. That
leads to poor performance, but is functional. Based upon the previous
thread Jiang pointed to, it seems like the real issue here isn't that
the node is memoryless, but that it's not even online yet? So NODE_DATA
access crashes?
Thanks,
Nish
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Peter Zijlstra @ 2014-07-21 20:06 UTC (permalink / raw)
To: Tony Luck
Cc: Nishanth Aravamudan, Jiang Liu, Andrew Morton, Mel Gorman,
David Rientjes, Mike Galbraith, Rafael J . Wysocki,
linux-mm@kvack.org, linux-hotplug, Linux Kernel Mailing List
In-Reply-To: <CA+8MBbK+ZdisT_yXh_jkWSd4hWEMisG614s4s0EyNV3j-7YOow@mail.gmail.com>
On Mon, Jul 21, 2014 at 10:41:59AM -0700, Tony Luck wrote:
> On Mon, Jul 21, 2014 at 10:23 AM, Nishanth Aravamudan
> <nacc@linux.vnet.ibm.com> wrote:
> > It seems like the issue is the order of onlining of resources on a
> > specific x86 platform?
>
> Yes. When we online a node the BIOS hits us with some ACPI hotplug events:
>
> First: Here are some new cpus
> Next: Here is some new memory
> Last; Here are some new I/O things (PCIe root ports, PCIe devices,
> IOAPICs, IOMMUs, ...)
>
> So there is a period where the node is memoryless - although that will generally
> be resolved when the memory hot plug event arrives ... that isn't guaranteed to
> occur (there might not be any memory on the node, or what memory there is
> may have failed self-test and been disabled).
Right, but we could 'easily' capture that in arch code and make it look
like it was done in a 'sane' order. No need to wreck the rest of the
kernel to support this particular BIOS fuckup.
^ permalink raw reply
* Re: [RFC Patch V1 15/30] mm, igb: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Nishanth Aravamudan @ 2014-07-21 21:09 UTC (permalink / raw)
To: Alexander Duyck
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki, Jeff Kirsher,
Jesse Brandeburg, Bruce Allan, Carolyn Wyborny, Don Skidmore,
Greg Rose, Alex Duyck, John Ronciak, Mitch Williams, Linux NICS,
Tony Luck, linux-mm, linux-hotplug, linux-kernel, e1000-devel,
Netdev
In-Reply-To: <CAKgT0UdZdbduP-=R7uRCxJVxt1yCDoHpnercnDoyrCbWNtx=6Q@mail.gmail.com>
On 21.07.2014 [12:53:33 -0700], Alexander Duyck wrote:
> I do agree the description should probably be changed. There shouldn't be
> any panics involved, only a performance impact as it will be reallocating
> always if it is on a node with no memory.
Yep, thanks for the review.
> My intention on this was to make certain that the memory used is from the
> closest node possible. As such I believe this change likely honours that.
Absolutely, just wanted to make it explicit that it's not a functional
fix, just a performance fix (presuming this shows up at all on systems
that have memoryless NUMA nodes).
I'd suggest an update to the comments, as well.
Thanks,
Nish
^ permalink raw reply
* Re: [RFC Patch V1 07/30] mm: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Jiang Liu @ 2014-07-23 3:16 UTC (permalink / raw)
To: Christoph Lameter, Tejun Heo
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Vladimir Davydov,
Johannes Weiner, Kirill A. Shutemov, Rik van Riel, Wanpeng Li,
Zhang Yanfei, Catalin Marinas, Jianyu Zhan, malc, Joonsoo Kim,
Fabian Frederick, Tony Luck, linux-mm, linux-hotplug,
linux-kernel
In-Reply-To: <alpine.DEB.2.11.1407111405280.5070@gentwo.org>
Hi Tejun and Christoph,
Thanks for your suggestions and discussion. Tejun really
gives a good point to hide memoryless node interface from normal
slab users. I will rework the patch set to go that direction.
Regards!
Gerry
On 2014/7/12 3:11, Christoph Lameter wrote:
> On Fri, 11 Jul 2014, Tejun Heo wrote:
>
>> On Fri, Jul 11, 2014 at 12:29:30PM -0500, Christoph Lameter wrote:
>>> GFP_THISNODE is mostly used by allocators that need memory from specific
>>> nodes. The use of numa_mem_id() there is useful because one will not
>>> get any memory at all when attempting to allocate from a memoryless
>>> node using GFP_THISNODE.
>>
>> As long as it's in allocator proper, it doesn't matter all that much
>> but the changes are clearly not contained, are they?
>
> Well there is a proliferation of memory allocators recently. NUMA is often
> a second thought in those.
>
^ permalink raw reply
* Re: [RFC Patch V1 09/30] mm, memcg: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Jiang Liu @ 2014-07-23 3:18 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Johannes Weiner, Tony Luck,
linux-mm, linux-hotplug, linux-kernel, cgroups
In-Reply-To: <20140718073614.GC21453@dhcp22.suse.cz>
Hi Michal,
Thanks for your comments! As discussed, we will
rework the patch set in another direction to hide memoryless
node from normal slab users.
Regards!
Gerry
On 2014/7/18 15:36, Michal Hocko wrote:
> On Fri 11-07-14 15:37:26, Jiang Liu wrote:
>> When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
>> may return a node without memory, and later cause system failure/panic
>> when calling kmalloc_node() and friends with returned node id.
>> So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
>> memory for the/current cpu.
>>
>> If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
>> is the same as cpu_to_node()/numa_node_id().
>
> The change makes difference only for really tiny memcgs. If we really
> have all pages on unevictable list or anon with no swap allowed and that
> is the reason why no node is set in scan_nodes mask then reclaiming
> memoryless node or any arbitrary close one doesn't make any difference.
> The current memcg might not have any memory on that node at all.
>
> So the change doesn't make any practical difference and the changelog is
> misleading.
>
>> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
>> ---
>> mm/memcontrol.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
>> index a2c7bcb0e6eb..d6c4b7255ca9 100644
>> --- a/mm/memcontrol.c
>> +++ b/mm/memcontrol.c
>> @@ -1933,7 +1933,7 @@ int mem_cgroup_select_victim_node(struct mem_cgroup *memcg)
>> * we use curret node.
>> */
>> if (unlikely(node = MAX_NUMNODES))
>> - node = numa_node_id();
>> + node = numa_mem_id();
>>
>> memcg->last_scanned_node = node;
>> return node;
>> --
>> 1.7.10.4
>>
>
^ permalink raw reply
* Re: [RFC Patch V1 15/30] mm, igb: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Jiang Liu @ 2014-07-23 3:20 UTC (permalink / raw)
To: Nishanth Aravamudan, Alexander Duyck
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Jeff Kirsher,
Jesse Brandeburg, Bruce Allan, Carolyn Wyborny, Don Skidmore,
Greg Rose, Alex Duyck, John Ronciak, Mitch Williams, Linux NICS,
Tony Luck, linux-mm, linux-hotplug, linux-kernel, e1000-devel,
Netdev
In-Reply-To: <20140721210900.GI4156@linux.vnet.ibm.com>
Hi Nishanth and Alexander,
Thanks for review, will update the comments
in next version.
Regards!
Gerry
On 2014/7/22 5:09, Nishanth Aravamudan wrote:
> On 21.07.2014 [12:53:33 -0700], Alexander Duyck wrote:
>> I do agree the description should probably be changed. There shouldn't be
>> any panics involved, only a performance impact as it will be reallocating
>> always if it is on a node with no memory.
>
> Yep, thanks for the review.
>
>> My intention on this was to make certain that the memory used is from the
>> closest node possible. As such I believe this change likely honours that.
>
> Absolutely, just wanted to make it explicit that it's not a functional
> fix, just a performance fix (presuming this shows up at all on systems
> that have memoryless NUMA nodes).
>
> I'd suggest an update to the comments, as well.
>
> Thanks,
> Nish
>
^ permalink raw reply
* Re: [RFC Patch V1 21/30] mm, irqchip: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Jiang Liu @ 2014-07-23 3:47 UTC (permalink / raw)
To: Jason Cooper
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Thomas Gleixner,
Alexander Shiyan, Tony Luck, linux-mm, linux-hotplug,
linux-kernel, linux-arm-kernel
In-Reply-To: <20140718124038.GE24496@titan.lakedaemon.net>
Hi Jason,
Thanks for your review. According to review comments,
we need to rework the patch set in another direction and will
give up this patch.
Regards!
Gerry
On 2014/7/18 20:40, Jason Cooper wrote:
> On Fri, Jul 11, 2014 at 03:37:38PM +0800, Jiang Liu wrote:
>> When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
>> may return a node without memory, and later cause system failure/panic
>> when calling kmalloc_node() and friends with returned node id.
>> So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
>> memory for the/current cpu.
>>
>> If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
>> is the same as cpu_to_node()/numa_node_id().
>>
>> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
>> ---
>> drivers/irqchip/irq-clps711x.c | 2 +-
>> drivers/irqchip/irq-gic.c | 2 +-
>> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> Do you have anything depending on this? Can apply it to irqchip? If
> you need to keep it with other changes,
>
> Acked-by: Jason Cooper <jason@lakedaemon.net>
>
> But please do let me know if I can take it.
>
> thx,
>
> Jason.
>
^ permalink raw reply
* Re: [RFC Patch V1 28/30] mm: Update _mem_id_[] for every possible CPU when memory configuration chan
From: Jiang Liu @ 2014-07-23 8:16 UTC (permalink / raw)
To: Nishanth Aravamudan
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Rik van Riel, Johannes Weiner,
Srivatsa S. Bhat, Dave Hansen, Tony Luck, linux-mm, linux-hotplug,
linux-kernel
In-Reply-To: <20140721174754.GE4156@linux.vnet.ibm.com>
On 2014/7/22 1:47, Nishanth Aravamudan wrote:
> On 11.07.2014 [15:37:45 +0800], Jiang Liu wrote:
>> Current kernel only updates _mem_id_[cpu] for onlined CPUs when memory
>> configuration changes. So kernel may allocate memory from remote node
>> for a CPU if the CPU is still in absent or offline state even if the
>> node associated with the CPU has already been onlined.
>
> This just sounds like the topology information is being updated at the
> wrong place/time? That is, the memory is online, the CPU is being
> brought online, but isn't associated with any node?
Hi Nishanth,
Yes, that's the case.
>
>> This patch tries to improve performance by updating _mem_id_[cpu] for
>> each possible CPU when memory configuration changes, thus kernel could
>> always allocate from local node once the node is onlined.
>
> Ok, what is the impact? Do you actually see better performance?
No real data to support this yet, just with code analysis.
Regards!
Gerry
>
>> We check node_online(cpu_to_node(cpu)) because:
>> 1) local_memory_node(nid) needs to access NODE_DATA(nid)
>> 2) try_offline_node(nid) just zeroes out NODE_DATA(nid) instead of free it
>>
>> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
>> ---
>> mm/page_alloc.c | 10 +++++-----
>> 1 file changed, 5 insertions(+), 5 deletions(-)
>>
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index 0ea758b898fd..de86e941ed57 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -3844,13 +3844,13 @@ static int __build_all_zonelists(void *data)
>> /*
>> * We now know the "local memory node" for each node--
>> * i.e., the node of the first zone in the generic zonelist.
>> - * Set up numa_mem percpu variable for on-line cpus. During
>> - * boot, only the boot cpu should be on-line; we'll init the
>> - * secondary cpus' numa_mem as they come on-line. During
>> - * node/memory hotplug, we'll fixup all on-line cpus.
>> + * Set up numa_mem percpu variable for all possible cpus
>> + * if associated node has been onlined.
>> */
>> - if (cpu_online(cpu))
>> + if (node_online(cpu_to_node(cpu)))
>> set_cpu_numa_mem(cpu, local_memory_node(cpu_to_node(cpu)));
>> + else
>> + set_cpu_numa_mem(cpu, NUMA_NO_NODE);
>> #endif
>
>
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Jiang Liu @ 2014-07-23 8:20 UTC (permalink / raw)
To: Nishanth Aravamudan, Tony Luck
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, linux-mm@kvack.org,
linux-hotplug, Linux Kernel Mailing List
In-Reply-To: <20140721175736.GG4156@linux.vnet.ibm.com>
On 2014/7/22 1:57, Nishanth Aravamudan wrote:
> On 21.07.2014 [10:41:59 -0700], Tony Luck wrote:
>> On Mon, Jul 21, 2014 at 10:23 AM, Nishanth Aravamudan
>> <nacc@linux.vnet.ibm.com> wrote:
>>> It seems like the issue is the order of onlining of resources on a
>>> specific x86 platform?
>>
>> Yes. When we online a node the BIOS hits us with some ACPI hotplug events:
>>
>> First: Here are some new cpus
>
> Ok, so during this period, you might get some remote allocations. Do you
> know the topology of these CPUs? That is they belong to a
> (soon-to-exist) NUMA node? Can you online that currently offline NUMA
> node at this point (so that NODE_DATA()) resolves, etc.)?
Hi Nishanth,
We have method to get the NUMA information about the CPU, and
patch "[RFC Patch V1 30/30] x86, NUMA: Online node earlier when doing
CPU hot-addition" tries to solve this issue by onlining NUMA node
as early as possible. Actually we are trying to enable memoryless node
as you have suggested.
Regards!
Gerry
>
>> Next: Here is some new memory
>
> And then update the NUMA topology at this point? That is,
> set_cpu_numa_node/mem as appropriate so the underlying allocators do the
> right thing?
>
>> Last; Here are some new I/O things (PCIe root ports, PCIe devices,
>> IOAPICs, IOMMUs, ...)
>>
>> So there is a period where the node is memoryless - although that will
>> generally be resolved when the memory hot plug event arrives ... that
>> isn't guaranteed to occur (there might not be any memory on the node,
>> or what memory there is may have failed self-test and been disabled).
>
> Right, but the allocator(s) generally does the right thing already in
> the face of memoryless nodes -- they fallback to the nearest node. That
> leads to poor performance, but is functional. Based upon the previous
> thread Jiang pointed to, it seems like the real issue here isn't that
> the node is memoryless, but that it's not even online yet? So NODE_DATA
> access crashes?
>
> Thanks,
> Nish
>
^ permalink raw reply
* Re: [Patch Part3 V4 16/21] iommu/vt-d: Implement DMAR unit hotplug framework
From: Joerg Roedel @ 2014-07-23 15:34 UTC (permalink / raw)
To: Jiang Liu
Cc: David Woodhouse, Yinghai Lu, Bjorn Helgaas, Dan Williams,
Vinod Koul, Rafael J . Wysocki, Ashok Raj, Yijing Wang, Tony Luck,
iommu, linux-pci, linux-hotplug, linux-kernel, dmaengine
In-Reply-To: <1405059585-10620-17-git-send-email-jiang.liu@linux.intel.com>
On Fri, Jul 11, 2014 at 02:19:40PM +0800, Jiang Liu wrote:
> On Intel platforms, an IO Hub (PCI/PCIe host bridge) may contain DMAR
> units, so we need to support DMAR hotplug when supporting PCI host
> bridge hotplug on Intel platforms.
>
> According to Section 8.8 "Remapping Hardware Unit Hot Plug" in "Intel
> Virtualization Technology for Directed IO Architecture Specification
> Rev 2.2", ACPI BIOS should implement ACPI _DSM method under the ACPI
> object for the PCI host bridge to support DMAR hotplug.
>
> This patch introduces interfaces to parse ACPI _DSM method for
> DMAR unit hotplug. It also implements state machines for DMAR unit
> hot-addition and hot-removal.
>
> The PCI host bridge hotplug driver should call dmar_hotplug_hotplug()
> before scanning PCI devices connected for hot-addition and after
> destroying all PCI devices for hot-removal.
>
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> ---
> drivers/iommu/dmar.c | 268 +++++++++++++++++++++++++++++++++--
> drivers/iommu/intel-iommu.c | 78 +++++++++-
> drivers/iommu/intel_irq_remapping.c | 5 +
> include/linux/dmar.h | 33 +++++
> 4 files changed, 370 insertions(+), 14 deletions(-)
This patch triggers a compile error with allnoconfig on amd64:
CC arch/x86/kernel/pci-dma.o
In file included from arch/x86/kernel/pci-dma.c:3:0:
include/linux/dmar.h:168:35: error: unknown type name ‘acpi_handle’
static inline int dmar_device_add(acpi_handle handle)
^
include/linux/dmar.h:173:38: error: unknown type name ‘acpi_handle’
static inline int dmar_device_remove(acpi_handle handle)
^
make[2]: *** [arch/x86/kernel/pci-dma.o] Error 1
make[1]: *** [arch/x86/kernel] Error 2
make: *** [arch/x86] Error 2
^ permalink raw reply
* iommu/vt-d: Fix build error caused by unknown definition of acpi_handle
From: Jiang Liu @ 2014-07-24 2:33 UTC (permalink / raw)
To: Joerg Roedel, David Woodhouse, Yinghai Lu, Bjorn Helgaas,
Dan Williams, Vinod Koul, Rafael J . Wysocki
Cc: Jiang Liu, Ashok Raj, Yijing Wang, Tony Luck, iommu, linux-pci,
linux-hotplug, linux-kernel, dmaengine
In-Reply-To: <20140723153420.GC14017@8bytes.org>
When both CONFIG_ACPI and CONFIG_DMAR_TABLE are disabled, commit
"Implement DMAR unit hotplug framework" causes build failure as below:
CC arch/x86/kernel/pci-dma.o
In file included from arch/x86/kernel/pci-dma.c:3:0:
include/linux/dmar.h:168:35: error: unknown type name ‘acpi_handle’
static inline int dmar_device_add(acpi_handle handle)
^
include/linux/dmar.h:173:38: error: unknown type name ‘acpi_handle’
static inline int dmar_device_remove(acpi_handle handle)
^
make[2]: *** [arch/x86/kernel/pci-dma.o] Error 1
make[1]: *** [arch/x86/kernel] Error 2
make: *** [arch/x86] Error 2
Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
---
Hi Joerg,
Could you please help to merge or fold this patch to fix the
build error?
Regards!
Gerry
---
include/linux/dmar.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/linux/dmar.h b/include/linux/dmar.h
index 9c06bb4b5b14..594d4ac79e75 100644
--- a/include/linux/dmar.h
+++ b/include/linux/dmar.h
@@ -165,12 +165,12 @@ static inline int dmar_ir_hotplug(struct dmar_drhd_unit *dmaru, bool insert)
#else /* CONFIG_DMAR_TABLE */
-static inline int dmar_device_add(acpi_handle handle)
+static inline int dmar_device_add(void *handle)
{
return 0;
}
-static inline int dmar_device_remove(acpi_handle handle)
+static inline int dmar_device_remove(void *handle)
{
return 0;
}
--
1.7.10.4
^ permalink raw reply related
* udev 215 creates inactive MD devices upon stopping them
From: Sebastian Parschauer @ 2014-07-24 14:48 UTC (permalink / raw)
To: linux-hotplug; +Cc: Linux RAID, artur.paszkiewicz, Francis Moreau
Hi,
as discussed on linux-raid, please fix the bug that udev 215 creates
inactive MD devices upon stopping them.
Reference: http://www.spinics.net/lists/raid/msg46676.html
Reported-by: Francis Moreau <francis.moro@gmail.com>
An open() call to /dev/mdX after creating it with mknod is enough to
create such inactive MD device.
According to Artur the issue is caused by this change in udev:
> commit 3ebdb81ef088afd3b4c72b516beb5610f8c93a0d
> Author: Kay Sievers <kay at vrfy.org>
> Date: Sun Apr 13 19:54:27 2014 -0700
>
> udev: serialize/synchronize block device event handling with file locks
>
> http://cgit.freedesktop.org/systemd/systemd/commit/?id>bdb81ef088afd3b4c72b516beb5610f8c93a0d
>
> It seems that they have already disabled this for dm for some reason,
> but not for md:
>
> commit e918a1b5a94f270186dca59156354acd2a596494
> Author: Kay Sievers <kay@vrfy.org>
> Date: Tue Jun 3 16:49:38 2014 +0200
>
> udev: exclude device-mapper from block device ownership event locking
>
> http://cgit.freedesktop.org/systemd/systemd/commit/?idé18a1b5a94f270186dca59156354acd2a596494
>
Thanks,
Sebastian
^ permalink raw reply
* Re: [RFC Patch V1 29/30] mm, x86: Enable memoryless node support to better support CPU/memory hotplu
From: Nishanth Aravamudan @ 2014-07-24 23:26 UTC (permalink / raw)
To: Jiang Liu
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, x86, Rafael J. Wysocki, Len Brown, Pavel Machek,
Toshi Kani, Igor Mammedov, Borislav Petkov, Paul Gortmaker,
Tang Chen, Zhang Yanfei, Lans Zhang, Tony Luck, linux-mm,
linux-hotplug, linux-kernel, Ingo Molnar, linux-pm
In-Reply-To: <1405064267-11678-30-git-send-email-jiang.liu@linux.intel.com>
On 11.07.2014 [15:37:46 +0800], Jiang Liu wrote:
> With current implementation, all CPUs within a NUMA node will be
> assocaited with another NUMA node if the node has no memory installed.
<snip>
> ---
> arch/x86/Kconfig | 3 +++
> arch/x86/kernel/acpi/boot.c | 5 ++++-
> arch/x86/kernel/smpboot.c | 2 ++
> arch/x86/mm/numa.c | 42 +++++++++++++++++++++++++++++++++++-------
> 4 files changed, 44 insertions(+), 8 deletions(-)
>
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index a8f749ef0fdc..f35b25b88625 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1887,6 +1887,9 @@ config USE_PERCPU_NUMA_NODE_ID
> def_bool y
> depends on NUMA
>
> +config HAVE_MEMORYLESS_NODES
> + def_bool NUMA
> +
> config ARCH_ENABLE_SPLIT_PMD_PTLOCK
> def_bool y
> depends on X86_64 || X86_PAE
> diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
> index 86281ffb96d6..3b5641703a49 100644
> --- a/arch/x86/kernel/acpi/boot.c
> +++ b/arch/x86/kernel/acpi/boot.c
> @@ -612,6 +612,8 @@ static void acpi_map_cpu2node(acpi_handle handle, int cpu, int physid)
> if (nid != -1) {
> set_apicid_to_node(physid, nid);
> numa_set_node(cpu, nid);
> + if (node_online(nid))
> + set_cpu_numa_mem(cpu, local_memory_node(nid));
How common is it for this method to be called for a CPU on an offline
node? Aren't you fixing this in the next patch (so maybe the order
should be changed?)?
> }
> #endif
> }
> @@ -644,9 +646,10 @@ int acpi_unmap_lsapic(int cpu)
> {
> #ifdef CONFIG_ACPI_NUMA
> set_apicid_to_node(per_cpu(x86_cpu_to_apicid, cpu), NUMA_NO_NODE);
> + set_cpu_numa_mem(cpu, NUMA_NO_NODE);
> #endif
>
> - per_cpu(x86_cpu_to_apicid, cpu) = -1;
> + per_cpu(x86_cpu_to_apicid, cpu) = BAD_APICID;
I think this is an unrelated change?
> set_cpu_present(cpu, false);
> num_processors--;
>
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index 5492798930ef..4a5437989ffe 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -162,6 +162,8 @@ static void smp_callin(void)
> __func__, cpuid);
> }
>
> + set_numa_mem(local_memory_node(cpu_to_node(cpuid)));
> +
Note that you might hit the same issue I reported on powerpc, if
smp_callin() is part of smp_init(). The waitqueue initialization code
depends on cpu_to_node() [and eventually cpu_to_mem()] to be initialized
quite early.
Thanks,
Nish
^ permalink raw reply
* Re: [RFC Patch V1 30/30] x86, NUMA: Online node earlier when doing CPU hot-addition
From: Nishanth Aravamudan @ 2014-07-24 23:30 UTC (permalink / raw)
To: Jiang Liu
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Rafael J. Wysocki, Len Brown,
Pavel Machek, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86,
Tony Luck, linux-mm, linux-hotplug, linux-kernel, linux-pm
In-Reply-To: <1405064267-11678-31-git-send-email-jiang.liu@linux.intel.com>
On 11.07.2014 [15:37:47 +0800], Jiang Liu wrote:
> With typical CPU hot-addition flow on x86, PCI host bridges embedded
> in physical processor are always associated with NOMA_NO_NODE, which
> may cause sub-optimal performance.
> 1) Handle CPU hot-addition notification
> acpi_processor_add()
> acpi_processor_get_info()
> acpi_processor_hotadd_init()
> acpi_map_lsapic()
> 1.a) acpi_map_cpu2node()
>
> 2) Handle PCI host bridge hot-addition notification
> acpi_pci_root_add()
> pci_acpi_scan_root()
> 2.a) if (node != NUMA_NO_NODE && !node_online(node)) node = NUMA_NO_NODE;
>
> 3) Handle memory hot-addition notification
> acpi_memory_device_add()
> acpi_memory_enable_device()
> add_memory()
> 3.a) node_set_online();
>
> 4) Online CPUs through sysfs interfaces
> cpu_subsys_online()
> cpu_up()
> try_online_node()
> 4.a) node_set_online();
>
> So associated node is always in offline state because it is onlined
> until step 3.a or 4.a.
>
> We could improve performance by online node at step 1.a. This change
> also makes the code symmetric. Nodes are always created when handling
> CPU/memory hot-addition events instead of handling user requests from
> sysfs interfaces, and are destroyed when handling CPU/memory hot-removal
> events.
It seems like this patch has little to nothing to do with the rest of
the series and can be sent on its own?
> It also close a race window caused by kmalloc_node(cpu_to_node(cpu)),
To be clear, the race is that on some x86 platforms, there is a period
of time where a node ID returned by cpu_to_node() is offline.
<snip>
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> ---
> arch/x86/kernel/acpi/boot.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
> index 3b5641703a49..00c2ed507460 100644
> --- a/arch/x86/kernel/acpi/boot.c
> +++ b/arch/x86/kernel/acpi/boot.c
> @@ -611,6 +611,7 @@ static void acpi_map_cpu2node(acpi_handle handle, int cpu, int physid)
> nid = acpi_get_node(handle);
> if (nid != -1) {
> set_apicid_to_node(physid, nid);
> + try_online_node(nid);
try_online_node() seems like it can fail? I assume it's a pretty rare
case, but should the return code be checked?
If it does fail, it seems like there are pretty serious problems and we
shouldn't be onlining this CPU, etc.?
> numa_set_node(cpu, nid);
> if (node_online(nid))
> set_cpu_numa_mem(cpu, local_memory_node(nid));
Which means you can remove this check presuming try_online_node()
returned 0.
Thanks,
Nish
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Nishanth Aravamudan @ 2014-07-24 23:32 UTC (permalink / raw)
To: Jiang Liu
Cc: Tony Luck, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
linux-mm@kvack.org, linux-hotplug, Linux Kernel Mailing List
In-Reply-To: <53CF7048.20302@linux.intel.com>
On 23.07.2014 [16:20:24 +0800], Jiang Liu wrote:
>
>
> On 2014/7/22 1:57, Nishanth Aravamudan wrote:
> > On 21.07.2014 [10:41:59 -0700], Tony Luck wrote:
> >> On Mon, Jul 21, 2014 at 10:23 AM, Nishanth Aravamudan
> >> <nacc@linux.vnet.ibm.com> wrote:
> >>> It seems like the issue is the order of onlining of resources on a
> >>> specific x86 platform?
> >>
> >> Yes. When we online a node the BIOS hits us with some ACPI hotplug events:
> >>
> >> First: Here are some new cpus
> >
> > Ok, so during this period, you might get some remote allocations. Do you
> > know the topology of these CPUs? That is they belong to a
> > (soon-to-exist) NUMA node? Can you online that currently offline NUMA
> > node at this point (so that NODE_DATA()) resolves, etc.)?
> Hi Nishanth,
> We have method to get the NUMA information about the CPU, and
> patch "[RFC Patch V1 30/30] x86, NUMA: Online node earlier when doing
> CPU hot-addition" tries to solve this issue by onlining NUMA node
> as early as possible. Actually we are trying to enable memoryless node
> as you have suggested.
Ok, it seems like you have two sets of patches then? One is to fix the
NUMA information timing (30/30 only). The rest of the patches are
general discussions about where cpu_to_mem() might be used instead of
cpu_to_node(). However, based upon Tejun's feedback, it seems like
rather than force all callers to use cpu_to_mem(), we should be looking
at the core VM to ensure fallback is occuring appropriately when
memoryless nodes are present.
Do you have a specific situation, once you've applied 30/30, where
kmalloc_node() leads to an Oops?
Thanks,
Nish
^ permalink raw reply
* Re: [RFC Patch V1 29/30] mm, x86: Enable memoryless node support to better support CPU/memory hotplu
From: Jiang Liu @ 2014-07-25 1:41 UTC (permalink / raw)
To: Nishanth Aravamudan
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, x86, Rafael J. Wysocki, Len Brown, Pavel Machek,
Toshi Kani, Igor Mammedov, Borislav Petkov, Paul Gortmaker,
Tang Chen, Zhang Yanfei, Lans Zhang, Tony Luck, linux-mm,
linux-hotplug, linux-kernel, Ingo Molnar, linux-pm
In-Reply-To: <20140724232605.GB24458@linux.vnet.ibm.com>
On 2014/7/25 7:26, Nishanth Aravamudan wrote:
> On 11.07.2014 [15:37:46 +0800], Jiang Liu wrote:
>> With current implementation, all CPUs within a NUMA node will be
>> assocaited with another NUMA node if the node has no memory installed.
>
> <snip>
>
>> ---
>> arch/x86/Kconfig | 3 +++
>> arch/x86/kernel/acpi/boot.c | 5 ++++-
>> arch/x86/kernel/smpboot.c | 2 ++
>> arch/x86/mm/numa.c | 42 +++++++++++++++++++++++++++++++++++-------
>> 4 files changed, 44 insertions(+), 8 deletions(-)
>>
>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>> index a8f749ef0fdc..f35b25b88625 100644
>> --- a/arch/x86/Kconfig
>> +++ b/arch/x86/Kconfig
>> @@ -1887,6 +1887,9 @@ config USE_PERCPU_NUMA_NODE_ID
>> def_bool y
>> depends on NUMA
>>
>> +config HAVE_MEMORYLESS_NODES
>> + def_bool NUMA
>> +
>> config ARCH_ENABLE_SPLIT_PMD_PTLOCK
>> def_bool y
>> depends on X86_64 || X86_PAE
>> diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
>> index 86281ffb96d6..3b5641703a49 100644
>> --- a/arch/x86/kernel/acpi/boot.c
>> +++ b/arch/x86/kernel/acpi/boot.c
>> @@ -612,6 +612,8 @@ static void acpi_map_cpu2node(acpi_handle handle, int cpu, int physid)
>> if (nid != -1) {
>> set_apicid_to_node(physid, nid);
>> numa_set_node(cpu, nid);
>> + if (node_online(nid))
>> + set_cpu_numa_mem(cpu, local_memory_node(nid));
>
> How common is it for this method to be called for a CPU on an offline
> node? Aren't you fixing this in the next patch (so maybe the order
> should be changed?)?
Hi Nishanth,
For physical CPU hot-addition instead of logical CPU online through
sysfs, the node is always in offline state.
In v2, I have reordered the patch set so patch 30 goes first.
>
>> }
>> #endif
>> }
>> @@ -644,9 +646,10 @@ int acpi_unmap_lsapic(int cpu)
>> {
>> #ifdef CONFIG_ACPI_NUMA
>> set_apicid_to_node(per_cpu(x86_cpu_to_apicid, cpu), NUMA_NO_NODE);
>> + set_cpu_numa_mem(cpu, NUMA_NO_NODE);
>> #endif
>>
>> - per_cpu(x86_cpu_to_apicid, cpu) = -1;
>> + per_cpu(x86_cpu_to_apicid, cpu) = BAD_APICID;
>
> I think this is an unrelated change?
Thanks for reminder, it's unrelated to support memoryless node.
>
>> set_cpu_present(cpu, false);
>> num_processors--;
>>
>> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
>> index 5492798930ef..4a5437989ffe 100644
>> --- a/arch/x86/kernel/smpboot.c
>> +++ b/arch/x86/kernel/smpboot.c
>> @@ -162,6 +162,8 @@ static void smp_callin(void)
>> __func__, cpuid);
>> }
>>
>> + set_numa_mem(local_memory_node(cpu_to_node(cpuid)));
>> +
>
> Note that you might hit the same issue I reported on powerpc, if
> smp_callin() is part of smp_init(). The waitqueue initialization code
> depends on cpu_to_node() [and eventually cpu_to_mem()] to be initialized
> quite early.
Thanks for reminder. Patch 29/30 together will setup cpu_to_mem() array
when enumerating CPUs for hot-adding events, so it should be ready
for use when onlining those CPUs.
Regards!
Gerry
>
> Thanks,
> Nish
>
^ permalink raw reply
* Re: [RFC Patch V1 30/30] x86, NUMA: Online node earlier when doing CPU hot-addition
From: Jiang Liu @ 2014-07-25 1:43 UTC (permalink / raw)
To: Nishanth Aravamudan
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Rafael J. Wysocki, Len Brown,
Pavel Machek, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86,
Tony Luck, linux-mm, linux-hotplug, linux-kernel, linux-pm
In-Reply-To: <20140724233027.GC24458@linux.vnet.ibm.com>
On 2014/7/25 7:30, Nishanth Aravamudan wrote:
> On 11.07.2014 [15:37:47 +0800], Jiang Liu wrote:
>> With typical CPU hot-addition flow on x86, PCI host bridges embedded
>> in physical processor are always associated with NOMA_NO_NODE, which
>> may cause sub-optimal performance.
>> 1) Handle CPU hot-addition notification
>> acpi_processor_add()
>> acpi_processor_get_info()
>> acpi_processor_hotadd_init()
>> acpi_map_lsapic()
>> 1.a) acpi_map_cpu2node()
>>
>> 2) Handle PCI host bridge hot-addition notification
>> acpi_pci_root_add()
>> pci_acpi_scan_root()
>> 2.a) if (node != NUMA_NO_NODE && !node_online(node)) node = NUMA_NO_NODE;
>>
>> 3) Handle memory hot-addition notification
>> acpi_memory_device_add()
>> acpi_memory_enable_device()
>> add_memory()
>> 3.a) node_set_online();
>>
>> 4) Online CPUs through sysfs interfaces
>> cpu_subsys_online()
>> cpu_up()
>> try_online_node()
>> 4.a) node_set_online();
>>
>> So associated node is always in offline state because it is onlined
>> until step 3.a or 4.a.
>>
>> We could improve performance by online node at step 1.a. This change
>> also makes the code symmetric. Nodes are always created when handling
>> CPU/memory hot-addition events instead of handling user requests from
>> sysfs interfaces, and are destroyed when handling CPU/memory hot-removal
>> events.
>
> It seems like this patch has little to nothing to do with the rest of
> the series and can be sent on its own?
>
>> It also close a race window caused by kmalloc_node(cpu_to_node(cpu)),
>
> To be clear, the race is that on some x86 platforms, there is a period
> of time where a node ID returned by cpu_to_node() is offline.
>
> <snip>
>
>> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
>> ---
>> arch/x86/kernel/acpi/boot.c | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
>> index 3b5641703a49..00c2ed507460 100644
>> --- a/arch/x86/kernel/acpi/boot.c
>> +++ b/arch/x86/kernel/acpi/boot.c
>> @@ -611,6 +611,7 @@ static void acpi_map_cpu2node(acpi_handle handle, int cpu, int physid)
>> nid = acpi_get_node(handle);
>> if (nid != -1) {
>> set_apicid_to_node(physid, nid);
>> + try_online_node(nid);
>
> try_online_node() seems like it can fail? I assume it's a pretty rare
> case, but should the return code be checked?
Good suggestion, I should split out this patch to fix the crash.
>
> If it does fail, it seems like there are pretty serious problems and we
> shouldn't be onlining this CPU, etc.?
>
>> numa_set_node(cpu, nid);
>> if (node_online(nid))
>> set_cpu_numa_mem(cpu, local_memory_node(nid));
>
> Which means you can remove this check presuming try_online_node()
> returned 0.
Yes, that's true.
>
> Thanks,
> Nish
>
^ permalink raw reply
* Re: [RFC Patch V1 30/30] x86, NUMA: Online node earlier when doing CPU hot-addition
From: Jiang Liu @ 2014-07-25 1:44 UTC (permalink / raw)
To: Nishanth Aravamudan
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Rafael J. Wysocki, Len Brown,
Pavel Machek, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, x86,
Tony Luck, linux-mm, linux-hotplug, linux-kernel, linux-pm
In-Reply-To: <20140724233027.GC24458@linux.vnet.ibm.com>
On 2014/7/25 7:30, Nishanth Aravamudan wrote:
> On 11.07.2014 [15:37:47 +0800], Jiang Liu wrote:
>> With typical CPU hot-addition flow on x86, PCI host bridges embedded
>> in physical processor are always associated with NOMA_NO_NODE, which
>> may cause sub-optimal performance.
>> 1) Handle CPU hot-addition notification
>> acpi_processor_add()
>> acpi_processor_get_info()
>> acpi_processor_hotadd_init()
>> acpi_map_lsapic()
>> 1.a) acpi_map_cpu2node()
>>
>> 2) Handle PCI host bridge hot-addition notification
>> acpi_pci_root_add()
>> pci_acpi_scan_root()
>> 2.a) if (node != NUMA_NO_NODE && !node_online(node)) node = NUMA_NO_NODE;
>>
>> 3) Handle memory hot-addition notification
>> acpi_memory_device_add()
>> acpi_memory_enable_device()
>> add_memory()
>> 3.a) node_set_online();
>>
>> 4) Online CPUs through sysfs interfaces
>> cpu_subsys_online()
>> cpu_up()
>> try_online_node()
>> 4.a) node_set_online();
>>
>> So associated node is always in offline state because it is onlined
>> until step 3.a or 4.a.
>>
>> We could improve performance by online node at step 1.a. This change
>> also makes the code symmetric. Nodes are always created when handling
>> CPU/memory hot-addition events instead of handling user requests from
>> sysfs interfaces, and are destroyed when handling CPU/memory hot-removal
>> events.
>
> It seems like this patch has little to nothing to do with the rest of
> the series and can be sent on its own?
>
>> It also close a race window caused by kmalloc_node(cpu_to_node(cpu)),
>
> To be clear, the race is that on some x86 platforms, there is a period
> of time where a node ID returned by cpu_to_node() is offline.
>
> <snip>
>
>> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
>> ---
>> arch/x86/kernel/acpi/boot.c | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
>> index 3b5641703a49..00c2ed507460 100644
>> --- a/arch/x86/kernel/acpi/boot.c
>> +++ b/arch/x86/kernel/acpi/boot.c
>> @@ -611,6 +611,7 @@ static void acpi_map_cpu2node(acpi_handle handle, int cpu, int physid)
>> nid = acpi_get_node(handle);
>> if (nid != -1) {
>> set_apicid_to_node(physid, nid);
>> + try_online_node(nid);
>
> try_online_node() seems like it can fail? I assume it's a pretty rare
> case, but should the return code be checked?
>
> If it does fail, it seems like there are pretty serious problems and we
> shouldn't be onlining this CPU, etc.?
>
>> numa_set_node(cpu, nid);
>> if (node_online(nid))
>> set_cpu_numa_mem(cpu, local_memory_node(nid));
>
> Which means you can remove this check presuming try_online_node()
> returned 0.
Good suggestion, will try to enhance the error handling path.
>
> Thanks,
> Nish
>
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Jiang Liu @ 2014-07-25 1:50 UTC (permalink / raw)
To: Nishanth Aravamudan
Cc: Tony Luck, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
linux-mm@kvack.org, linux-hotplug, Linux Kernel Mailing List
In-Reply-To: <20140724233230.GD24458@linux.vnet.ibm.com>
On 2014/7/25 7:32, Nishanth Aravamudan wrote:
> On 23.07.2014 [16:20:24 +0800], Jiang Liu wrote:
>>
>>
>> On 2014/7/22 1:57, Nishanth Aravamudan wrote:
>>> On 21.07.2014 [10:41:59 -0700], Tony Luck wrote:
>>>> On Mon, Jul 21, 2014 at 10:23 AM, Nishanth Aravamudan
>>>> <nacc@linux.vnet.ibm.com> wrote:
>>>>> It seems like the issue is the order of onlining of resources on a
>>>>> specific x86 platform?
>>>>
>>>> Yes. When we online a node the BIOS hits us with some ACPI hotplug events:
>>>>
>>>> First: Here are some new cpus
>>>
>>> Ok, so during this period, you might get some remote allocations. Do you
>>> know the topology of these CPUs? That is they belong to a
>>> (soon-to-exist) NUMA node? Can you online that currently offline NUMA
>>> node at this point (so that NODE_DATA()) resolves, etc.)?
>> Hi Nishanth,
>> We have method to get the NUMA information about the CPU, and
>> patch "[RFC Patch V1 30/30] x86, NUMA: Online node earlier when doing
>> CPU hot-addition" tries to solve this issue by onlining NUMA node
>> as early as possible. Actually we are trying to enable memoryless node
>> as you have suggested.
>
> Ok, it seems like you have two sets of patches then? One is to fix the
> NUMA information timing (30/30 only). The rest of the patches are
> general discussions about where cpu_to_mem() might be used instead of
> cpu_to_node(). However, based upon Tejun's feedback, it seems like
> rather than force all callers to use cpu_to_mem(), we should be looking
> at the core VM to ensure fallback is occuring appropriately when
> memoryless nodes are present.
>
> Do you have a specific situation, once you've applied 30/30, where
> kmalloc_node() leads to an Oops?
Hi Nishanth,
After following the two threads related to support of memoryless
node and digging more code, I realized my first version path set is an
overkill. As Tejun has pointed out, we shouldn't expose the detail of
memoryless node to normal user, but there are still some special users
who need the detail. So I have tried to summarize it as:
1) Arch code should online corresponding NUMA node before onlining any
CPU or memory, otherwise it may cause invalid memory access when
accessing NODE_DATA(nid).
2) For normal memory allocations without __GFP_THISNODE setting in the
gfp_flags, we should prefer numa_node_id()/cpu_to_node() instead of
numa_mem_id()/cpu_to_mem() because the latter loses hardware topology
information as pointed out by Tejun:
A - B - X - C - D
Where X is the memless node. numa_mem_id() on X would return
either B or C, right? If B or C can't satisfy the allocation,
the allocator would fallback to A from B and D for C, both of
which aren't optimal. It should first fall back to C or B
respectively, which the allocator can't do anymoe because the
information is lost when the caller side performs numa_mem_id().
3) For memory allocation with __GFP_THISNODE setting in gfp_flags,
numa_node_id()/cpu_to_node() should be used if caller only wants to
allocate from local memory, otherwise numa_mem_id()/cpu_to_mem()
should be used if caller wants to allocate from the nearest node.
4) numa_mem_id()/cpu_to_mem() should be used if caller wants to check
whether a page is allocated from the nearest node.
And my v2 patch set is based on above rules.
Any suggestions here?
Regards!
Gerry
>
> Thanks,
> Nish
>
^ permalink raw reply
* Re: [RFC Patch V1 22/30] mm, of: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Grant Likely @ 2014-07-28 13:30 UTC (permalink / raw)
To: Nishanth Aravamudan, Jiang Liu
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Rob Herring, Tony Luck,
linux-mm, linux-hotplug, linux-kernel, devicetree
In-Reply-To: <20140721175241.GF4156@linux.vnet.ibm.com>
On Mon, 21 Jul 2014 10:52:41 -0700, Nishanth Aravamudan <nacc@linux.vnet.ibm.com> wrote:
> On 11.07.2014 [15:37:39 +0800], Jiang Liu wrote:
> > When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
> > may return a node without memory, and later cause system failure/panic
> > when calling kmalloc_node() and friends with returned node id.
> > So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
> > memory for the/current cpu.
> >
> > If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
> > is the same as cpu_to_node()/numa_node_id().
> >
> > Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> > ---
> > drivers/of/base.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/of/base.c b/drivers/of/base.c
> > index b9864806e9b8..40d4772973ad 100644
> > --- a/drivers/of/base.c
> > +++ b/drivers/of/base.c
> > @@ -85,7 +85,7 @@ EXPORT_SYMBOL(of_n_size_cells);
> > #ifdef CONFIG_NUMA
> > int __weak of_node_to_nid(struct device_node *np)
> > {
> > - return numa_node_id();
> > + return numa_mem_id();
> > }
> > #endif
>
> Um, NAK. of_node_to_nid() returns the NUMA node ID for a given device
> tree node. The default should be the physically local NUMA node, not the
> nearest memory-containing node.
That description doesn't match the code. This patch only changes the
default implementation of of_node_to_nid() which doesn't take the device
node into account *at all* when returning a node ID. Just look at the
diff.
I think this patch is correct, and it doesn't affect the override
versions provided by powerpc and sparc.
g.
>
> I think the general direction of this patchset is good -- what NUMA
> information do we actually are about at each callsite. But the execution
> is blind and doesn't consider at all what the code is actually doing.
> The changelogs are all identical and don't actually provide any
> information about what errors this (or any) specific patch are
> resolving.
>
> Thanks,
> Nish
>
^ permalink raw reply
* Re: [RFC Patch V1 22/30] mm, of: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Nishanth Aravamudan @ 2014-07-28 19:26 UTC (permalink / raw)
To: Grant Likely
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki, Rob Herring,
Tony Luck, linux-mm, linux-hotplug, linux-kernel, devicetree
In-Reply-To: <20140728133040.854F5C4095E@trevor.secretlab.ca>
On 28.07.2014 [07:30:40 -0600], Grant Likely wrote:
> On Mon, 21 Jul 2014 10:52:41 -0700, Nishanth Aravamudan <nacc@linux.vnet.ibm.com> wrote:
> > On 11.07.2014 [15:37:39 +0800], Jiang Liu wrote:
> > > When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
> > > may return a node without memory, and later cause system failure/panic
> > > when calling kmalloc_node() and friends with returned node id.
> > > So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
> > > memory for the/current cpu.
> > >
> > > If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
> > > is the same as cpu_to_node()/numa_node_id().
> > >
> > > Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> > > ---
> > > drivers/of/base.c | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/of/base.c b/drivers/of/base.c
> > > index b9864806e9b8..40d4772973ad 100644
> > > --- a/drivers/of/base.c
> > > +++ b/drivers/of/base.c
> > > @@ -85,7 +85,7 @@ EXPORT_SYMBOL(of_n_size_cells);
> > > #ifdef CONFIG_NUMA
> > > int __weak of_node_to_nid(struct device_node *np)
> > > {
> > > - return numa_node_id();
> > > + return numa_mem_id();
> > > }
> > > #endif
> >
> > Um, NAK. of_node_to_nid() returns the NUMA node ID for a given device
> > tree node. The default should be the physically local NUMA node, not the
> > nearest memory-containing node.
>
> That description doesn't match the code. This patch only changes the
> default implementation of of_node_to_nid() which doesn't take the device
> node into account *at all* when returning a node ID. Just look at the
> diff.
I meant that of_node_to_nid() seems to be used throughout the call-sites
to indicate caller locality. We want to keep using cpu_to_node() there,
and fallback appropriately in the MM (when allocations occur offnode due
to memoryless nodes), not indicate memory-specific topology the caller
itself. There was a long thread between between Tejun and I that
discussed what we are trying for: https://lkml.org/lkml/2014/7/18/278
I understand that the code unconditionally returns current's NUMA node
ID right now (ignoring the device node). That seems correct, to me, for
something like:
of_device_add:
/* device_add will assume that this device is on the same node as
* the parent. If there is no parent defined, set the node
* explicitly */
if (!ofdev->dev.parent)
set_dev_node(&ofdev->dev, of_node_to_nid(ofdev->dev.of_node));
I don't think we want the default implementation to set the NUMA node of
a dev to the nearest NUMA node with memory?
> I think this patch is correct, and it doesn't affect the override
> versions provided by powerpc and sparc.
Yes, agreed, so maybe it doesn't matter. I guess my point was simply
that it only seems reasonable to change callers of cpu_to_node() to
cpu_to_mem() that aren't in the core MM is if they care about memoryless
nodes explicitly. I don't think the OF code does, so I don't think it
should change.
Sorry for my premature NAK and lack of clarity in my explanation.
-Nish
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox