* Re: [RFC Patch V1 07/30] mm: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Christoph Lameter @ 2014-07-11 15:58 UTC (permalink / raw)
To: Tejun Heo
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Vladimir Davydov, Johannes Weiner, Kirill A. Shutemov,
Rik van Riel, Wanpeng Li, Zhang Yanfei, Catalin Marinas,
Jianyu Zhan, malc, Joonsoo Kim, Fabian Frederick, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <20140711152156.GB29137@htj.dyndns.org>
On Fri, 11 Jul 2014, Tejun Heo wrote:
> Hello,
>
> On Fri, Jul 11, 2014 at 10:13:57AM -0500, Christoph Lameter wrote:
> > Allocators typically fall back but they wont in some cases if you say
> > that you want memory from a particular node. A GFP_THISNODE would force a
> > failure of the alloc. In other cases it should fall back. I am not sure
> > that all allocations obey these conventions though.
>
> But, GFP_THISNODE + numa_mem_id() is identical to numa_node_id() +
> nearest node with memory fallback. Is there any case where the user
> would actually want to always fail if it's on the memless node?
GFP_THISNODE allocatios must fail if there is no memory available on
the node. No fallback allowed.
If the allocator performs caching for a particular node (like SLAB) then
the allocator *cannnot* accept memory from another node and the alloc via
the page allocator must fail so that the allocator can then pick another
node for keeping track of the allocations.
> Even if that's the case, there's no reason to burden everyone with
> this distinction. Most users just wanna say "I'm on this node.
> Please allocate considering that". There's nothing wrong with using
> numa_node_id() for that.
Well yes that speaks for this patch.
^ permalink raw reply
* Re: [RFC Patch V1 07/30] mm: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Tejun Heo @ 2014-07-11 16:01 UTC (permalink / raw)
To: Christoph Lameter
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Vladimir Davydov, Johannes Weiner, Kirill A. Shutemov,
Rik van Riel, Wanpeng Li, Zhang Yanfei, Catalin Marinas,
Jianyu Zhan, malc, Joonsoo Kim, Fabian Frederick, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <alpine.DEB.2.11.1407111056060.27349@gentwo.org>
On Fri, Jul 11, 2014 at 10:58:52AM -0500, Christoph Lameter wrote:
> > But, GFP_THISNODE + numa_mem_id() is identical to numa_node_id() +
> > nearest node with memory fallback. Is there any case where the user
> > would actually want to always fail if it's on the memless node?
>
> GFP_THISNODE allocatios must fail if there is no memory available on
> the node. No fallback allowed.
I don't know. The intention is that the caller wants something on
this node or the caller will fail or fallback ourselves, right? For
most use cases just considering the nearest memory node as "local" for
memless nodes should work and serve the intentions of the users close
enough. Whether that'd be better or we'd be better off with something
else depends on the details for sure.
Thanks.
--
tejun
^ permalink raw reply
* Re: [RFC Patch V1 07/30] mm: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Christoph Lameter @ 2014-07-11 16:04 UTC (permalink / raw)
To: Tejun Heo
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Vladimir Davydov, Johannes Weiner, Kirill A. Shutemov,
Rik van Riel, Wanpeng Li, Zhang Yanfei, Catalin Marinas,
Jianyu Zhan, malc, Joonsoo Kim, Fabian Frederick, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <20140711155838.GB30865@htj.dyndns.org>
On Fri, 11 Jul 2014, Tejun Heo wrote:
> On Fri, Jul 11, 2014 at 10:55:59AM -0500, Christoph Lameter wrote:
> > > Where X is the memless node. num_mem_id() on X would return either B
> > > or C, right? If B or C can't satisfy the allocation, the allocator
> > > would fallback to A from B and D for C, both of which aren't optimal.
> > > It should first fall back to C or B respectively, which the allocator
> > > can't do anymoe because the information is lost when the caller side
> > > performs numa_mem_id().
> >
> > True but the advantage is that the numa_mem_id() allows the use of a
> > consitent sort of "local" node which increases allocator performance due
> > to the abillity to cache objects from that node.
>
> But the allocator can do the mapping the same. I really don't see why
> we'd push the distinction to the individual users.
The "users" (I guess you mean general kernel code/drivers) can use various
memory allocators which will do the right thing internally regarding
GFP_THISNODE. They do not need to worry too much about this unless there
are reasons beyond optimizing NUMA placement to need memory from a
particuylar node (f.e. a device that requires memory from a numa node that
is local to the PCI bus where the hardware resides).
^ permalink raw reply
* Re: [RFC Patch V1 07/30] mm: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Christoph Lameter @ 2014-07-11 16:19 UTC (permalink / raw)
To: Tejun Heo
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Vladimir Davydov, Johannes Weiner, Kirill A. Shutemov,
Rik van Riel, Wanpeng Li, Zhang Yanfei, Catalin Marinas,
Jianyu Zhan, malc, Joonsoo Kim, Fabian Frederick, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <20140711160152.GC30865@htj.dyndns.org>
On Fri, 11 Jul 2014, Tejun Heo wrote:
> On Fri, Jul 11, 2014 at 10:58:52AM -0500, Christoph Lameter wrote:
> > > But, GFP_THISNODE + numa_mem_id() is identical to numa_node_id() +
> > > nearest node with memory fallback. Is there any case where the user
> > > would actually want to always fail if it's on the memless node?
> >
> > GFP_THISNODE allocatios must fail if there is no memory available on
> > the node. No fallback allowed.
>
> I don't know. The intention is that the caller wants something on
> this node or the caller will fail or fallback ourselves, right? For
> most use cases just considering the nearest memory node as "local" for
> memless nodes should work and serve the intentions of the users close
> enough. Whether that'd be better or we'd be better off with something
> else depends on the details for sure.
Yes that works. But if we want a consistent node to allocate from (and
avoid the fallbacks) then we need this patch. I think this is up to those
needing memoryless nodes to figure out what semantics they need.
^ permalink raw reply
* Re: [RFC Patch V1 07/30] mm: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Tejun Heo @ 2014-07-11 16:24 UTC (permalink / raw)
To: Christoph Lameter
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Vladimir Davydov, Johannes Weiner, Kirill A. Shutemov,
Rik van Riel, Wanpeng Li, Zhang Yanfei, Catalin Marinas,
Jianyu Zhan, malc, Joonsoo Kim, Fabian Frederick, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <alpine.DEB.2.11.1407111117560.27592@gentwo.org>
On Fri, Jul 11, 2014 at 11:19:14AM -0500, Christoph Lameter wrote:
> Yes that works. But if we want a consistent node to allocate from (and
> avoid the fallbacks) then we need this patch. I think this is up to those
> needing memoryless nodes to figure out what semantics they need.
I'm not following what you're saying. Are you saying that we need to
spread numa_mem_id() all over the place for GFP_THISNODE users on
memless nodes? There aren't that many users of GFP_THISNODE.
Wouldn't it make far more sense to just change them? Or just
introduce a new GFP flag GFP_CLOSE_OR_BUST which allows falling back
to the nearest local node for memless nodes. There's no reason to
leak this information outside allocator proper.
Thanks.
--
tejun
^ permalink raw reply
* Re: [RFC Patch V1 07/30] mm: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Christoph Lameter @ 2014-07-11 17:29 UTC (permalink / raw)
To: Tejun Heo
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Vladimir Davydov, Johannes Weiner, Kirill A. Shutemov,
Rik van Riel, Wanpeng Li, Zhang Yanfei, Catalin Marinas,
Jianyu Zhan, malc, Joonsoo Kim, Fabian Frederick, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <20140711162451.GD30865@htj.dyndns.org>
On Fri, 11 Jul 2014, Tejun Heo wrote:
> On Fri, Jul 11, 2014 at 11:19:14AM -0500, Christoph Lameter wrote:
> > Yes that works. But if we want a consistent node to allocate from (and
> > avoid the fallbacks) then we need this patch. I think this is up to those
> > needing memoryless nodes to figure out what semantics they need.
>
> I'm not following what you're saying. Are you saying that we need to
> spread numa_mem_id() all over the place for GFP_THISNODE users on
> memless nodes? There aren't that many users of GFP_THISNODE.
GFP_THISNODE is mostly used by allocators that need memory from specific
nodes. The use of numa_mem_id() there is useful because one will not
get any memory at all when attempting to allocate from a memoryless
node using GFP_THISNODE.
I meant that the relying on fallback to the neighboring nodes without
GFP_THISNODE using numa_node_id() is one approach that may prevent memory
allocators from caching objects for that node because every allocation may
choose a different neighboring node. And the other is the use of
numa_mem_id() which will always use a specific node and avoid fallback to
different node.
The choice is up to those having an interest in memoryless nodes. Which
again I find a pretty strange thing to have that has already proven itself
difficult to maintain in the kernel given the the notion of memory
nodes that should have memory but surprisingly have none. Then there are
the esoteric fallback conditions and special cases introduced. Its a mess.
The best solution may be to just get rid of the whole thing and require
all processors to have a node with memory that is local to them. Current
"memoryless" hardware can simply decide on bootup to pick a memory node
that is local and thus we do not have to deal with it in the core.
^ permalink raw reply
* Re: [RFC Patch V1 07/30] mm: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Tejun Heo @ 2014-07-11 18:28 UTC (permalink / raw)
To: Christoph Lameter
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Vladimir Davydov, Johannes Weiner, Kirill A. Shutemov,
Rik van Riel, Wanpeng Li, Zhang Yanfei, Catalin Marinas,
Jianyu Zhan, malc, Joonsoo Kim, Fabian Frederick, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <alpine.DEB.2.11.1407111220410.4511@gentwo.org>
Hello,
On Fri, Jul 11, 2014 at 12:29:30PM -0500, Christoph Lameter wrote:
> GFP_THISNODE is mostly used by allocators that need memory from specific
> nodes. The use of numa_mem_id() there is useful because one will not
> get any memory at all when attempting to allocate from a memoryless
> node using GFP_THISNODE.
As long as it's in allocator proper, it doesn't matter all that much
but the changes are clearly not contained, are they?
Also, unless this is done where the falling back is actually
happening, numa_mem_id() seems like the wrong interface because you
end up losing information of the originating node. Given that this
isn't a wide spread use case, maybe we can do with something like
numa_mem_id() as a compromise but if we're doing that let's at least
make it clear that it's something ugly (give it an ugly name, not
something as generic as numa_mem_id()) and not expose it outside
allocators.
Thanks.
--
tejun
^ permalink raw reply
* Re: [RFC Patch V1 07/30] mm: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Christoph Lameter @ 2014-07-11 19:11 UTC (permalink / raw)
To: Tejun Heo
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Vladimir Davydov, Johannes Weiner, Kirill A. Shutemov,
Rik van Riel, Wanpeng Li, Zhang Yanfei, Catalin Marinas,
Jianyu Zhan, malc, Joonsoo Kim, Fabian Frederick, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <20140711182814.GE30865@htj.dyndns.org>
On Fri, 11 Jul 2014, Tejun Heo wrote:
> On Fri, Jul 11, 2014 at 12:29:30PM -0500, Christoph Lameter wrote:
> > GFP_THISNODE is mostly used by allocators that need memory from specific
> > nodes. The use of numa_mem_id() there is useful because one will not
> > get any memory at all when attempting to allocate from a memoryless
> > node using GFP_THISNODE.
>
> As long as it's in allocator proper, it doesn't matter all that much
> but the changes are clearly not contained, are they?
Well there is a proliferation of memory allocators recently. NUMA is often
a second thought in those.
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Dave Hansen @ 2014-07-11 20:02 UTC (permalink / raw)
To: Greg KH, Jiang Liu
Cc: Peter Zijlstra, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Rafael J . Wysocki, Tony Luck, linux-mm,
linux-hotplug, linux-kernel
In-Reply-To: <20140711153314.GA6155@kroah.com>
On 07/11/2014 08:33 AM, Greg KH wrote:
> On Fri, Jul 11, 2014 at 10:29:56AM +0200, Peter Zijlstra wrote:
>> > On Fri, Jul 11, 2014 at 03:37:17PM +0800, Jiang Liu wrote:
>>> > > Any comments are welcomed!
>> >
>> > Why would anybody _ever_ have a memoryless node? That's ridiculous.
> I'm with Peter here, why would this be a situation that we should even
> support? Are there machines out there shipping like this?
This is orthogonal to the problem Jiang Liu is solving, but...
The IBM guys have been hitting the CPU-less and memoryless node issues
forever, but that's mostly because their (traditional) hypervisor had
good NUMA support and ran multi-node guests.
I've never seen it in practice on x86 mostly because the hypervisors
don't have good NUMA support. I honestly think this is something x86 is
going to have to handle eventually anyway. It's essentially a resource
fragmentation problem, and there are going to be times where a guest
needs to be spun up and hypervisor has nodes with either no spare memory
or no spare CPUs.
The hypervisor has 3 choices in this case:
1. Lie about the NUMA layout
2. Waste the resources
3. Tell the guest how it's actually arranged
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Andi Kleen @ 2014-07-11 20:20 UTC (permalink / raw)
To: Greg KH
Cc: Jiang Liu, Peter Zijlstra, Andrew Morton, Mel Gorman,
David Rientjes, Mike Galbraith, Rafael J . Wysocki, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <20140711153314.GA6155@kroah.com>
Greg KH <gregkh@linuxfoundation.org> writes:
> On Fri, Jul 11, 2014 at 10:29:56AM +0200, Peter Zijlstra wrote:
>> On Fri, Jul 11, 2014 at 03:37:17PM +0800, Jiang Liu wrote:
>> > Any comments are welcomed!
>>
>> Why would anybody _ever_ have a memoryless node? That's ridiculous.
>
> I'm with Peter here, why would this be a situation that we should even
> support? Are there machines out there shipping like this?
We've always had memory nodes.
A classic case in the old days was a two socket system where someone
didn't populate any DIMMs on the second socket.
There are other cases too.
-Andi
--
ak@linux.intel.com -- Speaking for myself only
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Peter Zijlstra @ 2014-07-11 20:51 UTC (permalink / raw)
To: Andi Kleen
Cc: Greg KH, Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Rafael J . Wysocki, Tony Luck, linux-mm,
linux-hotplug, linux-kernel
In-Reply-To: <8761j3ve8s.fsf@tassilo.jf.intel.com>
On Fri, Jul 11, 2014 at 01:20:51PM -0700, Andi Kleen wrote:
> Greg KH <gregkh@linuxfoundation.org> writes:
>
> > On Fri, Jul 11, 2014 at 10:29:56AM +0200, Peter Zijlstra wrote:
> >> On Fri, Jul 11, 2014 at 03:37:17PM +0800, Jiang Liu wrote:
> >> > Any comments are welcomed!
> >>
> >> Why would anybody _ever_ have a memoryless node? That's ridiculous.
> >
> > I'm with Peter here, why would this be a situation that we should even
> > support? Are there machines out there shipping like this?
>
> We've always had memory nodes.
>
> A classic case in the old days was a two socket system where someone
> didn't populate any DIMMs on the second socket.
That's a obvious; don't do that then case. Its silly.
> There are other cases too.
Are there any sane ones?
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Andi Kleen @ 2014-07-11 21:58 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Andi Kleen, Greg KH, Jiang Liu, Andrew Morton, Mel Gorman,
David Rientjes, Mike Galbraith, Rafael J . Wysocki, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <20140711205106.GB20603@laptop.programming.kicks-ass.net>
On Fri, Jul 11, 2014 at 10:51:06PM +0200, Peter Zijlstra wrote:
> On Fri, Jul 11, 2014 at 01:20:51PM -0700, Andi Kleen wrote:
> > Greg KH <gregkh@linuxfoundation.org> writes:
> >
> > > On Fri, Jul 11, 2014 at 10:29:56AM +0200, Peter Zijlstra wrote:
> > >> On Fri, Jul 11, 2014 at 03:37:17PM +0800, Jiang Liu wrote:
> > >> > Any comments are welcomed!
> > >>
> > >> Why would anybody _ever_ have a memoryless node? That's ridiculous.
> > >
> > > I'm with Peter here, why would this be a situation that we should even
> > > support? Are there machines out there shipping like this?
> >
> > We've always had memory nodes.
> >
> > A classic case in the old days was a two socket system where someone
> > didn't populate any DIMMs on the second socket.
>
> That's a obvious; don't do that then case. Its silly.
True. We should recommend that anyone running Linux will email you
for approval of their configuration first.
> > There are other cases too.
>
> Are there any sane ones
Yes.
-Andi
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Jiri Kosina @ 2014-07-11 22:40 UTC (permalink / raw)
To: Greg KH
Cc: Jiang Liu, Peter Zijlstra, Andrew Morton, Mel Gorman,
David Rientjes, Mike Galbraith, Rafael J . Wysocki, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <20140711153314.GA6155@kroah.com>
On Fri, 11 Jul 2014, Greg KH wrote:
> > On Fri, Jul 11, 2014 at 03:37:17PM +0800, Jiang Liu wrote:
> > > Any comments are welcomed!
> >
> > Why would anybody _ever_ have a memoryless node? That's ridiculous.
>
> I'm with Peter here, why would this be a situation that we should even
> support? Are there machines out there shipping like this?
I am pretty sure I've seen ppc64 machine with memoryless NUMA node.
--
Jiri Kosina
SUSE Labs
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: H. Peter Anvin @ 2014-07-11 23:51 UTC (permalink / raw)
To: Andi Kleen, Greg KH
Cc: Jiang Liu, Peter Zijlstra, Andrew Morton, Mel Gorman,
David Rientjes, Mike Galbraith, Rafael J . Wysocki, Tony Luck,
linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <8761j3ve8s.fsf@tassilo.jf.intel.com>
On 07/11/2014 01:20 PM, Andi Kleen wrote:
> Greg KH <gregkh@linuxfoundation.org> writes:
>
>> On Fri, Jul 11, 2014 at 10:29:56AM +0200, Peter Zijlstra wrote:
>>> On Fri, Jul 11, 2014 at 03:37:17PM +0800, Jiang Liu wrote:
>>>> Any comments are welcomed!
>>>
>>> Why would anybody _ever_ have a memoryless node? That's ridiculous.
>>
>> I'm with Peter here, why would this be a situation that we should even
>> support? Are there machines out there shipping like this?
>
> We've always had memory nodes.
>
> A classic case in the old days was a two socket system where someone
> didn't populate any DIMMs on the second socket.
>
> There are other cases too.
>
Yes, like a node controller-based system where the system can be
populated with either memory cards or CPU cards, for example. Now you
can have both memoryless nodes and memory-only nodes...
Memory-only nodes also happen in real life. In some cases they are done
by permanently putting low-frequency CPUs to sleep for their memory
controllers.
-hpa
^ permalink raw reply
* Re: [RFC Patch V1 01/30] mm, kernel: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Jens Axboe @ 2014-07-12 12:32 UTC (permalink / raw)
To: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Dipankar Sarma, Paul E. McKenney, Balbir Singh, Thomas Gleixner,
Frederic Weisbecker, Jan Kara, Ingo Molnar, Christoph Hellwig,
Srivatsa S. Bhat, Roman Gushchin, Xie XiuQi
Cc: Tony Luck, linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <1405064267-11678-2-git-send-email-jiang.liu@linux.intel.com>
On 2014-07-11 09:37, Jiang Liu wrote:
> When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
> may return a node without memory, and later cause system failure/panic
> when calling kmalloc_node() and friends with returned node id.
> So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
> memory for the/current cpu.
>
> If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
> is the same as cpu_to_node()/numa_node_id().
I think blk-mq requires some of the same help, as do other places in the
block layer. I'll take a look at that.
As for you smp.c bits here:
Acked-by: Jens Axboe <axboe@fb.com>
--
Jens Axboe
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: David Rientjes @ 2014-07-15 1:18 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Andi Kleen, Greg KH, Jiang Liu, Andrew Morton, Mel Gorman,
Mike Galbraith, Rafael J . Wysocki, Tony Luck, linux-mm,
linux-hotplug, linux-kernel
In-Reply-To: <20140711205106.GB20603@laptop.programming.kicks-ass.net>
On Fri, 11 Jul 2014, Peter Zijlstra wrote:
> > There are other cases too.
>
> Are there any sane ones?
>
They are specifically allowed by the ACPI specification to be able to
include only cpus, I/O, networking cards, etc.
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: David Rientjes @ 2014-07-15 1:19 UTC (permalink / raw)
To: Jiri Kosina
Cc: Greg KH, Jiang Liu, Peter Zijlstra, Andrew Morton, Mel Gorman,
Mike Galbraith, Rafael J . Wysocki, Tony Luck,
Nishanth Aravamudan, linux-mm, linux-hotplug, linux-kernel
In-Reply-To: <alpine.LRH.2.00.1407120039120.17906@twin.jikos.cz>
On Sat, 12 Jul 2014, Jiri Kosina wrote:
> I am pretty sure I've seen ppc64 machine with memoryless NUMA node.
>
Yes, Nishanth Aravamudan (now cc'd) has been working diligently on the
problems that have been encountered, including problems in generic kernel
code, on powerpc with memoryless nodes.
^ permalink raw reply
* Re: [Patch Part3 V4 21/21] pci, ACPI, iommu: Enhance pci_root to support DMAR device hotplug
From: Bjorn Helgaas @ 2014-07-16 18:48 UTC (permalink / raw)
To: Jiang Liu
Cc: Joerg Roedel, David Woodhouse, Yinghai Lu, Dan Williams,
Vinod Koul, Rafael J . Wysocki, Rafael J. Wysocki, Len Brown,
Ashok Raj, Yijing Wang, Tony Luck, open list:INTEL IOMMU (VT-d),
linux-pci@vger.kernel.org, linux hotplug mailing,
linux-kernel@vger.kernel.org, dmaengine,
linux-acpi@vger.kernel.org
In-Reply-To: <1405059585-10620-22-git-send-email-jiang.liu@linux.intel.com>
On Fri, Jul 11, 2014 at 12:19 AM, Jiang Liu <jiang.liu@linux.intel.com> wrote:
> Finally enhance pci_root driver to support DMAR device hotplug when
> hot-plugging PCI host bridges.
>
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
I assume you'll merge this along with the rest of the series via some
non-PCI tree.
> ---
> drivers/acpi/pci_root.c | 16 ++++++++++++++--
> 1 file changed, 14 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
> index d388f13d48b4..99c2b9761c12 100644
> --- a/drivers/acpi/pci_root.c
> +++ b/drivers/acpi/pci_root.c
> @@ -33,6 +33,7 @@
> #include <linux/pci.h>
> #include <linux/pci-acpi.h>
> #include <linux/pci-aspm.h>
> +#include <linux/dmar.h>
> #include <linux/acpi.h>
> #include <linux/slab.h>
> #include <acpi/apei.h> /* for acpi_hest_init() */
> @@ -511,6 +512,7 @@ static int acpi_pci_root_add(struct acpi_device *device,
> struct acpi_pci_root *root;
> acpi_handle handle = device->handle;
> int no_aspm = 0, clear_aspm = 0;
> + bool hotadd = system_state != SYSTEM_BOOTING;
>
> root = kzalloc(sizeof(struct acpi_pci_root), GFP_KERNEL);
> if (!root)
> @@ -557,6 +559,11 @@ static int acpi_pci_root_add(struct acpi_device *device,
> strcpy(acpi_device_class(device), ACPI_PCI_ROOT_CLASS);
> device->driver_data = root;
>
> + if (hotadd && dmar_device_add(handle)) {
> + result = -ENXIO;
> + goto end;
> + }
> +
> pr_info(PREFIX "%s [%s] (domain %04x %pR)\n",
> acpi_device_name(device), acpi_device_bid(device),
> root->segment, &root->secondary);
> @@ -583,7 +590,7 @@ static int acpi_pci_root_add(struct acpi_device *device,
> root->segment, (unsigned int)root->secondary.start);
> device->driver_data = NULL;
> result = -ENODEV;
> - goto end;
> + goto remove_dmar;
> }
>
> if (clear_aspm) {
> @@ -597,7 +604,7 @@ static int acpi_pci_root_add(struct acpi_device *device,
> if (device->wakeup.flags.run_wake)
> device_set_run_wake(root->bus->bridge, true);
>
> - if (system_state != SYSTEM_BOOTING) {
> + if (hotadd) {
> pcibios_resource_survey_bus(root->bus);
> pci_assign_unassigned_root_bus_resources(root->bus);
> }
> @@ -607,6 +614,9 @@ static int acpi_pci_root_add(struct acpi_device *device,
> pci_unlock_rescan_remove();
> return 1;
>
> +remove_dmar:
> + if (hotadd)
> + dmar_device_remove(handle);
> end:
> kfree(root);
> return result;
> @@ -625,6 +635,8 @@ static void acpi_pci_root_remove(struct acpi_device *device)
>
> pci_remove_root_bus(root->bus);
>
> + dmar_device_remove(device->handle);
> +
> pci_unlock_rescan_remove();
>
> kfree(root);
> --
> 1.7.10.4
>
^ permalink raw reply
* Re: [RFC Patch V1 09/30] mm, memcg: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Michal Hocko @ 2014-07-18 7:36 UTC (permalink / raw)
To: Jiang Liu
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Johannes Weiner, Tony Luck,
linux-mm, linux-hotplug, linux-kernel, cgroups
In-Reply-To: <1405064267-11678-10-git-send-email-jiang.liu@linux.intel.com>
On Fri 11-07-14 15:37:26, Jiang Liu wrote:
> When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
> may return a node without memory, and later cause system failure/panic
> when calling kmalloc_node() and friends with returned node id.
> So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
> memory for the/current cpu.
>
> If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
> is the same as cpu_to_node()/numa_node_id().
The change makes difference only for really tiny memcgs. If we really
have all pages on unevictable list or anon with no swap allowed and that
is the reason why no node is set in scan_nodes mask then reclaiming
memoryless node or any arbitrary close one doesn't make any difference.
The current memcg might not have any memory on that node at all.
So the change doesn't make any practical difference and the changelog is
misleading.
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> ---
> mm/memcontrol.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index a2c7bcb0e6eb..d6c4b7255ca9 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -1933,7 +1933,7 @@ int mem_cgroup_select_victim_node(struct mem_cgroup *memcg)
> * we use curret node.
> */
> if (unlikely(node = MAX_NUMNODES))
> - node = numa_node_id();
> + node = numa_mem_id();
>
> memcg->last_scanned_node = node;
> return node;
> --
> 1.7.10.4
>
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* Re: [RFC Patch V1 21/30] mm, irqchip: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Jason Cooper @ 2014-07-18 12:40 UTC (permalink / raw)
To: Jiang Liu
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Thomas Gleixner,
Alexander Shiyan, Tony Luck, linux-mm, linux-hotplug,
linux-kernel, linux-arm-kernel
In-Reply-To: <1405064267-11678-22-git-send-email-jiang.liu@linux.intel.com>
On Fri, Jul 11, 2014 at 03:37:38PM +0800, Jiang Liu wrote:
> When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
> may return a node without memory, and later cause system failure/panic
> when calling kmalloc_node() and friends with returned node id.
> So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
> memory for the/current cpu.
>
> If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
> is the same as cpu_to_node()/numa_node_id().
>
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> ---
> drivers/irqchip/irq-clps711x.c | 2 +-
> drivers/irqchip/irq-gic.c | 2 +-
> 2 files changed, 2 insertions(+), 2 deletions(-)
Do you have anything depending on this? Can apply it to irqchip? If
you need to keep it with other changes,
Acked-by: Jason Cooper <jason@lakedaemon.net>
But please do let me know if I can take it.
thx,
Jason.
^ permalink raw reply
* Re: [RFC Patch V1 01/30] mm, kernel: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Nishanth Aravamudan @ 2014-07-21 17:15 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Dipankar Sarma, Balbir Singh, Thomas Gleixner, Jens Axboe,
Frederic Weisbecker, Jan Kara, Ingo Molnar, Christoph Hellwig,
Srivatsa S. Bhat, Roman Gushchin, Xie XiuQi, Tony Luck, linux-mm,
linux-hotplug, linux-kernel
In-Reply-To: <20140711151405.GK16041@linux.vnet.ibm.com>
Hi Paul,
On 11.07.2014 [08:14:05 -0700], Paul E. McKenney wrote:
> On Fri, Jul 11, 2014 at 03:37:18PM +0800, Jiang Liu wrote:
> > When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
> > may return a node without memory, and later cause system failure/panic
> > when calling kmalloc_node() and friends with returned node id.
> > So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
> > memory for the/current cpu.
> >
> > If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
> > is the same as cpu_to_node()/numa_node_id().
> >
> > Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
>
> For the rcutorture piece:
>
> Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>
> Or if you separate the kernel/rcu/rcutorture.c portion into a separate
> patch, I will queue it separately.
Just FYI, based upon a separate discussion with Tejun and others, it
seems to be preferred to avoid the proliferation of cpu_to_mem
throughout the kernel blindly. For kthread_create_on_node(), I'm going
to try and fix the underlying issue and so you, as the caller, should
still specify the NUMA node you are running the kthread on
(cpu_to_node), not where you expect the memory to come from
(cpu_to_mem).
Thanks,
Nish
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Nishanth Aravamudan @ 2014-07-21 17:23 UTC (permalink / raw)
To: Jiang Liu
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Tony Luck, linux-mm,
linux-hotplug, linux-kernel
In-Reply-To: <1405064267-11678-1-git-send-email-jiang.liu@linux.intel.com>
Hi Jiang,
On 11.07.2014 [15:37:17 +0800], Jiang Liu wrote:
> Previously we have posted a patch fix a memory crash issue caused by
> memoryless node on x86 platforms, please refer to
> http://comments.gmane.org/gmane.linux.kernel/1687425
>
> As suggested by David Rientjes, the most suitable fix for the issue
> should be to use cpu_to_mem() rather than cpu_to_node() in the caller.
> So this is the patchset according to David's suggestion.
Hrm, that is initially what David said, but then later on in the thread,
he specifically says he doesn't think memoryless nodes are the problem.
It seems like the issue is the order of onlining of resources on a
specifix x86 platform?
memoryless nodes in and of themselves don't cause the kernel to crash.
powerpc boots with them (both previously without
CONFIG_HAVE_MEMORYLESS_NODES and now with it) and is functional,
although it does lead to some performance issues I'm hoping to resolve.
In fact, David specifically says that the kernel crash you triggered
makes sense as cpu_to_node() points to an offline node?
In any case, a blind s/cpu_to_node/cpu_to_mem/ is not always correct.
There is a semantic difference and in some cases the allocator already
do the right thing under covers (falls back to nearest node) and in some
cases it doesn't.
Thanks,
Nish
^ permalink raw reply
* Re: [RFC Patch V1 01/30] mm, kernel: Use cpu_to_mem()/numa_mem_id() to support memoryless node
From: Paul E. McKenney @ 2014-07-21 17:33 UTC (permalink / raw)
To: Nishanth Aravamudan
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
Dipankar Sarma, Balbir Singh, Thomas Gleixner, Jens Axboe,
Frederic Weisbecker, Jan Kara, Ingo Molnar, Christoph Hellwig,
Srivatsa S. Bhat, Roman Gushchin, Xie XiuQi, Tony Luck, linux-mm,
linux-hotplug, linux-kernel
In-Reply-To: <20140721171527.GA4156@linux.vnet.ibm.com>
On Mon, Jul 21, 2014 at 10:15:27AM -0700, Nishanth Aravamudan wrote:
> Hi Paul,
>
> On 11.07.2014 [08:14:05 -0700], Paul E. McKenney wrote:
> > On Fri, Jul 11, 2014 at 03:37:18PM +0800, Jiang Liu wrote:
> > > When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
> > > may return a node without memory, and later cause system failure/panic
> > > when calling kmalloc_node() and friends with returned node id.
> > > So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
> > > memory for the/current cpu.
> > >
> > > If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
> > > is the same as cpu_to_node()/numa_node_id().
> > >
> > > Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> >
> > For the rcutorture piece:
> >
> > Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> >
> > Or if you separate the kernel/rcu/rcutorture.c portion into a separate
> > patch, I will queue it separately.
>
> Just FYI, based upon a separate discussion with Tejun and others, it
> seems to be preferred to avoid the proliferation of cpu_to_mem
> throughout the kernel blindly. For kthread_create_on_node(), I'm going
> to try and fix the underlying issue and so you, as the caller, should
> still specify the NUMA node you are running the kthread on
> (cpu_to_node), not where you expect the memory to come from
> (cpu_to_mem).
Even better!!! ;-)
Thanx, Paul
^ permalink raw reply
* Re: [RFC Patch V1 17/30] mm, intel_powerclamp: Use cpu_to_mem()/numa_mem_id() to support memoryless
From: Nishanth Aravamudan @ 2014-07-21 17:38 UTC (permalink / raw)
To: Jiang Liu
Cc: Andrew Morton, Mel Gorman, David Rientjes, Mike Galbraith,
Peter Zijlstra, Rafael J . Wysocki, Zhang Rui, Eduardo Valentin,
Tony Luck, linux-mm, linux-hotplug, linux-kernel, linux-pm
In-Reply-To: <1405064267-11678-18-git-send-email-jiang.liu@linux.intel.com>
On 11.07.2014 [15:37:34 +0800], Jiang Liu wrote:
> When CONFIG_HAVE_MEMORYLESS_NODES is enabled, cpu_to_node()/numa_node_id()
> may return a node without memory, and later cause system failure/panic
> when calling kmalloc_node() and friends with returned node id.
> So use cpu_to_mem()/numa_mem_id() instead to get the nearest node with
> memory for the/current cpu.
You used the same changelog for all of the patches, it seems. But the
interface below (kthread_create_on_node) doesn't go into kmalloc_node?
kthread_create_on_node eventually sets the value used by
tsk_fork_get_node(), which is used by alloc_task_struct_node() and
alloc_thread_info_node(). The first uses kmem_cache_alloc_node() and the
second, depending on the relative sizes of THREAD_SIZE and PAGE_SIZE
uses either alloc_kmem_pages_node() or kmem_cache_alloc_node().
kmem_cache_alloc_node() goes into the appropriate slab allocator which
on SLUB for instance, goes down into __alloc_pages_nodemask. But no
failure occurs when memoryless nodes are present, you just get memory
that is remote from the node specified? Similarly,
alloc_kmem_pages_node() calls into __alloc_pages with an appropriate
node_zonelist, which should provide for the correct fallback based upon
NUMA topology?
What system failure/panic did you see that is resolved by this patch?
> If CONFIG_HAVE_MEMORYLESS_NODES is disabled, cpu_to_mem()/numa_mem_id()
> is the same as cpu_to_node()/numa_node_id().
>
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
> ---
> drivers/thermal/intel_powerclamp.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/thermal/intel_powerclamp.c b/drivers/thermal/intel_powerclamp.c
> index 95cb7fc20e17..9d9be8cd1b50 100644
> --- a/drivers/thermal/intel_powerclamp.c
> +++ b/drivers/thermal/intel_powerclamp.c
> @@ -531,7 +531,7 @@ static int start_power_clamp(void)
>
> thread = kthread_create_on_node(clamp_thread,
> (void *) cpu,
> - cpu_to_node(cpu),
> + cpu_to_mem(cpu),
As Tejun has pointed out elsewhere, we lose context here about the
original node we were running on. That information is relevant for a few
reasons:
1) In the underlying allocator, we might not have memory *right now* to
satisfy a request, which, say, causes us to deactivate a slab
(CONFIG_SLUB). But that condition may be relieved in the future and we
want to use the correct node again then.
2) For topologies that are symmetrical around a memoryless node, we
could lose the correct fallback information when we specify a nearest
neighbor with memory.
Thanks,
Nish
^ permalink raw reply
* Re: [RFC Patch V1 00/30] Enable memoryless node on x86 platforms
From: Tony Luck @ 2014-07-21 17:41 UTC (permalink / raw)
To: Nishanth Aravamudan
Cc: Jiang Liu, Andrew Morton, Mel Gorman, David Rientjes,
Mike Galbraith, Peter Zijlstra, Rafael J . Wysocki,
linux-mm@kvack.org, linux-hotplug, Linux Kernel Mailing List
In-Reply-To: <20140721172331.GB4156@linux.vnet.ibm.com>
On Mon, Jul 21, 2014 at 10:23 AM, Nishanth Aravamudan
<nacc@linux.vnet.ibm.com> wrote:
> It seems like the issue is the order of onlining of resources on a
> specific x86 platform?
Yes. When we online a node the BIOS hits us with some ACPI hotplug events:
First: Here are some new cpus
Next: Here is some new memory
Last; Here are some new I/O things (PCIe root ports, PCIe devices,
IOAPICs, IOMMUs, ...)
So there is a period where the node is memoryless - although that will generally
be resolved when the memory hot plug event arrives ... that isn't guaranteed to
occur (there might not be any memory on the node, or what memory there is
may have failed self-test and been disabled).
-Tony
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox