Re: [RFC PATCH] dt:numa: adding numa node mapping for memory nodes.

Re: [RFC PATCH] dt:numa: adding numa node mapping for memory nodes.

[Kumar Gala]

On Sep 17, 2014, at 1:56 AM, Ganapatrao Kulkarni =
<ganapatrao.kulkarni@caviumnetworks.com> wrote:

> From: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>
>=20
> This patch adds property "nid" to memory node to provide the memory =
range to
> numa node id mapping.
>=20
> Signed-off-by: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>
>=20
> =97

Adding the PPC guys as they=92ve been doing NUMA on IBM Power Servers =
for years with OF/DT.  So we should really try and follow what they=92ve =
done.

> Documentation/devicetree/bindings/numa.txt | 58 =
++++++++++++++++++++++++++++++
> 1 file changed, 58 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/numa.txt
>=20
> diff --git a/Documentation/devicetree/bindings/numa.txt =
b/Documentation/devicetree/bindings/numa.txt
> new file mode 100644
> index 0000000..c4a94f2
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/numa.txt
> @@ -0,0 +1,58 @@
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D
> +numa id binding description
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D
> +
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D
> +1 - Introduction
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D
> +The device node  property "nid(numa node id)" can be added to memory
> +device node to map the range of memory addresses as defined in =
property "reg".
> +The property "nid" maps the memory range to the numa node id, which =
is used to
> +find the local and remory pages on numa aware systems.
> +
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D
> +2 - nid property
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D
> +Numa node id, "nid" is required property of memory device node for=20
> +numa enabled platforms.
> +
> +|------------------------------------------------------|
> +|Property Type  | Usage | Value Type | Definition      |
> +|------------------------------------------------------|
> +|  nid          |  R    |    <u32>   | Numa Node id    |
> +|               |       |            | for this memory |
> +|------------------------------------------------------|
> +
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D
> +4 - Example memory nodes with numa node id mapping
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D
> +
> +Example 1 (2 memory nodes, each mapped to a numa node.):
> +
> +	memory@00000000 {
> +		device_type =3D "memory";
> +		reg =3D <0x0 0x00000000 0x0 0x80000000>;
> +		nid =3D <0x0>;
> +	};
> +
> +	memory@10000000000 {
> +		device_type =3D "memory";
> +		reg =3D <0x100 0x00000000 0x0 0x80000000>;
> +		nid =3D <0x1>;
> +	};
> +
> +Example 2 (multiple memory ranges in each memory node and mapped to =
numa node):
> +
> +	memory@00000000 {
> +		device_type =3D "memory";
> +		reg =3D <0x0 0x00000000 0x0 0x80000000>,
> +		      <0x0 0x80000000 0x0 0x80000000>;
> +		nid =3D <0x0>;
> +	};
> +
> +	memory@10000000000 {
> +		device_type =3D "memory";
> +		reg =3D <0x100 0x00000000 0x0 0x80000000>,
> +		      <0x100 0x80000000 0x0 0x80000000>;
> +		nid =3D <0x1>;
> +	};
> --=20
> 1.8.1.4
>=20

--=20
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, =
hosted by The Linux Foundation

Re: [RFC PATCH] dt:numa: adding numa node mapping for memory nodes.

[Mark Rutland]

On Wed, Sep 17, 2014 at 04:37:30PM +0100, Kumar Gala wrote:
> 
> On Sep 17, 2014, at 1:56 AM, Ganapatrao Kulkarni <ganapatrao.kulkarni@caviumnetworks.com> wrote:
> 
> > From: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>
> > 
> > This patch adds property "nid" to memory node to provide the memory range to
> > numa node id mapping.
> > 
> > Signed-off-by: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>
> > 
> > —
> 
> Adding the PPC guys as they’ve been doing NUMA on IBM Power Servers
> for years with OF/DT.  So we should really try and follow what they’ve
> done.

Agreed.

> > Documentation/devicetree/bindings/numa.txt | 58 ++++++++++++++++++++++++++++++
> > 1 file changed, 58 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/numa.txt
> > 
> > diff --git a/Documentation/devicetree/bindings/numa.txt b/Documentation/devicetree/bindings/numa.txt
> > new file mode 100644
> > index 0000000..c4a94f2
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/numa.txt
> > @@ -0,0 +1,58 @@
> > +======================================================
> > +numa id binding description
> > +======================================================
> > +
> > +======================================================
> > +1 - Introduction
> > +======================================================
> > +The device node  property "nid(numa node id)" can be added to memory

Why the quotes?

> > +device node to map the range of memory addresses as defined in property "reg".
> > +The property "nid" maps the memory range to the numa node id, which is used to
> > +find the local and remory pages on numa aware systems.

What is a "numa node id", exactly, and how is the OS intended to use it?

I don't see how this can be of any use as-is.

Mark.

> > +
> > +======================================================
> > +2 - nid property
> > +======================================================
> > +Numa node id, "nid" is required property of memory device node for 
> > +numa enabled platforms.
> > +
> > +|------------------------------------------------------|
> > +|Property Type  | Usage | Value Type | Definition      |
> > +|------------------------------------------------------|
> > +|  nid          |  R    |    <u32>   | Numa Node id    |
> > +|               |       |            | for this memory |
> > +|------------------------------------------------------|
> > +
> > +========================================================
> > +4 - Example memory nodes with numa node id mapping
> > +========================================================
> > +
> > +Example 1 (2 memory nodes, each mapped to a numa node.):
> > +
> > +	memory@00000000 {
> > +		device_type = "memory";
> > +		reg = <0x0 0x00000000 0x0 0x80000000>;
> > +		nid = <0x0>;
> > +	};
> > +
> > +	memory@10000000000 {
> > +		device_type = "memory";
> > +		reg = <0x100 0x00000000 0x0 0x80000000>;
> > +		nid = <0x1>;
> > +	};
> > +
> > +Example 2 (multiple memory ranges in each memory node and mapped to numa node):
> > +
> > +	memory@00000000 {
> > +		device_type = "memory";
> > +		reg = <0x0 0x00000000 0x0 0x80000000>,
> > +		      <0x0 0x80000000 0x0 0x80000000>;
> > +		nid = <0x0>;
> > +	};
> > +
> > +	memory@10000000000 {
> > +		device_type = "memory";
> > +		reg = <0x100 0x00000000 0x0 0x80000000>,
> > +		      <0x100 0x80000000 0x0 0x80000000>;
> > +		nid = <0x1>;
> > +	};
> > -- 
> > 1.8.1.4
> > 
> 
> -- 
> Employee of Qualcomm Innovation Center, Inc.
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
> 
> --
> To unsubscribe from this list: send the line "unsubscribe devicetree" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [RFC PATCH] dt:numa: adding numa node mapping for memory nodes.

[Hanjun Guo]

On 2014-9-18 3:34, Mark Rutland wrote:
> On Wed, Sep 17, 2014 at 04:37:30PM +0100, Kumar Gala wrote:
>>
>> On Sep 17, 2014, at 1:56 AM, Ganapatrao Kulkarni <ganapatrao.kulkarni@caviumnetworks.com> wrote:
>>
>>> From: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>
>>>
>>> This patch adds property "nid" to memory node to provide the memory range to
>>> numa node id mapping.
>>>
>>> Signed-off-by: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>
>>>
>>> —
>>
>> Adding the PPC guys as they’ve been doing NUMA on IBM Power Servers
>> for years with OF/DT.  So we should really try and follow what they’ve
>> done.
> 
> Agreed.
> 
>>> Documentation/devicetree/bindings/numa.txt | 58 ++++++++++++++++++++++++++++++
>>> 1 file changed, 58 insertions(+)
>>> create mode 100644 Documentation/devicetree/bindings/numa.txt
>>>
>>> diff --git a/Documentation/devicetree/bindings/numa.txt b/Documentation/devicetree/bindings/numa.txt
>>> new file mode 100644
>>> index 0000000..c4a94f2
>>> --- /dev/null
>>> +++ b/Documentation/devicetree/bindings/numa.txt
>>> @@ -0,0 +1,58 @@
>>> +======================================================
>>> +numa id binding description
>>> +======================================================
>>> +
>>> +======================================================
>>> +1 - Introduction
>>> +======================================================
>>> +The device node  property "nid(numa node id)" can be added to memory
> 
> Why the quotes?
> 
>>> +device node to map the range of memory addresses as defined in property "reg".
>>> +The property "nid" maps the memory range to the numa node id, which is used to
>>> +find the local and remory pages on numa aware systems.
> 
> What is a "numa node id", exactly, and how is the OS intended to use it?

I think "Proximity Domain" would be more suitably, processors and memory or IOs
in the same domain will have better performance than crossing other domains.

Thanks
Hanjun

Re: [RFC PATCH] dt:numa: adding numa node mapping for memory nodes.

[Ganapatrao Kulkarni]

looks like previous email not had the email ids added by kumar gala.
adding the missing ids and sending again.

On Thu, Sep 18, 2014 at 9:32 AM, Ganapatrao Kulkarni
<gpkulkarni@gmail.com> wrote:
> On Thu, Sep 18, 2014 at 4:42 AM, Zi Shen Lim <zlim.lnx@gmail.com> wrote:
>> On Wed, Sep 17, 2014 at 2:48 PM, Nathan Lynch <Nathan_Lynch@mentor.com> wrote:
>>> On 09/17/2014 03:56 AM, Ganapatrao Kulkarni wrote:
>>>> From: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>
>>>>
>>>> This patch adds property "nid" to memory node to provide the memory range to
>>>> numa node id mapping.
>>>>
>>>> Signed-off-by: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>
>>>>
>>>> ---
>>>>  Documentation/devicetree/bindings/numa.txt | 58 ++++++++++++++++++++++++++++++
>>>>  1 file changed, 58 insertions(+)
>>>>  create mode 100644 Documentation/devicetree/bindings/numa.txt
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/numa.txt b/Documentation/devicetree/bindings/numa.txt
>>>> new file mode 100644
>>>> index 0000000..c4a94f2
>>>> --- /dev/null
>>>> +++ b/Documentation/devicetree/bindings/numa.txt
>>>> @@ -0,0 +1,58 @@
>>>> +======================================================
>>>> +numa id binding description
>>>> +======================================================
>>>> +
>>>> +======================================================
>>>> +1 - Introduction
>>>> +======================================================
>>>> +The device node  property "nid(numa node id)" can be added to memory
>>>> +device node to map the range of memory addresses as defined in property "reg".
>>>> +The property "nid" maps the memory range to the numa node id, which is used to
>>>> +find the local and remory pages on numa aware systems.
>>>
>>> "Local" and "remote" memory are notions that relate to some other
>>> resource -- typically a CPU, but also I/O resources on some systems.  It
>>> seems to me that a useful NUMA binding would at least specify a "nid"
>>> property, or something like it, for both cpu and memory nodes.  But this
>>> document speaks only of memory nodes.
> IMO, nid can be extended to cpu node also if the arch dont have any
> other way to do mapping, in fact this can be scaled to any node.
> for ARM arch, cpu mapping can be fetched from topology parsing and
> more over cpu mapping is required much later when secondary cpus comes
> online.
> i have added for memory, since this is parsed very early in the boot
> sequence and there is no support for dt based map at this moment(at
> least i am not aware off).
> numa distance matrix can also be defined, if the architecture has
> variable node distance other than just local/remote.
>>>
>>> As Kumar said, the device tree on powerpc server systems already has
>>> properties that express NUMA information.  If you can get hold of a copy
>>> of the PAPR (not ePAPR) from power.org, refer to the description of
>>> "ibm,associativity" and related properties.  I recall that it's a bit
>>> more complex than this proposal, though.
>>
>> I'm not able to find a link to the actual PAPR (not ePAPR)
>> specification. Anyone has a linky?
>>
>> I did find some code in tree, but couldn't find the bindings in
>> Documentation/devicetree/.
> yes, there is nothing in Documentation, if PPC are using since several
> years, then i humbly request to publish the documentation for numa
> bindings.
> No point in reinventing the wheel. Also i did not find any spec named
> PAPR, is it open spec, available to all like ePAPR?
>
>>
>> Seems like we'd care about form1?
>> -----8<-----
>> commit 41eab6f88f24124df89e38067b3766b7bef06ddb
>> Author: Anton Blanchard <anton@samba.org>
>> Date:   Sun May 16 20:22:31 2010 +0000
>>
>>     powerpc/numa: Use form 1 affinity to setup node distance
>>
>>     Form 1 affinity allows multiple entries in
>> ibm,associativity-reference-points
>>     which represent affinity domains in decreasing order of importance. The
>>     Linux concept of a node is always the first entry, but using the other
>>     values as an input to node_distance() allows the memory allocator to make
>>     better decisions on which node to go first when local memory has been
>>     exhausted.
>>
>>     We keep things simple and create an array indexed by NUMA node, capped at
>>     4 entries. Each time we lookup an associativity property we initialise
>>     the array which is overkill, but since we should only hit this path during
>>     boot it didn't seem worth adding a per node valid bit.
>>
>>     Signed-off-by: Anton Blanchard <anton@samba.org>
>>     Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
>> ----->8-----
>>
>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel