Re: [PATCH v5] Documentation, dt, numa: Add note to empty NUMA node

[Gavin Shan <gshan@redhat.com>]

Hi Rob,

On 7/2/21 10:02 AM, Gavin Shan wrote:
> On 7/2/21 3:25 AM, Rob Herring wrote:
>> On Mon, Jun 28, 2021 at 05:34:11PM +0800, Gavin Shan wrote:
>>> The empty memory nodes, where no memory resides in, are allowed.
>>> For these empty memory nodes, the 'len' of 'reg' property is zero.
>>> The NUMA node IDs are still valid and parsed, but memory may be
>>> added to them through hotplug afterwards. I finds difficulty to
>>> get where it's properly documented.
>>
>> This is already in use? If so, what platform(s)?
>>
> 
> It's not used yet, but will be used by QEMU once this patch is merged.
> In QEMU, ARM64 could have multiple empty memory nodes. The corresponding
> NUMA ID and distance map are still valid because memory may be added into
> these empty memory nodes in future.
> 
> For the QEMU case, the names of empty memory nodes are the biggest concern.
> According to device-tree specification, the name follows the format of
> 'memory@unit-address' and the 'unit-address' is equivalent to 'base-address'.
> However, the 'base-address' should be invalid one. In current QEMU implementation,
> the valid 'base-address' and 'unit-address' are provided to these empty
> memory nodes. Another issue in QEMU is trying to populate two empty memory
> nodes, which have same names. This leads to failure of device-tree population
> because of the duplicated memory node names, blocking VM from booting.
> 
>>> So lets add a section for empty memory nodes in NUMA binding
>>> document. Also, the 'unit-address', equivalent to 'base-address'
>>> in the 'reg' property of these empty memory nodes is suggested to
>>> be the summation of highest memory address plus the NUMA node ID.
>>
>> What purpose does this serve? The kernel won't do anything with it other
>> than validate the numa-node-id range.
>>
> 
> As mentioned above, the point is to have dummy, invalid and non-overlapped
> 'base-address' and 'unit-address' for these empty memory nodes, to avoid
> duplicated memory node names in devcie-tree.
> 
>>>
>>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>>> ---
>>> v5: Separate section for empty memory node
>>> ---
>>>   Documentation/devicetree/bindings/numa.txt | 61 +++++++++++++++++++++-
>>>   1 file changed, 60 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/Documentation/devicetree/bindings/numa.txt b/Documentation/devicetree/bindings/numa.txt
>>> index 21b35053ca5a..230c734af948 100644
>>> --- a/Documentation/devicetree/bindings/numa.txt
>>> +++ b/Documentation/devicetree/bindings/numa.txt
>>> @@ -103,7 +103,66 @@ Example:
>>>           };
>>>   ==============================================================================
>>> -4 - Example dts
>>> +4 - Empty memory nodes
>>> +==============================================================================
>>> +
>>> +Empty memory nodes, which no memory resides in, are allowed. The 'length'
>>> +field of the 'reg' property is zero, but the 'base-address' is a dummy
>>> +address and invalid. The 'base-address' could be the summation of highest
>>> +memory address plus the NUMA node ID. However, the NUMA node IDs and
>>> +distance maps are still valid and memory may be added into them through
>>> +hotplug afterwards.
>>> +
>>> +Example:
>>> +
>>> +    memory@0 {
>>> +        device_type = "memory";
>>> +        reg = <0x0 0x0 0x0 0x80000000>;
>>> +        numa-node-id = <0>;
>>> +    };
>>> +
>>> +    memory@0x80000000 {
>>
>> unit-address should not have '0x'.
>>
> 
> Ok. Lets fix it in v6 after it's agreed to add the section into the
> NUMA binding document. Actually, the '0x' is copied from the existing
> example in same document. After this patch is finalized, I will post
> separate patch to fix all wrong formats in same document as well.
> 

I posted v6 just now, where '0x' in 'unit-address' is dropped. After
this is merged, I will post followup patch to remove '0x' in 'unit-address'
for existing examples.

Thanks,
Gavin

>>> +        device_type = "memory";
>>> +        reg = <0x0 0x80000000 0x0 0x80000000>;
>>> +        numa-node-id = <1>;
>>> +    };
>>> +
>>> +    /* Empty memory node */
>>> +    memory@0x100000002 {
>>> +        device_type = "memory";
>>> +        reg = <0x1 0x2 0x0 0x0>;
>>> +        numa-node-id = <2>;
>>> +    };
>>> +
>>> +    /* Empty memory node */
>>> +    memory@0x100000003 {
>>> +        device_type = "memory";
>>> +        reg = <0x1 0x3 0x0 0x0>;
>>> +        numa-node-id = <3>;
>>> +    };
>>> +
>>> +    distance-map {
>>> +        compatible = "numa-distance-map-v1";
>>> +        distance-matrix = <0 0  10>,
>>> +                  <0 1  20>,
>>> +                  <0 2  40>,
>>> +                  <0 3  20>,
>>> +                  <1 0  20>,
>>> +                  <1 1  10>,
>>> +                  <1 2  20>,
>>> +                  <1 3  40>,
>>> +                  <2 0  40>,
>>> +                  <2 1  20>,
>>> +                  <2 2  10>,
>>> +                  <2 3  20>,
>>> +                  <3 0  20>,
>>> +                  <3 1  40>,
>>> +                  <3 2  20>,
>>> +                  <3 3  10>;
>>> +    };
>>> +
>>> +==============================================================================
>>> +5 - Example dts
>>>   ==============================================================================
>>>   Dual socket system consists of 2 boards connected through ccn bus and
>>> --