[Qemu-devel] [RFC PATCH v3 0/3] Add support for NUMA on ARM64

[Qemu-devel] [RFC PATCH v3 0/3] Add support for NUMA on ARM64

[Shannon Zhao]

Add support for NUMA on ARM64. Tested successfully running a guest
Linux kernel with the following patch applied:

- arm64:numa: adding numa support for arm64 platforms.
http://article.gmane.org/gmane.linux.ports.arm.kernel/382179

Changes v2 ... v3:
* update to use NUMA node property arm,associativity.

Changes v1 ... v2:
Take into account Peter's comments:
* rename virt_memory_init to arm_generate_memory_dtb
* move arm_generate_memory_dtb to boot.c and make it a common func
* use a struct numa_map to generate numa dtb

Example qemu command line:
qemu-system-aarch64 \
    -enable-kvm -smp 4\
    -kernel Image \
    -m 512 -machine virt,kernel_irqchip=on \
    -initrd guestfs.cpio.gz \
    -cpu host -nographic \
    -numa node,mem=256M,cpus=0-1,nodeid=0 \
    -numa node,mem=256M,cpus=2-3,nodeid=1 \
    -append "console=ttyAMA0 root=/dev/ram"

Todo:
1)The NUMA nodes information in DT is not finalized yet, so this
patch might need to be further modified to follow any changes in it.

2)Consider IO-NUMA as well

Please refer to the following url for NUMA DT node details:

- Documentation: arm64/arm: dt bindings for numa.
http://article.gmane.org/gmane.linux.ports.arm.kernel/382180

Example: 2 Node system each having 2 CPUs and a Memory

        arm,associativity-reference-points = <0x0 0x1>;

        memory@40000000 {
                reg = <0x0 0x40000000 0x0 0xfffffff>;
                device_type = "memory";
                arm,associativity = <0x0 0x0 0xffff>;
        };
        memory@50000000 {
                reg = <0x0 0x50000000 0x0 0xfffffff>;
                device_type = "memory";
                arm,associativity = <0x1 0x0 0xffff>;
        };

        cpus {
                #size-cells = <0x0>;
                #address-cells = <0x1>;

                cpu@0 {
                        reg = <0x0>;
                        enable-method = "psci";
                        compatible = "arm,arm-v8";
                        device_type = "cpu";
                        arm,associativity = <0x0 0x0 0x0>;
                };

                cpu@1 {
                        reg = <0x1>;
                        enable-method = "psci";
                        compatible = "arm,arm-v8";
                        device_type = "cpu";
                        arm,associativity = <0x0 0x0 0x1>;
                };

                cpu@2 {
                        reg = <0x2>;
                        enable-method = "psci";
                        compatible = "arm,arm-v8";
                        device_type = "cpu";
                        arm,associativity = <0x1 0x0 0x2>;
                };
                cpu@3 {
                        reg = <0x3>;
                        enable-method = "psci";
                        compatible = "arm,arm-v8";
                        device_type = "cpu";
                        arm,associativity = <0x1 0x0 0x3>;
                };
        }

- arm,associativity

The mapping is done using arm,associativity device property.
this property needs to be present in every device node which needs to to be
mapped to numa nodes.

arm,associativity property is set of 32-bit integers. representing the
board id, socket id and core id.

ex:
	/* board 0, socket 0, core 0 */
	arm,associativity = <0 0 0x000>;

	/* board 1, socket 0, core 8 */
	arm,associativity = <1 0 0x08>;
	
- arm,associativity-reference-points

This property is a set of 32-bit integers, each representing an index into
the arm,associativity nodes. The first integer is the most significant
NUMA boundary and the following are progressively less significant boundaries.
There can be more than one level of NUMA.

Ex:
	arm,associativity-reference-points = <0 1>;
	The board Id(index 0) used first to calculate the associativity (node
	distance), then follows the  socket id(index 1).

	arm,associativity-reference-points = <1 0>;
	The socket Id(index 1) used first to calculate the associativity,
	then follows the board id(index 0).

	arm,associativity-reference-points = <0>;
	Only the board Id(index 0) used to calculate the associativity.

	arm,associativity-reference-points = <1>;
	Only socket Id(index 1) used to calculate the associativity.

Shannon Zhao (3):
  hw/arm/virt: Use memory_region_allocate_system_memory to allocate
    memory
  hw/arm/virt: Don't add memory node in creat_fdt
  hw/arm/boot: Add arm_generate_memory_dtb to generate memory dtb
    according to NUMA topology

 hw/arm/boot.c |   80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 hw/arm/virt.c |    7 +---
 2 files changed, 79 insertions(+), 8 deletions(-)

[Qemu-devel] [RFC PATCH v3 1/3] hw/arm/virt: Use memory_region_allocate_system_memory to allocate memory

[Shannon Zhao]

Use memory_region_allocate_system_memory to allocate memory.
The function is sensitive to NUMA and can allocate memory
for NUMA topology.

Signed-off-by: Shannon Zhao <zhaoshenglong@huawei.com>
---
 hw/arm/virt.c |    5 ++---
 1 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 2353440..fdafa79 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -627,9 +627,8 @@ static void machvirt_init(MachineState *machine)
     fdt_add_cpu_nodes(vbi);
     fdt_add_psci_node(vbi);
 
-    memory_region_init_ram(ram, NULL, "mach-virt.ram", machine->ram_size,
-                           &error_abort);
-    vmstate_register_ram_global(ram);
+    memory_region_allocate_system_memory(ram, NULL, "mach-virt.ram",
+                                         machine->ram_size);
     memory_region_add_subregion(sysmem, vbi->memmap[VIRT_MEM].base, ram);
 
     create_flash(vbi);
-- 
1.7.1

[Qemu-devel] [RFC PATCH v3 2/3] hw/arm/virt: Don't add memory node in creat_fdt

[Shannon Zhao]

To support memory NUMA, don't add memory node in creat_fdt.
But add it in a new function which takes into accout NUMA
topology.

Signed-off-by: Shannon Zhao <zhaoshenglong@huawei.com>
---
 hw/arm/virt.c |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index fdafa79..505cd29 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -190,8 +190,6 @@ static void create_fdt(VirtBoardInfo *vbi)
      * to fill in necessary properties later
      */
     qemu_fdt_add_subnode(fdt, "/chosen");
-    qemu_fdt_add_subnode(fdt, "/memory");
-    qemu_fdt_setprop_string(fdt, "/memory", "device_type", "memory");
 
     /* Clock node, for the benefit of the UART. The kernel device tree
      * binding documentation claims the PL011 node clock properties are
-- 
1.7.1

[Qemu-devel] [RFC PATCH v3 3/3] hw/arm/boot: Generate memory dtb according to NUMA topology

[Shannon Zhao]

Add a new function arm_generate_memory_dtb which is used to
generate memory dtb according to NUMA topology and set the
NUMA topology property of every cpu.

Signed-off-by: Shannon Zhao <zhaoshenglong@huawei.com>
---
 hw/arm/boot.c |   80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 files changed, 77 insertions(+), 3 deletions(-)

diff --git a/hw/arm/boot.c b/hw/arm/boot.c
index 52ebd8b..a39b2b4 100644
--- a/hw/arm/boot.c
+++ b/hw/arm/boot.c
@@ -312,6 +312,82 @@ static void set_kernel_args_old(const struct arm_boot_info *info)
     }
 }
 
+/*
+ * arm_generate_memory_dtb() - generate memory dtb according to NUMA topology
+ * @fdt: the pointer to device tree
+ * @binfo: struct describing the boot environment
+ * @acells: address-cells of device tree
+ * @scells: size-cells of device tree
+ *
+ * Returns:0 success,
+ * -1 on errors.
+ *
+ */
+static int arm_generate_memory_dtb(void *fdt, const struct arm_boot_info *binfo,
+                        uint32_t acells, uint32_t scells)
+{
+    CPUState *cpu;
+    int i = 0;
+
+    if (!nb_numa_nodes) {
+        qemu_fdt_add_subnode(fdt, "/memory");
+        qemu_fdt_setprop_string(fdt, "/memory", "device_type", "memory");
+        return qemu_fdt_setprop_sized_cells(fdt, "/memory", "reg",
+                                      acells, binfo->loader_start,
+                                      scells, binfo->ram_size);
+    }
+
+    struct {
+        uint64_t mem_map[6];
+        uint64_t cpu_map[6];
+    } numa_map;
+
+    hwaddr mem_base = binfo->loader_start;
+
+    for (i = 0; i < nb_numa_nodes; i++) {
+        /* Generate mem_map */
+        char *nodename;
+        nodename = g_strdup_printf("/memory@%" PRIx64, mem_base);
+        qemu_fdt_add_subnode(fdt, nodename);
+        qemu_fdt_setprop_string(fdt, nodename, "device_type", "memory");
+        qemu_fdt_setprop_sized_cells(fdt, nodename, "reg",
+                                  acells, mem_base,
+                                  scells, numa_info[i].node_mem - 1);
+        numa_map.mem_map[0] = 0x1;
+        numa_map.mem_map[1] = i;
+        numa_map.mem_map[2] = 0x1;
+        numa_map.mem_map[3] = 0x0;
+        numa_map.mem_map[4] = 0x1;
+        numa_map.mem_map[5] = 0xffff;
+
+        qemu_fdt_setprop_sized_cells_from_array(fdt, nodename,
+                "arm,associativity", 3, numa_map.mem_map);
+
+        mem_base += numa_info[i].node_mem;
+        g_free(nodename);
+
+        /* Generate cpu_map */
+        CPU_FOREACH(cpu) {
+            if (test_bit(cpu->cpu_index, numa_info[i].node_cpu)) {
+                numa_map.cpu_map[0] = 0x1;
+                numa_map.cpu_map[1] = i;
+                numa_map.cpu_map[2] = 0x1;
+                numa_map.cpu_map[3] = 0x0;
+                numa_map.cpu_map[4] = 0x1;
+                numa_map.cpu_map[5] = cpu->cpu_index;
+                nodename = g_strdup_printf("/cpus/cpu@%d", cpu->cpu_index);
+                qemu_fdt_setprop_sized_cells_from_array(fdt, nodename,
+                        "arm,associativity", 3, numa_map.cpu_map);
+                g_free(nodename);
+            }
+        }
+    }
+    qemu_fdt_setprop_sized_cells(fdt, "/",
+            "arm,associativity-reference-points", 1, 0, 1, 1);
+
+    return 0;
+}
+
 /**
  * load_dtb() - load a device tree binary image into memory
  * @addr:       the address to load the image at
@@ -387,9 +463,7 @@ static int load_dtb(hwaddr addr, const struct arm_boot_info *binfo,
         goto fail;
     }
 
-    rc = qemu_fdt_setprop_sized_cells(fdt, "/memory", "reg",
-                                      acells, binfo->loader_start,
-                                      scells, binfo->ram_size);
+    rc = arm_generate_memory_dtb(fdt, binfo, acells, scells);
     if (rc < 0) {
         fprintf(stderr, "couldn't set /memory/reg\n");
         goto fail;
-- 
1.7.1

Re: [Qemu-devel] [RFC PATCH v3 2/3] hw/arm/virt: Don't add memory node in creat_fdt

[Peter Maydell]

On 6 January 2015 at 05:57, Shannon Zhao <zhaoshenglong@huawei.com> wrote:
> To support memory NUMA, don't add memory node in creat_fdt.
> But add it in a new function which takes into accout NUMA
> topology.
>
> Signed-off-by: Shannon Zhao <zhaoshenglong@huawei.com>
> ---
>  hw/arm/virt.c |    2 --
>  1 files changed, 0 insertions(+), 2 deletions(-)
>
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index fdafa79..505cd29 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -190,8 +190,6 @@ static void create_fdt(VirtBoardInfo *vbi)
>       * to fill in necessary properties later
>       */
>      qemu_fdt_add_subnode(fdt, "/chosen");
> -    qemu_fdt_add_subnode(fdt, "/memory");
> -    qemu_fdt_setprop_string(fdt, "/memory", "device_type", "memory");
>
>      /* Clock node, for the benefit of the UART. The kernel device tree
>       * binding documentation claims the PL011 node clock properties are

This patch will break bisection of all platforms using device tree;
you need to keep things working at all points in your patch series,
not just at the end when all the patches are applied.

thanks
-- PMM

Re: [Qemu-devel] [RFC PATCH v3 3/3] hw/arm/boot: Generate memory dtb according to NUMA topology

[Peter Maydell]

On 6 January 2015 at 05:57, Shannon Zhao <zhaoshenglong@huawei.com> wrote:
> Add a new function arm_generate_memory_dtb which is used to
> generate memory dtb according to NUMA topology and set the
> NUMA topology property of every cpu.
>
> Signed-off-by: Shannon Zhao <zhaoshenglong@huawei.com>

This looks a lot nicer than the previous patchset. It
doesn't seem to me like there's much point in reviewing it
in detail until the kernel/dtb folks have confirmed the
dtb bindings for NUMA and committed the documentation into
the kernel tree, though.

thanks
-- PMM

Re: [Qemu-devel] [RFC PATCH v3 2/3] hw/arm/virt: Don't add memory node in creat_fdt

[Shannon Zhao]

On 2015/1/6 17:55, Peter Maydell wrote:
> On 6 January 2015 at 05:57, Shannon Zhao <zhaoshenglong@huawei.com> wrote:
>> To support memory NUMA, don't add memory node in creat_fdt.
>> But add it in a new function which takes into accout NUMA
>> topology.
>>
>> Signed-off-by: Shannon Zhao <zhaoshenglong@huawei.com>
>> ---
>>  hw/arm/virt.c |    2 --
>>  1 files changed, 0 insertions(+), 2 deletions(-)
>>
>> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
>> index fdafa79..505cd29 100644
>> --- a/hw/arm/virt.c
>> +++ b/hw/arm/virt.c
>> @@ -190,8 +190,6 @@ static void create_fdt(VirtBoardInfo *vbi)
>>       * to fill in necessary properties later
>>       */
>>      qemu_fdt_add_subnode(fdt, "/chosen");
>> -    qemu_fdt_add_subnode(fdt, "/memory");
>> -    qemu_fdt_setprop_string(fdt, "/memory", "device_type", "memory");
>>
>>      /* Clock node, for the benefit of the UART. The kernel device tree
>>       * binding documentation claims the PL011 node clock properties are
> 
> This patch will break bisection of all platforms using device tree;
> you need to keep things working at all points in your patch series,
> not just at the end when all the patches are applied.
> 
Hi Peter,

Thanks for your remind. I'll make the patch 2 and 3 together in next version.

Thanks,
Shannon

Re: [Qemu-devel] [RFC PATCH v3 3/3] hw/arm/boot: Generate memory dtb according to NUMA topology

[Shannon Zhao]

On 2015/1/6 17:58, Peter Maydell wrote:
> On 6 January 2015 at 05:57, Shannon Zhao <zhaoshenglong@huawei.com> wrote:
>> Add a new function arm_generate_memory_dtb which is used to
>> generate memory dtb according to NUMA topology and set the
>> NUMA topology property of every cpu.
>>
>> Signed-off-by: Shannon Zhao <zhaoshenglong@huawei.com>
> 
> This looks a lot nicer than the previous patchset. It
> doesn't seem to me like there's much point in reviewing it
> in detail until the kernel/dtb folks have confirmed the
> dtb bindings for NUMA and committed the documentation into
> the kernel tree, though.
> 

Good :) I will also stay tuned on that.

Thanks,
Shannon