[PATCH 0/2] NUMA emulation for arm64

[PATCH 0/2] NUMA emulation for arm64

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>

This series adds a very simple NUMA emulation implementation and enables
selecting it on arm64 platforms.

Obvious question is why? Short answer - it can bring a significant performance
uplift on Raspberry Pi 5.

Longer answer is that splitting the physical RAM into chunks, and utilising an
allocation policy such as interleaving, can enable the BCM2721 memory controller
to better utilise parallelism in physical memory chip organisation.

In more conrete numbers, testing with Geekbench 6 shows that splitting into four
emulated NUMA nodes can uplift the single core score of the benchmark by around
6%, and the multi-core by around 18%.

Code is quite simple and new functionality can be enabled using the new
NUMA_EMULATION Kconfig option and then at runtime using the existing (shared
with other platforms) numa=fake=<N> kernel boot argument.

Note however that the default allocation policy is not interleaving and further
steps are required to "unlock" the performance uplift.

Simplest method is probably to launch test programs via the
"numactl --interleave=all COMMAND" wrapper, but it is also possible to change
the system wide policy via systemd configuration.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: “Rafael J. Wysocki" <rafael@kernel.org>

Maíra Canal (2):
  numa: Add simple generic NUMA emulation
  arm64/numa: Add NUMA emulation for ARM64

 arch/arm64/Kconfig            | 10 ++++++
 drivers/base/Kconfig          |  7 ++++
 drivers/base/Makefile         |  1 +
 drivers/base/arch_numa.c      |  6 ++++
 drivers/base/numa_emulation.c | 67 +++++++++++++++++++++++++++++++++++
 drivers/base/numa_emulation.h | 21 +++++++++++
 6 files changed, 112 insertions(+)
 create mode 100644 drivers/base/numa_emulation.c
 create mode 100644 drivers/base/numa_emulation.h

-- 
2.44.0

[PATCH 1/2] numa: Add simple generic NUMA emulation

[Tvrtko Ursulin]

From: Maíra Canal <mcanal@igalia.com>

Add some common code for splitting the memory into N emulated NUMA memory
nodes.

Individual architecture can then enable selecting this option and use the
existing numa=fake=<N> kernel argument to enable it.

Memory is always split into equally sized chunks.

Signed-off-by: Maíra Canal <mcanal@igalia.com>
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Co-developed-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: “Rafael J. Wysocki" <rafael@kernel.org>
---
 drivers/base/Kconfig          |  7 ++++
 drivers/base/Makefile         |  1 +
 drivers/base/arch_numa.c      |  6 ++++
 drivers/base/numa_emulation.c | 67 +++++++++++++++++++++++++++++++++++
 drivers/base/numa_emulation.h | 21 +++++++++++
 5 files changed, 102 insertions(+)
 create mode 100644 drivers/base/numa_emulation.c
 create mode 100644 drivers/base/numa_emulation.h

diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index 2b8fd6bb7da0..1f60cd4dd057 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -230,6 +230,13 @@ config GENERIC_ARCH_NUMA
 	  Enable support for generic NUMA implementation. Currently, RISC-V
 	  and ARM64 use it.
 
+config GENERIC_ARCH_NUMA_EMULATION
+	bool
+	depends on GENERIC_ARCH_NUMA
+	help
+	  Enable NUMA emulation. Note that NUMA emulation will only be used if
+	  the machine has no NUMA node.
+
 config FW_DEVLINK_SYNC_STATE_TIMEOUT
 	bool "sync_state() behavior defaults to timeout instead of strict"
 	help
diff --git a/drivers/base/Makefile b/drivers/base/Makefile
index 3079bfe53d04..34fcf5bd7370 100644
--- a/drivers/base/Makefile
+++ b/drivers/base/Makefile
@@ -25,6 +25,7 @@ obj-$(CONFIG_DEV_COREDUMP) += devcoredump.o
 obj-$(CONFIG_GENERIC_MSI_IRQ) += platform-msi.o
 obj-$(CONFIG_GENERIC_ARCH_TOPOLOGY) += arch_topology.o
 obj-$(CONFIG_GENERIC_ARCH_NUMA) += arch_numa.o
+obj-$(CONFIG_GENERIC_ARCH_NUMA_EMULATION) += numa_emulation.o
 obj-$(CONFIG_ACPI) += physical_location.o
 
 obj-y			+= test/
diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c
index 5b59d133b6af..6ad08f681b3c 100644
--- a/drivers/base/arch_numa.c
+++ b/drivers/base/arch_numa.c
@@ -15,6 +15,8 @@
 
 #include <asm/sections.h>
 
+#include "numa_emulation.h"
+
 struct pglist_data *node_data[MAX_NUMNODES] __read_mostly;
 EXPORT_SYMBOL(node_data);
 nodemask_t numa_nodes_parsed __initdata;
@@ -30,6 +32,8 @@ static __init int numa_parse_early_param(char *opt)
 		return -EINVAL;
 	if (str_has_prefix(opt, "off"))
 		numa_off = true;
+	if (str_has_prefix(opt, "fake="))
+		return numa_emu_cmdline(opt + 5);
 
 	return 0;
 }
@@ -471,6 +475,8 @@ void __init arch_numa_init(void)
 			return;
 		if (acpi_disabled && !numa_init(of_numa_init))
 			return;
+		if (!numa_init(numa_emu_init))
+			return;
 	}
 
 	numa_init(dummy_numa_init);
diff --git a/drivers/base/numa_emulation.c b/drivers/base/numa_emulation.c
new file mode 100644
index 000000000000..df652fa8351b
--- /dev/null
+++ b/drivers/base/numa_emulation.c
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Simple NUMA emulation.
+ *
+ * Copyright © 2024 Raspberry Pi Ltd
+ *
+ * Author: Maíra Canal <mcanal@igalia.com>
+ * Author: Tvrtko Ursulin <tursulin@igalia.com>
+ */
+#include <linux/memblock.h>
+
+#include "numa_emulation.h"
+
+static unsigned int emu_nodes;
+
+int __init numa_emu_cmdline(char *str)
+{
+	int ret;
+
+	ret = kstrtouint(str, 10, &emu_nodes);
+	if (ret)
+		return ret;
+
+	if (emu_nodes > MAX_NUMNODES) {
+		pr_notice("numa=fake=%u too large, reducing to %u\n",
+			  emu_nodes, MAX_NUMNODES);
+		emu_nodes = MAX_NUMNODES;
+	}
+
+	return 0;
+}
+
+int __init numa_emu_init(void)
+{
+	phys_addr_t start, end;
+	unsigned long size;
+	unsigned int i;
+	int ret;
+
+	if (!emu_nodes)
+		return -EINVAL;
+
+	start = memblock_start_of_DRAM();
+	end = memblock_end_of_DRAM() - 1;
+
+	size = DIV_ROUND_DOWN_ULL(end - start + 1, emu_nodes);
+	size = PAGE_ALIGN_DOWN(size);
+
+	for (i = 0; i < emu_nodes; i++) {
+		u64 s, e;
+
+		s = start + i * size;
+		e = s + size - 1;
+
+		if (i == (emu_nodes - 1) && e != end)
+			e = end;
+
+		pr_info("Faking a node at [mem %pap-%pap]\n", &s, &e);
+		ret = numa_add_memblk(i, s, e + 1);
+		if (ret) {
+			pr_err("Failed to add fake NUMA node %d!\n", i);
+			break;
+		}
+	}
+
+	return ret;
+}
diff --git a/drivers/base/numa_emulation.h b/drivers/base/numa_emulation.h
new file mode 100644
index 000000000000..62b38215a2f0
--- /dev/null
+++ b/drivers/base/numa_emulation.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * NUMA emulation header
+ *
+ * Copyright © 2024 Raspberry Pi Ltd
+ */
+
+#ifdef CONFIG_GENERIC_ARCH_NUMA_EMULATION
+int numa_emu_cmdline(char *str);
+int __init numa_emu_init(void);
+#else
+static inline int numa_emu_cmdline(char *str)
+{
+	return -EINVAL;
+}
+
+static int __init numa_emu_init(void)
+{
+	return -EOPNOTSUPP;
+}
+#endif /* CONFIG_NUMA_EMU */
-- 
2.44.0

[PATCH 2/2] arm64/numa: Add NUMA emulation for ARM64

[Tvrtko Ursulin]

From: Maíra Canal <mcanal@igalia.com>

Allow selecting NUMA emulation on arm64.

Signed-off-by: Maíra Canal <mcanal@igalia.com>
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: “Rafael J. Wysocki" <rafael@kernel.org>
---
 arch/arm64/Kconfig | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 8e80df015bdd..49c1cdc545d3 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1507,6 +1507,16 @@ config NODES_SHIFT
 	  Specify the maximum number of NUMA Nodes available on the target
 	  system.  Increases memory reserved to accommodate various tables.
 
+config NUMA_EMULATION
+	bool "NUMA emulation"
+	depends on NUMA
+	select GENERIC_ARCH_NUMA_EMULATION
+	help
+	  Enable NUMA emulation support. A flat machine will be split into
+	  virtual nodes when booted with "numa=fake=N", where N is the number
+	  of nodes, the system RAM will be split into N equal chunks, and
+	  assigned to each node.
+
 source "kernel/Kconfig.hz"
 
 config ARCH_SPARSEMEM_ENABLE
-- 
2.44.0

Re: [PATCH 1/2] numa: Add simple generic NUMA emulation

[Greg Kroah-Hartman]

On Tue, Jun 25, 2024 at 01:58:02PM +0100, Tvrtko Ursulin wrote:
> From: Maíra Canal <mcanal@igalia.com>
> 
> Add some common code for splitting the memory into N emulated NUMA memory
> nodes.
> 
> Individual architecture can then enable selecting this option and use the
> existing numa=fake=<N> kernel argument to enable it.
> 
> Memory is always split into equally sized chunks.
> 
> Signed-off-by: Maíra Canal <mcanal@igalia.com>
> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
> Co-developed-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: “Rafael J. Wysocki" <rafael@kernel.org>
> ---
>  drivers/base/Kconfig          |  7 ++++
>  drivers/base/Makefile         |  1 +
>  drivers/base/arch_numa.c      |  6 ++++
>  drivers/base/numa_emulation.c | 67 +++++++++++++++++++++++++++++++++++
>  drivers/base/numa_emulation.h | 21 +++++++++++

Why not just properly describe the numa topology in your bootloader or
device tree and not need any such "fake" stuff at all?

Also, you are now asking me to maintain these new files, not something
I'm comfortable doing at all sorry.

greg k-h

Re: [PATCH 1/2] numa: Add simple generic NUMA emulation

[Greg Kroah-Hartman]

On Tue, Jun 25, 2024 at 01:58:02PM +0100, Tvrtko Ursulin wrote:
> From: Maíra Canal <mcanal@igalia.com>
> 
> Add some common code for splitting the memory into N emulated NUMA memory
> nodes.
> 
> Individual architecture can then enable selecting this option and use the
> existing numa=fake=<N> kernel argument to enable it.
> 
> Memory is always split into equally sized chunks.
> 
> Signed-off-by: Maíra Canal <mcanal@igalia.com>
> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
> Co-developed-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: “Rafael J. Wysocki" <rafael@kernel.org>
> ---
>  drivers/base/Kconfig          |  7 ++++
>  drivers/base/Makefile         |  1 +
>  drivers/base/arch_numa.c      |  6 ++++
>  drivers/base/numa_emulation.c | 67 +++++++++++++++++++++++++++++++++++
>  drivers/base/numa_emulation.h | 21 +++++++++++
>  5 files changed, 102 insertions(+)
>  create mode 100644 drivers/base/numa_emulation.c
>  create mode 100644 drivers/base/numa_emulation.h
> 
> diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
> index 2b8fd6bb7da0..1f60cd4dd057 100644
> --- a/drivers/base/Kconfig
> +++ b/drivers/base/Kconfig
> @@ -230,6 +230,13 @@ config GENERIC_ARCH_NUMA
>  	  Enable support for generic NUMA implementation. Currently, RISC-V
>  	  and ARM64 use it.
>  
> +config GENERIC_ARCH_NUMA_EMULATION
> +	bool
> +	depends on GENERIC_ARCH_NUMA
> +	help
> +	  Enable NUMA emulation. Note that NUMA emulation will only be used if
> +	  the machine has no NUMA node.
> +
>  config FW_DEVLINK_SYNC_STATE_TIMEOUT
>  	bool "sync_state() behavior defaults to timeout instead of strict"
>  	help
> diff --git a/drivers/base/Makefile b/drivers/base/Makefile
> index 3079bfe53d04..34fcf5bd7370 100644
> --- a/drivers/base/Makefile
> +++ b/drivers/base/Makefile
> @@ -25,6 +25,7 @@ obj-$(CONFIG_DEV_COREDUMP) += devcoredump.o
>  obj-$(CONFIG_GENERIC_MSI_IRQ) += platform-msi.o
>  obj-$(CONFIG_GENERIC_ARCH_TOPOLOGY) += arch_topology.o
>  obj-$(CONFIG_GENERIC_ARCH_NUMA) += arch_numa.o
> +obj-$(CONFIG_GENERIC_ARCH_NUMA_EMULATION) += numa_emulation.o
>  obj-$(CONFIG_ACPI) += physical_location.o
>  
>  obj-y			+= test/
> diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c
> index 5b59d133b6af..6ad08f681b3c 100644
> --- a/drivers/base/arch_numa.c
> +++ b/drivers/base/arch_numa.c
> @@ -15,6 +15,8 @@
>  
>  #include <asm/sections.h>
>  
> +#include "numa_emulation.h"
> +
>  struct pglist_data *node_data[MAX_NUMNODES] __read_mostly;
>  EXPORT_SYMBOL(node_data);
>  nodemask_t numa_nodes_parsed __initdata;
> @@ -30,6 +32,8 @@ static __init int numa_parse_early_param(char *opt)
>  		return -EINVAL;
>  	if (str_has_prefix(opt, "off"))
>  		numa_off = true;
> +	if (str_has_prefix(opt, "fake="))
> +		return numa_emu_cmdline(opt + 5);

You did not document this at all :(

Re: [PATCH 1/2] numa: Add simple generic NUMA emulation

[Tvrtko Ursulin]

Hi Greg,

On 26/06/2024 08:38, Greg Kroah-Hartman wrote:
> On Tue, Jun 25, 2024 at 01:58:02PM +0100, Tvrtko Ursulin wrote:
>> From: Maíra Canal <mcanal@igalia.com>
>>
>> Add some common code for splitting the memory into N emulated NUMA memory
>> nodes.
>>
>> Individual architecture can then enable selecting this option and use the
>> existing numa=fake=<N> kernel argument to enable it.
>>
>> Memory is always split into equally sized chunks.
>>
>> Signed-off-by: Maíra Canal <mcanal@igalia.com>
>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
>> Co-developed-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
>> Cc: Catalin Marinas <catalin.marinas@arm.com>
>> Cc: Will Deacon <will@kernel.org>
>> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>> Cc: “Rafael J. Wysocki" <rafael@kernel.org>
>> ---
>>   drivers/base/Kconfig          |  7 ++++
>>   drivers/base/Makefile         |  1 +
>>   drivers/base/arch_numa.c      |  6 ++++
>>   drivers/base/numa_emulation.c | 67 +++++++++++++++++++++++++++++++++++
>>   drivers/base/numa_emulation.h | 21 +++++++++++
> 
> Why not just properly describe the numa topology in your bootloader or
> device tree and not need any such "fake" stuff at all?
> 
> Also, you are now asking me to maintain these new files, not something
> I'm comfortable doing at all sorry.

Mostly because ae3c107cd8be ("numa: Move numa implementation to common 
code") and existing common code in drivers/base/arch_numa.c it appeared 
it could be acceptable to add the simple NUMA emulation into the common 
code too. Then building upon the same concept as on x86 where no need 
for firmware changes is needed for experimenting with different 
configurations.

Would folding into arch_numa.c so no new files are added address your 
concern, or your main issue is the emulation in general?

 >> +	if (str_has_prefix(opt, "fake="))
 >> +		return numa_emu_cmdline(opt + 5);
 >
 > You did not document this at all :(

That was indeed an oversight. Just need to "copy with edits" some stuff 
from Documentation/arch/x86/x86_64/boot-options.rst.

Regards,

Tvrtko

Re: [PATCH 1/2] numa: Add simple generic NUMA emulation

[Tvrtko Ursulin]

Hi Greg,

Gentle reminder on the opens from this thread. Let me re-summarise the 
question below:

On 26/06/2024 12:47, Tvrtko Ursulin wrote:
> 
> Hi Greg,
> 
> On 26/06/2024 08:38, Greg Kroah-Hartman wrote:
>> On Tue, Jun 25, 2024 at 01:58:02PM +0100, Tvrtko Ursulin wrote:
>>> From: Maíra Canal <mcanal@igalia.com>
>>>
>>> Add some common code for splitting the memory into N emulated NUMA 
>>> memory
>>> nodes.
>>>
>>> Individual architecture can then enable selecting this option and use 
>>> the
>>> existing numa=fake=<N> kernel argument to enable it.
>>>
>>> Memory is always split into equally sized chunks.
>>>
>>> Signed-off-by: Maíra Canal <mcanal@igalia.com>
>>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
>>> Co-developed-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
>>> Cc: Catalin Marinas <catalin.marinas@arm.com>
>>> Cc: Will Deacon <will@kernel.org>
>>> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>>> Cc: “Rafael J. Wysocki" <rafael@kernel.org>
>>> ---
>>>   drivers/base/Kconfig          |  7 ++++
>>>   drivers/base/Makefile         |  1 +
>>>   drivers/base/arch_numa.c      |  6 ++++
>>>   drivers/base/numa_emulation.c | 67 +++++++++++++++++++++++++++++++++++
>>>   drivers/base/numa_emulation.h | 21 +++++++++++
>>
>> Why not just properly describe the numa topology in your bootloader or
>> device tree and not need any such "fake" stuff at all?
>>
>> Also, you are now asking me to maintain these new files, not something
>> I'm comfortable doing at all sorry.
> 
> Mostly because ae3c107cd8be ("numa: Move numa implementation to common 
> code") and existing common code in drivers/base/arch_numa.c it appeared 
> it could be acceptable to add the simple NUMA emulation into the common 
> code too. Then building upon the same concept as on x86 where no need 
> for firmware changes is needed for experimenting with different 
> configurations.
> 
> Would folding into arch_numa.c so no new files are added address your 
> concern, or your main issue is the emulation in general?

Re-iterating and slightly re-formulating this question I see three options:

a)
Fold the new simple generic code into the existing arch_numa.c, 
addressing the "no new files" objection, if that was the main objection.

b)
Move completely into arch code - aka you don't want to see it under 
drivers/base at all, ever, regardless of how simple the new code is, or 
that common NUMA code is already there.

c)
Strong nack for either a) or b) - so "do it in the firmware" comment.

Trying to understand your position so we can progress this.

Thanks,

Tvrtko

> 
>  >> +    if (str_has_prefix(opt, "fake="))
>  >> +        return numa_emu_cmdline(opt + 5);
>  >
>  > You did not document this at all :(
> 
> That was indeed an oversight. Just need to "copy with edits" some stuff 
> from Documentation/arch/x86/x86_64/boot-options.rst.
> 
> Regards,
> 
> Tvrtko

Re: [PATCH 1/2] numa: Add simple generic NUMA emulation

[Tvrtko Ursulin]

[Please excuse the re-send, but as I heard nothing concern is it did not 
get lost in your busy mailbox.]

Hi Greg,

Gentle reminder on the opens from this thread. Let me re-summarise the 
question below:

On 26/06/2024 12:47, Tvrtko Ursulin wrote:
> 
> Hi Greg,
> 
> On 26/06/2024 08:38, Greg Kroah-Hartman wrote:
>> On Tue, Jun 25, 2024 at 01:58:02PM +0100, Tvrtko Ursulin wrote:
>>> From: Maíra Canal <mcanal@igalia.com>
>>>
>>> Add some common code for splitting the memory into N emulated NUMA 
>>> memory
>>> nodes.
>>>
>>> Individual architecture can then enable selecting this option and use 
>>> the
>>> existing numa=fake=<N> kernel argument to enable it.
>>>
>>> Memory is always split into equally sized chunks.
>>>
>>> Signed-off-by: Maíra Canal <mcanal@igalia.com>
>>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
>>> Co-developed-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
>>> Cc: Catalin Marinas <catalin.marinas@arm.com>
>>> Cc: Will Deacon <will@kernel.org>
>>> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>>> Cc: “Rafael J. Wysocki" <rafael@kernel.org>
>>> ---
>>>   drivers/base/Kconfig          |  7 ++++
>>>   drivers/base/Makefile         |  1 +
>>>   drivers/base/arch_numa.c      |  6 ++++
>>>   drivers/base/numa_emulation.c | 67 +++++++++++++++++++++++++++++++++++
>>>   drivers/base/numa_emulation.h | 21 +++++++++++
>>
>> Why not just properly describe the numa topology in your bootloader or
>> device tree and not need any such "fake" stuff at all?
>>
>> Also, you are now asking me to maintain these new files, not something
>> I'm comfortable doing at all sorry.
> 
> Mostly because ae3c107cd8be ("numa: Move numa implementation to common 
> code") and existing common code in drivers/base/arch_numa.c it appeared 
> it could be acceptable to add the simple NUMA emulation into the common 
> code too. Then building upon the same concept as on x86 where no need 
> for firmware changes is needed for experimenting with different 
> configurations.
> 
> Would folding into arch_numa.c so no new files are added address your 
> concern, or your main issue is the emulation in general?

Re-iterating and slightly re-formulating this question I see three options:

a)
Fold the new simple generic code into the existing arch_numa.c, 
addressing the "no new files" objection, if that was the main objection.

b)
Move completely into arch code - aka you don't want to see it under 
drivers/base at all, ever, regardless of how simple the new code is, or 
that common NUMA code is already there.

c)
Strong nack for either a) or b) - so "do it in the firmware" comment.

Trying to understand your position so we can progress this.

Thanks,

Tvrtko

> 
>  >> +    if (str_has_prefix(opt, "fake="))
>  >> +        return numa_emu_cmdline(opt + 5);
>  >
>  > You did not document this at all :(
> 
> That was indeed an oversight. Just need to "copy with edits" some stuff 
> from Documentation/arch/x86/x86_64/boot-options.rst.
> 
> Regards,
> 
> Tvrtko

Re: [PATCH 1/2] numa: Add simple generic NUMA emulation

[Jonathan Cameron]

On Thu, 8 Aug 2024 12:56:44 +0100
Tvrtko Ursulin <tvrtko.ursulin@igalia.com> wrote:

> [Please excuse the re-send, but as I heard nothing concern is it did not 
> get lost in your busy mailbox.]
> 
> Hi Greg,
> 
> Gentle reminder on the opens from this thread. Let me re-summarise the 
> question below:
> 
> On 26/06/2024 12:47, Tvrtko Ursulin wrote:
> > 
> > Hi Greg,
> > 
> > On 26/06/2024 08:38, Greg Kroah-Hartman wrote:  
> >> On Tue, Jun 25, 2024 at 01:58:02PM +0100, Tvrtko Ursulin wrote:  
> >>> From: Maíra Canal <mcanal@igalia.com>
> >>>
> >>> Add some common code for splitting the memory into N emulated NUMA 
> >>> memory
> >>> nodes.
> >>>
> >>> Individual architecture can then enable selecting this option and use 
> >>> the
> >>> existing numa=fake=<N> kernel argument to enable it.
> >>>
> >>> Memory is always split into equally sized chunks.
> >>>
> >>> Signed-off-by: Maíra Canal <mcanal@igalia.com>
> >>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
> >>> Co-developed-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
> >>> Cc: Catalin Marinas <catalin.marinas@arm.com>
> >>> Cc: Will Deacon <will@kernel.org>
> >>> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> >>> Cc: “Rafael J. Wysocki" <rafael@kernel.org>
> >>> ---
> >>>   drivers/base/Kconfig          |  7 ++++
> >>>   drivers/base/Makefile         |  1 +
> >>>   drivers/base/arch_numa.c      |  6 ++++
> >>>   drivers/base/numa_emulation.c | 67 +++++++++++++++++++++++++++++++++++
> >>>   drivers/base/numa_emulation.h | 21 +++++++++++  
> >>
> >> Why not just properly describe the numa topology in your bootloader or
> >> device tree and not need any such "fake" stuff at all?
> >>
> >> Also, you are now asking me to maintain these new files, not something
> >> I'm comfortable doing at all sorry.  
> > 
> > Mostly because ae3c107cd8be ("numa: Move numa implementation to common 
> > code") and existing common code in drivers/base/arch_numa.c it appeared 
> > it could be acceptable to add the simple NUMA emulation into the common 
> > code too. Then building upon the same concept as on x86 where no need 
> > for firmware changes is needed for experimenting with different 
> > configurations.
> > 
> > Would folding into arch_numa.c so no new files are added address your 
> > concern, or your main issue is the emulation in general?  
> 
> Re-iterating and slightly re-formulating this question I see three options:
> 
> a)
> Fold the new simple generic code into the existing arch_numa.c, 
> addressing the "no new files" objection, if that was the main objection.
> 
> b)
> Move completely into arch code - aka you don't want to see it under 
> drivers/base at all, ever, regardless of how simple the new code is, or 
> that common NUMA code is already there.
> 
> c)
> Strong nack for either a) or b) - so "do it in the firmware" comment.
> 
> Trying to understand your position so we can progress this.

See: 
https://lore.kernel.org/all/20240807064110.1003856-20-rppt@kernel.org/
and rest of thread 
https://lore.kernel.org/all/20240807064110.1003856-1-rppt@kernel.org/
[PATCH v4 00/26] mm: introduce numa_memblks

Much larger rework and unification set from Mike Rapoport
that happens to end up adding numa emulation as part of making
the x86 numa_memblk work for arm64 etc.

It's in mm-unstable now so getting some test coverage etc.

Sorry, I'd kind of assumed this also went to linux-mm so
the connection would have been made.

Jonathan


> 
> Thanks,
> 
> Tvrtko
> 
> >   
> >  >> +    if (str_has_prefix(opt, "fake="))
> >  >> +        return numa_emu_cmdline(opt + 5);  
> >  >
> >  > You did not document this at all :(  
> > 
> > That was indeed an oversight. Just need to "copy with edits" some stuff 
> > from Documentation/arch/x86/x86_64/boot-options.rst.
> > 
> > Regards,
> > 
> > Tvrtko  
> 

Re: [PATCH 1/2] numa: Add simple generic NUMA emulation

[Tvrtko Ursulin]

Hi Jonathan,

On 08/08/2024 17:27, Jonathan Cameron wrote:
> On Thu, 8 Aug 2024 12:56:44 +0100
> Tvrtko Ursulin <tvrtko.ursulin@igalia.com> wrote:
> 
>> [Please excuse the re-send, but as I heard nothing concern is it did not
>> get lost in your busy mailbox.]
>>
>> Hi Greg,
>>
>> Gentle reminder on the opens from this thread. Let me re-summarise the
>> question below:
>>
>> On 26/06/2024 12:47, Tvrtko Ursulin wrote:
>>>
>>> Hi Greg,
>>>
>>> On 26/06/2024 08:38, Greg Kroah-Hartman wrote:
>>>> On Tue, Jun 25, 2024 at 01:58:02PM +0100, Tvrtko Ursulin wrote:
>>>>> From: Maíra Canal <mcanal@igalia.com>
>>>>>
>>>>> Add some common code for splitting the memory into N emulated NUMA
>>>>> memory
>>>>> nodes.
>>>>>
>>>>> Individual architecture can then enable selecting this option and use
>>>>> the
>>>>> existing numa=fake=<N> kernel argument to enable it.
>>>>>
>>>>> Memory is always split into equally sized chunks.
>>>>>
>>>>> Signed-off-by: Maíra Canal <mcanal@igalia.com>
>>>>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
>>>>> Co-developed-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
>>>>> Cc: Catalin Marinas <catalin.marinas@arm.com>
>>>>> Cc: Will Deacon <will@kernel.org>
>>>>> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>>>>> Cc: “Rafael J. Wysocki" <rafael@kernel.org>
>>>>> ---
>>>>>    drivers/base/Kconfig          |  7 ++++
>>>>>    drivers/base/Makefile         |  1 +
>>>>>    drivers/base/arch_numa.c      |  6 ++++
>>>>>    drivers/base/numa_emulation.c | 67 +++++++++++++++++++++++++++++++++++
>>>>>    drivers/base/numa_emulation.h | 21 +++++++++++
>>>>
>>>> Why not just properly describe the numa topology in your bootloader or
>>>> device tree and not need any such "fake" stuff at all?
>>>>
>>>> Also, you are now asking me to maintain these new files, not something
>>>> I'm comfortable doing at all sorry.
>>>
>>> Mostly because ae3c107cd8be ("numa: Move numa implementation to common
>>> code") and existing common code in drivers/base/arch_numa.c it appeared
>>> it could be acceptable to add the simple NUMA emulation into the common
>>> code too. Then building upon the same concept as on x86 where no need
>>> for firmware changes is needed for experimenting with different
>>> configurations.
>>>
>>> Would folding into arch_numa.c so no new files are added address your
>>> concern, or your main issue is the emulation in general?
>>
>> Re-iterating and slightly re-formulating this question I see three options:
>>
>> a)
>> Fold the new simple generic code into the existing arch_numa.c,
>> addressing the "no new files" objection, if that was the main objection.
>>
>> b)
>> Move completely into arch code - aka you don't want to see it under
>> drivers/base at all, ever, regardless of how simple the new code is, or
>> that common NUMA code is already there.
>>
>> c)
>> Strong nack for either a) or b) - so "do it in the firmware" comment.
>>
>> Trying to understand your position so we can progress this.
> 
> See:
> https://lore.kernel.org/all/20240807064110.1003856-20-rppt@kernel.org/
> and rest of thread
> https://lore.kernel.org/all/20240807064110.1003856-1-rppt@kernel.org/
> [PATCH v4 00/26] mm: introduce numa_memblks
> 
> Much larger rework and unification set from Mike Rapoport
> that happens to end up adding numa emulation as part of making
> the x86 numa_memblk work for arm64 etc.
> 
> It's in mm-unstable now so getting some test coverage etc.
> 
> Sorry, I'd kind of assumed this also went to linux-mm so
> the connection would have been made.

This is great - I did not see it since I don't read linux-mm regularly!

I gave Mike's implementation a spin on top of RPi 6.11 kernel and it 
mostly works fine.

Is the decision that this is going in pretty much set, that is, high 
level acks are there?

One area to potentially improve is working around CMA areas when they 
are put by the firmware at a range which straddles two nodes. In my 
series, albeit not the one I yet posted, I have some code to fudge that 
so that CMA ends up wholly in one node and so CMA initialisation can 
succeed.

I can try and adapt that patch to this series and post as a RFC.

Then there are some odd things about NUMA, memory pressure and swap 
behaviour, but that is not specific to this series and not something I 
got to the bottom off just yet. Could be specific to my board and IO for 
instance.

Regards,

Tvrtko

Re: [PATCH 1/2] numa: Add simple generic NUMA emulation

[Mike Rapoport]

Hi Tvrtko,

On Mon, Aug 12, 2024 at 05:35:31PM +0100, Tvrtko Ursulin wrote:
> 
> Hi Jonathan,
> 
> On 08/08/2024 17:27, Jonathan Cameron wrote:
> > On Thu, 8 Aug 2024 12:56:44 +0100
> > Tvrtko Ursulin <tvrtko.ursulin@igalia.com> wrote:
> > > 
> > > c)
> > > Strong nack for either a) or b) - so "do it in the firmware" comment.
> > > 
> > > Trying to understand your position so we can progress this.
> > 
> > See:
> > https://lore.kernel.org/all/20240807064110.1003856-20-rppt@kernel.org/
> > and rest of thread
> > https://lore.kernel.org/all/20240807064110.1003856-1-rppt@kernel.org/
> > [PATCH v4 00/26] mm: introduce numa_memblks
> > 
> > Much larger rework and unification set from Mike Rapoport
> > that happens to end up adding numa emulation as part of making
> > the x86 numa_memblk work for arm64 etc.
> > 
> > It's in mm-unstable now so getting some test coverage etc.
> > 
> > Sorry, I'd kind of assumed this also went to linux-mm so
> > the connection would have been made.
> 
> This is great - I did not see it since I don't read linux-mm regularly!
> 
> I gave Mike's implementation a spin on top of RPi 6.11 kernel and it mostly
> works fine.
> 
> Is the decision that this is going in pretty much set, that is, high level
> acks are there?

It's in mm-unstable now

https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git/log/?h=mm-unstable

and I hope will go to Linus next merge window.
 
> One area to potentially improve is working around CMA areas when they are
> put by the firmware at a range which straddles two nodes. In my series,
> albeit not the one I yet posted, I have some code to fudge that so that CMA
> ends up wholly in one node and so CMA initialisation can succeed.
> 
> I can try and adapt that patch to this series and post as a RFC.
> 
> Then there are some odd things about NUMA, memory pressure and swap
> behaviour, but that is not specific to this series and not something I got
> to the bottom off just yet. Could be specific to my board and IO for
> instance.
> 
> Regards,
> 
> Tvrtko

-- 
Sincerely yours,
Mike.