[PATCH v3 0/7] virtio-mem: s390 support

[PATCH v3 0/7] virtio-mem: s390 support

[David Hildenbrand]

Let's finally add s390 support for virtio-mem; my last RFC was sent
4 years ago, and a lot changed in the meantime.

The latest QEMU series is available at [1], which contains some more
details and a usage example on s390 (last patch).

There is not too much in here: The biggest part is querying a new diag(500)
STORAGE_LIMIT hypercall to obtain the proper "max_physmem_end".

The last three patches are not strictly required but certainly nice-to-have.

Note that -- in contrast to standby memory -- virtio-mem memory must be
configured to be automatically onlined as soon as hotplugged. The easiest
approach is using the "memhp_default_state=" kernel parameter or by using
proper udev rules. More details can be found at [2].

I have reviving+upstreaming a systemd service to handle configuring
that on my todo list, but for some reason I keep getting distracted ...

I tested various things, including:
 * Various memory hotplug/hotunplug combinations
 * Device hotplug/hotunplug
 * /proc/iomem output
 * reboot
 * kexec
 * kdump: make sure we properly enter the "kdump mode" in the virtio-mem
   driver

kdump support for virtio-mem memory on s390 will be sent out separately.

v2 -> v3
* "s390/kdump: make is_kdump_kernel() consistently return "true" in kdump
   environments only"
 -> Sent out separately [3]
* "s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM memory
   devices"
 -> No query function for diag500 for now.
 -> Update comment above setup_ident_map_size().
 -> Optimize/rewrite diag500_storage_limit() [Heiko]
 -> Change handling in detect_physmem_online_ranges [Alexander]
 -> Improve documentation.
* "s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA"
 -> Added after testing on systems with CONFIG_NUMA=y

v1 -> v2:
* Document the new diag500 subfunction
* Use "s390" instead of "s390x" consistently

[1] https://lkml.kernel.org/r/20241008105455.2302628-1-david@redhat.com
[2] https://virtio-mem.gitlab.io/user-guide/user-guide-linux.html
[3] https://lkml.kernel.org/r/20241023090651.1115507-1-david@redhat.com

Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Thomas Huth <thuth@redhat.com>
Cc: Cornelia Huck <cohuck@redhat.com>
Cc: Janosch Frank <frankja@linux.ibm.com>
Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Cc: "Eugenio Pérez" <eperezma@redhat.com>
Cc: Eric Farman <farman@linux.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Jonathan Corbet <corbet@lwn.net>

David Hildenbrand (7):
  Documentation: s390-diag.rst: make diag500 a generic KVM hypercall
  Documentation: s390-diag.rst: document diag500(STORAGE LIMIT)
    subfunction
  s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM
    memory devices
  virtio-mem: s390 support
  lib/Kconfig.debug: default STRICT_DEVMEM to "y" on s390
  s390/sparsemem: reduce section size to 128 MiB
  s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA

 Documentation/virt/kvm/s390/s390-diag.rst | 35 +++++++++++++----
 arch/s390/boot/physmem_info.c             | 47 ++++++++++++++++++++++-
 arch/s390/boot/startup.c                  |  7 +++-
 arch/s390/include/asm/physmem_info.h      |  3 ++
 arch/s390/include/asm/sparsemem.h         | 10 ++++-
 drivers/virtio/Kconfig                    | 12 +++---
 lib/Kconfig.debug                         |  2 +-
 7 files changed, 98 insertions(+), 18 deletions(-)


base-commit: ae90f6a6170d7a7a1aa4fddf664fbd093e3023bc
-- 
2.46.1

[PATCH v3 1/7] Documentation: s390-diag.rst: make diag500 a generic KVM hypercall

[David Hildenbrand]

Let's make it a generic KVM hypercall, allowing other subfunctions to
be more independent of virtio.

While at it, document that unsupported/unimplemented subfunctions result
in a SPECIFICATION exception.

This is a preparation for documenting a new hypercall.

Signed-off-by: David Hildenbrand <david@redhat.com>
---
 Documentation/virt/kvm/s390/s390-diag.rst | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/Documentation/virt/kvm/s390/s390-diag.rst b/Documentation/virt/kvm/s390/s390-diag.rst
index ca85f030eb0b..48a326d41cc0 100644
--- a/Documentation/virt/kvm/s390/s390-diag.rst
+++ b/Documentation/virt/kvm/s390/s390-diag.rst
@@ -35,20 +35,24 @@ DIAGNOSE function codes not specific to KVM, please refer to the
 documentation for the s390 hypervisors defining them.
 
 
-DIAGNOSE function code 'X'500' - KVM virtio functions
------------------------------------------------------
+DIAGNOSE function code 'X'500' - KVM functions
+----------------------------------------------
 
-If the function code specifies 0x500, various virtio-related functions
-are performed.
+If the function code specifies 0x500, various KVM-specific functions
+are performed, including virtio functions.
 
-General register 1 contains the virtio subfunction code. Supported
-virtio subfunctions depend on KVM's userspace. Generally, userspace
-provides either s390-virtio (subcodes 0-2) or virtio-ccw (subcode 3).
+General register 1 contains the subfunction code. Supported subfunctions
+depend on KVM's userspace. Regarding virtio subfunctions, generally
+userspace provides either s390-virtio (subcodes 0-2) or virtio-ccw
+(subcode 3).
 
 Upon completion of the DIAGNOSE instruction, general register 2 contains
 the function's return code, which is either a return code or a subcode
 specific value.
 
+If the specified subfunction is not supported, a SPECIFICATION exception
+will be triggered.
+
 Subcode 0 - s390-virtio notification and early console printk
     Handled by userspace.
 
-- 
2.46.1

Re: [PATCH v3 1/7] Documentation: s390-diag.rst: make diag500 a generic KVM hypercall

[Heiko Carstens]

On Fri, Oct 25, 2024 at 04:14:46PM +0200, David Hildenbrand wrote:
> Let's make it a generic KVM hypercall, allowing other subfunctions to
> be more independent of virtio.
> 
> While at it, document that unsupported/unimplemented subfunctions result
> in a SPECIFICATION exception.
> 
> This is a preparation for documenting a new hypercall.
> 
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  Documentation/virt/kvm/s390/s390-diag.rst | 18 +++++++++++-------
>  1 file changed, 11 insertions(+), 7 deletions(-)

Reviewed-by: Heiko Carstens <hca@linux.ibm.com>

Re: [PATCH v3 1/7] Documentation: s390-diag.rst: make diag500 a generic KVM hypercall

[Eric Farman]

On Fri, 2024-10-25 at 16:14 +0200, David Hildenbrand wrote:
> Let's make it a generic KVM hypercall, allowing other subfunctions to
> be more independent of virtio.
> 
> While at it, document that unsupported/unimplemented subfunctions result
> in a SPECIFICATION exception.
> 
> This is a preparation for documenting a new hypercall.

s/hypercall/subfunction/ ?

> 
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  Documentation/virt/kvm/s390/s390-diag.rst | 18 +++++++++++-------
>  1 file changed, 11 insertions(+), 7 deletions(-)

Reviewed-by: Eric Farman <farman@linux.ibm.com>

Re: [PATCH v3 1/7] Documentation: s390-diag.rst: make diag500 a generic KVM hypercall

[David Hildenbrand]

On 31.10.24 15:56, Eric Farman wrote:
> On Fri, 2024-10-25 at 16:14 +0200, David Hildenbrand wrote:
>> Let's make it a generic KVM hypercall, allowing other subfunctions to
>> be more independent of virtio.
>>
>> While at it, document that unsupported/unimplemented subfunctions result
>> in a SPECIFICATION exception.
>>
>> This is a preparation for documenting a new hypercall.
> 
> s/hypercall/subfunction/ ?

Indeed, thanks! I assume this can be fixed up when applying, unless I 
have to resend for other reasons.

> 
>>
>> Signed-off-by: David Hildenbrand <david@redhat.com>
>> ---
>>   Documentation/virt/kvm/s390/s390-diag.rst | 18 +++++++++++-------
>>   1 file changed, 11 insertions(+), 7 deletions(-)
> 
> Reviewed-by: Eric Farman <farman@linux.ibm.com>
> 

Thanks!

-- 
Cheers,

David / dhildenb

[PATCH v3 2/7] Documentation: s390-diag.rst: document diag500(STORAGE LIMIT) subfunction

[David Hildenbrand]

Let's document our new diag500 subfunction that can be implemented by
userspace.

Signed-off-by: David Hildenbrand <david@redhat.com>
---
 Documentation/virt/kvm/s390/s390-diag.rst | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/Documentation/virt/kvm/s390/s390-diag.rst b/Documentation/virt/kvm/s390/s390-diag.rst
index 48a326d41cc0..3e4f9e3bef81 100644
--- a/Documentation/virt/kvm/s390/s390-diag.rst
+++ b/Documentation/virt/kvm/s390/s390-diag.rst
@@ -80,6 +80,23 @@ Subcode 3 - virtio-ccw notification
 
     See also the virtio standard for a discussion of this hypercall.
 
+Subcode 4 - storage-limit
+    Handled by userspace.
+
+    After completion of the DIAGNOSE call, general register 2 will
+    contain the storage limit: the maximum physical address that might be
+    used for storage throughout the lifetime of the VM.
+
+    The storage limit does not indicate currently usable storage, it may
+    include holes, standby storage and areas reserved for other means, such
+    as memory hotplug or virtio-mem devices. Other interfaces for detecting
+    actually usable storage, such as SCLP, must be used in conjunction with
+    this subfunction.
+
+    Note that the storage limit can be larger, but never smaller than the
+    maximum storage address indicated by SCLP via the "maximum storage
+    increment" and the "increment size".
+
 
 DIAGNOSE function code 'X'501 - KVM breakpoint
 ----------------------------------------------
-- 
2.46.1

Re: [PATCH v3 2/7] Documentation: s390-diag.rst: document diag500(STORAGE LIMIT) subfunction

[Heiko Carstens]

On Fri, Oct 25, 2024 at 04:14:47PM +0200, David Hildenbrand wrote:
> Let's document our new diag500 subfunction that can be implemented by
> userspace.
> 
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  Documentation/virt/kvm/s390/s390-diag.rst | 17 +++++++++++++++++
>  1 file changed, 17 insertions(+)

Reviewed-by: Heiko Carstens <hca@linux.ibm.com>

Re: [PATCH v3 2/7] Documentation: s390-diag.rst: document diag500(STORAGE LIMIT) subfunction

[Eric Farman]

On Fri, 2024-10-25 at 16:14 +0200, David Hildenbrand wrote:
> Let's document our new diag500 subfunction that can be implemented by
> userspace.
> 
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  Documentation/virt/kvm/s390/s390-diag.rst | 17 +++++++++++++++++
>  1 file changed, 17 insertions(+)

Reviewed-by: Eric Farman <farman@linux.ibm.com>

[PATCH v3 3/7] s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM memory devices

[David Hildenbrand]

To support memory devices under QEMU/KVM, such as virtio-mem,
we have to prepare our kernel virtual address space accordingly and
have to know the highest possible physical memory address we might see
later: the storage limit. The good old SCLP interface is not suitable for
this use case.

In particular, memory owned by memory devices has no relationship to
storage increments, it is always detected using the device driver, and
unaware OSes (no driver) must never try making use of that memory.
Consequently this memory is located outside of the "maximum storage
increment"-indicated memory range.

Let's use our new diag500 STORAGE_LIMIT subcode to query this storage
limit that can exceed the "maximum storage increment", and use the
existing interfaces (i.e., SCLP) to obtain information about the initial
memory that is not owned+managed by memory devices.

If a hypervisor does not support such memory devices, the address exposed
through diag500 STORAGE_LIMIT will correspond to the maximum storage
increment exposed through SCLP.

To teach kdump on s390 to include memory owned by memory devices, there
will be ways to query the relevant memory ranges from the device via a
driver running in special kdump mode (like virtio-mem already implements
to filter /proc/vmcore access so we don't end up reading from unplugged
device blocks).

Update setup_ident_map_size(), to clarify that there can be more than
just online and standby memory.

Tested-by: Mario Casquero <mcasquer@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
 arch/s390/boot/physmem_info.c        | 47 +++++++++++++++++++++++++++-
 arch/s390/boot/startup.c             |  7 +++--
 arch/s390/include/asm/physmem_info.h |  3 ++
 3 files changed, 54 insertions(+), 3 deletions(-)

diff --git a/arch/s390/boot/physmem_info.c b/arch/s390/boot/physmem_info.c
index 1d131a81cb8b..f3ea5dbff10b 100644
--- a/arch/s390/boot/physmem_info.c
+++ b/arch/s390/boot/physmem_info.c
@@ -109,6 +109,42 @@ static int diag260(void)
 	return 0;
 }
 
+#define DIAG500_SC_STOR_LIMIT 4
+
+static int diag500_storage_limit(unsigned long *max_physmem_end)
+{
+	unsigned long storage_limit;
+	unsigned long reg1, reg2;
+	psw_t old;
+
+	asm volatile(
+		"	mvc	0(16,%[psw_old]),0(%[psw_pgm])\n"
+		"	epsw	%[reg1],%[reg2]\n"
+		"	st	%[reg1],0(%[psw_pgm])\n"
+		"	st	%[reg2],4(%[psw_pgm])\n"
+		"	larl	%[reg1],1f\n"
+		"	stg	%[reg1],8(%[psw_pgm])\n"
+		"	lghi	1,%[subcode]\n"
+		"	lghi	2,0\n"
+		"	diag	2,4,0x500\n"
+		"1:	mvc	0(16,%[psw_pgm]),0(%[psw_old])\n"
+		"	lgr	%[slimit],2\n"
+		: [reg1] "=&d" (reg1),
+		  [reg2] "=&a" (reg2),
+		  [slimit] "=d" (storage_limit),
+		  "=Q" (get_lowcore()->program_new_psw),
+		  "=Q" (old)
+		: [psw_old] "a" (&old),
+		  [psw_pgm] "a" (&get_lowcore()->program_new_psw),
+		  [subcode] "i" (DIAG500_SC_STOR_LIMIT)
+		: "memory", "1", "2");
+	if (!storage_limit)
+		return -EINVAL;
+	/* Convert inclusive end to exclusive end */
+	*max_physmem_end = storage_limit + 1;
+	return 0;
+}
+
 static int tprot(unsigned long addr)
 {
 	unsigned long reg1, reg2;
@@ -157,7 +193,9 @@ unsigned long detect_max_physmem_end(void)
 {
 	unsigned long max_physmem_end = 0;
 
-	if (!sclp_early_get_memsize(&max_physmem_end)) {
+	if (!diag500_storage_limit(&max_physmem_end)) {
+		physmem_info.info_source = MEM_DETECT_DIAG500_STOR_LIMIT;
+	} else if (!sclp_early_get_memsize(&max_physmem_end)) {
 		physmem_info.info_source = MEM_DETECT_SCLP_READ_INFO;
 	} else {
 		max_physmem_end = search_mem_end();
@@ -170,6 +208,13 @@ void detect_physmem_online_ranges(unsigned long max_physmem_end)
 {
 	if (!sclp_early_read_storage_info()) {
 		physmem_info.info_source = MEM_DETECT_SCLP_STOR_INFO;
+	} else if (physmem_info.info_source == MEM_DETECT_DIAG500_STOR_LIMIT) {
+		unsigned long online_end;
+
+		if (!sclp_early_get_memsize(&online_end)) {
+			physmem_info.info_source = MEM_DETECT_SCLP_READ_INFO;
+			add_physmem_online_range(0, online_end);
+		}
 	} else if (!diag260()) {
 		physmem_info.info_source = MEM_DETECT_DIAG260;
 	} else if (max_physmem_end) {
diff --git a/arch/s390/boot/startup.c b/arch/s390/boot/startup.c
index c8f149ad77e5..76c33c7442df 100644
--- a/arch/s390/boot/startup.c
+++ b/arch/s390/boot/startup.c
@@ -182,12 +182,15 @@ static void kaslr_adjust_got(unsigned long offset)
  * Merge information from several sources into a single ident_map_size value.
  * "ident_map_size" represents the upper limit of physical memory we may ever
  * reach. It might not be all online memory, but also include standby (offline)
- * memory. "ident_map_size" could be lower then actual standby or even online
+ * memory or memory areas reserved for other means (e.g., memory devices such as
+ * virtio-mem).
+ *
+ * "ident_map_size" could be lower then actual standby/reserved or even online
  * memory present, due to limiting factors. We should never go above this limit.
  * It is the size of our identity mapping.
  *
  * Consider the following factors:
- * 1. max_physmem_end - end of physical memory online or standby.
+ * 1. max_physmem_end - end of physical memory online, standby or reserved.
  *    Always >= end of the last online memory range (get_physmem_online_end()).
  * 2. CONFIG_MAX_PHYSMEM_BITS - the maximum size of physical memory the
  *    kernel is able to support.
diff --git a/arch/s390/include/asm/physmem_info.h b/arch/s390/include/asm/physmem_info.h
index f45cfc8bc233..51b68a43e195 100644
--- a/arch/s390/include/asm/physmem_info.h
+++ b/arch/s390/include/asm/physmem_info.h
@@ -9,6 +9,7 @@ enum physmem_info_source {
 	MEM_DETECT_NONE = 0,
 	MEM_DETECT_SCLP_STOR_INFO,
 	MEM_DETECT_DIAG260,
+	MEM_DETECT_DIAG500_STOR_LIMIT,
 	MEM_DETECT_SCLP_READ_INFO,
 	MEM_DETECT_BIN_SEARCH
 };
@@ -107,6 +108,8 @@ static inline const char *get_physmem_info_source(void)
 		return "sclp storage info";
 	case MEM_DETECT_DIAG260:
 		return "diag260";
+	case MEM_DETECT_DIAG500_STOR_LIMIT:
+		return "diag500 storage limit";
 	case MEM_DETECT_SCLP_READ_INFO:
 		return "sclp read info";
 	case MEM_DETECT_BIN_SEARCH:
-- 
2.46.1

Re: [PATCH v3 3/7] s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM memory devices

[Heiko Carstens]

On Fri, Oct 25, 2024 at 04:14:48PM +0200, David Hildenbrand wrote:
> To support memory devices under QEMU/KVM, such as virtio-mem,
> we have to prepare our kernel virtual address space accordingly and
> have to know the highest possible physical memory address we might see
> later: the storage limit. The good old SCLP interface is not suitable for
> this use case.
> 
> In particular, memory owned by memory devices has no relationship to
> storage increments, it is always detected using the device driver, and
> unaware OSes (no driver) must never try making use of that memory.
> Consequently this memory is located outside of the "maximum storage
> increment"-indicated memory range.
> 
> Let's use our new diag500 STORAGE_LIMIT subcode to query this storage
> limit that can exceed the "maximum storage increment", and use the
> existing interfaces (i.e., SCLP) to obtain information about the initial
> memory that is not owned+managed by memory devices.
> 
> If a hypervisor does not support such memory devices, the address exposed
> through diag500 STORAGE_LIMIT will correspond to the maximum storage
> increment exposed through SCLP.
> 
> To teach kdump on s390 to include memory owned by memory devices, there
> will be ways to query the relevant memory ranges from the device via a
> driver running in special kdump mode (like virtio-mem already implements
> to filter /proc/vmcore access so we don't end up reading from unplugged
> device blocks).
> 
> Update setup_ident_map_size(), to clarify that there can be more than
> just online and standby memory.
> 
> Tested-by: Mario Casquero <mcasquer@redhat.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  arch/s390/boot/physmem_info.c        | 47 +++++++++++++++++++++++++++-
>  arch/s390/boot/startup.c             |  7 +++--
>  arch/s390/include/asm/physmem_info.h |  3 ++
>  3 files changed, 54 insertions(+), 3 deletions(-)

Looks like I couldn't convince you to implement a query subcode.
But anyway, let's move on.

Acked-by: Heiko Carstens <hca@linux.ibm.com>

However, I would like to see an Ack or review from Alexander Gordeev
or Vasily Gorbik for this patch.

Re: [PATCH v3 3/7] s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM memory devices

[David Hildenbrand]

On 30.10.24 10:23, Heiko Carstens wrote:
> On Fri, Oct 25, 2024 at 04:14:48PM +0200, David Hildenbrand wrote:
>> To support memory devices under QEMU/KVM, such as virtio-mem,
>> we have to prepare our kernel virtual address space accordingly and
>> have to know the highest possible physical memory address we might see
>> later: the storage limit. The good old SCLP interface is not suitable for
>> this use case.
>>
>> In particular, memory owned by memory devices has no relationship to
>> storage increments, it is always detected using the device driver, and
>> unaware OSes (no driver) must never try making use of that memory.
>> Consequently this memory is located outside of the "maximum storage
>> increment"-indicated memory range.
>>
>> Let's use our new diag500 STORAGE_LIMIT subcode to query this storage
>> limit that can exceed the "maximum storage increment", and use the
>> existing interfaces (i.e., SCLP) to obtain information about the initial
>> memory that is not owned+managed by memory devices.
>>
>> If a hypervisor does not support such memory devices, the address exposed
>> through diag500 STORAGE_LIMIT will correspond to the maximum storage
>> increment exposed through SCLP.
>>
>> To teach kdump on s390 to include memory owned by memory devices, there
>> will be ways to query the relevant memory ranges from the device via a
>> driver running in special kdump mode (like virtio-mem already implements
>> to filter /proc/vmcore access so we don't end up reading from unplugged
>> device blocks).
>>
>> Update setup_ident_map_size(), to clarify that there can be more than
>> just online and standby memory.
>>
>> Tested-by: Mario Casquero <mcasquer@redhat.com>
>> Signed-off-by: David Hildenbrand <david@redhat.com>
>> ---
>>   arch/s390/boot/physmem_info.c        | 47 +++++++++++++++++++++++++++-
>>   arch/s390/boot/startup.c             |  7 +++--
>>   arch/s390/include/asm/physmem_info.h |  3 ++
>>   3 files changed, 54 insertions(+), 3 deletions(-)
> 
> Looks like I couldn't convince you to implement a query subcode.

Well, you convinced me that it might be useful, but after waiting on 
feedback from the KVM folks ... which didn't happen I moved on. In the 
cover letter I have "No query function for diag500 for now."

My thinking was that if we go for a query subcode, maybe we'd start 
"anew" with a new diag and use "0=query" like all similar instructions I 
am aware of. And that is then a bigger rework ...

... and I am not particularly interested in extra work without a clear 
statement from KVM people what (a) if that work is required and; (b) 
what it should look like.

Thanks for the review Heiko!

-- 
Cheers,

David / dhildenb

Re: [PATCH v3 3/7] s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM memory devices

[Heiko Carstens]

On Wed, Oct 30, 2024 at 10:42:05AM +0100, David Hildenbrand wrote:
> On 30.10.24 10:23, Heiko Carstens wrote:
> > Looks like I couldn't convince you to implement a query subcode.
> 
> Well, you convinced me that it might be useful, but after waiting on
> feedback from the KVM folks ... which didn't happen I moved on. In the cover
> letter I have "No query function for diag500 for now."
> 
> My thinking was that if we go for a query subcode, maybe we'd start "anew"
> with a new diag and use "0=query" like all similar instructions I am aware
> of. And that is then a bigger rework ...
> 
> ... and I am not particularly interested in extra work without a clear
> statement from KVM people what (a) if that work is required and; (b) what it
> should look like.

Yes, it is all good. Let's just move on.

Re: [PATCH v3 3/7] s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM memory devices

[Alexander Gordeev]

On Fri, Oct 25, 2024 at 04:14:48PM +0200, David Hildenbrand wrote:
> To support memory devices under QEMU/KVM, such as virtio-mem,
> we have to prepare our kernel virtual address space accordingly and
> have to know the highest possible physical memory address we might see
> later: the storage limit. The good old SCLP interface is not suitable for
> this use case.
> 
> In particular, memory owned by memory devices has no relationship to
> storage increments, it is always detected using the device driver, and
> unaware OSes (no driver) must never try making use of that memory.
> Consequently this memory is located outside of the "maximum storage
> increment"-indicated memory range.
> 
> Let's use our new diag500 STORAGE_LIMIT subcode to query this storage
> limit that can exceed the "maximum storage increment", and use the
> existing interfaces (i.e., SCLP) to obtain information about the initial
> memory that is not owned+managed by memory devices.
> 
> If a hypervisor does not support such memory devices, the address exposed
> through diag500 STORAGE_LIMIT will correspond to the maximum storage
> increment exposed through SCLP.
> 
> To teach kdump on s390 to include memory owned by memory devices, there
> will be ways to query the relevant memory ranges from the device via a
> driver running in special kdump mode (like virtio-mem already implements
> to filter /proc/vmcore access so we don't end up reading from unplugged
> device blocks).
> 
> Update setup_ident_map_size(), to clarify that there can be more than
> just online and standby memory.
> 
> Tested-by: Mario Casquero <mcasquer@redhat.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  arch/s390/boot/physmem_info.c        | 47 +++++++++++++++++++++++++++-
>  arch/s390/boot/startup.c             |  7 +++--
>  arch/s390/include/asm/physmem_info.h |  3 ++
>  3 files changed, 54 insertions(+), 3 deletions(-)

Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>

Re: [PATCH v3 3/7] s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM memory devices

[David Hildenbrand]

On 30.10.24 15:32, Alexander Gordeev wrote:
> On Fri, Oct 25, 2024 at 04:14:48PM +0200, David Hildenbrand wrote:
>> To support memory devices under QEMU/KVM, such as virtio-mem,
>> we have to prepare our kernel virtual address space accordingly and
>> have to know the highest possible physical memory address we might see
>> later: the storage limit. The good old SCLP interface is not suitable for
>> this use case.
>>
>> In particular, memory owned by memory devices has no relationship to
>> storage increments, it is always detected using the device driver, and
>> unaware OSes (no driver) must never try making use of that memory.
>> Consequently this memory is located outside of the "maximum storage
>> increment"-indicated memory range.
>>
>> Let's use our new diag500 STORAGE_LIMIT subcode to query this storage
>> limit that can exceed the "maximum storage increment", and use the
>> existing interfaces (i.e., SCLP) to obtain information about the initial
>> memory that is not owned+managed by memory devices.
>>
>> If a hypervisor does not support such memory devices, the address exposed
>> through diag500 STORAGE_LIMIT will correspond to the maximum storage
>> increment exposed through SCLP.
>>
>> To teach kdump on s390 to include memory owned by memory devices, there
>> will be ways to query the relevant memory ranges from the device via a
>> driver running in special kdump mode (like virtio-mem already implements
>> to filter /proc/vmcore access so we don't end up reading from unplugged
>> device blocks).
>>
>> Update setup_ident_map_size(), to clarify that there can be more than
>> just online and standby memory.
>>
>> Tested-by: Mario Casquero <mcasquer@redhat.com>
>> Signed-off-by: David Hildenbrand <david@redhat.com>
>> ---
>>   arch/s390/boot/physmem_info.c        | 47 +++++++++++++++++++++++++++-
>>   arch/s390/boot/startup.c             |  7 +++--
>>   arch/s390/include/asm/physmem_info.h |  3 ++
>>   3 files changed, 54 insertions(+), 3 deletions(-)
> 
> Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
> 

Thanks Alexander!

-- 
Cheers,

David / dhildenb

[PATCH v3 4/7] virtio-mem: s390 support

[David Hildenbrand]

Now that s390 code is prepared for memory devices that reside above the
maximum storage increment exposed through SCLP, everything is in place
to unlock virtio-mem support.

As virtio-mem in Linux currently supports logically onlining/offlining
memory in pageblock granularity, we have an effective hot(un)plug
granularity of 1 MiB on s390.

As virito-mem adds/removes individual Linux memory blocks (256MB), we
will currently never use gigantic pages in the identity mapping.

It is worth noting that neither storage keys nor storage attributes (e.g.,
data / nodat) are touched when onlining memory blocks, which is good
because we are not supposed to touch these parts for unplugged device
blocks that are logically offline in Linux.

We will currently never initialize storage keys for virtio-mem
memory -- IOW, storage_key_init_range() is never called. It could be added
in the future when plugging device blocks. But as that function
essentially does nothing without modifying the code (changing
PAGE_DEFAULT_ACC), that's just fine for now.

kexec should work as intended and just like on other architectures that
support virtio-mem: we will never place kexec binaries on virtio-mem
memory, and never indicate virtio-mem memory to the 2nd kernel. The
device driver in the 2nd kernel can simply reset the device --
turning all memory unplugged, to then start plugging memory and adding
them to Linux, without causing trouble because the memory is already
used elsewhere.

The special s390 kdump mode, whereby the 2nd kernel creates the ELF
core header, won't currently dump virtio-mem memory. The virtio-mem
driver has a special kdump mode, from where we can detect memory ranges
to dump. Based on this, support for dumping virtio-mem memory can be
added in the future fairly easily.

Acked-by: Michael S. Tsirkin <mst@redhat.com>
Tested-by: Mario Casquero <mcasquer@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
 drivers/virtio/Kconfig | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
index 42a48ac763ee..2eb747311bfd 100644
--- a/drivers/virtio/Kconfig
+++ b/drivers/virtio/Kconfig
@@ -122,7 +122,7 @@ config VIRTIO_BALLOON
 
 config VIRTIO_MEM
 	tristate "Virtio mem driver"
-	depends on X86_64 || ARM64 || RISCV
+	depends on X86_64 || ARM64 || RISCV || S390
 	depends on VIRTIO
 	depends on MEMORY_HOTPLUG
 	depends on MEMORY_HOTREMOVE
@@ -132,11 +132,11 @@ config VIRTIO_MEM
 	 This driver provides access to virtio-mem paravirtualized memory
 	 devices, allowing to hotplug and hotunplug memory.
 
-	 This driver currently only supports x86-64 and arm64. Although it
-	 should compile on other architectures that implement memory
-	 hot(un)plug, architecture-specific and/or common
-	 code changes may be required for virtio-mem, kdump and kexec to work as
-	 expected.
+	 This driver currently supports x86-64, arm64, riscv and s390.
+	 Although it should compile on other architectures that implement
+	 memory hot(un)plug, architecture-specific and/or common
+	 code changes may be required for virtio-mem, kdump and kexec to
+	 work as expected.
 
 	 If unsure, say M.
 
-- 
2.46.1

Re: [PATCH v3 4/7] virtio-mem: s390 support

[Heiko Carstens]

On Fri, Oct 25, 2024 at 04:14:49PM +0200, David Hildenbrand wrote:
> Now that s390 code is prepared for memory devices that reside above the
> maximum storage increment exposed through SCLP, everything is in place
> to unlock virtio-mem support.
...
> Acked-by: Michael S. Tsirkin <mst@redhat.com>
> Tested-by: Mario Casquero <mcasquer@redhat.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  drivers/virtio/Kconfig | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)

Acked-by: Heiko Carstens <hca@linux.ibm.com>

[PATCH v3 5/7] lib/Kconfig.debug: default STRICT_DEVMEM to "y" on s390

[David Hildenbrand]

virtio-mem currently depends on !DEVMEM | STRICT_DEVMEM. Let's default
STRICT_DEVMEM to "y" just like we do for arm64 and x86.

There could be ways in the future to filter access to virtio-mem device
memory even without STRICT_DEVMEM, but for now let's just keep it
simple.

Tested-by: Mario Casquero <mcasquer@redhat.com>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
 lib/Kconfig.debug | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 7312ae7c3cc5..fcad505e7c8b 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1905,7 +1905,7 @@ config STRICT_DEVMEM
 	bool "Filter access to /dev/mem"
 	depends on MMU && DEVMEM
 	depends on ARCH_HAS_DEVMEM_IS_ALLOWED || GENERIC_LIB_DEVMEM_IS_ALLOWED
-	default y if PPC || X86 || ARM64
+	default y if PPC || X86 || ARM64 || S390
 	help
 	  If this option is disabled, you allow userspace (root) access to all
 	  of memory, including kernel and userspace memory. Accidental
-- 
2.46.1

[PATCH v3 6/7] s390/sparsemem: reduce section size to 128 MiB

[David Hildenbrand]

Ever since commit 421c175c4d609 ("[S390] Add support for memory hot-add.")
we've been using a section size of 256 MiB on s390 and 32 MiB on s390.
Before that, we were using a section size of 32 MiB on both
architectures.

Now that we have a new mechanism to expose additional memory to a VM --
virtio-mem -- reduce the section size to 128 MiB to allow for more
flexibility and reduce the metadata overhead when dealing with hot(un)plug
granularity smaller than 256 MiB.

128 MiB has been used by x86-64 since the very beginning. arm64 with 4k
base pages switched to 128 MiB as well: it's just big enough on these
architectures to allows for using a huge page (2 MiB) in the vmemmap in
sane setups with sizeof(struct page) == 64 bytes and a huge page mapping
in the direct mapping, while still allowing for small hot(un)plug
granularity.

For s390, we could even switch to a 64 MiB section size, as our huge page
size is 1 MiB: but the smaller the section size, the more sections we'll
have to manage especially on bigger machines. Making it consistent with
x86-64 and arm64 feels like the right thing for now.

Note that the smallest memory hot(un)plug granularity is also limited by
the memory block size, determined by extracting the memory increment
size from SCLP. Under QEMU/KVM, implementing virtio-mem, we expose 0;
therefore, we'll end up with a memory block size of 128 MiB with a
128 MiB section size.

Tested-by: Mario Casquero <mcasquer@redhat.com>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
 arch/s390/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/s390/include/asm/sparsemem.h b/arch/s390/include/asm/sparsemem.h
index c549893602ea..ff628c50afac 100644
--- a/arch/s390/include/asm/sparsemem.h
+++ b/arch/s390/include/asm/sparsemem.h
@@ -2,7 +2,7 @@
 #ifndef _ASM_S390_SPARSEMEM_H
 #define _ASM_S390_SPARSEMEM_H
 
-#define SECTION_SIZE_BITS	28
+#define SECTION_SIZE_BITS	27
 #define MAX_PHYSMEM_BITS	CONFIG_MAX_PHYSMEM_BITS
 
 #endif /* _ASM_S390_SPARSEMEM_H */
-- 
2.46.1

[PATCH v3 7/7] s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA

[David Hildenbrand]

virtio-mem uses memory_add_physaddr_to_nid() to determine the NID to use
for memory it adds.

We currently fallback to the dummy implementation in mm/numa.c with
CONFIG_NUMA, which will end up triggering an undesired pr_info_once():

	Unknown online node for memory at 0x100000000, assuming node 0

On s390, we map all cpus and memory to node 0, so let's add a simple
memory_add_physaddr_to_nid() implementation that does exactly that,
but without complaining.

Signed-off-by: David Hildenbrand <david@redhat.com>
---
 arch/s390/include/asm/sparsemem.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/s390/include/asm/sparsemem.h b/arch/s390/include/asm/sparsemem.h
index ff628c50afac..6377b7ea8a40 100644
--- a/arch/s390/include/asm/sparsemem.h
+++ b/arch/s390/include/asm/sparsemem.h
@@ -5,4 +5,12 @@
 #define SECTION_SIZE_BITS	27
 #define MAX_PHYSMEM_BITS	CONFIG_MAX_PHYSMEM_BITS
 
+#ifdef CONFIG_NUMA
+static inline int memory_add_physaddr_to_nid(u64 addr)
+{
+	return 0;
+}
+#define memory_add_physaddr_to_nid memory_add_physaddr_to_nid
+#endif /* CONFIG_NUMA */
+
 #endif /* _ASM_S390_SPARSEMEM_H */
-- 
2.46.1

Re: [PATCH v3 7/7] s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA

[Heiko Carstens]

On Fri, Oct 25, 2024 at 04:14:52PM +0200, David Hildenbrand wrote:
> virtio-mem uses memory_add_physaddr_to_nid() to determine the NID to use
> for memory it adds.
> 
> We currently fallback to the dummy implementation in mm/numa.c with
> CONFIG_NUMA, which will end up triggering an undesired pr_info_once():
> 
> 	Unknown online node for memory at 0x100000000, assuming node 0
> 
> On s390, we map all cpus and memory to node 0, so let's add a simple
> memory_add_physaddr_to_nid() implementation that does exactly that,
> but without complaining.
> 
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  arch/s390/include/asm/sparsemem.h | 8 ++++++++
>  1 file changed, 8 insertions(+)
...

> diff --git a/arch/s390/include/asm/sparsemem.h b/arch/s390/include/asm/sparsemem.h
> index ff628c50afac..6377b7ea8a40 100644
> --- a/arch/s390/include/asm/sparsemem.h
> +++ b/arch/s390/include/asm/sparsemem.h
> @@ -5,4 +5,12 @@
>  #define SECTION_SIZE_BITS	27
>  #define MAX_PHYSMEM_BITS	CONFIG_MAX_PHYSMEM_BITS
>  
> +#ifdef CONFIG_NUMA
> +static inline int memory_add_physaddr_to_nid(u64 addr)
> +{
> +	return 0;
> +}
> +#define memory_add_physaddr_to_nid memory_add_physaddr_to_nid
> +#endif /* CONFIG_NUMA */
> +

We would need to have the same for phys_to_target_node(), even though
it looks like this won't be used on s390 currently.

Anyway, I'll add that, if I don't forget about it :)

For this patch:
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>

Re: [PATCH v3 0/7] virtio-mem: s390 support

[Heiko Carstens]

On Fri, Oct 25, 2024 at 04:14:45PM +0200, David Hildenbrand wrote:
> Let's finally add s390 support for virtio-mem; my last RFC was sent
> 4 years ago, and a lot changed in the meantime.
> 
> The latest QEMU series is available at [1], which contains some more
> details and a usage example on s390 (last patch).
> 
> There is not too much in here: The biggest part is querying a new diag(500)
> STORAGE_LIMIT hypercall to obtain the proper "max_physmem_end".

...

> David Hildenbrand (7):
>   Documentation: s390-diag.rst: make diag500 a generic KVM hypercall
>   Documentation: s390-diag.rst: document diag500(STORAGE LIMIT)
>     subfunction
>   s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM
>     memory devices
>   virtio-mem: s390 support
>   lib/Kconfig.debug: default STRICT_DEVMEM to "y" on s390
>   s390/sparsemem: reduce section size to 128 MiB
>   s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA
> 
>  Documentation/virt/kvm/s390/s390-diag.rst | 35 +++++++++++++----
>  arch/s390/boot/physmem_info.c             | 47 ++++++++++++++++++++++-
>  arch/s390/boot/startup.c                  |  7 +++-
>  arch/s390/include/asm/physmem_info.h      |  3 ++
>  arch/s390/include/asm/sparsemem.h         | 10 ++++-
>  drivers/virtio/Kconfig                    | 12 +++---
>  lib/Kconfig.debug                         |  2 +-
>  7 files changed, 98 insertions(+), 18 deletions(-)

I'll apply the whole series as soon as there are ACKs for the third
patch, and from the KVM guys for the whole series.
Christian, Janosch, Claudio?

Re: [PATCH v3 0/7] virtio-mem: s390 support

[Christian Borntraeger]

Am 30.10.24 um 10:34 schrieb Heiko Carstens:
> On Fri, Oct 25, 2024 at 04:14:45PM +0200, David Hildenbrand wrote:
>> Let's finally add s390 support for virtio-mem; my last RFC was sent
>> 4 years ago, and a lot changed in the meantime.
>>
>> The latest QEMU series is available at [1], which contains some more
>> details and a usage example on s390 (last patch).
>>
>> There is not too much in here: The biggest part is querying a new diag(500)
>> STORAGE_LIMIT hypercall to obtain the proper "max_physmem_end".
> 
> ...
> 
>> David Hildenbrand (7):
>>    Documentation: s390-diag.rst: make diag500 a generic KVM hypercall
>>    Documentation: s390-diag.rst: document diag500(STORAGE LIMIT)
>>      subfunction
>>    s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM
>>      memory devices
>>    virtio-mem: s390 support
>>    lib/Kconfig.debug: default STRICT_DEVMEM to "y" on s390
>>    s390/sparsemem: reduce section size to 128 MiB
>>    s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA
>>
>>   Documentation/virt/kvm/s390/s390-diag.rst | 35 +++++++++++++----
>>   arch/s390/boot/physmem_info.c             | 47 ++++++++++++++++++++++-
>>   arch/s390/boot/startup.c                  |  7 +++-
>>   arch/s390/include/asm/physmem_info.h      |  3 ++
>>   arch/s390/include/asm/sparsemem.h         | 10 ++++-
>>   drivers/virtio/Kconfig                    | 12 +++---
>>   lib/Kconfig.debug                         |  2 +-
>>   7 files changed, 98 insertions(+), 18 deletions(-)
> 
> I'll apply the whole series as soon as there are ACKs for the third
> patch, and from the KVM guys for the whole series.
> Christian, Janosch, Claudio?

Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com>
for the series.

@Eric Farman,
Was someone from your team planning to look into this (testing, review whatever)?

Re: [PATCH v3 0/7] virtio-mem: s390 support

[Eric Farman]

On Wed, 2024-10-30 at 17:49 +0100, Christian Borntraeger wrote:
> 
> Am 30.10.24 um 10:34 schrieb Heiko Carstens:
> > On Fri, Oct 25, 2024 at 04:14:45PM +0200, David Hildenbrand wrote:
> > > Let's finally add s390 support for virtio-mem; my last RFC was sent
> > > 4 years ago, and a lot changed in the meantime.
> > > 
> > > The latest QEMU series is available at [1], which contains some more
> > > details and a usage example on s390 (last patch).
> > > 
> > > There is not too much in here: The biggest part is querying a new diag(500)
> > > STORAGE_LIMIT hypercall to obtain the proper "max_physmem_end".
> > 
> > ...
> > 
> > > David Hildenbrand (7):
> > >    Documentation: s390-diag.rst: make diag500 a generic KVM hypercall
> > >    Documentation: s390-diag.rst: document diag500(STORAGE LIMIT)
> > >      subfunction
> > >    s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM
> > >      memory devices
> > >    virtio-mem: s390 support
> > >    lib/Kconfig.debug: default STRICT_DEVMEM to "y" on s390
> > >    s390/sparsemem: reduce section size to 128 MiB
> > >    s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA
> > > 
> > >   Documentation/virt/kvm/s390/s390-diag.rst | 35 +++++++++++++----
> > >   arch/s390/boot/physmem_info.c             | 47 ++++++++++++++++++++++-
> > >   arch/s390/boot/startup.c                  |  7 +++-
> > >   arch/s390/include/asm/physmem_info.h      |  3 ++
> > >   arch/s390/include/asm/sparsemem.h         | 10 ++++-
> > >   drivers/virtio/Kconfig                    | 12 +++---
> > >   lib/Kconfig.debug                         |  2 +-
> > >   7 files changed, 98 insertions(+), 18 deletions(-)
> > 
> > I'll apply the whole series as soon as there are ACKs for the third
> > patch, and from the KVM guys for the whole series.
> > Christian, Janosch, Claudio?
> 
> Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com>
> for the series.
> 
> @Eric Farman,
> Was someone from your team planning to look into this (testing, review whatever)?

The guy that was testing it just got back from a long holiday. I'll ping him and see if he can chime
in here.

Re: [PATCH v3 0/7] virtio-mem: s390 support

[Sumanth Korikkar]

On Fri, Oct 25, 2024 at 04:14:45PM +0200, David Hildenbrand wrote:
> Let's finally add s390 support for virtio-mem; my last RFC was sent
> 4 years ago, and a lot changed in the meantime.
> 
> The latest QEMU series is available at [1], which contains some more
> details and a usage example on s390 (last patch).
> 
> There is not too much in here: The biggest part is querying a new diag(500)
> STORAGE_LIMIT hypercall to obtain the proper "max_physmem_end".
> 
> The last three patches are not strictly required but certainly nice-to-have.
> 
> Note that -- in contrast to standby memory -- virtio-mem memory must be
> configured to be automatically onlined as soon as hotplugged. The easiest
> approach is using the "memhp_default_state=" kernel parameter or by using
> proper udev rules. More details can be found at [2].
> 
> I have reviving+upstreaming a systemd service to handle configuring
> that on my todo list, but for some reason I keep getting distracted ...
> 
> I tested various things, including:
>  * Various memory hotplug/hotunplug combinations
>  * Device hotplug/hotunplug
>  * /proc/iomem output
>  * reboot
>  * kexec
>  * kdump: make sure we properly enter the "kdump mode" in the virtio-mem
>    driver
> 
> kdump support for virtio-mem memory on s390 will be sent out separately.
> 
> v2 -> v3
> * "s390/kdump: make is_kdump_kernel() consistently return "true" in kdump
>    environments only"
>  -> Sent out separately [3]
> * "s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM memory
>    devices"
>  -> No query function for diag500 for now.
>  -> Update comment above setup_ident_map_size().
>  -> Optimize/rewrite diag500_storage_limit() [Heiko]
>  -> Change handling in detect_physmem_online_ranges [Alexander]
>  -> Improve documentation.
> * "s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA"
>  -> Added after testing on systems with CONFIG_NUMA=y
> 
> v1 -> v2:
> * Document the new diag500 subfunction
> * Use "s390" instead of "s390x" consistently
> 
> [1] https://lkml.kernel.org/r/20241008105455.2302628-1-david@redhat.com
> [2] https://virtio-mem.gitlab.io/user-guide/user-guide-linux.html
> [3] https://lkml.kernel.org/r/20241023090651.1115507-1-david@redhat.com
> 
> Cc: Heiko Carstens <hca@linux.ibm.com>
> Cc: Vasily Gorbik <gor@linux.ibm.com>
> Cc: Alexander Gordeev <agordeev@linux.ibm.com>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> Cc: Sven Schnelle <svens@linux.ibm.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Cornelia Huck <cohuck@redhat.com>
> Cc: Janosch Frank <frankja@linux.ibm.com>
> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Cc: "Eugenio P?rez" <eperezma@redhat.com>
> Cc: Eric Farman <farman@linux.ibm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Jonathan Corbet <corbet@lwn.net>
> 
> David Hildenbrand (7):
>   Documentation: s390-diag.rst: make diag500 a generic KVM hypercall
>   Documentation: s390-diag.rst: document diag500(STORAGE LIMIT)
>     subfunction
>   s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM
>     memory devices
>   virtio-mem: s390 support
>   lib/Kconfig.debug: default STRICT_DEVMEM to "y" on s390
>   s390/sparsemem: reduce section size to 128 MiB
>   s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA
> 
>  Documentation/virt/kvm/s390/s390-diag.rst | 35 +++++++++++++----
>  arch/s390/boot/physmem_info.c             | 47 ++++++++++++++++++++++-
>  arch/s390/boot/startup.c                  |  7 +++-
>  arch/s390/include/asm/physmem_info.h      |  3 ++
>  arch/s390/include/asm/sparsemem.h         | 10 ++++-
>  drivers/virtio/Kconfig                    | 12 +++---
>  lib/Kconfig.debug                         |  2 +-
>  7 files changed, 98 insertions(+), 18 deletions(-)
> 
> 
> base-commit: ae90f6a6170d7a7a1aa4fddf664fbd093e3023bc
> -- 
> 2.46.1
> 

Tested successfully various memory hotplug operations on lpar.

Tested-by: Sumanth Korikkar <sumanthk@linux.ibm.com>

Re: [PATCH v3 0/7] virtio-mem: s390 support

[Sumanth Korikkar]

On Thu, Oct 31, 2024 at 05:27:06PM +0100, Sumanth Korikkar wrote:
> On Fri, Oct 25, 2024 at 04:14:45PM +0200, David Hildenbrand wrote:
> > Let's finally add s390 support for virtio-mem; my last RFC was sent
> > 4 years ago, and a lot changed in the meantime.
> > 
> > The latest QEMU series is available at [1], which contains some more
> > details and a usage example on s390 (last patch).
> > 
> > There is not too much in here: The biggest part is querying a new diag(500)
> > STORAGE_LIMIT hypercall to obtain the proper "max_physmem_end".
> > 
> > The last three patches are not strictly required but certainly nice-to-have.
> > 
> > Note that -- in contrast to standby memory -- virtio-mem memory must be
> > configured to be automatically onlined as soon as hotplugged. The easiest
> > approach is using the "memhp_default_state=" kernel parameter or by using
> > proper udev rules. More details can be found at [2].
> > 
> > I have reviving+upstreaming a systemd service to handle configuring
> > that on my todo list, but for some reason I keep getting distracted ...
> > 
> > I tested various things, including:
> >  * Various memory hotplug/hotunplug combinations
> >  * Device hotplug/hotunplug
> >  * /proc/iomem output
> >  * reboot
> >  * kexec
> >  * kdump: make sure we properly enter the "kdump mode" in the virtio-mem
> >    driver
> > 
> > kdump support for virtio-mem memory on s390 will be sent out separately.
> > 
> > v2 -> v3
> > * "s390/kdump: make is_kdump_kernel() consistently return "true" in kdump
> >    environments only"
> >  -> Sent out separately [3]
> > * "s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM memory
> >    devices"
> >  -> No query function for diag500 for now.
> >  -> Update comment above setup_ident_map_size().
> >  -> Optimize/rewrite diag500_storage_limit() [Heiko]
> >  -> Change handling in detect_physmem_online_ranges [Alexander]
> >  -> Improve documentation.
> > * "s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA"
> >  -> Added after testing on systems with CONFIG_NUMA=y
> > 
> > v1 -> v2:
> > * Document the new diag500 subfunction
> > * Use "s390" instead of "s390x" consistently
> > 
> > [1] https://lkml.kernel.org/r/20241008105455.2302628-1-david@redhat.com
> > [2] https://virtio-mem.gitlab.io/user-guide/user-guide-linux.html
> > [3] https://lkml.kernel.org/r/20241023090651.1115507-1-david@redhat.com
> > 
> > Cc: Heiko Carstens <hca@linux.ibm.com>
> > Cc: Vasily Gorbik <gor@linux.ibm.com>
> > Cc: Alexander Gordeev <agordeev@linux.ibm.com>
> > Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> > Cc: Sven Schnelle <svens@linux.ibm.com>
> > Cc: Thomas Huth <thuth@redhat.com>
> > Cc: Cornelia Huck <cohuck@redhat.com>
> > Cc: Janosch Frank <frankja@linux.ibm.com>
> > Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
> > Cc: "Michael S. Tsirkin" <mst@redhat.com>
> > Cc: Jason Wang <jasowang@redhat.com>
> > Cc: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Cc: "Eugenio P?rez" <eperezma@redhat.com>
> > Cc: Eric Farman <farman@linux.ibm.com>
> > Cc: Andrew Morton <akpm@linux-foundation.org>
> > Cc: Jonathan Corbet <corbet@lwn.net>
> > 
> > David Hildenbrand (7):
> >   Documentation: s390-diag.rst: make diag500 a generic KVM hypercall
> >   Documentation: s390-diag.rst: document diag500(STORAGE LIMIT)
> >     subfunction
> >   s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM
> >     memory devices
> >   virtio-mem: s390 support
> >   lib/Kconfig.debug: default STRICT_DEVMEM to "y" on s390
> >   s390/sparsemem: reduce section size to 128 MiB
> >   s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA
> > 
> >  Documentation/virt/kvm/s390/s390-diag.rst | 35 +++++++++++++----
> >  arch/s390/boot/physmem_info.c             | 47 ++++++++++++++++++++++-
> >  arch/s390/boot/startup.c                  |  7 +++-
> >  arch/s390/include/asm/physmem_info.h      |  3 ++
> >  arch/s390/include/asm/sparsemem.h         | 10 ++++-
> >  drivers/virtio/Kconfig                    | 12 +++---
> >  lib/Kconfig.debug                         |  2 +-
> >  7 files changed, 98 insertions(+), 18 deletions(-)
> > 
> > 
> > base-commit: ae90f6a6170d7a7a1aa4fddf664fbd093e3023bc
> > -- 
> > 2.46.1
> > 
> 
> Tested successfully various memory hotplug operations on lpar.
> 
Just to be more precise, tested memory hotplug/hotunplug combinations +
Device hotplug/hotunplug operations on guest.

> Tested-by: Sumanth Korikkar <sumanthk@linux.ibm.com>

Re: [PATCH v3 0/7] virtio-mem: s390 support

[David Hildenbrand]

On 31.10.24 17:47, Sumanth Korikkar wrote:
> On Thu, Oct 31, 2024 at 05:27:06PM +0100, Sumanth Korikkar wrote:
>> On Fri, Oct 25, 2024 at 04:14:45PM +0200, David Hildenbrand wrote:
>>> Let's finally add s390 support for virtio-mem; my last RFC was sent
>>> 4 years ago, and a lot changed in the meantime.
>>>
>>> The latest QEMU series is available at [1], which contains some more
>>> details and a usage example on s390 (last patch).
>>>
>>> There is not too much in here: The biggest part is querying a new diag(500)
>>> STORAGE_LIMIT hypercall to obtain the proper "max_physmem_end".
>>>
>>> The last three patches are not strictly required but certainly nice-to-have.
>>>
>>> Note that -- in contrast to standby memory -- virtio-mem memory must be
>>> configured to be automatically onlined as soon as hotplugged. The easiest
>>> approach is using the "memhp_default_state=" kernel parameter or by using
>>> proper udev rules. More details can be found at [2].
>>>
>>> I have reviving+upstreaming a systemd service to handle configuring
>>> that on my todo list, but for some reason I keep getting distracted ...
>>>
>>> I tested various things, including:
>>>   * Various memory hotplug/hotunplug combinations
>>>   * Device hotplug/hotunplug
>>>   * /proc/iomem output
>>>   * reboot
>>>   * kexec
>>>   * kdump: make sure we properly enter the "kdump mode" in the virtio-mem
>>>     driver
>>>
>>> kdump support for virtio-mem memory on s390 will be sent out separately.
>>>
>>> v2 -> v3
>>> * "s390/kdump: make is_kdump_kernel() consistently return "true" in kdump
>>>     environments only"
>>>   -> Sent out separately [3]
>>> * "s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM memory
>>>     devices"
>>>   -> No query function for diag500 for now.
>>>   -> Update comment above setup_ident_map_size().
>>>   -> Optimize/rewrite diag500_storage_limit() [Heiko]
>>>   -> Change handling in detect_physmem_online_ranges [Alexander]
>>>   -> Improve documentation.
>>> * "s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA"
>>>   -> Added after testing on systems with CONFIG_NUMA=y
>>>
>>> v1 -> v2:
>>> * Document the new diag500 subfunction
>>> * Use "s390" instead of "s390x" consistently
>>>
>>> [1] https://lkml.kernel.org/r/20241008105455.2302628-1-david@redhat.com
>>> [2] https://virtio-mem.gitlab.io/user-guide/user-guide-linux.html
>>> [3] https://lkml.kernel.org/r/20241023090651.1115507-1-david@redhat.com
>>>
>>> Cc: Heiko Carstens <hca@linux.ibm.com>
>>> Cc: Vasily Gorbik <gor@linux.ibm.com>
>>> Cc: Alexander Gordeev <agordeev@linux.ibm.com>
>>> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
>>> Cc: Sven Schnelle <svens@linux.ibm.com>
>>> Cc: Thomas Huth <thuth@redhat.com>
>>> Cc: Cornelia Huck <cohuck@redhat.com>
>>> Cc: Janosch Frank <frankja@linux.ibm.com>
>>> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com>
>>> Cc: "Michael S. Tsirkin" <mst@redhat.com>
>>> Cc: Jason Wang <jasowang@redhat.com>
>>> Cc: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>> Cc: "Eugenio P?rez" <eperezma@redhat.com>
>>> Cc: Eric Farman <farman@linux.ibm.com>
>>> Cc: Andrew Morton <akpm@linux-foundation.org>
>>> Cc: Jonathan Corbet <corbet@lwn.net>
>>>
>>> David Hildenbrand (7):
>>>    Documentation: s390-diag.rst: make diag500 a generic KVM hypercall
>>>    Documentation: s390-diag.rst: document diag500(STORAGE LIMIT)
>>>      subfunction
>>>    s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM
>>>      memory devices
>>>    virtio-mem: s390 support
>>>    lib/Kconfig.debug: default STRICT_DEVMEM to "y" on s390
>>>    s390/sparsemem: reduce section size to 128 MiB
>>>    s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA
>>>
>>>   Documentation/virt/kvm/s390/s390-diag.rst | 35 +++++++++++++----
>>>   arch/s390/boot/physmem_info.c             | 47 ++++++++++++++++++++++-
>>>   arch/s390/boot/startup.c                  |  7 +++-
>>>   arch/s390/include/asm/physmem_info.h      |  3 ++
>>>   arch/s390/include/asm/sparsemem.h         | 10 ++++-
>>>   drivers/virtio/Kconfig                    | 12 +++---
>>>   lib/Kconfig.debug                         |  2 +-
>>>   7 files changed, 98 insertions(+), 18 deletions(-)
>>>
>>>
>>> base-commit: ae90f6a6170d7a7a1aa4fddf664fbd093e3023bc
>>> -- 
>>> 2.46.1
>>>
>>
>> Tested successfully various memory hotplug operations on lpar.
>>
> Just to be more precise, tested memory hotplug/hotunplug combinations +
> Device hotplug/hotunplug operations on guest.

Thanks a bunch!

-- 
Cheers,

David / dhildenb

Re: [PATCH v3 0/7] virtio-mem: s390 support

[Heiko Carstens]

On Fri, Oct 25, 2024 at 04:14:45PM +0200, David Hildenbrand wrote:
> Let's finally add s390 support for virtio-mem; my last RFC was sent
> 4 years ago, and a lot changed in the meantime.

...

> David Hildenbrand (7):
>   Documentation: s390-diag.rst: make diag500 a generic KVM hypercall
>   Documentation: s390-diag.rst: document diag500(STORAGE LIMIT)
>     subfunction
>   s390/physmem_info: query diag500(STORAGE LIMIT) to support QEMU/KVM
>     memory devices
>   virtio-mem: s390 support
>   lib/Kconfig.debug: default STRICT_DEVMEM to "y" on s390
>   s390/sparsemem: reduce section size to 128 MiB
>   s390/sparsemem: provide memory_add_physaddr_to_nid() with CONFIG_NUMA
> 
>  Documentation/virt/kvm/s390/s390-diag.rst | 35 +++++++++++++----
>  arch/s390/boot/physmem_info.c             | 47 ++++++++++++++++++++++-
>  arch/s390/boot/startup.c                  |  7 +++-
>  arch/s390/include/asm/physmem_info.h      |  3 ++
>  arch/s390/include/asm/sparsemem.h         | 10 ++++-
>  drivers/virtio/Kconfig                    | 12 +++---
>  lib/Kconfig.debug                         |  2 +-
>  7 files changed, 98 insertions(+), 18 deletions(-)

Series applied. Thanks a lot!