Re: [PATCH 0/4] Support dynamic (de)configuration of memory

David Hildenbrand
2025-10-07T16:11:21Z

On 26.09.25 15:15, Sumanth Korikkar wrote:
> Hi,

Hi,

>
> Patchset provides a new interface for dynamic configuration and
> deconfiguration of hotplug memory on s390, allowing with/without
> memmap_on_memory support. It is a follow up on the discussion with David
> when introducing memmap_on_memory support for s390 and support dynamic
> (de)configuration of memory:
> https://lore.kernel.org/all/ee492da8-74b4-4a97-8b24-73e07257f01d@redhat.com/
> https://lore.kernel.org/all/20241202082732.3959803-1-sumanthk@linux.ibm.com/
>
> The original motivation for introducing memmap_on_memory on s390 was to
> avoid using online memory to store struct pages metadata, particularly
> for standby memory blocks. This became critical in cases where there was
> an imbalance between standby and online memory, potentially leading to
> boot failures due to insufficient memory for metadata allocation.
>
> To address this, memmap_on_memory was utilized on s390. However, in its
> current form, it adds struct pages metadata at the start of each memory
> block at the time of addition (only standby memory), and this
> configuration is static. It cannot be changed at runtime (When the user
> needs continuous physical memory).
>
> Inorder to provide more flexibility to the user and overcome the above
> limitation, add an option to dynamically configure and deconfigure
> hotpluggable memory block with/without memmap_on_memory.

This will cleanly add/remove the memory, including the directmap and
other tracking data, so I like it.

>
> With the new interface, s390 will not add all possible hotplug memory in
> advance, like before, to make it visible in sysfs for online/offline
> actions. Instead, before memory block can be set online, it has to be
> configured via a new interface in /sys/firmware/memory/memoryX/config,
> which makes s390 similar to others. i.e. Adding of hotpluggable memory is
> controlled by the user instead of adding it at boottime.

Before I dig into the details, will onlining/offling still trigger
hypervisor action, or does that now really happen when memory is
added/removed?

That would be really nice, because it would remove the whole need for
"standby" memory, and having to treat hotplugged memory differently
under LPAR/z/VM than anywhere else (-> keep it offline).

>
> s390 kernel sysfs interface to configure/deconfigure memory with
> memmap_on_memory (with upcoming lsmem changes):
>
> * Initial memory layout:
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE SIZE STATE BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x7fffffff 2G online 0-15 yes no
> 0x80000000-0xffffffff 2G offline 16-31 no yes

Could we instead modify "STATE" to reflect that it is "not added" / "not
configured" / "disabled" etc?

Like

lsmem -o RANGE,SIZE,STATE,BLOCK,MEMMAP_ON_MEMORY
RANGE SIZE STATE BLOCK
0x00000000-0x7fffffff 2G online 0-15
0x80000000-0xffffffff 2G disabled 16-31

Or is that an attempt to maintain backwards compatibility?

>
> * Configure memory
> echo 1 > /sys/firmware/memory/memory16/config

The granularity here is also memory_block_size_bytes(), correct?

> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE SIZE STATE BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x7fffffff 2G online 0-15 yes no
> 0x80000000-0x87ffffff 128M offline 16 yes yes
> 0x88000000-0xffffffff 1.9G offline 17-31 no yes
>
> * Deconfigure memory
> echo 0 > /sys/firmware/memory/memory16/config
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE SIZE STATE BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x7fffffff 2G online 0-15 yes no
> 0x80000000-0xffffffff 2G offline 16-31 no yes
>
> * Enable memmap_on_memory and online it.
> (Deconfigure first)
> echo 0 > /sys/devices/system/memory/memory5/online
> echo 0 > /sys/firmware/memory/memory5/config
>
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE SIZE STATE BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x27ffffff 640M online 0-4 yes no
> 0x28000000-0x2fffffff 128M offline 5 no no
> 0x30000000-0x7fffffff 1.3G online 6-15 yes no
> 0x80000000-0xffffffff 2G offline 16-31 no yes
>
> (Enable memmap_on_memory and online it)
> echo 1 > /sys/firmware/memory/memory5/memmap_on_memory
> echo 1 > /sys/firmware/memory/memory5/config
> echo 1 > /sys/devices/system/memory/memory5/online

I guess the use for memmap_on_memory would now be limited to making
hotplug more likely to succeed in OOM scenarios.

>
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE SIZE STATE BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x27ffffff 640M online 0-4 yes no
> 0x28000000-0x2fffffff 128M online 5 yes yes
> 0x30000000-0x7fffffff 1.3G online 6-15 yes no
> 0x80000000-0xffffffff 2G offline 16-31 no yes
>
> * Disable memmap_on_memory and online it.
> (Deconfigure first)
> echo 0 > /sys/devices/system/memory/memory5/online
> echo 0 > /sys/firmware/memory/memory5/config
>
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE SIZE STATE BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x27ffffff 640M online 0-4 yes no
> 0x28000000-0x2fffffff 128M offline 5 no yes
> 0x30000000-0x7fffffff 1.3G online 6-15 yes no
> 0x80000000-0xffffffff 2G offline 16-31 no yes
>
> (Disable memmap_on_memory and online it)
> echo 0 > /sys/firmware/memory/memory5/memmap_on_memory
> echo 1 > /sys/firmware/memory/memory5/config
> echo 1 > /sys/devices/system/memory/memory5/online
>
> lsmem -o RANGE,SIZE,STATE,BLOCK,CONFIGURED,MEMMAP_ON_MEMORY
> RANGE SIZE STATE BLOCK CONFIGURED MEMMAP_ON_MEMORY
> 0x00000000-0x7fffffff 2G online 0-15 yes no
> 0x80000000-0xffffffff 2G offline 16-31 no yes
>
> * Userspace changes:
> lsmem/chmem tool is also changed to use the new interface. I will send
> it to util-linux soon.
>
> Patch 1 adds support for removal of boot-allocated memory blocks.
>
> Patch 2 provides option to dynamically configure and deconfigure memory
> with/without memmap_on_memory.
>
> Patch 3 removes MHP_OFFLINE_INACCESSIBLE from s390. The mhp flag was
> used to mark memory as not accessible until memory hotplug online phase
> begins. However, with patch 2, it is no longer essential. Memory can be
> brought to accessible state before adding memory, as the memory is added
> during runttime now instead of boottime.

Nice.

>
> Patch 4 removes the MEM_PREPARE_ONLINE/MEM_FINISH_OFFLINE notifiers. It
> is no longer needed. Memory can be brought to accessible state before
> adding memory now, with runtime (de)configuration of memory.

Nice.

--
Cheers

David / dhildenb