DAX system-ram usage by the kernel

DAX system-ram usage by the kernel

[Shesha Bhushan Sreenivasamurthy]

Hello All,
    When a CXL memory is assigned as system-ram using daxctl after boot, can kernel allocate memory from this memory range which cannot be moved and prevent this device from being made offline ?

Thanks,
Shesha.

Re: DAX system-ram usage by the kernel

[Shesha Bhushan Sreenivasamurthy]

From: Shesha Bhushan Sreenivasamurthy <sheshas@marvell.com>
Sent: Thursday, February 2, 2023 9:35 AM
To: linux-cxl@vger.kernel.org <linux-cxl@vger.kernel.org>
Subject: DAX system-ram usage by the kernel 
 
Hello All,
    When a CXL memory is assigned as system-ram using daxctl after boot, can kernel allocate memory from this memory range which cannot be moved and prevent this device from being made offline ?

After some reading, i found that this memory's valid zone will be "movable" and not "normal". Therefore, kernel's kmalloc, mmp and page-tables will only allocate from normal zone. Correct me if I am wrong.

Thanks,
Shesha.

Re: DAX system-ram usage by the kernel

[Gregory Price]

On Thu, Feb 02, 2023 at 05:56:21PM +0000, Shesha Bhushan Sreenivasamurthy wrote:
> 
> 
> 
> From: Shesha Bhushan Sreenivasamurthy <sheshas@marvell.com>
> Sent: Thursday, February 2, 2023 9:35 AM
> To: linux-cxl@vger.kernel.org <linux-cxl@vger.kernel.org>
> Subject: DAX system-ram usage by the kernel 
>  
> Hello All,
>     When a CXL memory is assigned as system-ram using daxctl after boot, can kernel allocate memory from this memory range which cannot be moved and prevent this device from being made offline ?
> 
> After some reading, i found that this memory's valid zone will be "movable" and not "normal". Therefore, kernel's kmalloc, mmp and page-tables will only allocate from normal zone. Correct me if I am wrong.
> 
> Thanks,
> Shesha.

There are a variety of reasons that moveable zones can fail to offline.

https://docs.kernel.org/admin-guide/mm/memory-hotplug.html#memory-offlining-and-zone-movable

RE: DAX system-ram usage by the kernel

[Dan Williams]

Shesha Bhushan Sreenivasamurthy wrote:
> Hello All,
>     When a CXL memory is assigned as system-ram using daxctl after
> boot, can kernel allocate memory from this memory range which cannot
> be moved and prevent this device from being made offline ?

Yes. Even with ZONE_MOVABLE there is no 100% guarantee that Linux will
not pin pages in there that prevent removal. The command:

  daxctl offline-memory $daxdev

...automates trying to offline all memblocks associate with $daxdev. If
that fails it means a page was busy in the kernel.

If a configuration needs hard guarantees then memory can not be onlined
at all and must be accessed via device-mappings.  If the configuration
can tolerate "more often than not successful removal" then ZONE_MOVABLE
targets that. Note though that there is a ratio of ZONE_NORMAL needed to
support increasing ZONE_MOVABLE. See the "auto-movable" setting for
"online_policy" in Documentation/admin-guide/mm/memory-hotplug.rst for
more details.