RE: Questions about CXL device (type 3 memory) hotplug

[Dan Williams <dan.j.williams@intel.com>]

Yasunori Gotou (Fujitsu) wrote:
> > Yasunori Gotou (Fujitsu) wrote:
[..]
> > > If a block hotremove sequence fails in the device, its user would like
> > > to keep the device online to postpone replacing it or select other device for
> > device pooling. (vice vesa).
> > > I don't find which component handle this situation.
> > 
> > It depends on how the memory is onlined and whether it gets pinned by the
> > kernel. As long as all of the memory is onlined to ZONE_MOVABLE then there is
> > a good chance to be able to get it back. However, ZONE_MOVABLE is not a
> > guarantee that memory can be removed later, and ZONE_MOVABLE requires
> > some ratio of ZONE_NORMAL memory to be present to make it usable. See
> > "Zone Imbalances" in Documentation/admin-guide/mm/memory-hotplug.rst.
> 
> I know it. Probably, I'm the first person who proposed that kernel divides its memory into
> movable and not movable area. (IIRC, it was BOF at Ottawa Linux Symposium 2004 or 2005).
> Actually, my name is still remain in git blame in the empty lines of the document.
> ----
> ac3332c44767b Documentation/admin-guide/mm/memory-hotplug.rst (David Hildenbrand     2021-09-07 19:54:49 -0700   4) Memory Hot(Un)Plug
> ac3332c44767b Documentation/admin-guide/mm/memory-hotplug.rst (David Hildenbrand     2021-09-07 19:54:49 -0700   5) ==================
> 6867c9310d5da Documentation/memory-hotplug.txt                (Yasunori Goto         2007-08-10 13:00:59 -0700   6)
>  :
> ---
> I'm glad to see many people have enhanced it after leaving from working for memory-hotplug 😊.

Nice! Yeah, I have noticed that most times when I think I need something
new for memory hotplug and CXL I run into David Hildenbrand's work
associated with virtio-mem.

> In my understanding, one of the big reason of memory hotplug failure is long term pin user pages
> like Infiniband RDMA, and I guess that or any similar features may have same problem.
> Many CXL devices like smartNIC will have such feature.
> Because It has ambivalent requirements.
> - To achieve fast data transfer, such feature want to skip the kernel layer and pin user pages
>  to transfer data directly. The most of CXL Device like Smart NIC will want to use it.
> - On the other hand, kernel has responsibility of such area management. Memory hotplug is one example of it,
>  and it will be important for CXL memory pool.
> 
> I think it is same with the issue FS-DAX vs. RDMA, and On Demand Paging is only one solution for it.
> I expect ODP may helpful for memory hotplug too.

It's going to be interesting. Yes, as memory becomes more dynamic, long
term page pinning is going to become more and more painful. It's even
worse because it's not just RDMA that causes the problem it's also any
device assigment to a guest VM that wants to pin all host pages backing
guest memory.

> About ratio problem between ZONE_NORMAL and ZONE_MOVABLE,
> I think user/platform will configure that DDR DRAM will be ZONE_NORMAL, and CXL memory pool will
> be ZONE_MOVABLE. It is easy for them to understand.

While that it is easy to understand, I worry that is in conflict with
one of the main value propositions of CXL which is vastly expanded
memory capacity. The conflict comes if the capacity of inexpensive CXL
outpaces the ZONE_NORMAL requirements that can be satisified from
locally attached DDR.