Re: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL

[Dragan Stancevic <dragan@stancevic.com>]

On 2/20/23 19:41, Kyungsan Kim wrote:
> CXL is a promising technology that leads to fundamental changes in computing architecture.
> To facilitate adoption and widespread of CXL memory, we are developing a memory tiering solution, called SMDK[1][2].
> Using SMDK and CXL RAM device, our team has been working with industry and academic partners over last year.
> Also, thanks to many researcher's effort, CXL adoption stage is gradually moving forward from basic enablement to real-world composite usecases.
> At this moment, based on the researches and experiences gained working on SMDK, we would like to suggest a session at LSF/MM/BFP this year
> to propose possible Linux MM changes with a brief of SMDK.
> 
> Adam Manzanares kindly adviced me that it is preferred to discuss implementation details on given problem and consensus at LSF/MM/BFP.
> Considering the adoption stage of CXL technology, however, let me suggest a design level discussion on the two MM expansions of SMDK this year.
> When we have design consensus with participants, we want to continue follow-up discussions with additional implementation details, hopefully.
> 
>   
> 1. A new zone, ZONE_EXMEM
> We added ZONE_EXMEM to manage CXL RAM device(s), separated from ZONE_NORMAL for usual DRAM due to the three reasons below.

Hi Kyungsan-

I read through your links and I am very interested in this 
talk/discussion from the perspective of cloud/virtualization hypervisor 
loads.

The problem that I am starting to tackle is clustering of hypervisors 
over cxl.mem for high availability of virtual machines. Or live 
migration of virtual machines between hypervisors using cxl.mem [1].


So I was wondering, with regards to the ZONE_XMEM, has any thought been 
given to the shared memory across virtual hierarchies [2], where you 
have cxl.mem access over cxl switches by multiple VH connections. It 
seems to me that there might be a need for differentiation of direct 
cxl.mem and switched cxl.mem. At least from the point of view where you 
have multiple hypervisors sharing the memory over a switch. Where they 
would potentially have to synchronize state/metadata about the memory.


[1] A high-level explanation is at http://nil-migration.org
[2] Compute Express Link Specification r3.0, v1.0 8/1/22, Page 51, 
figure 1-4, black color scheme circle(3) and bars.


--
Peace can only come as a natural consequence
of universal enlightenment -Dr. Nikola Tesla