Re: Question about CXL emulation in QEMU

Re: Question about CXL emulation in QEMU

[Jonathan Cameron]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb18030", Size: 3255 bytes --]

On Fri, 5 Jan 2024 19:05:58 +0800
"ÖÜͯ" <273415826@qq.com> wrote:

> Dear Experts,
> 

+CC linux-cxl as a lot more CXL focused folk there who may be interested
in this discussion.

> I am writing to seek your assistance about CXL emulation in QEMU. I
> am Zhou Tong and I am researching how to use QEMU to simulate CXL
> over Ethernet¡£
> 
> I want to implement remote registration of CXL.mem devices based on
> the QOM model. The general idea is: the CXL slave side notifies the
> master side of the size of the CXL memory and other control
> information through Ethernet, and the master side registers the CXL
> device locally based on the control information. When the master
> accesses the CXL device, KVM is responsible for intercepting the
> action of accessing the memory, and encapsulates the CXL message and
> forwards it to the slave through Ethernet£¬

Qemu is not going to see anything that looks like actual CXL messages,
just memory reads and writes - no different to any other RAM.

>ultimately achieving
> remote CXL memory access.. Ask the experts how to register the CXL
> device locally based on the control information without occupying the
> HVA resources of the master host (OR display the CXL device).

This sounds like it will run into the same problem we have with KVM
and the existing emulation of CXL in qemu.  To trap those operations
you'll need to treat it different from normal memory. That generally
limits you to not putting in instructions in that memory. Even in TCG
the performance is shocking as a result.

You could potentially add a caching layer and rely on page faulting
if you are happy with page granularity.  So any writes would result
in a page fault which QEMU could then handle - reads would fault first
time and fetch into a cache.

There are papers on how to do it... (google quickly gave me this)
https://eric-keller.github.io/papers/2020/FluidMem_ICDCS2020.pdf

I've been meaning to look at improving the perf and enabling KVM usage
for simple interleave configurations - particularly as that's one
potential model for virtualization of dynamic capacity devices but
that won't directly help you.

Perhaps there is existing RAM focused work for remote memory on KVM?

I think there are some existing commercial solutions for doing similar
for conventional RAM and the problem is fundamentally the same - I guess
you care about CXL to get a standard Host software stack?

I'm not sure why you'd get HVA resource usage or the host seeing the CXL
device.  This part is just the same as normal CXL device emulation,
it will absorb GPA space etc on the VM host and the PCIe config space etc
for the CXL device will be emulated as normal.

Jonathan

> 
> 
> Thank you in advance for your attention to this email, and I eagerly
> look forward to any insights or advice you may be able to provide. If
> there is a more convenient time for us to discuss this matter
> further, please let me know, and I will be more than happy to
> accommodate your schedule.
> 
> 
> Once again, thank you for your impactful contributions to the
> open-source community, and I greatly appreciate your time and
> consideration.
> 
> 
> 
> Regard,
> 
> 
> Zhou Tong.
> 
> 
> 
> 
> 
> 
> 
> ÖÜͯ
> 273415826@qq.com
> 
> 
> 
> &nbsp;

Re: Question about CXL emulation in QEMU

[Jonathan Cameron via]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb18030", Size: 3256 bytes --]

On Fri, 5 Jan 2024 19:05:58 +0800
"ÖÜͯ" <273415826@qq.com> wrote:

> Dear Experts,
> 

+CC linux-cxl as a lot more CXL focused folk there who may be interested
in this discussion.

> I am writing to seek your assistance about CXL emulation in QEMU. I
> am Zhou Tong and I am researching how to use QEMU to simulate CXL
> over Ethernet¡£
> 
> I want to implement remote registration of CXL.mem devices based on
> the QOM model. The general idea is: the CXL slave side notifies the
> master side of the size of the CXL memory and other control
> information through Ethernet, and the master side registers the CXL
> device locally based on the control information. When the master
> accesses the CXL device, KVM is responsible for intercepting the
> action of accessing the memory, and encapsulates the CXL message and
> forwards it to the slave through Ethernet£¬

Qemu is not going to see anything that looks like actual CXL messages,
just memory reads and writes - no different to any other RAM.

>ultimately achieving
> remote CXL memory access.. Ask the experts how to register the CXL
> device locally based on the control information without occupying the
> HVA resources of the master host (OR display the CXL device).

This sounds like it will run into the same problem we have with KVM
and the existing emulation of CXL in qemu.  To trap those operations
you'll need to treat it different from normal memory. That generally
limits you to not putting in instructions in that memory. Even in TCG
the performance is shocking as a result.

You could potentially add a caching layer and rely on page faulting
if you are happy with page granularity.  So any writes would result
in a page fault which QEMU could then handle - reads would fault first
time and fetch into a cache.

There are papers on how to do it... (google quickly gave me this)
https://eric-keller.github.io/papers/2020/FluidMem_ICDCS2020.pdf

I've been meaning to look at improving the perf and enabling KVM usage
for simple interleave configurations - particularly as that's one
potential model for virtualization of dynamic capacity devices but
that won't directly help you.

Perhaps there is existing RAM focused work for remote memory on KVM?

I think there are some existing commercial solutions for doing similar
for conventional RAM and the problem is fundamentally the same - I guess
you care about CXL to get a standard Host software stack?

I'm not sure why you'd get HVA resource usage or the host seeing the CXL
device.  This part is just the same as normal CXL device emulation,
it will absorb GPA space etc on the VM host and the PCIe config space etc
for the CXL device will be emulated as normal.

Jonathan

> 
> 
> Thank you in advance for your attention to this email, and I eagerly
> look forward to any insights or advice you may be able to provide. If
> there is a more convenient time for us to discuss this matter
> further, please let me know, and I will be more than happy to
> accommodate your schedule.
> 
> 
> Once again, thank you for your impactful contributions to the
> open-source community, and I greatly appreciate your time and
> consideration.
> 
> 
> 
> Regard,
> 
> 
> Zhou Tong.
> 
> 
> 
> 
> 
> 
> 
> ÖÜͯ
> 273415826@qq.com
> 
> 
> 
> &nbsp;

Re: Question about CXL emulation in QEMU

[Alex Bennée]

"周童" <273415826@qq.com> writes:

> Dear Experts,

(add maintainers to CC)

>
> I am writing to seek your assistance about CXL emulation in QEMU. I am Zhou Tong and I am researching how to use QEMU
> to simulate CXL over Ethernet。
>
> I want to implement remote registration of CXL.mem devices based on the QOM model. The general idea is: the CXL slave
> side notifies the master side of the size of the CXL memory and other control information through Ethernet, and the master
> side registers the CXL device locally based on the control information. When the master accesses the CXL device, KVM is
> responsible for intercepting the action of accessing the memory, and encapsulates the CXL message and forwards it to the
> slave through Ethernet，ultimately achieving remote CXL memory access.. Ask the experts how to register the CXL device
> locally based on the control information without occupying the HVA resources of the master host (OR display the CXL
> device).
>
> Thank you in advance for your attention to this email, and I eagerly look forward to any insights or advice you may be able to
> provide. If there is a more convenient time for us to discuss this matter further, please let me know, and I will be more than
> happy to accommodate your schedule.
>
> Once again, thank you for your impactful contributions to the open-source community, and I greatly appreciate your time and
> consideration.
>
> Regard,
>
> Zhou Tong.
>
> -------------------------------------------------------------------------------------------------------------------------
>
>  *  周童  
>    273415826@qq.com  
> * 

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro