Re: CXL Namespaces of ACPI disappearing in Qemu demo

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

Hi, Gregory
I am sorry to disturb you, but there was still an ignored problem about CXL, 
Although I had sent an email to jonathan (maybe he is busy recently so he forgot to reply),  
the Link is : https://lists.nongnu.org/archive/html/qemu-arm/2023-08/msg00278.html 

Maybe the core question is that how should we use CXL components with a standard PCIe system：
1) In Qemu, since the boards or machines in qemu can only use pxb-cxl (attached on pcie.0) to add
cxl host bridge, therefore, users should avoid to assign an occupied bus number by other devices
to pxb-cxl.

2) In real hardware, CXL components should use independent CXL root/tree (ACPI0017&ACPI0016)
to separate from the namespace of default pcie root/domain.

I would be grateful if you have some free time to help check this issue : )




[-- Attachment #2: Type: text/html, Size: 3050 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Gregory Price]

On Tue, Aug 22, 2023 at 03:22:02PM +0800, Yuquan Wang wrote:
> Hi, Gregory
> I am sorry to disturb you, but there was still an ignored problem about CXL, 
> Although I had sent an email to jonathan (maybe he is busy recently so he forgot to reply),  
> the Link is : https://lists.nongnu.org/archive/html/qemu-arm/2023-08/msg00278.html 
> 
> Maybe the core question is that how should we use CXL components with a standard PCIe system：
> 1) In Qemu, since the boards or machines in qemu can only use pxb-cxl (attached on pcie.0) to add
> cxl host bridge, therefore, users should avoid to assign an occupied bus number by other devices
> to pxb-cxl.
> 
> 2) In real hardware, CXL components should use independent CXL root/tree (ACPI0017&ACPI0016)
> to separate from the namespace of default pcie root/domain.
> 
> I would be grateful if you have some free time to help check this issue : )
> 

Generally speaking, ACPI questions are outside the scope of my
knowledge, but to the extent that i have debugged QEMU for CXL I can say
that #1 is correct - the host bridge adapter should not share PCI bus
id's with anything else or you will have issues.  This can be
frustrating if you don't know what the magic numbers are.

for my work, i have been using this when i need to attach two pxb-cxl
deivces

qemu-system-x86_64 \
-drive file=./cxl.qcow2,format=qcow2,index=0,media=disk,id=hd \
-m 4G,slots=4,maxmem=8G \
-smp 4 \
-machine type=q35,cxl=on,hmat=on \
-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=191 \


In regards to real hardware, I'm honestly not knowledgable enough about
ACPI or hardware to provide an intelligible answer unfortunately.

~Gregory

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 384 bytes --]

Hi, Gregory
On 2023-08-24 03:44,  Gregory wrote:

In regards to real hardware, I'm honestly not knowledgable enough about
ACPI or hardware to provide an intelligible answer unfortunately.

Never mind. 

Your precious guidance and replies along with Jonathan's are meaningful for me, because I was 
just touching CXL field in work as a fresh graduate.

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 1205 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Wed, 23 Aug 2023 15:44:35 -0400
Gregory Price <gregory.price@memverge.com> wrote:

> On Tue, Aug 22, 2023 at 03:22:02PM +0800, Yuquan Wang wrote:
> > Hi, Gregory
> > I am sorry to disturb you, but there was still an ignored problem about CXL, 
> > Although I had sent an email to jonathan (maybe he is busy recently so he forgot to reply),  
> > the Link is : https://lists.nongnu.org/archive/html/qemu-arm/2023-08/msg00278.html 
> > 
> > Maybe the core question is that how should we use CXL components with a standard PCIe system：
> > 1) In Qemu, since the boards or machines in qemu can only use pxb-cxl (attached on pcie.0) to add
> > cxl host bridge, therefore, users should avoid to assign an occupied bus number by other devices
> > to pxb-cxl.
> > 
> > 2) In real hardware, CXL components should use independent CXL root/tree (ACPI0017&ACPI0016)
> > to separate from the namespace of default pcie root/domain.
> > 
> > I would be grateful if you have some free time to help check this issue : )
> >   
> 
> Generally speaking, ACPI questions are outside the scope of my
> knowledge, but to the extent that i have debugged QEMU for CXL I can say
> that #1 is correct - the host bridge adapter should not share PCI bus
> id's with anything else or you will have issues.  This can be
> frustrating if you don't know what the magic numbers are.
> 
> for my work, i have been using this when i need to attach two pxb-cxl
> deivces
> 
> qemu-system-x86_64 \
> -drive file=./cxl.qcow2,format=qcow2,index=0,media=disk,id=hd \
> -m 4G,slots=4,maxmem=8G \
> -smp 4 \
> -machine type=q35,cxl=on,hmat=on \
> -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> -device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=191 \
> 
> 
> In regards to real hardware, I'm honestly not knowledgable enough about
> ACPI or hardware to provide an intelligible answer unfortunately.
> 
> ~Gregory

One additional thing to note. Nothing stops you putting a PCI device on a CXL
root port, or downstream switch port.  Some of the reporting is probably wrong
at the moment, but I've messed around with NVME below one and that 'works' fine
as Linux doesn't read the relevant registers anyway.

Jonathan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

Hi, Jonathan

Due to my poor experience and knowledge on cxl development history, I am sorry to continue
to ask some very simple and fundamental questions  : (

In hw/arm/virt :
[VIRT_CXL_HOST] =           { 0x0, 64 * KiB * 16 }, /* 16 UID */

It seems like the specific space for MMIO of cxl host bridges.  Why not just use the existing "VIRT_PCIE_MMIO" 
space for them? 

I would be grateful if you have some free time to help check this issue : )

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 1910 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Mon, 4 Sep 2023 18:27:10 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> 
Hi Yuquan

Given this question isn't just ARM specific included qemu-devel in the cc list as
that gets much wider reading than qemu-arm.

> Due to my poor experience and knowledge on cxl development history, I am sorry to continue
> to ask some very simple and fundamental questions  : (
> 
> In hw/arm/virt :
> [VIRT_CXL_HOST] =           { 0x0, 64 * KiB * 16 }, /* 16 UID */
> 
> It seems like the specific space for MMIO of cxl host bridges.  Why not just use the existing "VIRT_PCIE_MMIO" 
> space for them? 

At the system design level, MMIO space of Root complex register space via RCRB
does not map in a similar fashion to PCIE MMIO space (which is handled via
address decoding in the PCIE fabric). It is much more similar to MMIO for platform
devices - as such the implementation handles in like a platform device (well 16 of
them which seemed enough for any sane usecase).


Now the next bit I've only quickly explored so may have some errors!

For instance on a GPEX the main MMIO region is an alias to the sysbus mmio region 1.
https://elixir.bootlin.com/qemu/latest/source/hw/arm/virt.c#L1452
That region is then mapped to a generic pci_root_bus for use by bar
mappings etc.

https://elixir.bootlin.com/qemu/latest/source/hw/pci-host/gpex.c#L136

So in theory we could make some space for the CXL root bridge RCRB registers
but it would make various generic paths more complex.  In a real system
those registers are likely to be far from the PCI MMIO space anyway so the
way it's modeled is probably more realistic than pushing the RCRB into the
existing allocation.

I hope that clarifies our reasoning for handling this MMIO region separately.

Jonathan

> 
> I would be grateful if you have some free time to help check this issue : )
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Mon, 4 Sep 2023 18:27:10 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> 
Hi Yuquan

Given this question isn't just ARM specific included qemu-devel in the cc list as
that gets much wider reading than qemu-arm.

> Due to my poor experience and knowledge on cxl development history, I am sorry to continue
> to ask some very simple and fundamental questions  : (
> 
> In hw/arm/virt :
> [VIRT_CXL_HOST] =           { 0x0, 64 * KiB * 16 }, /* 16 UID */
> 
> It seems like the specific space for MMIO of cxl host bridges.  Why not just use the existing "VIRT_PCIE_MMIO" 
> space for them? 

At the system design level, MMIO space of Root complex register space via RCRB
does not map in a similar fashion to PCIE MMIO space (which is handled via
address decoding in the PCIE fabric). It is much more similar to MMIO for platform
devices - as such the implementation handles in like a platform device (well 16 of
them which seemed enough for any sane usecase).


Now the next bit I've only quickly explored so may have some errors!

For instance on a GPEX the main MMIO region is an alias to the sysbus mmio region 1.
https://elixir.bootlin.com/qemu/latest/source/hw/arm/virt.c#L1452
That region is then mapped to a generic pci_root_bus for use by bar
mappings etc.

https://elixir.bootlin.com/qemu/latest/source/hw/pci-host/gpex.c#L136

So in theory we could make some space for the CXL root bridge RCRB registers
but it would make various generic paths more complex.  In a real system
those registers are likely to be far from the PCI MMIO space anyway so the
way it's modeled is probably more realistic than pushing the RCRB into the
existing allocation.

I hope that clarifies our reasoning for handling this MMIO region separately.

Jonathan

> 
> I would be grateful if you have some free time to help check this issue : )
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

Hi, Jonathan
On 2023-09-04 20:43,  jonathan.cameron wrote:
> 
> At the system design level, MMIO space of Root complex register space via RCRB
> does not map in a similar fashion to PCIE MMIO space (which is handled via
> address decoding in the PCIE fabric). It is much more similar to MMIO for platform
> devices - as such the implementation handles in like a platform device (well 16 of
> them which seemed enough for any sane usecase).
> 
> 

Oh,thanks! According to above, therefore, the core factor is the implementation of RCRB.

> 
> So in theory we could make some space for the CXL root bridge RCRB registers
> but it would make various generic paths more complex.  In a real system
> those registers are likely to be far from the PCI MMIO space anyway so the
> way it's modeled is probably more realistic than pushing the RCRB into the
> existing allocation.
> 

Here implies that all CXL root bridge will use RCRB registers.

From Table 8-17 and Figure 9-14 in CXL3.0 specification, I understood that only RCH DP &
RCD UP will use RCRBs, and CXL host bridges VH mode will use other way to realize
the CHBCR. I had tried to find more explanation in CXL spec, but I haven't found. Hence 
this is why I am confused.

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2493 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Tue, 5 Sep 2023 18:45:02 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-04 20:43,  jonathan.cameron wrote:
> > 
> > At the system design level, MMIO space of Root complex register space via RCRB
> > does not map in a similar fashion to PCIE MMIO space (which is handled via
> > address decoding in the PCIE fabric). It is much more similar to MMIO for platform
> > devices - as such the implementation handles in like a platform device (well 16 of
> > them which seemed enough for any sane usecase).
> > 
> >   
> 
> Oh,thanks! According to above, therefore, the core factor is the implementation of RCRB.
> 
> > 
> > So in theory we could make some space for the CXL root bridge RCRB registers
> > but it would make various generic paths more complex.  In a real system
> > those registers are likely to be far from the PCI MMIO space anyway so the
> > way it's modeled is probably more realistic than pushing the RCRB into the
> > existing allocation.
> >   
> 
> Here implies that all CXL root bridge will use RCRB registers.
> 
> From Table 8-17 and Figure 9-14 in CXL3.0 specification, I understood that only RCH DP &
> RCD UP will use RCRBs, and CXL host bridges VH mode will use other way to realize
> the CHBCR. I had tried to find more explanation in CXL spec, but I haven't found. Hence 
> this is why I am confused.

Ah. That distinction is a bit messy.  Both an RCRB and CHBCR (CXL Host Bridge Component
Registers) are similar in the sense that they are mapped in host memory space. 
As I understand it the distinction is more about the format / contents of that memory
than how you access them. As an aside, they are described by a static ACPI table,
so they can't be in the MMIO space used for BARs etc.

Jonathan


> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Tue, 5 Sep 2023 18:45:02 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-04 20:43,  jonathan.cameron wrote:
> > 
> > At the system design level, MMIO space of Root complex register space via RCRB
> > does not map in a similar fashion to PCIE MMIO space (which is handled via
> > address decoding in the PCIE fabric). It is much more similar to MMIO for platform
> > devices - as such the implementation handles in like a platform device (well 16 of
> > them which seemed enough for any sane usecase).
> > 
> >   
> 
> Oh,thanks! According to above, therefore, the core factor is the implementation of RCRB.
> 
> > 
> > So in theory we could make some space for the CXL root bridge RCRB registers
> > but it would make various generic paths more complex.  In a real system
> > those registers are likely to be far from the PCI MMIO space anyway so the
> > way it's modeled is probably more realistic than pushing the RCRB into the
> > existing allocation.
> >   
> 
> Here implies that all CXL root bridge will use RCRB registers.
> 
> From Table 8-17 and Figure 9-14 in CXL3.0 specification, I understood that only RCH DP &
> RCD UP will use RCRBs, and CXL host bridges VH mode will use other way to realize
> the CHBCR. I had tried to find more explanation in CXL spec, but I haven't found. Hence 
> this is why I am confused.

Ah. That distinction is a bit messy.  Both an RCRB and CHBCR (CXL Host Bridge Component
Registers) are similar in the sense that they are mapped in host memory space. 
As I understand it the distinction is more about the format / contents of that memory
than how you access them. As an aside, they are described by a static ACPI table,
so they can't be in the MMIO space used for BARs etc.

Jonathan


> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

Hi, Jonathan
On 2023-09-05 22:34,  jonathan.cameron wrote:
> 
> As I understand it the distinction is more about the format / contents of that memory
> than how you access them. 

Yes, RCH DP RCRB includes registers from PCIe Type 1 Config Header and
PCIe capabilities and extended capabilities while CHBCR includes component registers 
with the same layout and discovery mechanism in other CXL components.

> As an aside, they are described by a static ACPI table,
> so they can't be in the MMIO space used for BARs etc.
> 

In CXL 3.0 Spec, the Figure 9-14 (CXL Link/Protocol Register Mapping in a CXL VH)
shows that CHBCR is mapped by "Host Proprietary Static Bar". According to your guidance,
it is not a standard PCIe BAR using PCIe MMIO Space, so I understand it is a special BAR for 
MMIO of a platform device?


Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2501 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Wed, 6 Sep 2023 19:22:19 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-05 22:34,  jonathan.cameron wrote:
> > 
> > As I understand it the distinction is more about the format / contents of that memory
> > than how you access them.   
> 
> Yes, RCH DP RCRB includes registers from PCIe Type 1 Config Header and
> PCIe capabilities and extended capabilities while CHBCR includes component registers 
> with the same layout and discovery mechanism in other CXL components.
> 
> > As an aside, they are described by a static ACPI table,
> > so they can't be in the MMIO space used for BARs etc.
> >   
> 
> In CXL 3.0 Spec, the Figure 9-14 (CXL Link/Protocol Register Mapping in a CXL VH)
> shows that CHBCR is mapped by "Host Proprietary Static Bar". According to your guidance,
> it is not a standard PCIe BAR using PCIe MMIO Space, so I understand it is a special BAR for 
> MMIO of a platform device?

Hmm. Host proprietary so basically you can in theory do anything you like.

In QEMU emulation at least it's not in the PCIe MMIO space.  I'd not rule out other
implementations putting it somewhere in that space.  For now I'm not seeing
a) Anything that says our choice is invalid.
b) Any advantage in making the handling of PCIe MMIO space more complex to shoe
   horn this in there.

Jonathan
> 
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Wed, 6 Sep 2023 19:22:19 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-05 22:34,  jonathan.cameron wrote:
> > 
> > As I understand it the distinction is more about the format / contents of that memory
> > than how you access them.   
> 
> Yes, RCH DP RCRB includes registers from PCIe Type 1 Config Header and
> PCIe capabilities and extended capabilities while CHBCR includes component registers 
> with the same layout and discovery mechanism in other CXL components.
> 
> > As an aside, they are described by a static ACPI table,
> > so they can't be in the MMIO space used for BARs etc.
> >   
> 
> In CXL 3.0 Spec, the Figure 9-14 (CXL Link/Protocol Register Mapping in a CXL VH)
> shows that CHBCR is mapped by "Host Proprietary Static Bar". According to your guidance,
> it is not a standard PCIe BAR using PCIe MMIO Space, so I understand it is a special BAR for 
> MMIO of a platform device?

Hmm. Host proprietary so basically you can in theory do anything you like.

In QEMU emulation at least it's not in the PCIe MMIO space.  I'd not rule out other
implementations putting it somewhere in that space.  For now I'm not seeing
a) Anything that says our choice is invalid.
b) Any advantage in making the handling of PCIe MMIO space more complex to shoe
   horn this in there.

Jonathan
> 
> 
> Many thanks
> Yuquan

Re: A confusion about cxl.mem in CXL drivers

[Jonathan Cameron]

On Wed, 20 Sep 2023 18:24:46 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> 
> There jumped a silly confusion when I am analyzing cxl drivers: 
> Since Host to access the cxl device memory and to access its mmio registers
> both use load/store instructions, how system software distinguish 
> with cxl.io and cxl.mem protocols?
> 
> Many thanks
> Yuquan

Hi Yuquan

I'm afraid I don't really understand the question.  So I'm guessing a bit whilst
trying to answer.

Is it about the kernel side of things, or more on what we are doing in QEMU
where we use iomem regions for both CXL.io and CXL.mem? For CXL.mem that
is done in QEMU to give us the ability to do fine grained address decoding
(below page level) but it doesn't in practice matter to the OS on top. It's
just an implementation detail in QEMU. There are knock on effects if you run with
KVM though as instructions can end up being read from the memory
(so that's not advised!).
However for both CXL.io and CXL.mem it is a host physical address range
and how the OS deals with it is dependent on how it is mapped.

So from driver side of things, the CXL.IO stuff is either in ECAM (for config
space) or mapped as PCIe BARs.  The CXL.mem stuff is mapped via the Host Physical
Addresses described in a CXL Fixed Memory Window.

So the right type of access is used based on the underlying hardware performing
the routing for the appropriate Host Physical Address range.  Same applies
on top of QEMU.

Jonathan

Re: A confusion about cxl.mem in CXL drivers

[Yuquan Wang]

On 2023-09-20 20:19,  jonathan.cameron wrote:

Thanks for your patient explanation.

>
> So from driver side of things, the CXL.IO stuff is either in ECAM (for config
> space) or mapped as PCIe BARs.  The CXL.mem stuff is mapped via the Host Physical
> Addresses described in a CXL Fixed Memory Window.
> 
> So the right type of access is used based on the underlying hardware performing
> the routing for the appropriate Host Physical Address range.  Same applies
> on top of QEMU.
> 

Therefore, from the view of kernel side, the kernel do not need to distinguish the physical cxl subprotocol
(cxl.io,cxl.cache,cxl.mem). In fact, the underlying hardware would directly finish this work so system software
don't care about it.

According to the instance of qemu virt machine, my understanding is below:
1) CXL.IO: finding, setting and enumerating CXL ECAM/BARs (programmed in cxl_acpi, cxl_pci drivers)
    Underlying hardware performing the routing : PCIe RC
    
2) CXL.MEM: host is going to access the memory mapped in CFMWs 
    Underlying hardware performing the routing : HDM decoders

Many thanks
Yuquan

Re: A confusion about cxl.mem in CXL drivers

[Yuquan Wang]

Sorry for resending this mail as I still confused about it.

On 2023-09-20 20:19,  jonathan.cameron wrote:

Thanks for your patient explanation.

>
> So from driver side of things, the CXL.IO stuff is either in ECAM (for config
> space) or mapped as PCIe BARs.  The CXL.mem stuff is mapped via the Host Physical
> Addresses described in a CXL Fixed Memory Window.
> 
> So the right type of access is used based on the underlying hardware performing
> the routing for the appropriate Host Physical Address range.  Same applies
> on top of QEMU.
> 

Therefore, from the view of kernel side, the kernel do not need to distinguish the physical cxl subprotocol
(cxl.io,cxl.cache,cxl.mem). In fact, the underlying hardware would directly finish this work so system software
don't care about it.

According to the instance of qemu virt machine, my understanding is below:
1) CXL.IO: finding, setting and enumerating CXL ECAM/BARs (programmed in cxl_acpi, cxl_pci drivers)
    Underlying hardware performing the routing : PCIe RC
    
2) CXL.MEM: host is going to access the memory mapped in CFMWs 
    Underlying hardware performing the routing : HDM decoders

Many thanks
Yuquan

Re: A confusion about cxl.mem in CXL drivers

[Jonathan Cameron]

On Sun, 8 Oct 2023 14:50:36 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Sorry for resending this mail as I still confused about it.
> 
> On 2023-09-20 20:19,  jonathan.cameron wrote:
> 
> Thanks for your patient explanation.
> 
> >
> > So from driver side of things, the CXL.IO stuff is either in ECAM (for config
> > space) or mapped as PCIe BARs.  The CXL.mem stuff is mapped via the Host Physical
> > Addresses described in a CXL Fixed Memory Window.
> > 
> > So the right type of access is used based on the underlying hardware performing
> > the routing for the appropriate Host Physical Address range.  Same applies
> > on top of QEMU.
> >   
> 
> Therefore, from the view of kernel side, the kernel do not need to distinguish the physical cxl subprotocol
> (cxl.io,cxl.cache,cxl.mem). In fact, the underlying hardware would directly finish this work so system software
> don't care about it.

The kernel does distinguish between these and may need to do different things to ensure
ordering of reads and writes etc.  They all work on a simple system by reading and writing
from host physical addresses (after mapping them into a virtual address in the kernel).

CXL.io is two things:
1.PCI config space accesses so those will typically be done via ECAM which is
a special region of the host physical address space where reads and writes resolve in PCI
config space transactions.
2. PCI Bar access.  Just like any normal PCI device these end up mapped to specific
host addresses programmed into the EP either by firmware or the kernel during bus enumeration.
As with PCI config reads and writes, accessing this memory address range results in the
right protocol messages being sent over the PCI bus.

CXL.cache is only relevant to accelerators - today we only support memory devices.
There is a Type 2 prototype around from Ira, but we haven't upstreamed it.

CXL.mem is accessed by the kernel via a host physical address range.  We program various
encoders in CXL.io space and make use of fixed system address decoders (described by the
ACPI CEDT table) to work out how to route a particular host physical address.  The CXL
root complex (or some protocol translation agent near that will route to the device in
question).

So the kernel basically needs to know what region to send read and writes to.  Note that
not all systems have simple memory based access to some of these protocols but most
systems likely to run CXL probably do.

> 
> According to the instance of qemu virt machine, my understanding is below:
> 1) CXL.IO: finding, setting and enumerating CXL ECAM/BARs (programmed in cxl_acpi, cxl_pci drivers)
>     Underlying hardware performing the routing : PCIe RC

Yes.

>     
> 2) CXL.MEM: host is going to access the memory mapped in CFMWs 
>     Underlying hardware performing the routing : HDM decoders

+ the host decoders which are described by CXL Fixed memory windows.

> 
> Many thanks
> Yuquan
> 
>