[PATCH RFC 0/5] hw/cxl: Type 2 Device RFC

[PATCH RFC 0/5] hw/cxl: Type 2 Device RFC

[Ira Weiny]

Type 2 devices are not yet a reality.  Developing core kernel support
is difficult without some test device to model against.

Define a type 2 device 'cxl-accel'.  This device is derived from the
type 3 device and retains all that functionality for now.

Mock up a couple of accelerator features (Back Invalidate [BI] and
Unordered IO [UIO]) as examples for the RFC.  These have no
functionality other than to report the features as present for software
to key off of.

Defining these devices in qemu can be done with the following example:

...
  -device cxl-accel,bus=sw0p0,volatile-memdev=cxl-ac-mem5,id=cxl-dev5,sn=0xCAFE0005
...

NOTE: I'm leaving off Michael Tsirkin for now because this is really
rough and I'm mainly sending it out because it was talked about in the
CXL community call on 5/16.

Not-Yet-Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
Ira Weiny (5):
      hw/cxl: Use define for build bug detection
      hw/cxl: Refactor component register initialization
      hw/cxl: Derive a CXL accelerator device from Type-3
      hw/cxl/accel: Add Back-Invalidate decoder capbility structure
      hw/cxl: Add UIO HDM decoder register fields

 docs/system/devices/cxl.rst    | 11 ++++++
 hw/cxl/cxl-component-utils.c   | 80 +++++++++++++++++++-----------------------
 hw/mem/cxl_type3.c             | 39 ++++++++++++++++++++
 include/hw/cxl/cxl_component.h | 51 +++++++++++++++++++--------
 include/hw/cxl/cxl_device.h    | 16 +++++++++
 include/hw/pci/pci_ids.h       |  1 +
 6 files changed, 141 insertions(+), 57 deletions(-)
---
base-commit: 8eb2a03258313f404ca0c8609a8f9009b9b4318c
change-id: 20230517-rfc-type2-dev-c2d661a29d96

Best regards,
-- 
Ira Weiny <ira.weiny@intel.com>

[PATCH RFC 1/5] hw/cxl: Use define for build bug detection

[Ira Weiny]

Magic numbers can be confusing.

Use the range size define for CXL.cachemem rather than a magic number.
Update/add spec references.

Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
 include/hw/cxl/cxl_component.h | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
index 52b6a2d67f40..bca2b756c202 100644
--- a/include/hw/cxl/cxl_component.h
+++ b/include/hw/cxl/cxl_component.h
@@ -10,7 +10,7 @@
 #ifndef CXL_COMPONENT_H
 #define CXL_COMPONENT_H
 
-/* CXL 2.0 - 8.2.4 */
+/* CXL 3.0 - 8.2.3 */
 #define CXL2_COMPONENT_IO_REGION_SIZE 0x1000
 #define CXL2_COMPONENT_CM_REGION_SIZE 0x1000
 #define CXL2_COMPONENT_BLOCK_SIZE 0x10000
@@ -173,7 +173,9 @@ HDM_DECODER_INIT(3);
     (CXL_IDE_REGISTERS_OFFSET + CXL_IDE_REGISTERS_SIZE)
 #define CXL_SNOOP_REGISTERS_SIZE   0x8
 
-QEMU_BUILD_BUG_MSG((CXL_SNOOP_REGISTERS_OFFSET + CXL_SNOOP_REGISTERS_SIZE) >= 0x1000,
+/* CXL 3.0 8.2.3 Table 8-21 */
+QEMU_BUILD_BUG_MSG((CXL_SNOOP_REGISTERS_OFFSET +
+                    CXL_SNOOP_REGISTERS_SIZE) >= CXL2_COMPONENT_CM_REGION_SIZE,
                    "No space for registers");
 
 typedef struct component_registers {

-- 
2.40.0

Re: [PATCH RFC 1/5] hw/cxl: Use define for build bug detection

[Jonathan Cameron via]

On Wed, 17 May 2023 19:45:54 -0700
Ira Weiny <ira.weiny@intel.com> wrote:

> Magic numbers can be confusing.
> 
> Use the range size define for CXL.cachemem rather than a magic number.
> Update/add spec references.
> 
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>

I guess we should do a scrub to move all refs to 3.0 soon
given it's horrible having a mixture of spec versions for the references.

For future specs, we should only do this when sufficient X.Y references
have started to appear - I think that's true for r3.0 now.

Jonathan

> ---
>  include/hw/cxl/cxl_component.h | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
> index 52b6a2d67f40..bca2b756c202 100644
> --- a/include/hw/cxl/cxl_component.h
> +++ b/include/hw/cxl/cxl_component.h
> @@ -10,7 +10,7 @@
>  #ifndef CXL_COMPONENT_H
>  #define CXL_COMPONENT_H
>  
> -/* CXL 2.0 - 8.2.4 */
> +/* CXL 3.0 - 8.2.3 */
>  #define CXL2_COMPONENT_IO_REGION_SIZE 0x1000
>  #define CXL2_COMPONENT_CM_REGION_SIZE 0x1000
>  #define CXL2_COMPONENT_BLOCK_SIZE 0x10000
> @@ -173,7 +173,9 @@ HDM_DECODER_INIT(3);
>      (CXL_IDE_REGISTERS_OFFSET + CXL_IDE_REGISTERS_SIZE)
>  #define CXL_SNOOP_REGISTERS_SIZE   0x8
>  
> -QEMU_BUILD_BUG_MSG((CXL_SNOOP_REGISTERS_OFFSET + CXL_SNOOP_REGISTERS_SIZE) >= 0x1000,
> +/* CXL 3.0 8.2.3 Table 8-21 */
> +QEMU_BUILD_BUG_MSG((CXL_SNOOP_REGISTERS_OFFSET +
> +                    CXL_SNOOP_REGISTERS_SIZE) >= CXL2_COMPONENT_CM_REGION_SIZE,
>                     "No space for registers");
>  
>  typedef struct component_registers {
> 

Re: [PATCH RFC 1/5] hw/cxl: Use define for build bug detection

[Ira Weiny]

Jonathan Cameron wrote:
> On Wed, 17 May 2023 19:45:54 -0700
> Ira Weiny <ira.weiny@intel.com> wrote:
> 
> > Magic numbers can be confusing.
> > 
> > Use the range size define for CXL.cachemem rather than a magic number.
> > Update/add spec references.
> > 
> > Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> 
> I guess we should do a scrub to move all refs to 3.0 soon
> given it's horrible having a mixture of spec versions for the references.
> 
> For future specs, we should only do this when sufficient X.Y references
> have started to appear - I think that's true for r3.0 now.

For the kernel side I think Dan is taking the 'if you are updating it then
update the spec' but otherwise leave it be.  So since I'm touching the
code I updated it.

I agree, it is a pain to have to look at the 2.0 spec but you can do it.

Ira

Re: [PATCH RFC 1/5] hw/cxl: Use define for build bug detection

[Jonathan Cameron via]

On Thu, 18 May 2023 13:19:12 -0700
Ira Weiny <ira.weiny@intel.com> wrote:

> Jonathan Cameron wrote:
> > On Wed, 17 May 2023 19:45:54 -0700
> > Ira Weiny <ira.weiny@intel.com> wrote:
> >   
> > > Magic numbers can be confusing.
> > > 
> > > Use the range size define for CXL.cachemem rather than a magic number.
> > > Update/add spec references.
> > > 
> > > Signed-off-by: Ira Weiny <ira.weiny@intel.com>  
> > 
> > I guess we should do a scrub to move all refs to 3.0 soon
> > given it's horrible having a mixture of spec versions for the references.
> > 
> > For future specs, we should only do this when sufficient X.Y references
> > have started to appear - I think that's true for r3.0 now.  
> 
> For the kernel side I think Dan is taking the 'if you are updating it then
> update the spec' but otherwise leave it be.  So since I'm touching the
> code I updated it.
> 
> I agree, it is a pain to have to look at the 2.0 spec but you can do it.

Only if you are either a member of the consortium, or happened to have
grabbed a copy in the past I think.  I've had people mentioning they can't
get it today.

Jonathan

> 
> Ira

Re: [PATCH RFC 1/5] hw/cxl: Use define for build bug detection

[Ira Weiny]

Jonathan Cameron wrote:
> On Thu, 18 May 2023 13:19:12 -0700
> Ira Weiny <ira.weiny@intel.com> wrote:
> 
> > Jonathan Cameron wrote:
> > > On Wed, 17 May 2023 19:45:54 -0700
> > > Ira Weiny <ira.weiny@intel.com> wrote:
> > >   
> > > > Magic numbers can be confusing.
> > > > 
> > > > Use the range size define for CXL.cachemem rather than a magic number.
> > > > Update/add spec references.
> > > > 
> > > > Signed-off-by: Ira Weiny <ira.weiny@intel.com>  
> > > 
> > > I guess we should do a scrub to move all refs to 3.0 soon
> > > given it's horrible having a mixture of spec versions for the references.
> > > 
> > > For future specs, we should only do this when sufficient X.Y references
> > > have started to appear - I think that's true for r3.0 now.  
> > 
> > For the kernel side I think Dan is taking the 'if you are updating it then
> > update the spec' but otherwise leave it be.  So since I'm touching the
> > code I updated it.
> > 
> > I agree, it is a pain to have to look at the 2.0 spec but you can do it.
> 
> Only if you are either a member of the consortium, or happened to have
> grabbed a copy in the past I think.  I've had people mentioning they can't
> get it today.
> 

Oh.  :-(  That seems unfortunate.  I've emailed the CXL admins.  Perhaps
they can post an archive page or something?

Ira

[PATCH RFC 2/5] hw/cxl: Refactor component register initialization

[Ira Weiny]

CXL 3.0 8.2.4 Table 8-22 defines which capabilities are mandatory, not
permitted, or optional for each type of device.

cxl_component_register_init_common() uses a rather odd 'fall through'
mechanism to define each component register set.  This assumes that any
device or capability being added builds on the previous devices
capabilities.  This is not true as there are mutually exclusive
capabilities defined.  For example, downstream ports can not have snoop
but it can have Back Invalidate capable decoders.

Refactor this code to make it easier to add individual capabilities as
defined by a device type.  Any capability which is not specified by the
type is left NULL'ed out which complies with the packed nature of the
register array.

Update all spec references to 3.0.

No functional changes should be seen with this patch.

Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
 hw/cxl/cxl-component-utils.c   | 73 ++++++++++++++++--------------------------
 include/hw/cxl/cxl_component.h | 36 ++++++++++++++-------
 2 files changed, 52 insertions(+), 57 deletions(-)

diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index 0e23e132ed3a..7949d12b7351 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -198,43 +198,19 @@ void cxl_component_register_init_common(uint32_t *reg_state, uint32_t *write_msk
 {
     int caps = 0;
 
-    /*
-     * In CXL 2.0 the capabilities required for each CXL component are such that,
-     * with the ordering chosen here, a single number can be used to define
-     * which capabilities should be provided.
-     */
-    switch (type) {
-    case CXL2_DOWNSTREAM_PORT:
-    case CXL2_DEVICE:
-        /* RAS, Link */
-        caps = 2;
-        break;
-    case CXL2_UPSTREAM_PORT:
-    case CXL2_TYPE3_DEVICE:
-    case CXL2_LOGICAL_DEVICE:
-        /* + HDM */
-        caps = 3;
-        break;
-    case CXL2_ROOT_PORT:
-        /* + Extended Security, + Snoop */
-        caps = 5;
-        break;
-    default:
-        abort();
-    }
-
     memset(reg_state, 0, CXL2_COMPONENT_CM_REGION_SIZE);
 
     /* CXL Capability Header Register */
     ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ID, 1);
     ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, VERSION, 1);
     ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, CACHE_MEM_VERSION, 1);
-    ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ARRAY_SIZE, caps);
 
 #define init_cap_reg(reg, id, version)                                        \
-    QEMU_BUILD_BUG_ON(CXL_##reg##_REGISTERS_OFFSET == 0);                     \
+    QEMU_BUILD_BUG_ON(CXL_##reg##_CAP_HDR_IDX == 0);                          \
     do {                                                                      \
-        int which = R_CXL_##reg##_CAPABILITY_HEADER;                          \
+        int which = CXL_##reg##_CAP_HDR_IDX;                                  \
+        if (CXL_##reg##_CAP_HDR_IDX > caps)                                   \
+            caps = CXL_##reg##_CAP_HDR_IDX;                                   \
         reg_state[which] = FIELD_DP32(reg_state[which],                       \
                                       CXL_##reg##_CAPABILITY_HEADER, ID, id); \
         reg_state[which] =                                                    \
@@ -245,25 +221,32 @@ void cxl_component_register_init_common(uint32_t *reg_state, uint32_t *write_msk
                        CXL_##reg##_REGISTERS_OFFSET);                         \
     } while (0)
 
-    init_cap_reg(RAS, 2, 2);
-    ras_init_common(reg_state, write_msk);
-
-    init_cap_reg(LINK, 4, 2);
-
-    if (caps < 3) {
-        return;
-    }
-
-    init_cap_reg(HDM, 5, 1);
-    hdm_init_common(reg_state, write_msk, type);
-
-    if (caps < 5) {
-        return;
+    /* CXL 3.0 8.2.4 Table 8-22 */
+    switch (type) {
+    case CXL2_ROOT_PORT:
+        /* + Extended Security, + Snoop */
+        init_cap_reg(EXTSEC, 6, 1);
+        init_cap_reg(SNOOP, 8, 1);
+        /* FALL THROUGH */
+    case CXL2_UPSTREAM_PORT:
+    case CXL2_TYPE3_DEVICE:
+    case CXL2_LOGICAL_DEVICE:
+        /* + HDM */
+        init_cap_reg(HDM, 5, 1);
+        hdm_init_common(reg_state, write_msk, type);
+        /* FALL THROUGH */
+    case CXL2_DOWNSTREAM_PORT:
+    case CXL2_DEVICE:
+        /* RAS, Link */
+        init_cap_reg(RAS, 2, 2);
+        ras_init_common(reg_state, write_msk);
+        init_cap_reg(LINK, 4, 2);
+        break;
+    default:
+        abort();
     }
 
-    init_cap_reg(EXTSEC, 6, 1);
-    init_cap_reg(SNOOP, 8, 1);
-
+    ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ARRAY_SIZE, caps);
 #undef init_cap_reg
 }
 
diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
index bca2b756c202..7c08c02c5e9d 100644
--- a/include/hw/cxl/cxl_component.h
+++ b/include/hw/cxl/cxl_component.h
@@ -31,29 +31,41 @@ enum reg_type {
 };
 
 /*
- * Capability registers are defined at the top of the CXL.cache/mem region and
- * are packed. For our purposes we will always define the caps in the same
- * order.
- * CXL 2.0 - 8.2.5 Table 142 for details.
+ * CXL 3.0 - 8.2.4 Table 8-22 and 8-23
+ *
+ * Capability registers are defined at the top of the CXL.cache/mem region.
+ * They are defined to be packed and at variable offsets.  However, NULL
+ * capabilities can be added to the packed array.  To facilitate easier access
+ * within the QEMU code, define these at specified offsets.  Then NULL out any
+ * capabilities for devices which don't (or can't) have a particular capability
+ * (see cxl_component_register_init_common).  NULL capabilities are to be
+ * ignored by software.
+ *
+ * 'offsets' are based on index's which can then be used to report the array
+ * size in CXL Capability Header Register (index/offset 0).
+ *
+ * See CXL 3.0 Table 8-25 for an example of allowing a 'NULL' header.
  */
 
-/* CXL 2.0 - 8.2.5.1 */
+/* CXL 3.0 - 8.2.4.1 */
 REG32(CXL_CAPABILITY_HEADER, 0)
     FIELD(CXL_CAPABILITY_HEADER, ID, 0, 16)
     FIELD(CXL_CAPABILITY_HEADER, VERSION, 16, 4)
     FIELD(CXL_CAPABILITY_HEADER, CACHE_MEM_VERSION, 20, 4)
     FIELD(CXL_CAPABILITY_HEADER, ARRAY_SIZE, 24, 8)
 
-#define CXLx_CAPABILITY_HEADER(type, offset)                  \
-    REG32(CXL_##type##_CAPABILITY_HEADER, offset)             \
+/* CXL 3.0 - 8.2.4.2 on... */
+#define CXLx_CAPABILITY_HEADER(type, idx)                     \
+    enum { CXL_##type##_CAP_HDR_IDX = idx };                  \
+    REG32(CXL_##type##_CAPABILITY_HEADER, (idx * 0x4))        \
         FIELD(CXL_##type##_CAPABILITY_HEADER, ID, 0, 16)      \
         FIELD(CXL_##type##_CAPABILITY_HEADER, VERSION, 16, 4) \
         FIELD(CXL_##type##_CAPABILITY_HEADER, PTR, 20, 12)
-CXLx_CAPABILITY_HEADER(RAS, 0x4)
-CXLx_CAPABILITY_HEADER(LINK, 0x8)
-CXLx_CAPABILITY_HEADER(HDM, 0xc)
-CXLx_CAPABILITY_HEADER(EXTSEC, 0x10)
-CXLx_CAPABILITY_HEADER(SNOOP, 0x14)
+CXLx_CAPABILITY_HEADER(RAS, 1)
+CXLx_CAPABILITY_HEADER(LINK, 2)
+CXLx_CAPABILITY_HEADER(HDM, 3)
+CXLx_CAPABILITY_HEADER(EXTSEC, 4)
+CXLx_CAPABILITY_HEADER(SNOOP, 5)
 
 /*
  * Capability structures contain the actual registers that the CXL component

-- 
2.40.0

[PATCH RFC 3/5] hw/cxl: Derive a CXL accelerator device from Type-3

[Ira Weiny]

It is desirable to have a test accelerator device to present various
accelerator features such as Back-Invalidate (BI) registers to OS
software for testing.

BI accelerator devices require memory that can be easily modeled as a
sub-class of Type-3 device.

Derive 'cxl-accel' from cxl-type3.  Add documentation for such a device.

Follow on patches will add BI registers and other simulation of the
accelerator device.

Adding devices qemu can be done with the following example:

...
  -device cxl-accel,bus=sw0p0,volatile-memdev=cxl-ac-mem5,id=cxl-dev5,sn=0xCAFE0005
...

Not-Yet-Signed-off-by: Ira Weiny <ira.weiny@intel.com>

---
The device ID and class code are completely made up by me.  As discussed
in the last community call perhaps these could be declared in some more
official capacity?
---
 docs/system/devices/cxl.rst | 11 +++++++++++
 hw/mem/cxl_type3.c          | 28 ++++++++++++++++++++++++++++
 include/hw/cxl/cxl_device.h | 16 ++++++++++++++++
 include/hw/pci/pci_ids.h    |  1 +
 4 files changed, 56 insertions(+)

diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst
index 95900252c56a..5bc931be44b3 100644
--- a/docs/system/devices/cxl.rst
+++ b/docs/system/devices/cxl.rst
@@ -321,6 +321,17 @@ A very simple setup with just one directly attached CXL Type 3 Volatile Memory d
   -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \
   -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G
 
+A very simple setup with just one directly attached CXL Type 2 Volatile Memory
+Accelerator device::
+
+  qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \
+  ...
+  -object memory-backend-ram,id=vmem0,share=on,size=256M \
+  -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
+  -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
+  -device cxl-accel,bus=root_port13,volatile-memdev=vmem0,id=cxl-accel0 \
+  -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G
+
 The same volatile setup may optionally include an LSA region::
 
   qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index 3e63dbd83551..c7eafd76d1ea 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -1691,3 +1691,31 @@ static void ct3d_registers(void)
 }
 
 type_init(ct3d_registers);
+
+static void cxl_accel_class_init(ObjectClass *oc, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(oc);
+    PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+
+    pc->class_id = PCI_CLASS_CXL_QEMU_ACCEL;
+    pc->vendor_id = PCI_VENDOR_ID_INTEL;
+    pc->device_id = 0xd94; /* LVF for now */
+    pc->revision = 1;
+
+    dc->desc = "CXL Accelerator Device (Type 2)";
+}
+
+static const TypeInfo cxl_accel_dev_info = {
+    .name = TYPE_CXL_ACCEL,
+    .parent = TYPE_CXL_TYPE3,
+    .class_size = sizeof(struct CXLAccelClass),
+    .class_init = cxl_accel_class_init,
+    .instance_size = sizeof(CXLAccelDev),
+};
+
+static void cxl_accel_dev_registers(void)
+{
+    type_register_static(&cxl_accel_dev_info);
+}
+
+type_init(cxl_accel_dev_registers);
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index cd7f28dba884..f7f6688ee6e2 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -432,6 +432,22 @@ struct CXLType3Class {
     bool (*set_cacheline)(CXLType3Dev *ct3d, uint64_t dpa_offset, uint8_t *data);
 };
 
+/*
+ * Accel devices are a type3 device but with additional functionality.
+ */
+struct CXLAccelDev {
+    /* Private: Must be first */
+    CXLType3Dev parent_obj;
+};
+
+struct CXLAccelClass {
+    /* Private: Must be first */
+    CXLType3Class parent_class;
+};
+
+#define TYPE_CXL_ACCEL "cxl-accel"
+OBJECT_DECLARE_TYPE(CXLAccelDev, CXLAccelClass, CXL_ACCEL)
+
 struct CSWMBCCIDev {
     PCIDevice parent_obj;
     CXLComponentState cxl_cstate;
diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
index e4386ebb2038..2dbf350ebba4 100644
--- a/include/hw/pci/pci_ids.h
+++ b/include/hw/pci/pci_ids.h
@@ -54,6 +54,7 @@
 #define PCI_CLASS_MEMORY_RAM             0x0500
 #define PCI_CLASS_MEMORY_FLASH           0x0501
 #define PCI_CLASS_MEMORY_CXL             0x0502
+#define PCI_CLASS_CXL_QEMU_ACCEL         0x0503
 #define PCI_CLASS_MEMORY_OTHER           0x0580
 
 #define PCI_BASE_CLASS_BRIDGE            0x06

-- 
2.40.0

[PATCH RFC 4/5] hw/cxl/accel: Add Back-Invalidate decoder capbility structure

[Ira Weiny]

The presence of the Back-Invalidate (BI) decoder capability structure
indicates a CXL downstream port, root port, or device supports the BI
messages.

Add the BI capability structure to the accelerator device.

Not-Yet-Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
 hw/cxl/cxl-component-utils.c   |  5 +++++
 hw/mem/cxl_type3.c             | 11 +++++++++++
 include/hw/cxl/cxl_component.h | 11 +++++++++--
 3 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index 7949d12b7351..a9efa252b4ae 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -228,6 +228,7 @@ void cxl_component_register_init_common(uint32_t *reg_state, uint32_t *write_msk
         init_cap_reg(EXTSEC, 6, 1);
         init_cap_reg(SNOOP, 8, 1);
         /* FALL THROUGH */
+    case CXL3_TYPE2_DEVICE:
     case CXL2_UPSTREAM_PORT:
     case CXL2_TYPE3_DEVICE:
     case CXL2_LOGICAL_DEVICE:
@@ -246,6 +247,10 @@ void cxl_component_register_init_common(uint32_t *reg_state, uint32_t *write_msk
         abort();
     }
 
+    if (type == CXL3_TYPE2_DEVICE) {
+        init_cap_reg(BI_DECODER, 12, 1);
+    }
+
     ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ARRAY_SIZE, caps);
 #undef init_cap_reg
 }
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index c7eafd76d1ea..95fdaaa18f37 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -1692,6 +1692,16 @@ static void ct3d_registers(void)
 
 type_init(ct3d_registers);
 
+static void accel_reset(DeviceState *dev)
+{
+    CXLAccelDev *acceld = CXL_ACCEL(dev);
+    uint32_t *reg_state = acceld->parent_obj.cxl_cstate.crb.cache_mem_registers;
+    uint32_t *write_msk = acceld->parent_obj.cxl_cstate.crb.cache_mem_regs_write_mask;
+
+    cxl_component_register_init_common(reg_state, write_msk, CXL3_TYPE2_DEVICE);
+    cxl_device_register_init_common(&acceld->parent_obj.cxl_dstate);
+}
+
 static void cxl_accel_class_init(ObjectClass *oc, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(oc);
@@ -1703,6 +1713,7 @@ static void cxl_accel_class_init(ObjectClass *oc, void *data)
     pc->revision = 1;
 
     dc->desc = "CXL Accelerator Device (Type 2)";
+    dc->reset = accel_reset;
 }
 
 static const TypeInfo cxl_accel_dev_info = {
diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
index 7c08c02c5e9d..a5b5512aed94 100644
--- a/include/hw/cxl/cxl_component.h
+++ b/include/hw/cxl/cxl_component.h
@@ -28,6 +28,7 @@ enum reg_type {
     CXL2_UPSTREAM_PORT,
     CXL2_DOWNSTREAM_PORT,
     CXL3_SWITCH_MAILBOX_CCI,
+    CXL3_TYPE2_DEVICE,
 };
 
 /*
@@ -66,6 +67,7 @@ CXLx_CAPABILITY_HEADER(LINK, 2)
 CXLx_CAPABILITY_HEADER(HDM, 3)
 CXLx_CAPABILITY_HEADER(EXTSEC, 4)
 CXLx_CAPABILITY_HEADER(SNOOP, 5)
+CXLx_CAPABILITY_HEADER(BI_DECODER, 6)
 
 /*
  * Capability structures contain the actual registers that the CXL component
@@ -185,9 +187,14 @@ HDM_DECODER_INIT(3);
     (CXL_IDE_REGISTERS_OFFSET + CXL_IDE_REGISTERS_SIZE)
 #define CXL_SNOOP_REGISTERS_SIZE   0x8
 
+/* CXL 3.0 8.2.4.26 - CXL BI Decoder Capability Structure */
+#define CXL_BI_DECODER_REGISTERS_OFFSET \
+    (CXL_SNOOP_REGISTERS_OFFSET + CXL_SNOOP_REGISTERS_SIZE)
+#define CXL_BI_DECODER_REGISTERS_SIZE   0xC
+
 /* CXL 3.0 8.2.3 Table 8-21 */
-QEMU_BUILD_BUG_MSG((CXL_SNOOP_REGISTERS_OFFSET +
-                    CXL_SNOOP_REGISTERS_SIZE) >= CXL2_COMPONENT_CM_REGION_SIZE,
+QEMU_BUILD_BUG_MSG((CXL_BI_DECODER_REGISTERS_OFFSET +
+                    CXL_BI_DECODER_REGISTERS_SIZE) >= CXL2_COMPONENT_CM_REGION_SIZE,
                    "No space for registers");
 
 typedef struct component_registers {

-- 
2.40.0

[PATCH RFC 5/5] hw/cxl: Add UIO HDM decoder register fields

[Ira Weiny]

HDM decoders optionally support Unordered IO (UIO) access.  Devices
indicate UIO support by setting the capable bit.  Software can then set
up to UIO decoder count HDM's as UIO enabled when configuring the HDMs
on the device.

Define the UIO capable bit and decoder count.  Default type 2 devices to
support UIO for testing.

Not-Yet-Signed-off-by: Ira Weiny <ira.weiny@intel.com>
---
 hw/cxl/cxl-component-utils.c   | 6 ++++++
 include/hw/cxl/cxl_component.h | 2 ++
 2 files changed, 8 insertions(+)

diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index a9efa252b4ae..252b2beb2110 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -173,6 +173,12 @@ static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk,
     ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, INTERLEAVE_256B, 1);
     ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, INTERLEAVE_4K, 1);
     ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, POISON_ON_ERR_CAP, 0);
+    if (type == CXL3_TYPE2_DEVICE) {
+        ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, UIO_CAPABLE, 1);
+        ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, UIO_DECODER_CNT,
+                         decoder_count);
+    }
+
     ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_GLOBAL_CONTROL,
                      HDM_DECODER_ENABLE, 0);
     write_msk[R_CXL_HDM_DECODER_GLOBAL_CONTROL] = 0x3;
diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
index a5b5512aed94..7c24e699ef80 100644
--- a/include/hw/cxl/cxl_component.h
+++ b/include/hw/cxl/cxl_component.h
@@ -162,6 +162,8 @@ REG32(CXL_HDM_DECODER_CAPABILITY, CXL_HDM_REGISTERS_OFFSET)
     FIELD(CXL_HDM_DECODER_CAPABILITY, INTERLEAVE_256B, 8, 1)
     FIELD(CXL_HDM_DECODER_CAPABILITY, INTERLEAVE_4K, 9, 1)
     FIELD(CXL_HDM_DECODER_CAPABILITY, POISON_ON_ERR_CAP, 10, 1)
+    FIELD(CXL_HDM_DECODER_CAPABILITY, UIO_CAPABLE, 13, 1)
+    FIELD(CXL_HDM_DECODER_CAPABILITY, UIO_DECODER_CNT, 16, 4)
 REG32(CXL_HDM_DECODER_GLOBAL_CONTROL, CXL_HDM_REGISTERS_OFFSET + 4)
     FIELD(CXL_HDM_DECODER_GLOBAL_CONTROL, POISON_ON_ERR_EN, 0, 1)
     FIELD(CXL_HDM_DECODER_GLOBAL_CONTROL, HDM_DECODER_ENABLE, 1, 1)

-- 
2.40.0

Re: [PATCH RFC 0/5] hw/cxl: Type 2 Device RFC

[Cédric Le Goater]

Hello,

On 5/18/23 04:45, Ira Weiny wrote:
> Type 2 devices are not yet a reality.  Developing core kernel support
> is difficult without some test device to model against.
> 
> Define a type 2 device 'cxl-accel'.  This device is derived from the
> type 3 device and retains all that functionality for now.
> 
> Mock up a couple of accelerator features (Back Invalidate [BI] and
> Unordered IO [UIO]) as examples for the RFC.  These have no
> functionality other than to report the features as present for software
> to key off of.
> 
> Defining these devices in qemu can be done with the following example:
> 
> ...
>    -device cxl-accel,bus=sw0p0,volatile-memdev=cxl-ac-mem5,id=cxl-dev5,sn=0xCAFE0005
> ...
> 
> NOTE: I'm leaving off Michael Tsirkin for now because this is really
> rough and I'm mainly sending it out because it was talked about in the
> CXL community call on 5/16.
> 
> Not-Yet-Signed-off-by: Ira Weiny <ira.weiny@intel.com>


A recent proposal to add support in VFIO for CXL passthrough uses
this emulated device and a sample driver for a proof of concept.
For this effort, it would be very useful to have a QEMU model for
a CXL type2 device, even partially implemented.

I haven't found any updates of this series. What are the plans for
upstream today ?

For vfio-cxl, see :
   
* [RFC] vfio: introduce vfio-cxl to support CXL type-2 accelerator passthrough
   https://lore.kernel.org/kvm/20240920223446.1908673-1-zhiw@nvidia.com
* [RFC] Introduce vfio-cxl to support CXL type-2 device passthrough
   https://lore.kernel.org/all/20240921071440.1915876-1-zhiw@nvidia.com/


Thanks,

C.

   

> ---
> Ira Weiny (5):
>        hw/cxl: Use define for build bug detection
>        hw/cxl: Refactor component register initialization
>        hw/cxl: Derive a CXL accelerator device from Type-3
>        hw/cxl/accel: Add Back-Invalidate decoder capbility structure
>        hw/cxl: Add UIO HDM decoder register fields
> 
>   docs/system/devices/cxl.rst    | 11 ++++++
>   hw/cxl/cxl-component-utils.c   | 80 +++++++++++++++++++-----------------------
>   hw/mem/cxl_type3.c             | 39 ++++++++++++++++++++
>   include/hw/cxl/cxl_component.h | 51 +++++++++++++++++++--------
>   include/hw/cxl/cxl_device.h    | 16 +++++++++
>   include/hw/pci/pci_ids.h       |  1 +
>   6 files changed, 141 insertions(+), 57 deletions(-)
> ---
> base-commit: 8eb2a03258313f404ca0c8609a8f9009b9b4318c
> change-id: 20230517-rfc-type2-dev-c2d661a29d96
> 
> Best regards,

Re: [PATCH RFC 0/5] hw/cxl: Type 2 Device RFC

[Zhi Wang]

On 17/10/2024 19.57, Cédric Le Goater wrote:
> External email: Use caution opening links or attachments
> 
> 
> Hello,
> 
> On 5/18/23 04:45, Ira Weiny wrote:
>> Type 2 devices are not yet a reality.  Developing core kernel support
>> is difficult without some test device to model against.
>>
>> Define a type 2 device 'cxl-accel'.  This device is derived from the
>> type 3 device and retains all that functionality for now.
>>
>> Mock up a couple of accelerator features (Back Invalidate [BI] and
>> Unordered IO [UIO]) as examples for the RFC.  These have no
>> functionality other than to report the features as present for software
>> to key off of.
>>
>> Defining these devices in qemu can be done with the following example:
>>
>> ...
>>    -device 
>> cxl-accel,bus=sw0p0,volatile-memdev=cxl-ac-mem5,id=cxl-dev5,sn=0xCAFE0005
>> ...
>>
>> NOTE: I'm leaving off Michael Tsirkin for now because this is really
>> rough and I'm mainly sending it out because it was talked about in the
>> CXL community call on 5/16.
>>
>> Not-Yet-Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> 
> 
> A recent proposal to add support in VFIO for CXL passthrough uses
> this emulated device and a sample driver for a proof of concept.
> For this effort, it would be very useful to have a QEMU model for
> a CXL type2 device, even partially implemented.
> 
> I haven't found any updates of this series. What are the plans for
> upstream today ?
>

I was discussing this with Ira and Jonathan in the LPC and CXL discord 
groups. (Irq and Jonathan, please feel free to correct me) I think we 
all thought that having the emulated device support in QEMU and a sample 
CXL type-2 driver in the upstream kernel could be valuable for

1) people who wish to contribute and propose changes, like refactor, and 
code tweaks related to CXL type-2 support in the kernel. They can 
validate their patches to check if they break something without a CXL 
type-2 device.

2) people who wish to contribute on solving problems of CXL type-2 
generic code, e.g. loading sequences of modules... They can get involved 
without a real HW.

My gut feeling is I can start to work with folks to get it into the 
mainline together with the sample driver when CXL type-2 support is 
merged. So people can play with it.

3) people who would like to try on vfio-cxl.

What would be nice to include in the patchset V1 in my mind:

(Ira and Jonathan and folks please feel free to comment)

Must have:
  - HDM decoder support (>1 HDM decoder). (which I have done it in my 
tree, so the CXL core can create a CXL region)

Nice to have:
  - CXLType2 device system reset handler. As what mentioned in my cover 
letter, a system reset handler is required to reset the device state. 
Linux kernel CXL core assumes the HW is pre-configured by FW if it sees 
CXL.mem is enabled when enumerating the device. So I have to kill QEMU 
instead of resetting the virtual machine when debugging.

  - CXLType2 device in the patch should be derived from PCIDevice 
(current it is derived from the CXLType3 device, which carries quite 
some unnecessary stuff to present to the guest)

  - Refactor of the current QEMU FWMS code to deliver the guest access 
to the virtual memory backend to the correct emulated CXL device (which 
is currently hardcoded to connect to cxl-type3 device in QEMU. Without 
this, the access to the CXL region from the guest seems problematic but 
creating CXL region is still fine.)

Thanks,
Zhi.

> For vfio-cxl, see :
> 
> * [RFC] vfio: introduce vfio-cxl to support CXL type-2 accelerator 
> passthrough
>    https://lore.kernel.org/kvm/20240920223446.1908673-1-zhiw@nvidia.com
> * [RFC] Introduce vfio-cxl to support CXL type-2 device passthrough
>    https://lore.kernel.org/all/20240921071440.1915876-1-zhiw@nvidia.com/
> 
> 
> Thanks,
> 
> C.
> 
> 
> 
>> ---
>> Ira Weiny (5):
>>        hw/cxl: Use define for build bug detection
>>        hw/cxl: Refactor component register initialization
>>        hw/cxl: Derive a CXL accelerator device from Type-3
>>        hw/cxl/accel: Add Back-Invalidate decoder capbility structure
>>        hw/cxl: Add UIO HDM decoder register fields
>>
>>   docs/system/devices/cxl.rst    | 11 ++++++
>>   hw/cxl/cxl-component-utils.c   | 80 
>> +++++++++++++++++++-----------------------
>>   hw/mem/cxl_type3.c             | 39 ++++++++++++++++++++
>>   include/hw/cxl/cxl_component.h | 51 +++++++++++++++++++--------
>>   include/hw/cxl/cxl_device.h    | 16 +++++++++
>>   include/hw/pci/pci_ids.h       |  1 +
>>   6 files changed, 141 insertions(+), 57 deletions(-)
>> ---
>> base-commit: 8eb2a03258313f404ca0c8609a8f9009b9b4318c
>> change-id: 20230517-rfc-type2-dev-c2d661a29d96
>>
>> Best regards,
> 

Re: [PATCH RFC 0/5] hw/cxl: Type 2 Device RFC

[Alejandro Lucero Palau]

On 10/18/24 15:49, Zhi Wang wrote:
> On 17/10/2024 19.57, Cédric Le Goater wrote:
>> External email: Use caution opening links or attachments
>>
>>
>> Hello,
>>
>> On 5/18/23 04:45, Ira Weiny wrote:
>>> Type 2 devices are not yet a reality.  Developing core kernel support
>>> is difficult without some test device to model against.
>>>
>>> Define a type 2 device 'cxl-accel'.  This device is derived from the
>>> type 3 device and retains all that functionality for now.
>>>
>>> Mock up a couple of accelerator features (Back Invalidate [BI] and
>>> Unordered IO [UIO]) as examples for the RFC.  These have no
>>> functionality other than to report the features as present for software
>>> to key off of.
>>>
>>> Defining these devices in qemu can be done with the following example:
>>>
>>> ...
>>>     -device
>>> cxl-accel,bus=sw0p0,volatile-memdev=cxl-ac-mem5,id=cxl-dev5,sn=0xCAFE0005
>>> ...
>>>
>>> NOTE: I'm leaving off Michael Tsirkin for now because this is really
>>> rough and I'm mainly sending it out because it was talked about in the
>>> CXL community call on 5/16.
>>>
>>> Not-Yet-Signed-off-by: Ira Weiny <ira.weiny@intel.com>
>>
>> A recent proposal to add support in VFIO for CXL passthrough uses
>> this emulated device and a sample driver for a proof of concept.
>> For this effort, it would be very useful to have a QEMU model for
>> a CXL type2 device, even partially implemented.
>>
>> I haven't found any updates of this series. What are the plans for
>> upstream today ?
>>
> I was discussing this with Ira and Jonathan in the LPC and CXL discord
> groups. (Irq and Jonathan, please feel free to correct me) I think we
> all thought that having the emulated device support in QEMU and a sample
> CXL type-2 driver in the upstream kernel could be valuable for
>
> 1) people who wish to contribute and propose changes, like refactor, and
> code tweaks related to CXL type-2 support in the kernel. They can
> validate their patches to check if they break something without a CXL
> type-2 device.
>
> 2) people who wish to contribute on solving problems of CXL type-2
> generic code, e.g. loading sequences of modules... They can get involved
> without a real HW.
>
> My gut feeling is I can start to work with folks to get it into the
> mainline together with the sample driver when CXL type-2 support is
> merged. So people can play with it.


I did use an emulated Type2 device for my initial development and I'm 
still using it for wider testing. It is basically same than the Type3 
with small changes. I think a proper solution should include command 
line options for configuring the device with flexibility or maybe 
referring to a file with that specific configuration. Note there exist 
optional functionalities like HDM decoder, and CXL.cache will need such 
a flexibility as well.

The RFC contains the driver for such emulated device implemented inside 
the tools/testing/cxl directory, although it has changed since then, but 
happy to share it. It is almost equal to the code inside efx_cxl.c along 
with the functions/macros for defining the driver.

FWIW, although I'm working on Type2 support, I really think qemu could 
help for development and testing complex CXL infrastructures, more for 
memory expanders aka Type3 than Type2. I know this requires a good 
effort but IMO, it will pay off.


> 3) people who would like to try on vfio-cxl.
>
> What would be nice to include in the patchset V1 in my mind:
>
> (Ira and Jonathan and folks please feel free to comment)
>
> Must have:
>    - HDM decoder support (>1 HDM decoder). (which I have done it in my
> tree, so the CXL core can create a CXL region)
>
> Nice to have:
>    - CXLType2 device system reset handler. As what mentioned in my cover
> letter, a system reset handler is required to reset the device state.
> Linux kernel CXL core assumes the HW is pre-configured by FW if it sees
> CXL.mem is enabled when enumerating the device. So I have to kill QEMU
> instead of resetting the virtual machine when debugging.
>
>    - CXLType2 device in the patch should be derived from PCIDevice
> (current it is derived from the CXLType3 device, which carries quite
> some unnecessary stuff to present to the guest)
>
>    - Refactor of the current QEMU FWMS code to deliver the guest access
> to the virtual memory backend to the correct emulated CXL device (which
> is currently hardcoded to connect to cxl-type3 device in QEMU. Without
> this, the access to the CXL region from the guest seems problematic but
> creating CXL region is still fine.)
>
> Thanks,
> Zhi.
>
>> For vfio-cxl, see :
>>
>> * [RFC] vfio: introduce vfio-cxl to support CXL type-2 accelerator
>> passthrough
>>     https://lore.kernel.org/kvm/20240920223446.1908673-1-zhiw@nvidia.com
>> * [RFC] Introduce vfio-cxl to support CXL type-2 device passthrough
>>     https://lore.kernel.org/all/20240921071440.1915876-1-zhiw@nvidia.com/
>>
>>
>> Thanks,
>>
>> C.
>>
>>
>>
>>> ---
>>> Ira Weiny (5):
>>>         hw/cxl: Use define for build bug detection
>>>         hw/cxl: Refactor component register initialization
>>>         hw/cxl: Derive a CXL accelerator device from Type-3
>>>         hw/cxl/accel: Add Back-Invalidate decoder capbility structure
>>>         hw/cxl: Add UIO HDM decoder register fields
>>>
>>>    docs/system/devices/cxl.rst    | 11 ++++++
>>>    hw/cxl/cxl-component-utils.c   | 80
>>> +++++++++++++++++++-----------------------
>>>    hw/mem/cxl_type3.c             | 39 ++++++++++++++++++++
>>>    include/hw/cxl/cxl_component.h | 51 +++++++++++++++++++--------
>>>    include/hw/cxl/cxl_device.h    | 16 +++++++++
>>>    include/hw/pci/pci_ids.h       |  1 +
>>>    6 files changed, 141 insertions(+), 57 deletions(-)
>>> ---
>>> base-commit: 8eb2a03258313f404ca0c8609a8f9009b9b4318c
>>> change-id: 20230517-rfc-type2-dev-c2d661a29d96
>>>
>>> Best regards,

Re: [PATCH RFC 0/5] hw/cxl: Type 2 Device RFC

[Jonathan Cameron via]

On Fri, 18 Oct 2024 16:25:58 +0100
Alejandro Lucero Palau <alucerop@amd.com> wrote:

> On 10/18/24 15:49, Zhi Wang wrote:
> > On 17/10/2024 19.57, Cédric Le Goater wrote:  
> >> External email: Use caution opening links or attachments
> >>
> >>
> >> Hello,
> >>
> >> On 5/18/23 04:45, Ira Weiny wrote:  
> >>> Type 2 devices are not yet a reality.  Developing core kernel support
> >>> is difficult without some test device to model against.
> >>>
> >>> Define a type 2 device 'cxl-accel'.  This device is derived from the
> >>> type 3 device and retains all that functionality for now.
> >>>
> >>> Mock up a couple of accelerator features (Back Invalidate [BI] and
> >>> Unordered IO [UIO]) as examples for the RFC.  These have no
> >>> functionality other than to report the features as present for software
> >>> to key off of.
> >>>
> >>> Defining these devices in qemu can be done with the following example:
> >>>
> >>> ...
> >>>     -device
> >>> cxl-accel,bus=sw0p0,volatile-memdev=cxl-ac-mem5,id=cxl-dev5,sn=0xCAFE0005
> >>> ...
> >>>
> >>> NOTE: I'm leaving off Michael Tsirkin for now because this is really
> >>> rough and I'm mainly sending it out because it was talked about in the
> >>> CXL community call on 5/16.
> >>>
> >>> Not-Yet-Signed-off-by: Ira Weiny <ira.weiny@intel.com>  
> >>
> >> A recent proposal to add support in VFIO for CXL passthrough uses
> >> this emulated device and a sample driver for a proof of concept.
> >> For this effort, it would be very useful to have a QEMU model for
> >> a CXL type2 device, even partially implemented.
> >>
> >> I haven't found any updates of this series. What are the plans for
> >> upstream today ?
> >>  
> > I was discussing this with Ira and Jonathan in the LPC and CXL discord
> > groups. (Irq and Jonathan, please feel free to correct me) I think we
> > all thought that having the emulated device support in QEMU and a sample
> > CXL type-2 driver in the upstream kernel could be valuable for
> >
> > 1) people who wish to contribute and propose changes, like refactor, and
> > code tweaks related to CXL type-2 support in the kernel. They can
> > validate their patches to check if they break something without a CXL
> > type-2 device.
> >
> > 2) people who wish to contribute on solving problems of CXL type-2
> > generic code, e.g. loading sequences of modules... They can get involved
> > without a real HW.
> >
> > My gut feeling is I can start to work with folks to get it into the
> > mainline together with the sample driver when CXL type-2 support is
> > merged. So people can play with it.  
> 
> 
> I did use an emulated Type2 device for my initial development and I'm 
> still using it for wider testing. It is basically same than the Type3 
> with small changes. I think a proper solution should include command 
> line options for configuring the device with flexibility or maybe 
> referring to a file with that specific configuration. Note there exist 
> optional functionalities like HDM decoder, and CXL.cache will need such 
> a flexibility as well.

In the first instance, I'd just make up a single reasonable configuration.
Later, it may make sense to make it flexible.


> 
> The RFC contains the driver for such emulated device implemented inside 
> the tools/testing/cxl directory, although it has changed since then, but 
> happy to share it. It is almost equal to the code inside efx_cxl.c along 
> with the functions/macros for defining the driver.
> 
> FWIW, although I'm working on Type2 support, I really think qemu could 
> help for development and testing complex CXL infrastructures, more for 
> memory expanders aka Type3 than Type2. I know this requires a good 
> effort but IMO, it will pay off.

I'm definitely not against adding support and Zhi's rough sketch sounds
about right.

> 
> > 3) people who would like to try on vfio-cxl.
> >
> > What would be nice to include in the patchset V1 in my mind:
> >
> > (Ira and Jonathan and folks please feel free to comment)
> >
> > Must have:
> >    - HDM decoder support (>1 HDM decoder). (which I have done it in my
> > tree, so the CXL core can create a CXL region)
> >
> > Nice to have:
> >    - CXLType2 device system reset handler. As what mentioned in my cover
> > letter, a system reset handler is required to reset the device state.
> > Linux kernel CXL core assumes the HW is pre-configured by FW if it sees
> > CXL.mem is enabled when enumerating the device. So I have to kill QEMU
> > instead of resetting the virtual machine when debugging.
> >
> >    - CXLType2 device in the patch should be derived from PCIDevice
> > (current it is derived from the CXLType3 device, which carries quite
> > some unnecessary stuff to present to the guest)
> >
> >    - Refactor of the current QEMU FWMS code to deliver the guest access
> > to the virtual memory backend to the correct emulated CXL device (which
> > is currently hardcoded to connect to cxl-type3 device in QEMU. Without
> > this, the access to the CXL region from the guest seems problematic but
> > creating CXL region is still fine.)
> >
> > Thanks,
> > Zhi.
> >  
> >> For vfio-cxl, see :
> >>
> >> * [RFC] vfio: introduce vfio-cxl to support CXL type-2 accelerator
> >> passthrough
> >>     https://lore.kernel.org/kvm/20240920223446.1908673-1-zhiw@nvidia.com
> >> * [RFC] Introduce vfio-cxl to support CXL type-2 device passthrough
> >>     https://lore.kernel.org/all/20240921071440.1915876-1-zhiw@nvidia.com/
> >>
> >>
> >> Thanks,
> >>
> >> C.
> >>
> >>
> >>  
> >>> ---
> >>> Ira Weiny (5):
> >>>         hw/cxl: Use define for build bug detection
> >>>         hw/cxl: Refactor component register initialization
> >>>         hw/cxl: Derive a CXL accelerator device from Type-3
> >>>         hw/cxl/accel: Add Back-Invalidate decoder capbility structure
> >>>         hw/cxl: Add UIO HDM decoder register fields
> >>>
> >>>    docs/system/devices/cxl.rst    | 11 ++++++
> >>>    hw/cxl/cxl-component-utils.c   | 80
> >>> +++++++++++++++++++-----------------------
> >>>    hw/mem/cxl_type3.c             | 39 ++++++++++++++++++++
> >>>    include/hw/cxl/cxl_component.h | 51 +++++++++++++++++++--------
> >>>    include/hw/cxl/cxl_device.h    | 16 +++++++++
> >>>    include/hw/pci/pci_ids.h       |  1 +
> >>>    6 files changed, 141 insertions(+), 57 deletions(-)
> >>> ---
> >>> base-commit: 8eb2a03258313f404ca0c8609a8f9009b9b4318c
> >>> change-id: 20230517-rfc-type2-dev-c2d661a29d96
> >>>
> >>> Best regards,