[PATCH] hw/acpi/cxl: Drop device-memory support from CFMWS entries

[PATCH] hw/acpi/cxl: Drop device-memory support from CFMWS entries

[Dan Williams]

While it was a reasonable idea to specify no window restricitions at the
outset of the CXL emulation support, it turns out that in practice a
platform will never follow the QEMU example of specifying simultaneous
support for HDM-H and HDM-D[B] in a single window.

HDM-D mandates extra bus cycles for host/device bias protocol, and HDM-DB
mandates extra bus cycles for back-invalidate protocol, so hardware must
be explicitly prepared for device-memory unlike host-only memory
(HDM-H).

In preparation for the kernel dropping support for windows that do not
select between device and host-only memory, move QEMU exclusively to
declaring host-only windows.

Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
 hw/acpi/cxl.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/hw/acpi/cxl.c b/hw/acpi/cxl.c
index 2bf8c0799359..defb289e2fef 100644
--- a/hw/acpi/cxl.c
+++ b/hw/acpi/cxl.c
@@ -103,8 +103,8 @@ static void cedt_build_cfmws(GArray *table_data, CXLState *cxls)
         /* Host Bridge Interleave Granularity */
         build_append_int_noprefix(table_data, fw->enc_int_gran, 4);
 
-        /* Window Restrictions */
-        build_append_int_noprefix(table_data, 0x0f, 2); /* No restrictions */
+        /* Window Restrictions: Host-only ram and pmem */
+        build_append_int_noprefix(table_data, 0x0e, 2);
 
         /* QTG ID */
         build_append_int_noprefix(table_data, 0, 2);

Re: [PATCH] hw/acpi/cxl: Drop device-memory support from CFMWS entries

[Jonathan Cameron via]

On Mon, 20 Mar 2023 23:08:31 -0700
Dan Williams <dan.j.williams@intel.com> wrote:

> While it was a reasonable idea to specify no window restricitions at the
> outset of the CXL emulation support, it turns out that in practice a
> platform will never follow the QEMU example of specifying simultaneous
> support for HDM-H and HDM-D[B] in a single window.
> 
> HDM-D mandates extra bus cycles for host/device bias protocol, and HDM-DB
> mandates extra bus cycles for back-invalidate protocol, so hardware must
> be explicitly prepared for device-memory unlike host-only memory
> (HDM-H).
> 
> In preparation for the kernel dropping support for windows that do not
> select between device and host-only memory, move QEMU exclusively to
> declaring host-only windows.
> 
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Hi Dan,

Can we have some spec references? I think the Protocol tables in
appendix C would work for that - but more specific examples called
out from them would be good.

I'm also not totally convinced it isn't a host implementation detail
- key here is that host bridge's are still part of the host so can
do impdef stuff as long as they look correct to CXL side and to
host side.

Time for some wild implementation conjecturing.

Imagine a host that has host bridges of above average smartness.
Those host bridges have HDM decoders (this doesn't work if not)

Host is using a single HPA window for HDM-D[B] and HDM-H.
The host bridge knows the target is HDM-H - it can get that form
the HDM decoder Target Type bit etc.  The HB can send (to the
rest of the Host) whatever replies are necessary / fill in extra
info to make it look like HDM-D[B] to the host interconnect protocol.

(after some fun with a white board we think you can make this efficient
 by potentially making the Host bridge HDM decoder setup visible to
 other parts of the host - relatively easy give lots of time allowed
 for a decoder commit).

Why would you do this?  Limited HPA space availability on the host
and wanting to be very flexible about use of the CXL windows.

Obviously this is all moot if there is a constraint we can point to
in a specification.

BTW. I'm carrying a patch (it's on the gitlab tree) that I haven't
posted yet that lets you configure this restriction at runtime as
a similar potential host implementation restriction occurs for
PMEM vs Volatile.  That is also needed to exercise the fun corners of
QTG etc.

Jonathan

> ---
>  hw/acpi/cxl.c |    4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/acpi/cxl.c b/hw/acpi/cxl.c
> index 2bf8c0799359..defb289e2fef 100644
> --- a/hw/acpi/cxl.c
> +++ b/hw/acpi/cxl.c
> @@ -103,8 +103,8 @@ static void cedt_build_cfmws(GArray *table_data, CXLState *cxls)
>          /* Host Bridge Interleave Granularity */
>          build_append_int_noprefix(table_data, fw->enc_int_gran, 4);
>  
> -        /* Window Restrictions */
> -        build_append_int_noprefix(table_data, 0x0f, 2); /* No restrictions */
> +        /* Window Restrictions: Host-only ram and pmem */
> +        build_append_int_noprefix(table_data, 0x0e, 2);
>  
>          /* QTG ID */
>          build_append_int_noprefix(table_data, 0, 2);
> 
> 

Re: [PATCH] hw/acpi/cxl: Drop device-memory support from CFMWS entries

[Dan Williams]

Jonathan Cameron wrote:
> On Mon, 20 Mar 2023 23:08:31 -0700
> Dan Williams <dan.j.williams@intel.com> wrote:
> 
> > While it was a reasonable idea to specify no window restricitions at the
> > outset of the CXL emulation support, it turns out that in practice a
> > platform will never follow the QEMU example of specifying simultaneous
> > support for HDM-H and HDM-D[B] in a single window.
> > 
> > HDM-D mandates extra bus cycles for host/device bias protocol, and HDM-DB
> > mandates extra bus cycles for back-invalidate protocol, so hardware must
> > be explicitly prepared for device-memory unlike host-only memory
> > (HDM-H).
> > 
> > In preparation for the kernel dropping support for windows that do not
> > select between device and host-only memory, move QEMU exclusively to
> > declaring host-only windows.
> > 
> > Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> Hi Dan,
> 
> Can we have some spec references? I think the Protocol tables in
> appendix C would work for that - but more specific examples called
> out from them would be good.

I presume our messages crossed in the ether:

https://lore.kernel.org/linux-cxl/641a018ed7fb8_269929455@dwillia2-xfh.jf.intel.com.notmuch/

...but yes, if I was still committed to this change, which I am not at
this point, some references from Appendix C and "3.3.11 Forward Progress
and Ordering Rules would be appropriate".

> I'm also not totally convinced it isn't a host implementation detail
> - key here is that host bridge's are still part of the host so can
> do impdef stuff as long as they look correct to CXL side and to
> host side.
> 
> Time for some wild implementation conjecturing.
> 
> Imagine a host that has host bridges of above average smartness.
> Those host bridges have HDM decoders (this doesn't work if not)
> 
> Host is using a single HPA window for HDM-D[B] and HDM-H.
> The host bridge knows the target is HDM-H - it can get that form
> the HDM decoder Target Type bit etc.  The HB can send (to the
> rest of the Host) whatever replies are necessary / fill in extra
> info to make it look like HDM-D[B] to the host interconnect protocol.
> 
> (after some fun with a white board we think you can make this efficient
>  by potentially making the Host bridge HDM decoder setup visible to
>  other parts of the host - relatively easy give lots of time allowed
>  for a decoder commit).
> 
> Why would you do this?  Limited HPA space availability on the host
> and wanting to be very flexible about use of the CXL windows.

Limited HPA space and fewer decode rules at the root level indeed sounds
compelling.

> Obviously this is all moot if there is a constraint we can point to
> in a specification.

At this time I don't have such a reference.

> BTW. I'm carrying a patch (it's on the gitlab tree) that I haven't
> posted yet that lets you configure this restriction at runtime as
> a similar potential host implementation restriction occurs for
> PMEM vs Volatile.  That is also needed to exercise the fun corners of
> QTG etc.

Sounds good.

RE: [PATCH] hw/acpi/cxl: Drop device-memory support from CFMWS entries

[Dan Williams]

Dan Williams wrote:
> While it was a reasonable idea to specify no window restricitions at the
> outset of the CXL emulation support, it turns out that in practice a
> platform will never follow the QEMU example of specifying simultaneous
> support for HDM-H and HDM-D[B] in a single window.
> 
> HDM-D mandates extra bus cycles for host/device bias protocol, and HDM-DB
> mandates extra bus cycles for back-invalidate protocol, so hardware must
> be explicitly prepared for device-memory unlike host-only memory
> (HDM-H).
> 
> In preparation for the kernel dropping support for windows that do not
> select between device and host-only memory, move QEMU exclusively to
> declaring host-only windows.

After an offline discussion determined that a sufficiently sophisticated
platform might be able to support mixed HDM-H and HDM-D[B] in the same
window, so the kernel is not going to drop this support.