Re: [Lsf-pc] [LSF/MM] CXL Boot to Bash - Section 4: Interleave

["Yuquan Wang" <wangyuquan1236@phytium.com.cn>]

[-- Attachment #1: Type: text/plain, Size: 4831 bytes --]

On 2025-03-27 10:51,  dan.j.williams, gourry wrote:
Yuquan Wang wrote:
> On Wed, Mar 26, 2025 at 08:53:14AM -0400, Gregory Price wrote:
> > On Wed, Mar 26, 2025 at 05:28:00PM +0800, Yuquan Wang wrote:
> > > > 
> > > > Notice at both the root and the host bridge, the Interleave Ways is 2.
> > > > There are two targets at each level.  The host bridge has a granularity
> > > > of 512 to capture its parent's ways and granularity (`2*256`).
> > > > 
> > > > Each decoder is programmed with the total number of targets (4) and the
> > > > overall granularity (256B).
> > > >
> > > 
> > > Sorry, I tried to set this topology on Qemu Virt and used:
> > > "cxl create-region -d decoder0.0 -t ram -m mem0,mem1,mem2,mem3"
> > > 
> > > but it failed with:
> > > "cxl region: validate_ways: Interleave ways 2 is less than number of memdevs specified: 4"
> > > 
> > > It seems like the CFMWs IW should be 4?
> > >
> > 
> > It has been a while since i've interacted with QEMU's interleave stuff,
> > but IIRC (at least back when I was working on it) most configurations
> > had 1 device per host bridge - in which case the CFMWS IW should be 4
> > with each of the host bridges described in it.
> > 
> > I'm not sure you can do multiple devices per host bridge without a
> > switch setup.
> >
> Qemu counld add 'cxl-rp' under a cxl host bridge. Below is my qemu
> command:
> 
> -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
> -device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \
> -device cxl-rp,port=1,bus=cxl.1,id=root_port1,chassis=0,slot=1 \
> -device cxl-type3,bus=root_port0,volatile-memdev=mem2,id=cxl-mem1 \
> -device cxl-type3,bus=root_port1,volatile-memdev=mem3,id=cxl-mem2 \
> -device pxb-cxl,bus_nr=20,bus=pcie.0,id=cxl.2 \
> -device cxl-rp,port=2,bus=cxl.2,id=root_port2,chassis=0,slot=2 \
> -device cxl-rp,port=3,bus=cxl.2,id=root_port3,chassis=0,slot=3 \
> -device cxl-type3,bus=root_port2,volatile-memdev=mem4,id=cxl-mem3 \
> -device cxl-type3,bus=root_port3,volatile-memdev=mem5,id=cxl-mem4 \
> -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=2G \
> 
> My lspci shows:
> 
> -+-[0000:00]-+-00.0  Red Hat, Inc. QEMU PCIe Host bridge
>  |           +-01.0  Red Hat, Inc. Virtio network device
>  |           +-02.0  Red Hat, Inc. Virtio block device
>  |           +-03.0  Red Hat, Inc. QEMU PCIe Expander bridge
>  |           \-04.0  Red Hat, Inc. QEMU PCIe Expander bridge
>  +-[0000:0c]-+-00.0-[0d]----00.0  Intel Corporation Device 0d93 (CXL)
>  |           \-01.0-[0e]----00.0  Intel Corporation Device 0d93 (CXL)
>  \-[0000:14]-+-00.0-[15]----00.0  Intel Corporation Device 0d93 (CXL)
>              \-01.0-[16]----00.0  Intel Corporation Device 0d93 (CXL)
> 
> My cxl list shows:
> [
>   {
>     "memdev":"mem1",
>     "ram_size":268435456,
>     "serial":0,
>     "host":"0000:15:00.0"
>   },
>   {
>     "memdev":"mem0",
>     "ram_size":268435456,
>     "serial":0,
>     "host":"0000:16:00.0"
>   },
>   {
>     "memdev":"mem2",
>     "ram_size":268435456,
>     "serial":0,
>     "host":"0000:0e:00.0"
>   },
>   {
>     "memdev":"mem3",
>     "ram_size":268435456,
>     "serial":0,
>     "host":"0000:0d:00.0"
>   }
> ]
> 
> Then:
> 
> # cxl create-region -d decoder0.0 -t ram -m mem0,mem1,mem2,mem3
> cxl region: validate_ways: Interleave ways 2 is less than number of memdevs specified: 4
> cxl region: cmd_create_region: created 0 regions
> 
> This case confuesed me :(
 
What is the output of:
 
    cxl list -M -d decoder0.0
 
...my expectation is that it only finds 2 potential endpoint devices
that are mapped by that decoder, I.e. QEMU did not produce a CFMWS that
interleaves the 2 host-bridges.
 
The error message could be improved to clarify that only endpoints
mapped by the given decoder are candidates to be members of a region.

# cxl list -d decoder0.0
[
  {
    "decoder":"decoder0.0",
    "resource":1099511627776,
    "size":2147483648,
    "interleave_ways":2,
    "interleave_granularity":256,
    "max_available_extent":2147483648,
    "pmem_capable":true,
    "volatile_capable":true,
    "accelmem_capable":true,
    "qos_class":0,
    "nr_targets":2
  }
]

CFMWS:

              Subtable Type : 01 [CXL Fixed Memory Window Structure]
              Reserved : 00
              Length : 002C
              Reserved : 00000000
              Window base address : 0000010000000000
              Window size : 0000000080000000
              Interleave Members : 01
              Interleave Arithmetic : 00
              Reserved : 0000
              Granularity : 00000000
              Restrictions : 000F
              QtgId : 0000
              First Target : 0000000C
              Next Target : 00000014

Yuquan

[-- Attachment #2: Type: text/html, Size: 9104 bytes --]