Re: [ISSUE]: cxl create region with one memdev fails when we have more than one HBs in the cxl topology setup

[Fan Ni <nifan.cxl@gmail.com>]

On Fri, Mar 07, 2025 at 12:28:30PM -0800, Dan Williams wrote:
> Fan Ni wrote:
> > [ISSUE]: Create region with one memdev failed when we have 2 HBs in cxl topology setup
> > 
> > I create a cxl topology with 2 HBs like below:
> >              CFMW
> >            /    \
> >         HB0     HB1
> >          |       |
> >         RP0     RP1
> >          |       |
> >         pmem    ram
> > 
> > qemu command line:
> > 
> > /home/fan/cxl/qemu/build/qemu-system-x86_64 -s  -kernel /home/fan/cxl/linux-fixes//arch/x86/boot/bzImage -append "root=/dev/sda rw console=ttyS0,115200 ignore_loglevel nokaslr cxl_acpi.dyndbg=+fplm cxl_pci.dyndbg=+fplm cxl_core.dyndbg=+fplm cxl_mem.dyndbg=+fplm cxl_pmem.dyndbg=+fplm cxl_port.dyndbg=+fplm cxl_region.dyndbg=+fplm cxl_test.dyndbg=+fplm cxl_mock.dyndbg=+fplm cxl_mock_mem.dyndbg=+fplm dax.dyndbg=+fplm dax_cxl.dyndbg=+fplm device_dax.dyndbg=+fplm" -smp 8 -accel kvm -serial mon:stdio  -nographic  -qmp tcp:localhost:4445,server,wait=off -netdev user,id=network0,hostfwd=tcp::2024-:22 -device e1000,netdev=network0  -monitor telnet:127.0.0.1:12346,server,nowait -drive file=/home/fan/cxl/images/qemu-image.img,index=0,media=disk,format=raw -machine q35,cxl=on -cpu qemu64,mce=on -m 8G,maxmem=64G,slots=8  -virtfs local,path=/opt/lib/modules,mount_tag=modshare,security_model=mapped  -virtfs local,path=/home/fan,mount_tag=homeshare,security_model=mapped -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 -device cxl-rp,port=13,bus=cxl.1,id=root_port13,chassis=0,slot=2 -object memory-backend-file,id=hmem0,share=on,mem-path=/tmp/host//hmem0.raw,size=512M -object memory-backend-file,id=lsa0,share=on,mem-path=/tmp/host//lsa0.raw,size=2M -device cxl-type3,bus=root_port13,memdev=hmem0,lsa=lsa0,id=cxl-memdev0,sn=3840 -device pxb-cxl,bus_nr=112,bus=pcie.0,id=cxl.2 -device cxl-rp,port=14,bus=cxl.2,id=root_port14,chassis=0,slot=3 -object memory-backend-file,id=hmem1,share=on,mem-path=/tmp/host//hmem1.raw,size=512M -device cxl-type3,bus=root_port14,volatile-memdev=hmem1,id=cxl-vmemdev1,sn=3841 -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8K 
> > 
> > 
> > -----------------
> > # cxl create-region -m -d decoder0.0 -w 1 mem1 -s 512M -t pmem
> > cxl region: create_region: region0: set_interleave_ways failed: Invalid argument
> > cxl region: cmd_create_region: created 0 regions
> > -----------------
> > 
> > The reason is for the root decoder, iw = 2, while for creating the region as
> > above, number of ways is 1, which fails the check here:
> > 
> > https://web.git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/tree/drivers/cxl/core/region.c?h=fixes#n491
> > 
> > -----------------
> > 	if (!is_power_of_2(val / cxld->interleave_ways) ||
> > 	    (val % cxld->interleave_ways)) {
> > 		dev_dbg(&cxlr->dev, "invalid interleave: %d\n", val);
> > 		return -EINVAL;
> > 	}
> > -----------------
> > 491		if (!is_power_of_2(val / cxld->interleave_ways) ||
> > (gdb) n
> > 493			dev_dbg(&cxlr->dev, "invalid interleave: %d\n", val);
> > (gdb) p val
> > $8 = 1
> > (gdb) p cxld->interleave_ways
> > $9 = 2
> > 
> > So how can we create a region with only one memdev when there are more than 1 HBs,
> > but we only want to touch device under one HB?
> 
> Then you would need a new CFMWS entry to allow that. On hardware
> platforms with multiple host bridges you will see a set like this
> following for a 2 host bridge platform with RAM + PMEM support:
> 
> CFMWS0: HB0, IW = 1, RAM
> CFMWS1: HB1, IW = 1, RAM
> CFMWS2: HB0, IW = 1, PMEM
> CFMWS3: HB1, IW = 1, PMEM
> CFMWS4: HB0 + HB1, IW = 2, RAM
> CFMWS5: HB0 + HB1, IW = 2, PMEM
> 
> So, in this case it looks like your CFMWS set ("decoders:root0") only
> includes something like CFMWS5, so you need to update QEMU to produce
> one or more of CFMWS[0-3] windows.

Thanks Dan for the detailed explanation. It is much clearer to me now
how this works.

Fan
> 
> [..]
> >     "decoders:root0":[
> >       {
> >         "decoder":"decoder0.0",
> >         "resource":79725330432,
> >         "size":4294967296,
> >         "interleave_ways":2,
> >         "interleave_granularity":8192,
> >         "max_available_extent":4294967296,
> >         "pmem_capable":true,
> >         "volatile_capable":true,
> >         "accelmem_capable":true,
> >         "qos_class":0,
> >         "nr_targets":2,
> >         "targets":[
> >           {
> >             "target":"pci0000:70",
> >             "alias":"ACPI0016:00",
> >             "position":1,
> >             "id":112
> >           },
> >           {
> >             "target":"pci0000:0c",
> >             "alias":"ACPI0016:01",
> >             "position":0,
> >             "id":12
> >           }
> >         ]
> >       }
> >     ]