Re: [ISSUE]: cxl create region with one memdev fails when we have more than one HBs in the cxl topology setup

[Dan Williams <dan.j.williams@intel.com>]

Fan Ni wrote:
> [ISSUE]: Create region with one memdev failed when we have 2 HBs in cxl topology setup
> 
> I create a cxl topology with 2 HBs like below:
>              CFMW
>            /    \
>         HB0     HB1
>          |       |
>         RP0     RP1
>          |       |
>         pmem    ram
> 
> qemu command line:
> 
> /home/fan/cxl/qemu/build/qemu-system-x86_64 -s  -kernel /home/fan/cxl/linux-fixes//arch/x86/boot/bzImage -append "root=/dev/sda rw console=ttyS0,115200 ignore_loglevel nokaslr cxl_acpi.dyndbg=+fplm cxl_pci.dyndbg=+fplm cxl_core.dyndbg=+fplm cxl_mem.dyndbg=+fplm cxl_pmem.dyndbg=+fplm cxl_port.dyndbg=+fplm cxl_region.dyndbg=+fplm cxl_test.dyndbg=+fplm cxl_mock.dyndbg=+fplm cxl_mock_mem.dyndbg=+fplm dax.dyndbg=+fplm dax_cxl.dyndbg=+fplm device_dax.dyndbg=+fplm" -smp 8 -accel kvm -serial mon:stdio  -nographic  -qmp tcp:localhost:4445,server,wait=off -netdev user,id=network0,hostfwd=tcp::2024-:22 -device e1000,netdev=network0  -monitor telnet:127.0.0.1:12346,server,nowait -drive file=/home/fan/cxl/images/qemu-image.img,index=0,media=disk,format=raw -machine q35,cxl=on -cpu qemu64,mce=on -m 8G,maxmem=64G,slots=8  -virtfs local,path=/opt/lib/modules,mount_tag=modshare,security_model=mapped  -virtfs local,path=/home/fan,mount_tag=homeshare,security_model=mapped -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 -device cxl-rp,port=13,bus=cxl.1,id=root_port13,chassis=0,slot=2 -object memory-backend-file,id=hmem0,share=on,mem-path=/tmp/host//hmem0.raw,size=512M -object memory-backend-file,id=lsa0,share=on,mem-path=/tmp/host//lsa0.raw,size=2M -device cxl-type3,bus=root_port13,memdev=hmem0,lsa=lsa0,id=cxl-memdev0,sn=3840 -device pxb-cxl,bus_nr=112,bus=pcie.0,id=cxl.2 -device cxl-rp,port=14,bus=cxl.2,id=root_port14,chassis=0,slot=3 -object memory-backend-file,id=hmem1,share=on,mem-path=/tmp/host//hmem1.raw,size=512M -device cxl-type3,bus=root_port14,volatile-memdev=hmem1,id=cxl-vmemdev1,sn=3841 -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8K 
> 
> 
> -----------------
> # cxl create-region -m -d decoder0.0 -w 1 mem1 -s 512M -t pmem
> cxl region: create_region: region0: set_interleave_ways failed: Invalid argument
> cxl region: cmd_create_region: created 0 regions
> -----------------
> 
> The reason is for the root decoder, iw = 2, while for creating the region as
> above, number of ways is 1, which fails the check here:
> 
> https://web.git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/tree/drivers/cxl/core/region.c?h=fixes#n491
> 
> -----------------
> 	if (!is_power_of_2(val / cxld->interleave_ways) ||
> 	    (val % cxld->interleave_ways)) {
> 		dev_dbg(&cxlr->dev, "invalid interleave: %d\n", val);
> 		return -EINVAL;
> 	}
> -----------------
> 491		if (!is_power_of_2(val / cxld->interleave_ways) ||
> (gdb) n
> 493			dev_dbg(&cxlr->dev, "invalid interleave: %d\n", val);
> (gdb) p val
> $8 = 1
> (gdb) p cxld->interleave_ways
> $9 = 2
> 
> So how can we create a region with only one memdev when there are more than 1 HBs,
> but we only want to touch device under one HB?

Then you would need a new CFMWS entry to allow that. On hardware
platforms with multiple host bridges you will see a set like this
following for a 2 host bridge platform with RAM + PMEM support:

CFMWS0: HB0, IW = 1, RAM
CFMWS1: HB1, IW = 1, RAM
CFMWS2: HB0, IW = 1, PMEM
CFMWS3: HB1, IW = 1, PMEM
CFMWS4: HB0 + HB1, IW = 2, RAM
CFMWS5: HB0 + HB1, IW = 2, PMEM

So, in this case it looks like your CFMWS set ("decoders:root0") only
includes something like CFMWS5, so you need to update QEMU to produce
one or more of CFMWS[0-3] windows.

[..]
>     "decoders:root0":[
>       {
>         "decoder":"decoder0.0",
>         "resource":79725330432,
>         "size":4294967296,
>         "interleave_ways":2,
>         "interleave_granularity":8192,
>         "max_available_extent":4294967296,
>         "pmem_capable":true,
>         "volatile_capable":true,
>         "accelmem_capable":true,
>         "qos_class":0,
>         "nr_targets":2,
>         "targets":[
>           {
>             "target":"pci0000:70",
>             "alias":"ACPI0016:00",
>             "position":1,
>             "id":112
>           },
>           {
>             "target":"pci0000:0c",
>             "alias":"ACPI0016:01",
>             "position":0,
>             "id":12
>           }
>         ]
>       }
>     ]