Re: [RFC PATCH 00/15] Region driver

[Dan Williams <dan.j.williams@intel.com>]

Jonathan Cameron wrote:
> ....
> 
> > > Hi Ben,
> > >
> > > I finally got around to actually trying this out on top of Dan's recent fix set
> > > (I rebased it from the cxl/preview branch on kernel.org).
> > >
> > > I'm not having much luck actually bring up a region.
> > >
> > > The patch set refers to configuring the end point decoders, but all their
> > > sysfs attributes are read only.  Am I missing a dependency somewhere or
> > > is the intent that this series is part of the solution only?
> > >
> > > I'm confused!  
> > 
> > There's a new series that's being reviewed internally before going to the list:
> > 
> > https://gitlab.com/bwidawsk/linux/-/tree/cxl_region-redux3
> > 
> > Given the proximity to the merge window opening and the need to get
> > the "mem_enabled" series staged, I asked Ben to hold it back from the
> > list for now.
> > 
> > There are some changes I am folding into it, but I hope to send it out
> > in the next few days after "mem_enabled" is finalized.
> 
> Hi Dan,
> 
> I switched from an earlier version of the region code over to a rebase of the tree.
> Two issues below you may already have fixed.
> 
> The second is a carry over from an earlier set so I haven't tested
> without it but looks like it's still valid.
> 
> Anyhow, thought it might save some cycles to preempt you sending
> out the series if these issues are still present.
> 
> Minimal testing so far on these with 2 hb, 2 rp, 4 directly connected
> devices, but once you post I'll test more extensively.  I've not
> really thought about the below much, so might not be best way to fix.
> 
> Found a bug in QEMU code as well (missing write masks for the
> target list registers) - will post fix for that shortly.

Hi Jonathan,

Tomorrow I'll post the tranche to the list, but wanted to let you and
others watching that that the 'preview' branch [1] now has the proposed
initial region support. Once the bots give the thumbs up I'll send it
along.

To date I've only tested it with cxl_test and an internal test vehicle.
The cxl_test script I used to setup and teardown a x8 interleave across
x2 host bridges and x4 switches is:

---

#!/bin/bash
modprobe cxl_test
udevadm settle
decoder=$(cxl list -b cxl_test -D -d root | jq -r ".[] |
          select(.pmem_capable == true) | 
          select(.nr_targets == 2) |
          .decoder")

readarray -t mem < <(cxl list -M -d $decoder | jq -r ".[].memdev")
readarray -t endpoint < <(cxl reserve-dpa -t pmem ${mem[*]} -s $((256<<20)) |
                          jq -r ".[] | .decoder.decoder")
region=$(cat /sys/bus/cxl/devices/$decoder/create_pmem_region)
echo $region > /sys/bus/cxl/devices/$decoder/create_pmem_region
uuidgen > /sys/bus/cxl/devices/$region/uuid
nr_targets=${#endpoint[@]}
echo $nr_targets > /sys/bus/cxl/devices/$region/interleave_ways
g=$(cat /sys/bus/cxl/devices/$decoder/interleave_granularity)
echo $g > /sys/bus/cxl/devices/$region/interleave_granularity
echo $((nr_targets * (256<<20))) > /sys/bus/cxl/devices/$region/size
port_dev0=$(cxl list -T -d $decoder | jq -r ".[] |
            .targets | .[] | select(.position == 0) | .target")
port_dev1=$(cxl list -T -d $decoder | jq -r ".[] |
            .targets | .[] | select(.position == 1) | .target")
readarray -t mem_sort0 < <(cxl list -M -p $port_dev0 | jq -r ".[] | .memdev")
readarray -t mem_sort1 < <(cxl list -M -p $port_dev1 | jq -r ".[] | .memdev")
mem_sort=()
mem_sort[0]=${mem_sort0[0]}
mem_sort[1]=${mem_sort1[0]}
mem_sort[2]=${mem_sort0[2]}
mem_sort[3]=${mem_sort1[2]}
mem_sort[4]=${mem_sort0[1]}
mem_sort[5]=${mem_sort1[1]}
mem_sort[6]=${mem_sort0[3]}
mem_sort[7]=${mem_sort1[3]}

#mem_sort[2]=${mem_sort0[0]}
#mem_sort[1]=${mem_sort1[0]}
#mem_sort[0]=${mem_sort0[2]}
#mem_sort[3]=${mem_sort1[2]}
#mem_sort[4]=${mem_sort0[1]}
#mem_sort[5]=${mem_sort1[1]}
#mem_sort[6]=${mem_sort0[3]}
#mem_sort[7]=${mem_sort1[3]}

endpoint=()
for i in ${mem_sort[@]}
do
        readarray -O ${#endpoint[@]} -t endpoint < <(cxl list -Di -d endpoint -m $i | jq -r ".[] |
                                                     select(.mode == \"pmem\") | .decoder")
done
pos=0
for i in ${endpoint[@]}
do
        echo $i > /sys/bus/cxl/devices/$region/target$pos
        pos=$((pos+1))
done
echo "$region added ${#endpoint[@]} targets: ${endpoint[@]}"

echo 1 > /sys/bus/cxl/devices/$region/commit
echo 0 > /sys/bus/cxl/devices/$region/commit

pos=0
for i in ${endpoint[@]}
do
        echo "" > /sys/bus/cxl/devices/$region/target$pos
        pos=$((pos+1))
done
readarray -t endpoint < <(cxl free-dpa -t pmem ${mem[*]} |
                          jq -r ".[] | .decoder.decoder")
echo "$region released ${#endpoint[@]} targets: ${endpoint[@]}"

---

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=preview