Re: [GIT preview] for-6.3/cxl-ram-region

[Gregory Price <gregory.price@memverge.com>]

On Mon, Jan 30, 2023 at 03:18:38PM -0800, Dan Williams wrote:
> Gregory Price wrote:
> > On Mon, Jan 30, 2023 at 12:10:08PM -0800, Dan Williams wrote:
> > > Gregory Price wrote:
> > > [..]
> > > > I found the same results.
> > > > echo mem0 > /sys/bus/cxl/devices/region0/target0
> > > > 
> > > > Not sure if bug/missing feature, but after attaching a device to the
> > > > target, you get no output when reading target0
> > > > 
> > > > ```
> > > > [root@fedora ~]# cat /sys/bus/cxl/devices/region0/target0
> > > > 
> > > > [root@fedora ~]#
> > > 
> > > Hmm, did you not get:
> > > 
> > > "-bash: echo: write error: Invalid argument"
> > > 
> > > ...at that step? Because targetX expects an endpoint decoder, not a
> > > memdev.
> > > 
> > 
> > ¯\_(ツ)_/¯ bug?
> > 
> > I went through the kernel code and thought it was looking for a memdev
> > but i guess i was wrong
> > 
> > // ... snip ...
> >     dev = bus_find_device_by_name(&cxl_bus_type, NULL, buf);
> >     if (!dev)
> >       return -ENODEV;
> > 
> >     if (!is_endpoint_decoder(dev)) {
> >       rc = -EINVAL;
> >       goto out;
> >     }
> > // ... snip ...
> 
> Oh, yes, it is a bug. I fixed it up when I address this 0day report:
> 
> https://lore.kernel.org/linux-cxl/202301281313.kVrIreUj-lkp@intel.com/
> 
> ...I posted another spin of that branch with that fixed up.


This worked for me, though it took me a bit to figure out how to wire
everythign up - still not sure this is entirely correct but this is the
string of commands that were required to successfully attach an endpoint
decoder to the root decoder.

Note: The root decoder has interleave_(granularity=256, ways=1), and the
region code appears to require the same granularity?  Does that mean
we're stuck to 256b granularity? (unless i misread the code, which i'm
about 75% sure i am).


Command string:

# Program the endpoint decoder for ram of size 1GB
echo ram > /sys/bus/cxl/devices/decoder2.0/mode
echo 0x40000000 > /sys/bus/cxl/devices/decoder2.0/dpa_size

# Create a region in the root decoder
echo region0 > /sys/bus/cxl/devices/decoder0.0/create_ram_region

# Configure that region with the same interleave granularity
# and ways as the root and endpoint decoders
echo 256 > /sys/bus/cxl/devices/region0/interleave_granularity
echo 1 > /sys/bus/cxl/devices/region0/interleave_ways
echo 0x40000000 > /sys/bus/cxl/devices/region0/size

# Link the endpoint decoder as a target in the region
echo decoder2.0 > /sys/bus/cxl/devices/region0/target0

# Commit the changes
echo 1 > /sys/bus/cxl/devices/region0/commit