Re: [GIT preview] for-6.3/cxl-ram-region

[Gregory Price <gregory.price@memverge.com>]

On Tue, Jan 31, 2023 at 02:56:16PM +0000, Jonathan Cameron wrote:
> On Mon, 30 Jan 2023 09:23:54 -0500
> Gregory Price <gregory.price@memverge.com> wrote:
> 
> > On Thu, Jan 26, 2023 at 07:34:24PM +0000, Jonathan Cameron wrote:
> > > Looks like decoders are programmed correctly as I can read and write from
> > > the HPA using devmem2.
> > > 
> > > This is against http://gitlab.com/jic23/qemu cxl-2023-01-26 which has been
> > >   
> > 
> > Johnathan, can you explain how you're accessing the memory? I don't
> > quite follow.
> 
> Command sequence is pretty similar to yours (not checked it's identical)
> and once commit is done, using a version of devmem2 (not sure it was this one)
> https://github.com/hackndev/tools/blob/master/devmem2.c 
> (if you want 64 bit read / write add the obvious additional parameter ;)
> 
> with appropriately loose kernel configuration that /dev/mem works
> 

Very cool, i have confirmed this all works *and* that we can actually
back the memory with a file and see changes

For readers, and sake of completeness:


Kernel:
https://github.com/l1k/linux/commits/doe
+
https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-6.3/cxl-ram-region
(merge cxl-ram-region into doe)


QEMU:
https://gitlab.com/jic23/qemu/-/tree/cxl-2023-01-26
(might need to revert last commit if you're not on arch)


QEMU Config: file_memexp.sh

sudo /opt/qemu-cxl/bin/qemu-system-x86_64 \
-drive file=/var/lib/libvirt/images/cxl.qcow2,format=qcow2,index=0,media=disk,id=hd \
-m 2G,slots=4,maxmem=4G \
-smp 4 \
-machine type=q35,accel=kvm,cxl=on \
-enable-kvm \
-nographic \
-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,port=0,slot=0 \
-object memory-backend-file,id=mem0,mem-path=/tmp/mem0,size=1G,share=true \
-device cxl-type3,bus=rp0,volatile-memdev=mem0,id=cxl-mem0 \
-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=1G


Mapping the memory in the host (still not confident about the interleave
settings, but it works)

map_memory.sh
# Program the endpoint decoder for ram of size 1GB
echo ram > /sys/bus/cxl/devices/decoder2.0/mode
echo 0x40000000 > /sys/bus/cxl/devices/decoder2.0/dpa_size

# Create a region in the root decoder
echo region0 > /sys/bus/cxl/devices/decoder0.0/create_ram_region

# Configure that region's interleave and size
echo 4096 > /sys/bus/cxl/devices/region0/interleave_granularity
echo 1 > /sys/bus/cxl/devices/region0/interleave_ways
echo 0x40000000 > /sys/bus/cxl/devices/region0/size

# Link the endpoint decoder as a target in the region
echo decoder2.0 > /sys/bus/cxl/devices/region0/target0

# Commit the changes
echo 1 > /sys/bus/cxl/devices/region0/commit



Hack up devmem2 (it uses strtoul, i needs to use strtoull)
https://github.com/hackndev/tools/blob/master/devmem2.c

Then run it

[root@fedora ~]# ./devmem2 0x290000000 w 0x87654321
/dev/mem opened.
Memory mapped at address 0x7fb8697fb000.
Value at address 0x290000000 (0x7fb8697fb000): 0x00000000
Written 0x87654321; readback 0x87654321


Validate the memory actually got written to the backing file:
[gourry@fedora build]$ xxd /tmp/mem0 | head
00000000: 2143 6587 0000 0000 0000 0000 0000 0000  !Ce.............



:]