Re: ndctl cxl test suite fails in arm64 QEMU

[Itaru Kitayama <itaru.kitayama@linux.dev>]

Hi Alison,

> On Feb 26, 2025, at 7:48, Alison Schofield <alison.schofield@intel.com> wrote:
> 
> On Wed, Feb 26, 2025 at 07:37:58AM +0900, Itaru Kitayama wrote:
>> Hi Marc,
>> 
>>> On Feb 26, 2025, at 2:08, Marc Herbert <Marc.Herbert@linux.intel.com> wrote:
>>> 
>>> On 2025-02-24 20:09, Itaru Kitayama wrote:
>>>> 
>>>> Has anyone noticed the ndctl cxl test suite failures I reported below on arm64, QEMU emulation?
>>>> 
>>>> https://github.com/pmem/ndctl/issues/278
>>>> 
>>>> I’m using Jonathan’s latest CXL capable QEMU [1], and the latest CXL kernel [2].
>>>> 
>>> 
>>> 
>>> Hi,
>>> 
>>> Sorry I wanted to spend more time and share a more detailed answer and
>>> of course I got side-tracked.
>>> 
>>> Do you have CONFIG_RANDOMIZE_MEMORY on?
>>> 
>>> If yes then please try this either this:
>>> 
>>> cd kernel
>>> ./scripts/config -d CONFIG_RANDOMIZE_MEMORY
>>> 
>>> ... or this:
>>> 
>>> --- a/run_qemu.sh
>>> +++ b/run_qemu.sh
>>> @@ -681,6 +681,7 @@ build_kernel_cmdline()
>>> "initcall_debug"
>>> "log_buf_len=20M"
>>> "memory_hotplug.memmap_on_memory=force"
>>> + "nokaslr"
>>> )
>>> if [[ $_arg_gdb == "on" ]]; then
>>> kcmd+=(
>> 
>> I chose the kernel configuration change and rebuilt the kernel, modules installed, then finally cxl_test kernel module installed on the emulated system, but I got worth results (occasionally, the kernel OOPS as well):
> 
> Yikes.
> 
> What is your ndctl --version

I’m using GitHub’s latest.

>  
> What is your run_qemu cmdline?

Unfortunately, it is x86 centric, at this moment it’s hard to use.
 
> I think you said kernel is cxl/next ?

Yes, I did.

> 
> The simplest one when I intend to use cxl-test is:
> run_qemu.sh --git-qemu --cxl-test --cxl-debug -r kmod

Instead, here are the options I use when emulating on arm64:

/home/itaru/projects/qemu/build/qemu-system-aarch64 \
         -M virt,virtualization=off,pflash0=rom,pflash1=efivars,gic-version=3,cxl=on,acpi=on -m 16384 \
         -cpu max \
         -smp 2 \
         -accel tcg \
         -nographic \
         -display none \
         -kernel $HOME/projects/linux/arch/arm64/boot/Image \
         -append "root=/dev/vda rw acpi=force" \
         -drive format=raw,if=none,file=/home/itaru/ubuntu24.img,id=hd0 \
         -device virtio-blk-pci,drive=hd0,serial="dummyserial" \
         -nodefaults \
         -device virtio-net-pci,netdev=net0 \
         -netdev user,id=net0,hostfwd=tcp::8024-:22 \
         -blockdev node-name=rom,driver=file,filename=edk2-aarch64-code.fd,read-only=true \
         -blockdev node-name=efivars,driver=file,filename=qemu-arm64-efivars.test \
         -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
         -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
         -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
         -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
         -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
         -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
         -serial mon:stdio \
         -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
         -fsdev local,security_model=none,path=/home/itaru,id=shr0

Thanks,
Itaru.

> 
>> 
>> Log of Meson test suite run on 2025-02-26T07:32:24.101461
>> 
>> Inherited environment: LANG=C.UTF-8 LS_COLORS='' TERM=xterm-256color PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin MAIL=/var/mail/root LOGNAME=root USER=root HOME=/root SHELL=/bin/bash SUDO_COMMAND='/usr/bin/meson test cxl-region-sysfs.sh' SUDO_USER=realm SUDO_UID=1000 SUDO_GID=1000 
>> 
>> ==================================== 1/1 =====================================
>> test:         ndctl:cxl / cxl-region-sysfs.sh
>> start time:   22:32:24
>> duration:     0.71s
>> result:       exit status 1
>> command:      NDCTL=/home/realm/projects/ndctl/build/ndctl/ndctl DAXCTL=/home/realm/projects/ndctl/build/daxctl/daxctl TEST_PATH=/home/realm/projects/ndctl/build/test UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MALLOC_PERTURB_=29 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 DATA_PATH=/home/realm/projects/ndctl/test MESON_TEST_ITERATION=1 LD_LIBRARY_PATH=/home/realm/projects/ndctl/build/daxctl/lib:/home/realm/projects/ndctl/build/ndctl/lib:/home/realm/projects/ndctl/build/cxl/lib /bin/bash /home/realm/projects/ndctl/test/cxl-region-sysfs.sh
>> ----------------------------------- stdout -----------------------------------
>> test/cxl-region-sysfs.sh: failed at line 40
>> ----------------------------------- stderr -----------------------------------
>> + trap 'err $LINENO' ERR
>> + check_prereq jq
>> + command -v jq
>> + modprobe -r cxl_test
>> + modprobe cxl_test
>> + rc=1
>> ++ ./cxl/cxl list -b cxl_test -D -d root
>> ++ jq -r '.[] |
>>  select(.pmem_capable == true) |
>>  select(.nr_targets == 2) |
>>  .decoder'
>>  Warning: no matching devices found
>> 
>> + decoder=
>> + readarray -t mem
>> ++ ./cxl/cxl list -M -d
>> ++ jq -r '.[].memdev'
>>  Error: switch `d' requires a value
>> 
>> usage: cxl list [<options>]
>> 
>>    -d, --decoder <decoder device name>
>>                          filter by CXL decoder device name(s) / class
>> + readarray -t endpoint
>> ++ ./cxl/cxl reserve-dpa -t pmem -s 268435456
>> ++ jq -r '.[] | .decoder.decoder'
>> 
>> usage: cxl reserve-dpa <mem0> [<mem1>..<memn>] [<options>]
>> 
>>    -b, --bus <bus name>  Limit operation to the specified bus
>>    -v, --verbose         turn on debug
>>    -S, --serial          use serial numbers to id memdevs
>>    -s, --size <size>     size in bytes (Default: all available capacity)
>>    -d, --decoder <decoder instance id>
>>                          override the automatic decoder selection
>>    -t, --type <type>     'pmem' or 'ram' (volatile) (Default: 'pmem')
>>    -f, --force           Attempt 'expected to fail' operations
>> 
>> ++ cat /sys/bus/cxl/devices//create_pmem_region
>> cat: /sys/bus/cxl/devices//create_pmem_region: No such file or directory
>> + region=
>> ++ err 40
>> +++ basename /home/realm/projects/ndctl/test/cxl-region-sysfs.sh
>> ++ echo test/cxl-region-sysfs.sh: failed at line 40
>> ++ '[' -n '' ']'
>> ++ exit 1
>> ==============================================================================
>> 
>> 
>> Summary of Failures:
>> 
>> 1/1 ndctl:cxl / cxl-region-sysfs.sh FAIL             0.71s   exit status 1
>> 
>> Ok:                 0   
>> Expected Fail:      0   
>> Fail:               1   
>> Unexpected Pass:    0   
>> Skipped:            0   
>> Timeout:            0