Re: ndctl cxl test suite fails in arm64 QEMU

[Itaru Kitayama <itaru.kitayama@linux.dev>]

> On Feb 27, 2025, at 3:45, Alison Schofield <alison.schofield@intel.com> wrote:
> 
> On Wed, Feb 26, 2025 at 08:53:12AM +0900, Itaru Kitayama wrote:
>> Hi Alison,
>> 
>>> On Feb 26, 2025, at 7:48, Alison Schofield <alison.schofield@intel.com> wrote:
>>> 
>>> On Wed, Feb 26, 2025 at 07:37:58AM +0900, Itaru Kitayama wrote:
>>>> Hi Marc,
>>>> 
>>>>> On Feb 26, 2025, at 2:08, Marc Herbert <Marc.Herbert@linux.intel.com> wrote:
>>>>> 
>>>>> On 2025-02-24 20:09, Itaru Kitayama wrote:
>>>>>> 
>>>>>> Has anyone noticed the ndctl cxl test suite failures I reported below on arm64, QEMU emulation?
>>>>>> 
>>>>>> https://github.com/pmem/ndctl/issues/278
>>>>>> 
>>>>>> I’m using Jonathan’s latest CXL capable QEMU [1], and the latest CXL kernel [2].
>>>>>> 
>>>>> 
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> Sorry I wanted to spend more time and share a more detailed answer and
>>>>> of course I got side-tracked.
>>>>> 
>>>>> Do you have CONFIG_RANDOMIZE_MEMORY on?
>>>>> 
>>>>> If yes then please try this either this:
>>>>> 
>>>>> cd kernel
>>>>> ./scripts/config -d CONFIG_RANDOMIZE_MEMORY
>>>>> 
>>>>> ... or this:
>>>>> 
>>>>> --- a/run_qemu.sh
>>>>> +++ b/run_qemu.sh
>>>>> @@ -681,6 +681,7 @@ build_kernel_cmdline()
>>>>> "initcall_debug"
>>>>> "log_buf_len=20M"
>>>>> "memory_hotplug.memmap_on_memory=force"
>>>>> + "nokaslr"
>>>>> )
>>>>> if [[ $_arg_gdb == "on" ]]; then
>>>>> kcmd+=(
>>>> 
>>>> I chose the kernel configuration change and rebuilt the kernel, modules installed, then finally cxl_test kernel module installed on the emulated system, but I got worth results (occasionally, the kernel OOPS as well):
>>> 
>>> Yikes.
>>> 
>>> What is your ndctl --version
>> 
>> I’m using GitHub’s latest.
>> 
>>> 
>>> What is your run_qemu cmdline?
>> 
>> Unfortunately, it is x86 centric, at this moment it’s hard to use.
>> 
>>> I think you said kernel is cxl/next ?
>> 
>> Yes, I did.
>> 
>>> 
>>> The simplest one when I intend to use cxl-test is:
>>> run_qemu.sh --git-qemu --cxl-test --cxl-debug -r kmod
>> 
>> Instead, here are the options I use when emulating on arm64:
>> 
>> /home/itaru/projects/qemu/build/qemu-system-aarch64 \
>>         -M virt,virtualization=off,pflash0=rom,pflash1=efivars,gic-version=3,cxl=on,acpi=on -m 16384 \
>>         -cpu max \
>>         -smp 2 \
>>         -accel tcg \
>>         -nographic \
>>         -display none \
>>         -kernel $HOME/projects/linux/arch/arm64/boot/Image \
>>         -append "root=/dev/vda rw acpi=force" \
>>         -drive format=raw,if=none,file=/home/itaru/ubuntu24.img,id=hd0 \
>>         -device virtio-blk-pci,drive=hd0,serial="dummyserial" \
>>         -nodefaults \
>>         -device virtio-net-pci,netdev=net0 \
>>         -netdev user,id=net0,hostfwd=tcp::8024-:22 \
>>         -blockdev node-name=rom,driver=file,filename=edk2-aarch64-code.fd,read-only=true \
>>         -blockdev node-name=efivars,driver=file,filename=qemu-arm64-efivars.test \
>>         -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
>>         -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
>>         -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
>>         -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
>>         -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
>>         -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
>>         -serial mon:stdio \
>>         -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
>>         -fsdev local,security_model=none,path=/home/itaru,id=shr0
>> 
>> Thanks,
>> Itaru.
> 
> Hi Itaru,
> 
> I am able to reproduce your original issue on x86. It occurs, like Marc
> suggested, when RANDOMIZE_MEMORY is enabled. (The nokaslr kernel param
> disables both RANDOMIZE_BASE and RANDOMIZE_MEMORY in x86.)
> 
> Poking around arm64 related configs, it doesn't have a direct equivalent
> to RANDOMIZE_MEMORY. Maybe ARM64_MODULE_PLTS or RANDOMIZE_MODULE_REGION_FULL,
> both of which seem to affect how modules are loaded into memory are 
> causing the problem.
> 
> Are you familiar with those, able to try those out?
> 
> --Alison
> 
>> 
>>> 
>>>> 
>>>> Log of Meson test suite run on 2025-02-26T07:32:24.101461
>>>> 
>>>> Inherited environment: LANG=C.UTF-8 LS_COLORS='' TERM=xterm-256color PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin MAIL=/var/mail/root LOGNAME=root USER=root HOME=/root SHELL=/bin/bash SUDO_COMMAND='/usr/bin/meson test cxl-region-sysfs.sh' SUDO_USER=realm SUDO_UID=1000 SUDO_GID=1000 
>>>> 
>>>> ==================================== 1/1 =====================================
>>>> test:         ndctl:cxl / cxl-region-sysfs.sh
>>>> start time:   22:32:24
>>>> duration:     0.71s
>>>> result:       exit status 1
>>>> command:      NDCTL=/home/realm/projects/ndctl/build/ndctl/ndctl DAXCTL=/home/realm/projects/ndctl/build/daxctl/daxctl TEST_PATH=/home/realm/projects/ndctl/build/test UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MALLOC_PERTURB_=29 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 DATA_PATH=/home/realm/projects/ndctl/test MESON_TEST_ITERATION=1 LD_LIBRARY_PATH=/home/realm/projects/ndctl/build/daxctl/lib:/home/realm/projects/ndctl/build/ndctl/lib:/home/realm/projects/ndctl/build/cxl/lib /bin/bash /home/realm/projects/ndctl/test/cxl-region-sysfs.sh
>>>> ----------------------------------- stdout -----------------------------------
>>>> test/cxl-region-sysfs.sh: failed at line 40
>>>> ----------------------------------- stderr -----------------------------------
>>>> + trap 'err $LINENO' ERR
>>>> + check_prereq jq
>>>> + command -v jq
>>>> + modprobe -r cxl_test
>>>> + modprobe cxl_test
>>>> + rc=1
>>>> ++ ./cxl/cxl list -b cxl_test -D -d root
>>>> ++ jq -r '.[] |
>>>> select(.pmem_capable == true) |
>>>> select(.nr_targets == 2) |
>>>> .decoder'
>>>> Warning: no matching devices found
>>>> 
>>>> + decoder=
>>>> + readarray -t mem
>>>> ++ ./cxl/cxl list -M -d
>>>> ++ jq -r '.[].memdev'
>>>> Error: switch `d' requires a value
>>>> 
>>>> usage: cxl list [<options>]
>>>> 
>>>>   -d, --decoder <decoder device name>
>>>>                         filter by CXL decoder device name(s) / class
>>>> + readarray -t endpoint
>>>> ++ ./cxl/cxl reserve-dpa -t pmem -s 268435456
>>>> ++ jq -r '.[] | .decoder.decoder'
>>>> 
>>>> usage: cxl reserve-dpa <mem0> [<mem1>..<memn>] [<options>]
>>>> 
>>>>   -b, --bus <bus name>  Limit operation to the specified bus
>>>>   -v, --verbose         turn on debug
>>>>   -S, --serial          use serial numbers to id memdevs
>>>>   -s, --size <size>     size in bytes (Default: all available capacity)
>>>>   -d, --decoder <decoder instance id>
>>>>                         override the automatic decoder selection
>>>>   -t, --type <type>     'pmem' or 'ram' (volatile) (Default: 'pmem')
>>>>   -f, --force           Attempt 'expected to fail' operations
>>>> 
>>>> ++ cat /sys/bus/cxl/devices//create_pmem_region
>>>> cat: /sys/bus/cxl/devices//create_pmem_region: No such file or directory
>>>> + region=
>>>> ++ err 40
>>>> +++ basename /home/realm/projects/ndctl/test/cxl-region-sysfs.sh
>>>> ++ echo test/cxl-region-sysfs.sh: failed at line 40
>>>> ++ '[' -n '' ']'
>>>> ++ exit 1
>>>> ==============================================================================
>>>> 
>>>> 
>>>> Summary of Failures:
>>>> 
>>>> 1/1 ndctl:cxl / cxl-region-sysfs.sh FAIL             0.71s   exit status 1
>>>> 
>>>> Ok:                 0   
>>>> Expected Fail:      0   
>>>> Fail:               1   
>>>> Unexpected Pass:    0   
>>>> Skipped:            0   
>>>> Timeout:            0   

Now I am having an Invalid argument. It seems only value 0 is accepted to the sysfs entry.

I paste the .config I am using.