Re: CXL 1.1 Support Plan

["Verma, Vishal L" <vishal.l.verma@intel.com>]

On Fri, 2021-10-08 at 18:43 +0000, John Groves wrote:
> On 10/7/21 4:30 PM, Verma, Vishal L wrote:
> > On Thu, 2021-09-30 at 22:17 +0000, John Groves wrote:
> > > I don't see your doc patch yet in the ndctl repo, but I recommend merging it there.
> > > 
> > > A related question: "daxctl reconfigure-device --mode=system-ram ..." works for me,
> > > with --force, but going the other way (--mode=devdax) fails.  But a reboot puts it back
> > > into devdax mode regardless of the pre-boot setting (i.e. --mode=system-ram reverts
> > > back to devdax on reboot).
> > Oh, I'm a bit confused. --force only applies when going from system-ram
> > to devdax -- it offlines the memory for you. Without force, you're
> > responsible for a prior 'daxctl offline-memory daxX.Y' step.
> > 
> > Going from devdax to system-ram should not need --force, and I don't
> > think force actually does anything there.
> 
> Hoping I've successfully de-mangled this message...
> 
> I definitely might be doing something wrong.  Here is a "typescript".
> 
> # grep dax /proc/iomem
>     880000000-107fffffff : *dax*0.0
> 
> # ls -al /dev/dax0.0
> crw------- 1 root root 252, 2 Oct  8 13:14 /dev/dax0.0
> 
> # numastat
>                            node0
> numa_hit                 2698949
> numa_miss                      0
> numa_foreign                   0
> interleave_hit             14565
> local_node               2698949
> other_node                     0
> 
> # daxctl reconfigure-device --mode=system-ram --region=0 dax0.0
> dax0.0: error: kernel policy will auto-online memory, aborting
> error reconfiguring devices: Device or resource busy
> reconfigured 0 devices

Ah yes - so thios points to either

  CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y

or 

  $ cat /sys/devices/system/memory/auto_online_blocks 
  online

daxctl wants to online the new memory in ZONE_MOVABLE by default, but
either of the above would race daxctl to online it in ZONE_NORMAL -
that's the warning you got above. 

> 
> # daxctl reconfigure-device --mode=system-ram --region=0 --force dax0.0
> 
> dax0.0:
>   WARNING: detected a race while onlining memory
>   Some memory may not be in the expected zone. It is
>   recommended to disable any other onlining mechanisms,
>   and retry. If onlining is to be left to other agents,
>   use the --no-online option to suppress this warning

I guess the force does work in this case :)

> dax0.0: all memory sections (256) already online
> [
>   {
>     "chardev":"dax0.0",
>     "size":34359738368,
>     "target_node":1,
>     "align":2097152,
>     "mode":"system-ram",
>     "online_memblocks":256,
>     "total_memblocks":256,
>     "movable":false

This indicates that the new memory went into ZONE_NORMAL.
That can make it hard to convert back to devdax, but..


[snip]

> Starting in the state where I left off above:
> 
> # daxctl reconfigure-device --mode=devdax dax0.0
> error reconfiguring devices: Device or resource busy
> reconfigured 0 devices
> 
> # daxctl reconfigure-device --mode=devdax --force dax0.0
> libdaxctl: offline_one_memblock: dax0.0: Failed to offline /sys/devices/system/node/node1/memory272/state: Device or resource busy
> dax0.0: failed to offline memory: Device or resource busy
> 
> error reconfiguring devices: Device or resource busy
> reconfigured 0 devices
> 
> # daxctl offline-memory dax0.0
> dax0.0: 4 memory sections already offline
> libdaxctl: offline_one_memblock: dax0.0: Failed to offline /sys/devices/system/node/node1/memory272/state: Device or resource busy
> dax0.0: failed to offline memory: Device or resource busy
> error offlining memory: Device or resource busy
> offlined memory for 0 devices

This confused me a bit - I wonder if 'daxctl offline memory' claiming
that it is already offline is a bug, as a subsequent reconfigure then
says 'failed to offline..'

What is this range backed by? Most of the testing I did with this was
using a pmem device that gets it's own 'target_node' (i.e. the new
memory ends up in a new numa node of its own).

If you carve out memory using memmap or efi_fake_mem, It would end up
getting hotplugged into an existing numa node. I wonder if that causes
problems with the hot-unplug. Let take another look at this.

> 
> # daxctl offline-memory --force dax0.0
>   Error: unknown option `force'
>  usage: daxctl offline-memory <device> [<options>]
> 
>     -r, --region <region-id>
>                           filter by region
>     -u, --human           use human friendly number formats
>     -v, --verbose         emit more debug messages
> 
>  
> 
> I have to reboot to get back to dax memory, though it's very possible that my
> [questionable] doc reading skills are at fault.
> 
> Thanks,
> John
> 
> 
> 
> 
> 
>