RE: [PATCH v4 0/9] dax/hmem, cxl: Coordinate Soft Reserved handling with CXL and HMEM

[<dan.j.williams@intel.com>]

Yasunori Gotou (Fujitsu) wrote:
[..]
> > > == plug - after PCI rescan cannot create hmem 6070000000-a06fffffff :
> > > CXL Window 1
> > >   6070000000-a06fffffff : region1
> > >
> > > kernel: cxl_region region1: config state: 0
> > > kernel: cxl_acpi ACPI0017:00: decoder0.1: created region1
> > > kernel: cxl_pci 0000:04:00.0: mem1:decoder10.0: __construct_region
> > > region1 res: [mem 0x6070000000-0xa06fffffff flags 0x200] iw: 1 ig:
> > > 4096
> > > kernel: cxl_mem mem1: decoder:decoder10.0 parent:0000:04:00.0
> > > port:endpoint10 range:0x6070000000-0xa06fffffff pos:0
> > > kernel: cxl region1: region sort successful
> > > kernel: cxl region1: mem1:endpoint10 decoder10.0 add: mem1:decoder10.0
> > > @ 0 next: none nr_eps: 1 nr_targets: 1
> > > kernel: cxl region1: pci0000:00:port2 decoder2.1 add: mem1:decoder10.0
> > > @ 0 next: mem1 nr_eps: 1 nr_targets: 1
> > > kernel: cxl region1: pci0000:00:port2 cxl_port_setup_targets expected
> > > iw: 1 ig: 4096 [mem 0x6070000000-0xa06fffffff flags 0x200]
> > > kernel: cxl region1: pci0000:00:port2 cxl_port_setup_targets got iw: 1
> > > ig: 256 state: disabled 0x6070000000:0xa06fffffff
> > 
> > Did the device get reset in the process? This looks like decoders bounced in an
> > inconsistent fashion from unplug to replug and autodiscovery.
> 
> You are correct.
> This environment does not support actual PCIe hotplug.
> Even if we perform PCIe hotplug emulation by manipulating sysfs, some CXL Decoder registers,
> which have read-only attributes, are not initialized.
> I confirmed about a month and a half ago that this was causing the hot-add process to fail.
> I suspect that such registers must be initialized by the hardware when a hot-add occurs.
> 
> I should have informed Wolski-san about this in advance. My apologies.

No worries, just wanted to understand what was happening, thanks for
confirming.

However, this does raise an important issue that tooling could solve. If
you are committed to unplugging a device and the decoders are locked
then tooling should probably arrange for a secondary bus reset to unlock
and disable those decoders. Otherwise, the kernel might have a hard time
guaranteeing that a removed device restores at the exact address it had
previously, especially when there is free CFMWS capacity.