Re: [PATCH v4 0/10] CXL Reset support for Type 2 devices

[<dan.j.williams@intel.com>]

Vikram Sethi wrote:
> Hi Dan,
> 
> >Once the protocol error handling series has landed there is a potential
> >to extend that with CXL Reset recovery. However, that needs a clear
> >error model defined as to which resets have a chance of recovering
> >*system* operation when CXL.cache/mem fails. In Terry's series, panic /
> >reboot is the recovery, not reset.
> 
> It's not just about CXL protocol error handling though. Type2 Device
> passthrough will be a common usecase for CXL reset (entire device is
> assigned to VM) across different VM assignment.

I understand. The point about CXL Protocol Error handling series is that
it at least enlightens the PCIe core about the presence of active CXL
links.

It is also the case that it much closer to being upstream than this set
which still has fundamental questions.

> The cache and mem of the device must be cleared via CXL reset for full
> device passthrough in addition to the PCIe/CXL.IO reset. Another
> usecase is when the device is reconfigured either in baremetal or in
> VM for passthrough usecase. Such usecases often require a reset of the
> device, and it's cache, mem and not a narrow "memregion" reset, so I'm
> not in favor of exposing it via CXL.mem sysfs entries. IMO, device
> sysfs attribute reset method is appropriate for type2 devices.

That case is already handled today with secondary bus reset to
completely reset an entire device. That path is problematic for CXL
because PCI reset has no idea about how to manage caches or handle
memory unplug.

Administrator is responsible for making sure that event is not a
surprise memory removal or cache protocol corruption event. That is a
level of explosiveness that PCI reset has never needed to consider.

CXL Reset wants to be more surgical than secondary bus reset. The only
way to be more surgical is to cooperate with the CXL core that knows
whether the explosives have been rendered inert.

> Finally, there is the error usecase, which in the common case is as
> simple as an uncorrected ECC error in the HDM. While not strictly
> necessary, it is common practice to reset the device in such cases, to
> recover the bad page/row via PPR on device reset. You want reset of
> the memory controller as part of the CXL reset here, FLR is not
> enough.

??

CXL memory repair is already upstream without CXL Reset, see
CONFIG_CXL_EDAC_MEM_REPAIR.

> Regarding scope, and "recoverability", we have significant experience
> with recovering "pre-CXL" coherent GB200 devices for device errors:
> memory ECC or other, by killing the application using device memory,
> unloading driver, resetting the device, and reloading driver. The CXL
> protocol error series can use CXL reset series, but I don't see that
> this series needs to wait for protocol error handling to be merged.  

I do want to get to the point where device memory error recovery is a
first class citizen. CXL Protocol Error handling just happens to be at
the top of the review queue and safe to assume it can be a foundation
for further RAS features.