RE: Questions about CXL device (type 3 memory) hotplug

[Dan Williams <dan.j.williams@intel.com>]

Yasunori Gotou (Fujitsu) wrote:
> > Yasunori Gotou (Fujitsu) wrote:
> > [..]
> > > One of what I mind here --was-- which documentation describes OS
> > triggered hotremove instead of PCIe trigger.
> > > Because many hardware/firmware developers don't know the circumstance
> > of Linux.
> > > They may want to implement same system not only for Linux but also for
> > > VMware or any other system, and may want to obey only the specification or
> > any similar documents.
> > > But I found " CXL* Type 3 Memory Device Software Guide: 2.13.7 OS
> > managed hot remove sequence"
> > >
> > https://cdrdv2-public.intel.com/643805/643805_CXL%20Memory%20Device
> > %20
> > > SW%20Guide_Rev1p0.pdf Then, I can talk with them by it. So, it was
> > > solved.
> > >
> > > My remain questions are the followings.
> > >
> > > Q6) Are there any way to hotremove from outside of servers now?
> > >     Currently, administrator seems to need to login a server and execute
> > offline and cxl disable commands
> > >     to remove memory in it, right? But in future, something software like
> > memory pool manager,
> > >     Fabric Manager, or any other management tools which can manage
> > many servers CXL devices
> > >     will want to remove each server's devices from outside.
> > >     But I'm not sure it can available or not yet now.
> > 
> > As far as I can see all of the PCI hotplug state machines just coordinate the
> > removal internal to themselves and the PCI bus core without any participation
> > from the impacted driver before the ->remove() event. The ->remove() event is
> > too late to cancel the hotplug. So the change here would be either an upcall to
> > userspace, or some permission request callback to the impacted driver. Since
> > this is a policy decision whether to allow a given CXL device to be removed that
> > leans towards a userspace upcall mechanism.
> 
> Ah, sorry... My description of question was not good.
> I understand that PCIe hotremove is not suitable for trigger of CXL memory.
> 
> What I would like to ask is "Are there any agent or daemon which gets a hotremove 
> request from outside of server and executes offline and cxl disable region without
> users operation?"
> I suppose such memory pool manager (or others) would like to ask the agent to
> execute such operation.
> (Probably, the agent need to get the request by REST API.)

No, there's no coordination between the kernel and userspace when the
attention button is pressed. So any coordinated removal must be handled
before the removal is attempted. I think it would be useful to have a
mode of operation where pressing the attention button just notifies
userspace and it handles the coordinated shutdown of the device.

If the question is having a management API to trigger removal I am not
aware of any work in this space.