From: <alucerop@amd.com>
To: <linux-cxl@vger.kernel.org>, <dave.jiang@intel.com>,
<dave@stgolabs.net>, <jic23@kernel.org>,
<alison.schofield@intel.com>, <vishal.l.verma@intel.com>
Cc: Alejandro Lucero <alucerop@amd.com>
Subject: [RFC 0/3] Type2 unwinding ordering change
Date: Wed, 18 Nov 2026 19:14:28 +0000 [thread overview]
Message-ID: <20261118191431.21781-1-alucerop@amd.com> (raw)
From: Alejandro Lucero <alucerop@amd.com>
This RFC requires more work mainly with the tracepoints and surely with
documentation which, I hope, will get better with help from reviewers.
Regarding the solution proposed for the described problem, I know there could
be other solutions, so I'll be happy if someone comes with a better one, but
my feeling is the current design based strongly on devm actions is adding
complexity which would be desirable to reduce somehow.
The summary
===========
Current Type2 support is safe but semantically wrong: a Type2 driver could not
be (temporarily) aware about the CXL region (the one creating the Host physical
range to work with) is not there anymore. It could because it depends on the
unwinding triggering point which changes the ordering of CXL objects unwinding.
It is safe because for current Type2 expectations the physical mapping will
remain: those HDM decoders building up the path and programmed by the BIOS will
not be reset.
While implementing multiPF support for Type2 I found this behaviour specially
annoying because potential concurrency problems due to this non deterministic
unwinding order of Type2 related objects. Adding to this the semantical problem
which will likely be a bigger problem when adhoc Type2 regions are supported, I
propose a change to how the current design works for the unwinding of objects
for a Type2 device.
I seize the necessity of explaining below the problem in detail for also
proposing to add to the CXL kernel documentation such (adjusted) explanation for
helping other to understand how the CXL subsystem works.
The long story
==============
Current CXL subsystem relies on the linux kernel device abstraction for
representing the different componentes building up the subsystem itself. The
core or root devices of the subsystem are created from data coming through ACPI
and are not discussed in this document, with the target being Type2 devices and
the binding/unbinding of those physical devices into the CXL subsystem.
Along with logical devices come CXL drivers which allow to perform this
binding/unbinding through usual kernel probe mechanisms. Moreover, current CXL
subsystem design relies on devres (managed resources) kernel functionality
linking actions to device unbinding from drivers helping with all the unwinding
required.
When a device is unbound the kernel will first invoke those devm actions linked
to the device and then, if there is no further references to the device kobject,
it will be released or freed. This implies actions linked to the unbinding
happen at the unbinding time but the final device release can happen at a later
time. Furthermore, those actions can lead to other related objects to be removed
as well as their existence depends on the current one being removed. For a Type2
device doing CXL initialization, these are the objects created with order from
left to right and with dependencies in the opposite direction:
cxl_memdev --> endpoint (cxl_port) --> cxl_decoder --> cxl_region
A CXL region defines the Host physical range where the device memory (HDM) is
mapped and someone, a client, will likely be using it somehow. For a Type2 this
client is the Type2 "generic" driver, like a network driver using CXL for some
kind of acceleration. For a Type3 the client could be the kernel memory
management or some user space functionality using such a physical range through
a dax device, with the driver taking over the device being the CXL core mem
driver. Note this last driver is also used in Type2 for the cxl_memdev created,
as next paragraph does explain.
The above object chain is created when a new memdev device is added to the
kernel device core and such a device binding to the cxl_mem driver. This will
trigger port(s) creation, at least one, a cxl_port device which will bind to
the cxl_port driver when added to the kernel device core. For a decoder only a
device is created (or as many as decoders advertises the Type2 device) but no
binding as there is no decoder driver to bind to. Finally, the region device
will be automatically created during initialization for current Type2 devices
expectation (this is called autodiscovery but work for supporting adhoc region
creation is in the way) and it will bind to the cxl_region driver. Once there
exists a driver bound to a device, it can be unbound through the sysfs files
created by default by the kernel device core.
Unwinding expectation
=====================
One could think a Type2 driver creating all those objects should control the
unwinding of all of them, not necessarily because some driver action but maybe
because the kernel signals the driver about that necessity. However, with the
current design those objects can end up being unbound through different paths
and not always the Type2 driver receives the signal or not in time to safely
stop using the CXL pathway created:
1) The pci device owning the Type2 CXL "device" unbinds from its driver.
2) The type2 driver is removed triggering the pci device unbinding.
3) The cxl memdev is unbound from the cxl mem driver through sysfs.
4) The cxl endpoint port is unbound from the cxl port driver through systfs.
5) The cxl_region is unbound from the cxl region driver through sysfs.
8) Root port linked to the device is unbound frmo the cxl port through sysfs.
9) cxl acpi kernel module is removed.
The first two cases go through the driver remove code path allowing a Type2
aware driver to stop using the CXL region memory range while all the CXL
objects are still there. Devm actions will start the CXL unwinding just after
that code path is executed with the cxl_memdev object being the first one
getting the unwinding signal, but the order here is not so relevant as there is
no client using the CXL region after the driver exit code path execution.
The next case makes ordering an issue. A cxl_memdev unbindng triggers several
demv release/clean actions:
- cxl_detach_ep
- endpoint_unregister_region
- cxld_unregister
- schedule_detach_memdev
As we can see, the chain-reaction leads to the endpoint unbinding, which in
turn leads to such endpoint unregistered from a region and that implying the
region unregistered itself and the related HDM decoder(s) "likely" being reset.
The related memdev is also scheduled for detachment. What about the Type2 driver
using the HPA linked to those objects?
Currently a Type2 driver is signalled to unbind when its memdev is going to be
removed. Following this previous chain, that happens when the memdev is going
to detach, but not when its detachment is scheduled. As stated earlier, that
implies the Type2 driver keeps using the mapping region after the objects
creating it have already disappeared. Linking a devm action between endpoint
and memdev is not so trivial and the locking issues arise quickly.
Using sysfs unbind for endpoint and region objects leads to the same problem.
Trying to change the devm chain could fix one case but easily destroying the
proper unwinding for the other ones leaving objects which should not be there
anymore.
This problem is solvable changing the way some devm action are used for Type2:
1) Any object unbinding should make the memdev to be detached first,
implying the Type2 driver will be removed as the first step in the
unwinding chain.
2) Make explicit the removal of endpoint and decoder from the memdev
detachment.
This solution implies to remove some of current devm actions linking region and
endpoint port unbinding to cxl memdev detach without allowing other actions to
happen independently of such memdev unbinding. Then some of those actions to
be invoked at the right time within the general unwinding of the chain.
For helping other people to study and visualize the problem, some tracepoints
are added hopefully showing the current unwinding and the new one after the
full patchset is applied.
Another consequence coming with this change is the state of those objects linked
to a memdev owned by a Type2 driver can only disappear through a specific order
facilitating handling potential concurrency of related functionlities like
supporting multipf where other non-PF0 functions will link to a cxl_memdev which
can onbly disappear through a well-protected and deterministic path precluding
problems with potential concurrency.
Alejandro Lucero (3):
cxl: add tracepoints to the unwinding code path
cxl: make explicit removal of endpoint and decoder for Type2
Documentation/cxl: add document about Type2 binding and unbinding
.../cxl/type2/cxl_type2_objects_unwinding.rst | 102 ++++++++++++++++++
drivers/cxl/core/memdev.c | 26 ++++-
drivers/cxl/core/port.c | 39 ++++++-
drivers/cxl/core/region.c | 42 +++++++-
drivers/cxl/cxl.h | 3 +
drivers/cxl/cxl_memdev_tracepoints.h | 58 ++++++++++
drivers/cxl/cxl_port_tracepoints.h | 91 ++++++++++++++++
drivers/cxl/cxl_region_tracepoints.h | 78 ++++++++++++++
drivers/cxl/cxlmem.h | 11 ++
drivers/cxl/port.c | 5 -
10 files changed, 443 insertions(+), 12 deletions(-)
create mode 100644 Documentation/driver-api/cxl/type2/cxl_type2_objects_unwinding.rst
create mode 100644 drivers/cxl/cxl_memdev_tracepoints.h
create mode 100644 drivers/cxl/cxl_port_tracepoints.h
create mode 100644 drivers/cxl/cxl_region_tracepoints.h
base-commit: 9d66f26fefacebfeed15de31c1123451a17ff119
--
2.34.1
next reply other threads:[~2026-11-18 17:44 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 19:14 alucerop [this message]
2026-11-18 19:14 ` [RFC 1/3] cxl: add tracepoints to the unwinding code path alucerop
2026-11-18 18:00 ` sashiko-bot
2026-11-18 19:14 ` [RFC 2/3] cxl: make explicit removal of endpoint and decoder for Type2 alucerop
2026-11-18 18:09 ` sashiko-bot
2026-11-18 19:14 ` [RFC 3/3] Documentation/cxl: add document about Type2 binding and unbinding alucerop
2026-11-18 18:04 ` sashiko-bot
2026-11-18 23:39 ` [RFC 0/3] Type2 unwinding ordering change Dave Jiang
2026-11-19 8:13 ` Lucero Palau, Alejandro
2026-11-19 10:00 ` Lucero Palau, Alejandro
2026-11-20 15:17 ` Lucero Palau, Alejandro
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261118191431.21781-1-alucerop@amd.com \
--to=alucerop@amd.com \
--cc=alison.schofield@intel.com \
--cc=dave.jiang@intel.com \
--cc=dave@stgolabs.net \
--cc=jic23@kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.