[PATCH v4 0/2] cxl/memdev: Fix poison debugfs vs unbind deadlock

[PATCH v4 0/2] cxl/memdev: Fix poison debugfs vs unbind deadlock

[Guixin Liu]

Writing a memdev poison debugfs file while cxl_mem is being unbound
deadlocks. Patch 2 fixes it by not waiting for the device lock in those
handlers. Patch 1 adds the trylock guard it uses.

Patch 2 does not build without patch 1, so the two need to travel
together.

Testing:

Reproduced on a QEMU CXL topology whose type3 devices advertise poison
inject support:

  while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done &
  while :; do
      echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
      echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
  done

Without the fix the unbind wedges within seconds. The three tasks
involved, from /proc/<pid>/stack:

  writer, state S, holds the debugfs reference and waits for the lock
    cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
    debugfs_attr_write+0x61/0xb0
    full_proxy_write+0xfc/0x1c0
    vfs_write+0x1d4/0xe60

  unbind, state D, holds the lock and waits for the reference to drain
    remove_one+0x27f/0x3d0
    debugfs_remove+0x44/0x60
    release_nodes+0xfa/0x2c0
    devres_release_all+0x113/0x1a0
    device_unbind_cleanup+0x76/0x260
    device_release_driver_internal+0x3eb/0x540
    unbind_store+0xde/0x100

  cxl_port workqueue, state D, blocked on the same lock
    device_release_driver_internal+0x96/0x540
    detach_memdev+0x79/0xb0 [cxl_core]
    process_one_work+0x6b0/0xfb0

The writer is in interruptible sleep and can be killed; the unbind
cannot, and because cxl_bus_wq is an ordered workqueue the wedged
detach_memdev() blocks every other CXL bus work item behind it.

The same deadlock shows up with a pciehp hot-remove racing the writer
loop: the pciehp thread hits the same debugfs_remove() drain holding
the memdev lock, and the hot-remove wedges the same way. With both
patches applied the hot-remove flow completes normally.

With both patches applied, 46 unbind/bind cycles against the same writer
loop all completed, no task was left in D state, and the writer collected
10920 EBUSY returns from the contended trylock. Note that lockdep stays
quiet either way: one leg of the cycle is the debugfs active_users
completion rather than a lock it tracks.

v3 -> v4:
- reword the patch 2 commit message per Alison: enumerate the teardown
  paths that race a poison write into an unbind, and state that mixing
  poison writes with cxl_mem teardown is not a supported use of this
  debug ABI, though the fix keeps the cost of doing so to a failed write
- add the pciehp hot-remove reproduction, reported during v3 review,
  to the patch 2 commit message
- collect Jonathan's Reviewed-by on both patches (no code change)

v1 -> v2:
- add the device_trylock() guard and use ACQUIRE(device_try, ...) instead
  of open-coding device_trylock()/device_unlock(), keeping the style the
  Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and
  the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was tested

v2 -> v3:
- rebase onto v7.3-rc2 (master), per Dave's request to send the series
  against Linus's tags rather than cxl/next

v1:
https://lore.kernel.org/linux-cxl/20260826125248.4003792-1-kanie@linux.alibaba.com/

v2:
https://lore.kernel.org/linux-cxl/20260831124809.889829-1-kanie@linux.alibaba.com/

v3:
https://lore.kernel.org/linux-cxl/20260910094017.4032170-1-kanie@linux.alibaba.com/

Guixin Liu (2):
  driver core: Add conditional guard support for device_trylock()
  cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

 drivers/cxl/mem.c      | 14 ++++++++++----
 include/linux/device.h |  1 +
 2 files changed, 11 insertions(+), 4 deletions(-)

-- 
2.43.7

[PATCH v4 1/2] driver core: Add conditional guard support for device_trylock()

[Guixin Liu]

Introduce a conditional guard version of device_trylock() for scenarios
that must not block on the device lock. device_trylock() returns 1 on
success like mutex_trylock(), so the default binary condition applies and
ACQUIRE_ERR() reports -EBUSY on contention.

Suggested-by: Shaikh Kamaluddin <shaikhkamal2012@gmail.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@oss.qualcomm.com>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
---
 include/linux/device.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/device.h b/include/linux/device.h
index aee79fd6b32b..a8d6cf76d137 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -1121,6 +1121,7 @@ static inline void device_unlock(struct device *dev)
 
 DEFINE_GUARD(device, struct device *, device_lock(_T), device_unlock(_T))
 DEFINE_GUARD_COND(device, _intr, device_lock_interruptible(_T), _RET == 0)
+DEFINE_GUARD_COND(device, _try, device_trylock(_T))
 
 static inline void device_lock_assert(struct device *dev)
 {
-- 
2.43.7

[PATCH v4 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

[Guixin Liu]

The poison debugfs handlers take the memdev device lock so that the
region lookup sees a stable cxlmd->dev.driver. debugfs holds a
reference on the file across the handler, and cxl_mem unbind removes
that file while holding the very same device lock, so a handler that
waits for the lock deadlocks against a concurrent unbind.

The trigger is any teardown that detaches cxl_mem from the memdev
while a poison write is in flight: unbinding or unloading cxl_mem or
cxl_pci, or removing the endpoint PCI device, including hot-remove.

Both tasks then hang. The unbind side is uninterruptible, and it also
blocks the memdev detach work, which runs on an ordered workqueue and so
stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of
waiting. An unbind that wins the race removes the file first and the
write fails with -ENOENT.

The poison inject and clear files are a debug ABI for expert users,
mostly device vendors, to test the poison capabilities of their
devices. Mixing a poison write with cxl_mem teardown is not a supported
use of the interface, but when it happens anyway the cost should be a
failed write, not a hung kernel.

Found by code inspection. Reproduced by writing inject_poison in a loop
while unbinding and rebinding cxl_mem, and confirmed fixed by the same
test. Also reproduced the same deadlock with those writes racing a
pciehp hot-remove of the memdev; with this patch the hot-remove flow
completes normally.

Fixes: 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")
Suggested-by: Shaikh Kamaluddin <shaikhkamal2012@gmail.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@oss.qualcomm.com>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
---
 drivers/cxl/mem.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 798e5c369cfc..3959ec963026 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -50,8 +50,13 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/*
+	 * Never wait for this lock: the debugfs proxy holds a file reference
+	 * across the callback and unbind removes the file under the same
+	 * device lock, so waiting here deadlocks against unbind.
+	 */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
 		return rc;
 
 	return cxl_inject_poison(cxlmd, dpa);
@@ -65,8 +70,9 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/* Never wait, per the inject path above. */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
 		return rc;
 
 	return cxl_clear_poison(cxlmd, dpa);
-- 
2.43.7

Re: [PATCH v4 0/2] cxl/memdev: Fix poison debugfs vs unbind deadlock

[Dave Jiang]

On 9/15/26 7:54 PM, Guixin Liu wrote:
> Writing a memdev poison debugfs file while cxl_mem is being unbound
> deadlocks. Patch 2 fixes it by not waiting for the device lock in those
> handlers. Patch 1 adds the trylock guard it uses.
> 
> Patch 2 does not build without patch 1, so the two need to travel
> together.
> 
> Testing:
> 
> Reproduced on a QEMU CXL topology whose type3 devices advertise poison
> inject support:
> 
>   while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done &
>   while :; do
>       echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
>       echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
>   done
> 
> Without the fix the unbind wedges within seconds. The three tasks
> involved, from /proc/<pid>/stack:
> 
>   writer, state S, holds the debugfs reference and waits for the lock
>     cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
>     debugfs_attr_write+0x61/0xb0
>     full_proxy_write+0xfc/0x1c0
>     vfs_write+0x1d4/0xe60
> 
>   unbind, state D, holds the lock and waits for the reference to drain
>     remove_one+0x27f/0x3d0
>     debugfs_remove+0x44/0x60
>     release_nodes+0xfa/0x2c0
>     devres_release_all+0x113/0x1a0
>     device_unbind_cleanup+0x76/0x260
>     device_release_driver_internal+0x3eb/0x540
>     unbind_store+0xde/0x100
> 
>   cxl_port workqueue, state D, blocked on the same lock
>     device_release_driver_internal+0x96/0x540
>     detach_memdev+0x79/0xb0 [cxl_core]
>     process_one_work+0x6b0/0xfb0
> 
> The writer is in interruptible sleep and can be killed; the unbind
> cannot, and because cxl_bus_wq is an ordered workqueue the wedged
> detach_memdev() blocks every other CXL bus work item behind it.
> 
> The same deadlock shows up with a pciehp hot-remove racing the writer
> loop: the pciehp thread hits the same debugfs_remove() drain holding
> the memdev lock, and the hot-remove wedges the same way. With both
> patches applied the hot-remove flow completes normally.
> 
> With both patches applied, 46 unbind/bind cycles against the same writer
> loop all completed, no task was left in D state, and the writer collected
> 10920 EBUSY returns from the contended trylock. Note that lockdep stays
> quiet either way: one leg of the cycle is the debugfs active_users
> completion rather than a lock it tracks.
> 
> v3 -> v4:
> - reword the patch 2 commit message per Alison: enumerate the teardown
>   paths that race a poison write into an unbind, and state that mixing
>   poison writes with cxl_mem teardown is not a supported use of this
>   debug ABI, though the fix keeps the cost of doing so to a failed write
> - add the pciehp hot-remove reproduction, reported during v3 review,
>   to the patch 2 commit message
> - collect Jonathan's Reviewed-by on both patches (no code change)
> 
> v1 -> v2:
> - add the device_trylock() guard and use ACQUIRE(device_try, ...) instead
>   of open-coding device_trylock()/device_unlock(), keeping the style the
>   Fixes: commit established (Shaikh Kamaluddin)
> - cut the changelog down to the failing condition, the consequence and
>   the fix; the call graph and the reproducer live here instead
> - say how the issue was found and how it was tested
> 
> v2 -> v3:
> - rebase onto v7.3-rc2 (master), per Dave's request to send the series
>   against Linus's tags rather than cxl/next
> 
> v1:
> https://lore.kernel.org/linux-cxl/20260826125248.4003792-1-kanie@linux.alibaba.com/
> 
> v2:
> https://lore.kernel.org/linux-cxl/20260831124809.889829-1-kanie@linux.alibaba.com/
> 
> v3:
> https://lore.kernel.org/linux-cxl/20260910094017.4032170-1-kanie@linux.alibaba.com/
> 
> Guixin Liu (2):
>   driver core: Add conditional guard support for device_trylock()
>   cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind
> 
>  drivers/cxl/mem.c      | 14 ++++++++++----
>  include/linux/device.h |  1 +
>  2 files changed, 11 insertions(+), 4 deletions(-)
> 

For the series
Reviewed-by: Dave Jiang <dave.jiang@intel.com>

Greg,
I can take the patches through the CXL tree if you ack the first patch. Thanks!

Re: [PATCH v4 0/2] cxl/memdev: Fix poison debugfs vs unbind deadlock

[Greg Kroah-Hartman]

On Fri, Sep 18, 2026 at 08:26:28AM -0700, Dave Jiang wrote:
> 
> 
> On 9/15/26 7:54 PM, Guixin Liu wrote:
> > Writing a memdev poison debugfs file while cxl_mem is being unbound
> > deadlocks. Patch 2 fixes it by not waiting for the device lock in those
> > handlers. Patch 1 adds the trylock guard it uses.
> > 
> > Patch 2 does not build without patch 1, so the two need to travel
> > together.
> > 
> > Testing:
> > 
> > Reproduced on a QEMU CXL topology whose type3 devices advertise poison
> > inject support:
> > 
> >   while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done &
> >   while :; do
> >       echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
> >       echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
> >   done
> > 
> > Without the fix the unbind wedges within seconds. The three tasks
> > involved, from /proc/<pid>/stack:
> > 
> >   writer, state S, holds the debugfs reference and waits for the lock
> >     cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
> >     debugfs_attr_write+0x61/0xb0
> >     full_proxy_write+0xfc/0x1c0
> >     vfs_write+0x1d4/0xe60
> > 
> >   unbind, state D, holds the lock and waits for the reference to drain
> >     remove_one+0x27f/0x3d0
> >     debugfs_remove+0x44/0x60
> >     release_nodes+0xfa/0x2c0
> >     devres_release_all+0x113/0x1a0
> >     device_unbind_cleanup+0x76/0x260
> >     device_release_driver_internal+0x3eb/0x540
> >     unbind_store+0xde/0x100
> > 
> >   cxl_port workqueue, state D, blocked on the same lock
> >     device_release_driver_internal+0x96/0x540
> >     detach_memdev+0x79/0xb0 [cxl_core]
> >     process_one_work+0x6b0/0xfb0
> > 
> > The writer is in interruptible sleep and can be killed; the unbind
> > cannot, and because cxl_bus_wq is an ordered workqueue the wedged
> > detach_memdev() blocks every other CXL bus work item behind it.
> > 
> > The same deadlock shows up with a pciehp hot-remove racing the writer
> > loop: the pciehp thread hits the same debugfs_remove() drain holding
> > the memdev lock, and the hot-remove wedges the same way. With both
> > patches applied the hot-remove flow completes normally.
> > 
> > With both patches applied, 46 unbind/bind cycles against the same writer
> > loop all completed, no task was left in D state, and the writer collected
> > 10920 EBUSY returns from the contended trylock. Note that lockdep stays
> > quiet either way: one leg of the cycle is the debugfs active_users
> > completion rather than a lock it tracks.
> > 
> > v3 -> v4:
> > - reword the patch 2 commit message per Alison: enumerate the teardown
> >   paths that race a poison write into an unbind, and state that mixing
> >   poison writes with cxl_mem teardown is not a supported use of this
> >   debug ABI, though the fix keeps the cost of doing so to a failed write
> > - add the pciehp hot-remove reproduction, reported during v3 review,
> >   to the patch 2 commit message
> > - collect Jonathan's Reviewed-by on both patches (no code change)
> > 
> > v1 -> v2:
> > - add the device_trylock() guard and use ACQUIRE(device_try, ...) instead
> >   of open-coding device_trylock()/device_unlock(), keeping the style the
> >   Fixes: commit established (Shaikh Kamaluddin)
> > - cut the changelog down to the failing condition, the consequence and
> >   the fix; the call graph and the reproducer live here instead
> > - say how the issue was found and how it was tested
> > 
> > v2 -> v3:
> > - rebase onto v7.3-rc2 (master), per Dave's request to send the series
> >   against Linus's tags rather than cxl/next
> > 
> > v1:
> > https://lore.kernel.org/linux-cxl/20260826125248.4003792-1-kanie@linux.alibaba.com/
> > 
> > v2:
> > https://lore.kernel.org/linux-cxl/20260831124809.889829-1-kanie@linux.alibaba.com/
> > 
> > v3:
> > https://lore.kernel.org/linux-cxl/20260910094017.4032170-1-kanie@linux.alibaba.com/
> > 
> > Guixin Liu (2):
> >   driver core: Add conditional guard support for device_trylock()
> >   cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind
> > 
> >  drivers/cxl/mem.c      | 14 ++++++++++----
> >  include/linux/device.h |  1 +
> >  2 files changed, 11 insertions(+), 4 deletions(-)
> > 
> 
> For the series
> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
> 
> Greg,
> I can take the patches through the CXL tree if you ack the first patch. Thanks!

Please do!

Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

Re: [PATCH v4 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

[Alison Schofield]

On Wed, Sep 16, 2026 at 10:54:53AM +0800, Guixin Liu wrote:
> The poison debugfs handlers take the memdev device lock so that the
> region lookup sees a stable cxlmd->dev.driver. debugfs holds a
> reference on the file across the handler, and cxl_mem unbind removes
> that file while holding the very same device lock, so a handler that
> waits for the lock deadlocks against a concurrent unbind.
> 
> The trigger is any teardown that detaches cxl_mem from the memdev
> while a poison write is in flight: unbinding or unloading cxl_mem or
> cxl_pci, or removing the endpoint PCI device, including hot-remove.
> 
> Both tasks then hang. The unbind side is uninterruptible, and it also
> blocks the memdev detach work, which runs on an ordered workqueue and so
> stalls every other CXL bus work item.
> 
> Take the lock with the trylock guard and return -EBUSY instead of
> waiting. An unbind that wins the race removes the file first and the
> write fails with -ENOENT.
> 
> The poison inject and clear files are a debug ABI for expert users,
> mostly device vendors, to test the poison capabilities of their
> devices. Mixing a poison write with cxl_mem teardown is not a supported
> use of the interface, but when it happens anyway the cost should be a
> failed write, not a hung kernel.
> 
> Found by code inspection. Reproduced by writing inject_poison in a loop
> while unbinding and rebinding cxl_mem, and confirmed fixed by the same
> test. Also reproduced the same deadlock with those writes racing a
> pciehp hot-remove of the memdev; with this patch the hot-remove flow
> completes normally.

Reviewed-by: Alison Schofield <alison.schofield@intel.com>

Re: [PATCH v4 0/2] cxl/memdev: Fix poison debugfs vs unbind deadlock

[Dave Jiang]

On 9/15/26 7:54 PM, Guixin Liu wrote:
> Writing a memdev poison debugfs file while cxl_mem is being unbound
> deadlocks. Patch 2 fixes it by not waiting for the device lock in those
> handlers. Patch 1 adds the trylock guard it uses.
> 
> Patch 2 does not build without patch 1, so the two need to travel
> together.
> 
> Testing:
> 
> Reproduced on a QEMU CXL topology whose type3 devices advertise poison
> inject support:
> 
>   while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done &
>   while :; do
>       echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
>       echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
>   done
> 
> Without the fix the unbind wedges within seconds. The three tasks
> involved, from /proc/<pid>/stack:
> 
>   writer, state S, holds the debugfs reference and waits for the lock
>     cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
>     debugfs_attr_write+0x61/0xb0
>     full_proxy_write+0xfc/0x1c0
>     vfs_write+0x1d4/0xe60
> 
>   unbind, state D, holds the lock and waits for the reference to drain
>     remove_one+0x27f/0x3d0
>     debugfs_remove+0x44/0x60
>     release_nodes+0xfa/0x2c0
>     devres_release_all+0x113/0x1a0
>     device_unbind_cleanup+0x76/0x260
>     device_release_driver_internal+0x3eb/0x540
>     unbind_store+0xde/0x100
> 
>   cxl_port workqueue, state D, blocked on the same lock
>     device_release_driver_internal+0x96/0x540
>     detach_memdev+0x79/0xb0 [cxl_core]
>     process_one_work+0x6b0/0xfb0
> 
> The writer is in interruptible sleep and can be killed; the unbind
> cannot, and because cxl_bus_wq is an ordered workqueue the wedged
> detach_memdev() blocks every other CXL bus work item behind it.
> 
> The same deadlock shows up with a pciehp hot-remove racing the writer
> loop: the pciehp thread hits the same debugfs_remove() drain holding
> the memdev lock, and the hot-remove wedges the same way. With both
> patches applied the hot-remove flow completes normally.
> 
> With both patches applied, 46 unbind/bind cycles against the same writer
> loop all completed, no task was left in D state, and the writer collected
> 10920 EBUSY returns from the contended trylock. Note that lockdep stays
> quiet either way: one leg of the cycle is the debugfs active_users
> completion rather than a lock it tracks.
> 
> v3 -> v4:
> - reword the patch 2 commit message per Alison: enumerate the teardown
>   paths that race a poison write into an unbind, and state that mixing
>   poison writes with cxl_mem teardown is not a supported use of this
>   debug ABI, though the fix keeps the cost of doing so to a failed write
> - add the pciehp hot-remove reproduction, reported during v3 review,
>   to the patch 2 commit message
> - collect Jonathan's Reviewed-by on both patches (no code change)
> 
> v1 -> v2:
> - add the device_trylock() guard and use ACQUIRE(device_try, ...) instead
>   of open-coding device_trylock()/device_unlock(), keeping the style the
>   Fixes: commit established (Shaikh Kamaluddin)
> - cut the changelog down to the failing condition, the consequence and
>   the fix; the call graph and the reproducer live here instead
> - say how the issue was found and how it was tested
> 
> v2 -> v3:
> - rebase onto v7.3-rc2 (master), per Dave's request to send the series
>   against Linus's tags rather than cxl/next
> 
> v1:
> https://lore.kernel.org/linux-cxl/20260826125248.4003792-1-kanie@linux.alibaba.com/
> 
> v2:
> https://lore.kernel.org/linux-cxl/20260831124809.889829-1-kanie@linux.alibaba.com/
> 
> v3:
> https://lore.kernel.org/linux-cxl/20260910094017.4032170-1-kanie@linux.alibaba.com/
> 
> Guixin Liu (2):
>   driver core: Add conditional guard support for device_trylock()
>   cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind
> 
>  drivers/cxl/mem.c      | 14 ++++++++++----
>  include/linux/device.h |  1 +
>  2 files changed, 11 insertions(+), 4 deletions(-)
> 


Applied to cxl/next:
18f269585447
391bc7ab9be6