netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH net] net/mlx5: Avoid deadlock between PCI error recovery and health reporter
@ 2025-08-07 13:11 Gerd Bayer
  2025-08-10 11:51 ` Shay Drori
  0 siblings, 1 reply; 4+ messages in thread
From: Gerd Bayer @ 2025-08-07 13:11 UTC (permalink / raw)
  To: Saeed Mahameed, Leon Romanovsky, Tariq Toukan, Mark Bloch,
	Jakub Kicinski, Paolo Abeni, Moshe Shemesh, Shay Drori
  Cc: Niklas Schnelle, Alexandra Winter, Gerd Bayer, Andrew Lunn,
	David S. Miller, Eric Dumazet, Yuanyuan Zhong, Mohamed Khalfella,
	netdev, linux-rdma, linux-kernel

During error recovery testing a pair of tasks was reported to be hung
due to a dead-lock situation:

- mlx5_unload_one() trying to acquire devlink lock while the PCI error
  recovery code had acquired the pci_cfg_access_lock().
- mlx5_crdump_collect() trying to acquire the pci_cfg_access_lock()
  while devlink_health_report() had acquired the devlink lock.

Move the check for pci_channel_offline prior to acquiring the
pci_cfg_access_lock in mlx5_vsc_gw_lock since collecting the crdump will
fail anyhow while PCI error recovery is running.

Fixes: 33afbfcc105a ("net/mlx5: Stop waiting for PCI if pci channel is offline")
Signed-off-by: Gerd Bayer <gbayer@linux.ibm.com>
---

Hi all,

while the initial hit was recorded during "random" testing, where PCI
error recovery and poll_health() tripped almost simultaneously, I found
a way to reproduce a very similar hang at will on s390:

Inject a PCI error recovery event on a Physical Function <BDF> with
  zpcictl --reset-fw <BDF>

then request a crdump with
  devlink health dump show pci/<BDF> reporter fw_fatal

With the patch applied I didn't get the hang but kernel logs showed:
[  792.885743] mlx5_core 000a:00:00.0: mlx5_crdump_collect:51:(pid 1415): crdump: failed to lock vsc gw err -13

and the crdump request ended with:
kernel answers: Permission denied

Thanks, Gerd
---
 drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c
index 432c98f2626d..d2d3b57a57d5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c
@@ -73,16 +73,15 @@ int mlx5_vsc_gw_lock(struct mlx5_core_dev *dev)
 	u32 lock_val;
 	int ret;
 
+	if (pci_channel_offline(dev->pdev))
+		return -EACCES;
+
 	pci_cfg_access_lock(dev->pdev);
 	do {
 		if (retries > VSC_MAX_RETRIES) {
 			ret = -EBUSY;
 			goto pci_unlock;
 		}
-		if (pci_channel_offline(dev->pdev)) {
-			ret = -EACCES;
-			goto pci_unlock;
-		}
 
 		/* Check if semaphore is already locked */
 		ret = vsc_read(dev, VSC_SEMAPHORE_OFFSET, &lock_val);
-- 
2.48.1


^ permalink raw reply related	[flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-08-11 12:05 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-08-07 13:11 [PATCH net] net/mlx5: Avoid deadlock between PCI error recovery and health reporter Gerd Bayer
2025-08-10 11:51 ` Shay Drori
2025-08-11  7:29   ` Gerd Bayer
2025-08-11 12:04     ` Shay Drori

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).