[PATCH 0/2 V2] enable/disable IRQD_NO_BALANCING during reset

[PATCH 0/2 V2] enable/disable IRQD_NO_BALANCING during reset

[wenxiong]

From: Wen Xiong <wenxiong@linux.ibm.com>

Issue is:
Dynamic remove/add storage adapter(SCSI IPR and FC Qlogic) test
hits EEH on Powerpc.

This patchset fixes the issue with enable/disable IRQD_NO_BALANCING
during adapter reset in both of ipr and qla2xxx drivers.

V2: fixes the complier issue on x86 platform.

Wen Xiong (2):
  scsi/ipr: enable/disable IRQD_NO_BALANCING during reset
  scsi/qla2xxx: enable/disable IRQD_NO_BALANCING during reset

 drivers/scsi/ipr.c            | 28 +++++++++++++++++++++++++++-
 drivers/scsi/qla2xxx/qla_os.c | 30 ++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+), 1 deletion(-)

-- 
2.47.3

[PATCH 1/2 V2] scsi/ipr: enable/disable IRQD_NO_BALANCING during reset

[wenxiong]

From: Wen Xiong <wenxiong@linux.ibm.com>

Issue is:
Dynamic remove/add storage adapter(SCSI IPR and FC Qlogic) test
hits EEH on Powerpc.

EEH: [c00000000004f75c] __eeh_send_failure_event+0x7c/0x160
EEH: [c000000000048444] eeh_dev_check_failure.part.0+0x254/0x650
EEH: [c008000001650678] eeh_readl+0x60/0x90 [ipr]
EEH: [c00800000166746c] ipr_cancel_op+0x2b8/0x524 [ipr]
EEH: [c008000001656524] ipr_eh_abort+0x6c/0x130 [ipr]
EEH: [c000000000ab0d20] scmd_eh_abort_handler+0x140/0x440
EEH: [c00000000017e558] process_one_work+0x298/0x590
EEH: [c00000000017eef8] worker_thread+0xa8/0x620
EEH: [c00000000018be34] kthread+0x124/0x130
EEH: [c00000000000cd64] ret_from_kernel_thread+0x5c/0x64

We took a pcie bus trace and found out that a vector of msix is
cleared to 0 by irqbalance daemon.  If we disable irqbalance
daemon, we won't see the issue on both of adapters.

We enabled debug in ipr driver,
[   44.103071] ipr: Entering __ipr_remove
[   44.103083] ipr: Entering ipr_initiate_ioa_bringdown
[   44.103091] ipr: Entering ipr_reset_shutdown_ioa
[   44.103099] ipr: Leaving ipr_reset_shutdown_ioa
[   44.103105] ipr: Leaving ipr_initiate_ioa_bringdown
[   44.149918] ipr: Entering ipr_reset_ucode_download
[   44.149935] ipr: Entering ipr_reset_alert
[   44.150032] ipr: Entering ipr_reset_start_timer
[   44.150038] ipr: Leaving ipr_reset_alert
[   44.244343] scsi 1:2:3:0: alua: Detached
[   44.254300] ipr: Entering ipr_reset_start_bist
[   44.254320] ipr: Entering ipr_reset_start_timer
[   44.254325] ipr: Leaving ipr_reset_start_bist
[   44.364329] scsi 1:2:4:0: alua: Detached
[   45.134341] scsi 1:2:5:0: alua: Detached
[   45.860949] ipr: Entering ipr_reset_shutdown_ioa
[   45.860962] ipr: Leaving ipr_reset_shutdown_ioa
[   45.860966] ipr: Entering ipr_reset_alert
[   45.861028] ipr: Entering ipr_reset_start_timer
[   45.861035] ipr: Leaving ipr_reset_alert
[   45.964302] ipr: Entering ipr_reset_start_bist
[   45.964309] ipr: Entering ipr_reset_start_timer
[   45.964313] ipr: Leaving ipr_reset_start_bist
[   46.264301] ipr: Entering ipr_reset_bist_done
[   46.264309] ipr: Leaving ipr_reset_bist_done

During adapter reset, ipr device driver blocks config space access but
can't block MMIO access for msix entries.
There is very small window: irqbalance daemon kicks in during adapter
reset before ipr driver calls pci_restore_state(pdev) to restore msix
table.

irqbalance daemon reads back all 0 for that msix vector
in __pci_read_msi_msg().

irqbalance daemon:

msi_domain_set_affinity()
->irq_chip_set_affinity_patent()
->xive_irq_set_affinity()
->irq_chip_compose_msi_msg()
  ->pseries_msi_compose_msg()
  ->__pci_read_msi_msg(): read all 0 since didn't call pci_restore_state
->irq_chip_write_msi_msg()
  -> pci_write_msg_msi(): write 0 to the msix vector entry

When ipr driver call pci_restore_state(pdev) in
ipr_reset_restore_cfg_space(), the msix vector entry
has been cleared by irqbalance daemon in pci_write_msg_msix().

pci_restore_state()
->__pci_restore_msix_state()

Below is the MSIX table for ipr adapter after 'irqbalance"
dameon kicked in during adapter reset

Dump MSIx table: index=0 address_lo=c800 address_hi=10000000 msg_data=0
Dump MSIx table: index=1 address_lo=c810 address_hi=10000000 msg_data=0
Dump MSIx table: index=2 address_lo=c820 address_hi=10000000 msg_data=0
Dump MSIx table: index=3 address_lo=c830 address_hi=10000000 msg_data=0
Dump MSIx table: index=4 address_lo=c840 address_hi=10000000 msg_data=0
Dump MSIx table: index=5 address_lo=c850 address_hi=10000000 msg_data=0
Dump MSIx table: index=6 address_lo=c860 address_hi=10000000 msg_data=0
Dump MSIx table: index=7 address_lo=c870 address_hi=10000000 msg_data=0
Dump MSIx table: index=8 address_lo=0 address_hi=0 msg_data=0
---------> Hit EEH since msix vector of index=8 are 0

Dump MSIx table: index=9 address_lo=c890 address_hi=10000000 msg_data=0
Dump MSIx table: index=10 address_lo=c8a0 address_hi=10000000 msg_data=0
Dump MSIx table: index=11 address_lo=c8b0 address_hi=10000000 msg_data=0
Dump MSIx table: index=12 address_lo=c8c0 address_hi=10000000 msg_data=0
Dump MSIx table: index=13 address_lo=c8d0 address_hi=10000000 msg_data=0
Dump MSIx table: index=14 address_lo=c8e0 address_hi=10000000 msg_data=0
Dump MSIx table: index=15 address_lo=c8f0 address_hi=10000000 msg_data=0

[   46.264312] ipr: Entering ipr_reset_restore_cfg_space
[   46.267439] ipr: Entering ipr_fail_all_ops
[   46.267447] ipr: Leaving ipr_fail_all_ops
[   46.267451] ipr: Leaving ipr_reset_restore_cfg_space
[   46.267454] ipr: Entering ipr_ioa_bringdown_done
[   46.267458] ipr: Leaving ipr_ioa_bringdown_done
[   46.267467] ipr: Entering ipr_worker_thread
[   46.267470] ipr: Leaving ipr_worker_thread

We don't need to do irq balance during adapter reset.
This patch is enabled "IRQ_NO_BALANCING" bit before starting adapter
reset and disabled it after calling pci_restore_state(). irqbalance
daemon is disabled for this period of short time(~2s);

Thanks,
Wendy

Signed-off-by: Wen Xiong <wenxiong@linux.ibm.com>
Signed-off-by: Kyle Mahlkuch <Kyle.Mahlkuch@ibm.com>
---
 drivers/scsi/ipr.c | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/ipr.c b/drivers/scsi/ipr.c
index 4fb5654472d8..4eaca8777630 100644
--- a/drivers/scsi/ipr.c
+++ b/drivers/scsi/ipr.c
@@ -61,8 +61,8 @@
 #include <linux/hdreg.h>
 #include <linux/reboot.h>
 #include <linux/stringify.h>
+#include <linux/irq.h>
 #include <asm/io.h>
-#include <asm/irq.h>
 #include <asm/processor.h>
 #include <scsi/scsi.h>
 #include <scsi/scsi_host.h>
@@ -7843,6 +7843,30 @@ static int ipr_dump_mailbox_wait(struct ipr_cmnd *ipr_cmd)
 	return IPR_RC_JOB_RETURN;
 }
 
+/**
+ * ipr_set_affinity_nobalance
+ * @ioa_cfg:	ipr_ioa_cfg struct for an ipr device
+ * @flag:	bool
+ *	true: ensable "IRQ_NO_BALANCING" bit for msix interrupt
+ *	false: disable "IRQ_NO_BALANCING" bit for msix interrupt
+ * Description: This function will be called to disable/enable
+ *	"IRQ_NO_BALANCING" to avoid irqbalance daemon
+ *	kicking in during adapter reset.
+ **/
+static void ipr_set_affinity_nobalance(struct ipr_ioa_cfg *ioa_cfg, bool flag)
+{
+	int irq, i;
+
+	for (i = 0; i < ioa_cfg->nvectors; i++) {
+		irq = pci_irq_vector(ioa_cfg->pdev, i);
+
+		if (flag)
+			irq_set_status_flags(irq, IRQ_NO_BALANCING);
+		else
+			irq_clear_status_flags(irq, IRQ_NO_BALANCING);
+	}
+}
+
 /**
  * ipr_reset_restore_cfg_space - Restore PCI config space.
  * @ipr_cmd:	ipr command struct
@@ -7867,6 +7891,7 @@ static int ipr_reset_restore_cfg_space(struct ipr_cmnd *ipr_cmd)
 		return IPR_RC_JOB_CONTINUE;
 	}
 
+	ipr_set_affinity_nobalance(ioa_cfg, false);
 	ipr_fail_all_ops(ioa_cfg);
 
 	if (ioa_cfg->sis64) {
@@ -7946,6 +7971,7 @@ static int ipr_reset_start_bist(struct ipr_cmnd *ipr_cmd)
 		rc = pci_write_config_byte(ioa_cfg->pdev, PCI_BIST, PCI_BIST_START);
 
 	if (rc == PCIBIOS_SUCCESSFUL) {
+		ipr_set_affinity_nobalance(ioa_cfg, true);
 		ipr_cmd->job_step = ipr_reset_bist_done;
 		ipr_reset_start_timer(ipr_cmd, IPR_WAIT_FOR_BIST_TIMEOUT);
 		rc = IPR_RC_JOB_RETURN;
-- 
2.47.3

[PATCH 2/2 V2] scsi/qla2xxx: enable/disable IRQD_NO_BALANCING during reset

[wenxiong]

From: Wen Xiong <wenxiong@linux.ibm.com>

Issue is:
Dynamic remove/add of FC Qlogic storage adapters test hits EEH on Powerpc.

EEH: [c00000000004f77c] __eeh_send_failure_event+0x7c/0x160
EEH: [c000000000048464] eeh_dev_check_failure.part.0+0x254/0x660
EEH: [c000000000934e0c] __pci_read_msi_msg+0x1ac/0x280
EEH: [c000000000100f68] pseries_msi_compose_msg+0x28/0x40
EEH: [c00000000020e1cc] irq_chip_compose_msi_msg+0x5c/0x90
EEH: [c000000000214b1c] msi_domain_set_affinity+0xbc/0x100
EEH: [c000000000206be4] irq_do_set_affinity+0x214/0x2c0
EEH: [c000000000206e04] irq_set_affinity_locked+0x174/0x230
EEH: [c000000000207044] irq_set_affinity+0x64/0xa0
EEH: [c000000000212890] write_irq_affinity.constprop.0.isra.0+0x130/0x150
EEH: [c00000000068868c] proc_reg_write+0xfc/0x160
EEH: [c0000000005adb48] vfs_write+0xf8/0x4e0
EEH: [c0000000005ae234] ksys_write+0x84/0x140
EEH: [c00000000002e994] system_call_exception+0x164/0x310
EEH: [c00000000000bfe8] system_call_vectored_common+0xe8/0x278

We see that irqbalance daemon kicks in before invoking qla2xxx->slot_reset
during the EEH recovery process.

irqbalance daemon
->irq_set_affinity()
->msi_domain_set_affinity()
->irq_chip_set_affiinity_parent()
->xive_irq_set_affinity()
->pseries_msi_compose_ms()
->__pci_read_msi_msg()
->irq_chip_compose_msi_msg()

In __pci_read_msi_msg(), we see the first msix vector is set to all F by the
irqbalance daemon.
pci_write_msg_msix: index=0, lo=ffffffff hi=fffffff

We tried disabling the irqbalance daemon, then ran the DLPAR remove/add
test and the test completed succesfully.

We don't need to do irq balance during adapter reset.
This patch enables "IRQ_NO_BALANCING" bit before starting adapter
reset and disables the bit after calling pci_restore_state(). The irqbalance
daemon is disabled for this short period of ~2s.

Thanks,
Kyle

Signed-off-by: Kyle Mahlkuch <Kyle.Mahlkuch@ibm.com>
Signed-off-by: Wen Xiong <wenxiong@linux.ibm.com>
---
 drivers/scsi/qla2xxx/qla_os.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c
index 9a2f328200ab..c5e91f5779e8 100644
--- a/drivers/scsi/qla2xxx/qla_os.c
+++ b/drivers/scsi/qla2xxx/qla_os.c
@@ -17,6 +17,7 @@
 #include <linux/crash_dump.h>
 #include <linux/trace_events.h>
 #include <linux/trace.h>
+#include <linux/irq.h>
 
 #include <scsi/scsi_tcq.h>
 #include <scsi/scsicam.h>
@@ -7771,6 +7772,31 @@ static void qla_pci_error_cleanup(scsi_qla_host_t *vha)
 }
 
 
+/**
+ * qla2xxx_set_affinity_nobalance
+ * @pdev: pci_dev struct for a qla2xxx device
+ * @flag: bool
+ * true: enable "IRQ_NO_BALANCING" bit for msix interrupt
+ * false: disable "IRQ_NO_BALANCING" bit for msix interrupt
+ * Description: This function will be called to disable/enable
+ * "IRQ_NO_BALANCING" to avoid irqbalance daemon
+ * kicking in during adapter reset.
+ **/
+
+static void qla2xxx_set_affinity_nobalance(struct pci_dev *pdev, bool flag)
+{
+	int irq, i;
+
+	for (i = 0; i < QLA_BASE_VECTORS; i++) {
+		irq = pci_irq_vector(pdev, i);
+
+		if (flag)
+			irq_set_status_flags(irq, IRQ_NO_BALANCING);
+		else
+			irq_clear_status_flags(irq, IRQ_NO_BALANCING);
+	}
+}
+
 static pci_ers_result_t
 qla2xxx_pci_error_detected(struct pci_dev *pdev, pci_channel_state_t state)
 {
@@ -7789,6 +7815,8 @@ qla2xxx_pci_error_detected(struct pci_dev *pdev, pci_channel_state_t state)
 		goto out;
 	}
 
+	qla2xxx_set_affinity_nobalance(pdev, false);
+
 	switch (state) {
 	case pci_channel_io_normal:
 		qla_pci_set_eeh_busy(vha);
@@ -7935,6 +7963,8 @@ qla2xxx_pci_slot_reset(struct pci_dev *pdev)
 	ql_dbg(ql_dbg_aer, base_vha, 0x900e,
 	    "Slot Reset returning %x.\n", ret);
 
+	qla2xxx_set_affinity_nobalance(pdev, true);
+
 	return ret;
 }
 
-- 
2.47.3

Re: [PATCH 0/2 V2] enable/disable IRQD_NO_BALANCING during reset

[Martin K. Petersen]

Wen,

> Issue is: Dynamic remove/add storage adapter(SCSI IPR and FC Qlogic)
> test hits EEH on Powerpc.
>
> This patchset fixes the issue with enable/disable IRQD_NO_BALANCING
> during adapter reset in both of ipr and qla2xxx drivers.

Applied to 6.19/scsi-staging, thanks!

-- 
Martin K. Petersen

Re: [PATCH 0/2 V2] enable/disable IRQD_NO_BALANCING during reset

[Martin K. Petersen]

On Tue, 28 Oct 2025 09:24:25 -0500, wenxiong@linux.ibm.com wrote:

> Issue is:
> Dynamic remove/add storage adapter(SCSI IPR and FC Qlogic) test
> hits EEH on Powerpc.
> 
> This patchset fixes the issue with enable/disable IRQD_NO_BALANCING
> during adapter reset in both of ipr and qla2xxx drivers.
> 
> [...]

Applied to 6.19/scsi-queue, thanks!

[1/2] scsi/ipr: enable/disable IRQD_NO_BALANCING during reset
      https://git.kernel.org/mkp/scsi/c/6ac3484fb13b
[2/2] scsi/qla2xxx: enable/disable IRQD_NO_BALANCING during reset
      https://git.kernel.org/mkp/scsi/c/eaea513077cd

-- 
Martin K. Petersen