[PATCH] net/mlx5: poll mlx5 eq during irq migration

[PATCH] net/mlx5: poll mlx5 eq during irq migration

[Praveen Kumar Kannoju]

Interrupt lost scenario has been observed in multiple issues during IRQ
migration due to cpu scaling activity. This further led to the presence of
unhandled EQE's causing corresponding Mellanox transmission queues to
become full and get timedout. This patch overcomes this situation by
polling the EQ associated with the IRQ which undergoes migration, to
recover any unhandled EQE's and keep the transmission uninterrupted from
the corresponding queue.

Signed-off-by: Praveen Kumar Kannoju <praveen.kannoju@oracle.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/eq.c  | 41 +++++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/lib/eq.h  |  1 +
 2 files changed, 42 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
index 25499da177bc..4f0653305f46 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
@@ -22,6 +22,10 @@
 #include "devlink.h"
 #include "en_accel/ipsec.h"
 
+unsigned int mlx5_reap_eq_irq_aff_change;
+module_param(mlx5_reap_eq_irq_aff_change, int, 0644);
+MODULE_PARM_DESC(mlx5_reap_eq_irq_aff_change, "mlx5_reap_eq_irq_aff_change: 0 = Disable MLX5 EQ Reap upon IRQ affinity change, \
+		 1 = Enable MLX5 EQ Reap upon IRQ affinity change. Default=0");
 enum {
 	MLX5_EQE_OWNER_INIT_VAL	= 0x1,
 };
@@ -951,10 +955,36 @@ static int alloc_rmap(struct mlx5_core_dev *mdev) { return 0; }
 static void free_rmap(struct mlx5_core_dev *mdev) {}
 #endif
 
+void mlx5_eq_reap_irq_notify(struct irq_affinity_notify *notify, const cpumask_t *mask)
+{
+	u32 eqe_count;
+	struct mlx5_eq_comp *eq = container_of(notify, struct mlx5_eq_comp, notify);
+
+	if (mlx5_reap_eq_irq_aff_change) {
+		mlx5_core_warn(eq->core.dev, "irqn = 0x%x migration notified, EQ 0x%x: Cons = 0x%x\n",
+			       eq->core.irqn, eq->core.eqn, eq->core.cons_index);
+
+		while (!rtnl_trylock())
+			msleep(20);
+
+		eqe_count = mlx5_eq_poll_irq_disabled(eq);
+		if (eqe_count)
+			mlx5_core_warn(eq->core.dev, "Recovered %d eqes on EQ 0x%x\n",
+				       eqe_count, eq->core.eqn);
+		rtnl_unlock();
+	}
+}
+
+void mlx5_eq_reap_irq_release(struct kref *ref) {}
+
 static void destroy_comp_eq(struct mlx5_core_dev *dev, struct mlx5_eq_comp *eq, u16 vecidx)
 {
 	struct mlx5_eq_table *table = dev->priv.eq_table;
 
+	if (irq_set_affinity_notifier(eq->core.irqn, NULL))
+		mlx5_core_warn(dev, "failed to unset EQ 0x%x to irq 0x%x affinty\n",
+			       eq->core.eqn, eq->core.irqn);
+
 	xa_erase(&table->comp_eqs, vecidx);
 	mlx5_eq_disable(dev, &eq->core, &eq->irq_nb);
 	if (destroy_unmap_eq(dev, &eq->core))
@@ -990,6 +1020,7 @@ static int create_comp_eq(struct mlx5_core_dev *dev, u16 vecidx)
 	struct mlx5_irq *irq;
 	int nent;
 	int err;
+	int ret;
 
 	lockdep_assert_held(&table->comp_lock);
 	if (table->curr_comp_eqs == table->max_comp_eqs) {
@@ -1036,6 +1067,16 @@ static int create_comp_eq(struct mlx5_core_dev *dev, u16 vecidx)
 	if (err)
 		goto disable_eq;
 
+	eq->notify.notify = mlx5_eq_reap_irq_notify;
+	eq->notify.release = mlx5_eq_reap_irq_release;
+	ret = irq_set_affinity_notifier(eq->core.irqn, &eq->notify);
+	if (ret) {
+		mlx5_core_warn(dev, "mlx5_eq_reap_irq_nofifier: EQ 0x%x irqn = 0x%x irq_set_affinity_notifier failed: %d\n",
+			       eq->core.eqn, eq->core.irqn, ret);
+	}
+	mlx5_core_dbg(dev, "mlx5_eq_reap_irq_nofifier: EQ 0x%x irqn = 0x%x irq_set_affinity_notifier set.\n",
+		      eq->core.eqn, eq->core.irqn);
+
 	table->curr_comp_eqs++;
 	return eq->core.eqn;
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
index b1edc71ffc6d..669bacb9e390 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
@@ -46,6 +46,7 @@ struct mlx5_eq_comp {
 	struct notifier_block   irq_nb;
 	struct mlx5_eq_tasklet  tasklet_ctx;
 	struct list_head        list;
+	struct irq_affinity_notify notify;
 };
 
 static inline u32 eq_get_size(struct mlx5_eq *eq)
-- 
2.43.7

Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[Jason Gunthorpe]

On Wed, Mar 04, 2026 at 04:17:04PM +0000, Praveen Kumar Kannoju wrote:
> Interrupt lost scenario has been observed in multiple issues during IRQ
> migration due to cpu scaling activity. This further led to the presence of
> unhandled EQE's causing corresponding Mellanox transmission queues to
> become full and get timedout. This patch overcomes this situation by
> polling the EQ associated with the IRQ which undergoes migration, to
> recover any unhandled EQE's and keep the transmission uninterrupted from
> the corresponding queue.

What? This does not seem like something we should do like this.

IRQ migration is not supposed to loose interrupts, this seems like a
IRQ layer bug to me. If it is buggy and loosing interrupts it should
probably inject a spurious interrupt around these events so all
devices can be enjoy the bug fix.

Basically you need to explain with alot more detail why the IRQ was
lost, not just some hand wavey "migration something something"..

BTW there are known bugs in things like qemu that can loose interrupts
around changes to the MSI (and worse than that too), but I thought
they were all fixed now?

Jason

Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[kernel test robot]

Hi Praveen,

kernel test robot noticed the following build warnings:

[auto build test WARNING on net-next/main]
[also build test WARNING on net/main linus/master v6.16-rc1 next-20260304]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Praveen-Kumar-Kannoju/net-mlx5-poll-mlx5-eq-during-irq-migration/20260305-003505
base:   net-next/main
patch link:    https://lore.kernel.org/r/20260304161704.910564-1-praveen.kannoju%40oracle.com
patch subject: [PATCH] net/mlx5: poll mlx5 eq during irq migration
config: x86_64-rhel-9.4-ltp (https://download.01.org/0day-ci/archive/20260305/202603050528.5JWnahEr-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260305/202603050528.5JWnahEr-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202603050528.5JWnahEr-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> drivers/net/ethernet/mellanox/mlx5/core/eq.c:958:6: warning: no previous prototype for 'mlx5_eq_reap_irq_notify' [-Wmissing-prototypes]
     958 | void mlx5_eq_reap_irq_notify(struct irq_affinity_notify *notify, const cpumask_t *mask)
         |      ^~~~~~~~~~~~~~~~~~~~~~~
>> drivers/net/ethernet/mellanox/mlx5/core/eq.c:978:6: warning: no previous prototype for 'mlx5_eq_reap_irq_release' [-Wmissing-prototypes]
     978 | void mlx5_eq_reap_irq_release(struct kref *ref) {}
         |      ^~~~~~~~~~~~~~~~~~~~~~~~


vim +/mlx5_eq_reap_irq_notify +958 drivers/net/ethernet/mellanox/mlx5/core/eq.c

   957	
 > 958	void mlx5_eq_reap_irq_notify(struct irq_affinity_notify *notify, const cpumask_t *mask)
   959	{
   960		u32 eqe_count;
   961		struct mlx5_eq_comp *eq = container_of(notify, struct mlx5_eq_comp, notify);
   962	
   963		if (mlx5_reap_eq_irq_aff_change) {
   964			mlx5_core_warn(eq->core.dev, "irqn = 0x%x migration notified, EQ 0x%x: Cons = 0x%x\n",
   965				       eq->core.irqn, eq->core.eqn, eq->core.cons_index);
   966	
   967			while (!rtnl_trylock())
   968				msleep(20);
   969	
   970			eqe_count = mlx5_eq_poll_irq_disabled(eq);
   971			if (eqe_count)
   972				mlx5_core_warn(eq->core.dev, "Recovered %d eqes on EQ 0x%x\n",
   973					       eqe_count, eq->core.eqn);
   974			rtnl_unlock();
   975		}
   976	}
   977	
 > 978	void mlx5_eq_reap_irq_release(struct kref *ref) {}
   979	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[kernel test robot]

Hi Praveen,

kernel test robot noticed the following build warnings:

[auto build test WARNING on net-next/main]
[also build test WARNING on net/main linus/master v7.0-rc2 next-20260304]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Praveen-Kumar-Kannoju/net-mlx5-poll-mlx5-eq-during-irq-migration/20260305-003505
base:   net-next/main
patch link:    https://lore.kernel.org/r/20260304161704.910564-1-praveen.kannoju%40oracle.com
patch subject: [PATCH] net/mlx5: poll mlx5 eq during irq migration
config: x86_64-buildonly-randconfig-002-20260305 (https://download.01.org/0day-ci/archive/20260305/202603051647.fykhqQ3H-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260305/202603051647.fykhqQ3H-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202603051647.fykhqQ3H-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> drivers/net/ethernet/mellanox/mlx5/core/eq.c:958:6: warning: no previous prototype for function 'mlx5_eq_reap_irq_notify' [-Wmissing-prototypes]
     958 | void mlx5_eq_reap_irq_notify(struct irq_affinity_notify *notify, const cpumask_t *mask)
         |      ^
   drivers/net/ethernet/mellanox/mlx5/core/eq.c:958:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
     958 | void mlx5_eq_reap_irq_notify(struct irq_affinity_notify *notify, const cpumask_t *mask)
         | ^
         | static 
>> drivers/net/ethernet/mellanox/mlx5/core/eq.c:978:6: warning: no previous prototype for function 'mlx5_eq_reap_irq_release' [-Wmissing-prototypes]
     978 | void mlx5_eq_reap_irq_release(struct kref *ref) {}
         |      ^
   drivers/net/ethernet/mellanox/mlx5/core/eq.c:978:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
     978 | void mlx5_eq_reap_irq_release(struct kref *ref) {}
         | ^
         | static 
   2 warnings generated.


vim +/mlx5_eq_reap_irq_notify +958 drivers/net/ethernet/mellanox/mlx5/core/eq.c

   957	
 > 958	void mlx5_eq_reap_irq_notify(struct irq_affinity_notify *notify, const cpumask_t *mask)
   959	{
   960		u32 eqe_count;
   961		struct mlx5_eq_comp *eq = container_of(notify, struct mlx5_eq_comp, notify);
   962	
   963		if (mlx5_reap_eq_irq_aff_change) {
   964			mlx5_core_warn(eq->core.dev, "irqn = 0x%x migration notified, EQ 0x%x: Cons = 0x%x\n",
   965				       eq->core.irqn, eq->core.eqn, eq->core.cons_index);
   966	
   967			while (!rtnl_trylock())
   968				msleep(20);
   969	
   970			eqe_count = mlx5_eq_poll_irq_disabled(eq);
   971			if (eqe_count)
   972				mlx5_core_warn(eq->core.dev, "Recovered %d eqes on EQ 0x%x\n",
   973					       eqe_count, eq->core.eqn);
   974			rtnl_unlock();
   975		}
   976	}
   977	
 > 978	void mlx5_eq_reap_irq_release(struct kref *ref) {}
   979	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[kernel test robot]

Hi Praveen,

kernel test robot noticed the following build warnings:

[auto build test WARNING on net-next/main]
[also build test WARNING on net/main linus/master v7.0-rc2 next-20260304]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Praveen-Kumar-Kannoju/net-mlx5-poll-mlx5-eq-during-irq-migration/20260305-003505
base:   net-next/main
patch link:    https://lore.kernel.org/r/20260304161704.910564-1-praveen.kannoju%40oracle.com
patch subject: [PATCH] net/mlx5: poll mlx5 eq during irq migration
config: x86_64-rhel-9.4 (https://download.01.org/0day-ci/archive/20260305/202603051743.ceus9qzu-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260305/202603051743.ceus9qzu-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202603051743.ceus9qzu-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> drivers/net/ethernet/mellanox/mlx5/core/eq.c:958:6: warning: no previous prototype for 'mlx5_eq_reap_irq_notify' [-Wmissing-prototypes]
     958 | void mlx5_eq_reap_irq_notify(struct irq_affinity_notify *notify, const cpumask_t *mask)
         |      ^~~~~~~~~~~~~~~~~~~~~~~
>> drivers/net/ethernet/mellanox/mlx5/core/eq.c:978:6: warning: no previous prototype for 'mlx5_eq_reap_irq_release' [-Wmissing-prototypes]
     978 | void mlx5_eq_reap_irq_release(struct kref *ref) {}
         |      ^~~~~~~~~~~~~~~~~~~~~~~~


vim +/mlx5_eq_reap_irq_notify +958 drivers/net/ethernet/mellanox/mlx5/core/eq.c

   957	
 > 958	void mlx5_eq_reap_irq_notify(struct irq_affinity_notify *notify, const cpumask_t *mask)
   959	{
   960		u32 eqe_count;
   961		struct mlx5_eq_comp *eq = container_of(notify, struct mlx5_eq_comp, notify);
   962	
   963		if (mlx5_reap_eq_irq_aff_change) {
   964			mlx5_core_warn(eq->core.dev, "irqn = 0x%x migration notified, EQ 0x%x: Cons = 0x%x\n",
   965				       eq->core.irqn, eq->core.eqn, eq->core.cons_index);
   966	
   967			while (!rtnl_trylock())
   968				msleep(20);
   969	
   970			eqe_count = mlx5_eq_poll_irq_disabled(eq);
   971			if (eqe_count)
   972				mlx5_core_warn(eq->core.dev, "Recovered %d eqes on EQ 0x%x\n",
   973					       eqe_count, eq->core.eqn);
   974			rtnl_unlock();
   975		}
   976	}
   977	
 > 978	void mlx5_eq_reap_irq_release(struct kref *ref) {}
   979	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[kernel test robot]

Hi Praveen,

kernel test robot noticed the following build warnings:

[auto build test WARNING on net-next/main]
[also build test WARNING on net/main linus/master v7.0-rc2 next-20260304]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Praveen-Kumar-Kannoju/net-mlx5-poll-mlx5-eq-during-irq-migration/20260305-003505
base:   net-next/main
patch link:    https://lore.kernel.org/r/20260304161704.910564-1-praveen.kannoju%40oracle.com
patch subject: [PATCH] net/mlx5: poll mlx5 eq during irq migration
config: loongarch-randconfig-r121-20260305 (https://download.01.org/0day-ci/archive/20260305/202603051910.7oo8wCfc-lkp@intel.com/config)
compiler: loongarch64-linux-gcc (GCC) 15.2.0
sparse: v0.6.5-rc1
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260305/202603051910.7oo8wCfc-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202603051910.7oo8wCfc-lkp@intel.com/

sparse warnings: (new ones prefixed by >>)
>> drivers/net/ethernet/mellanox/mlx5/core/eq.c:25:14: sparse: sparse: symbol 'mlx5_reap_eq_irq_aff_change' was not declared. Should it be static?
>> drivers/net/ethernet/mellanox/mlx5/core/eq.c:958:6: sparse: sparse: symbol 'mlx5_eq_reap_irq_notify' was not declared. Should it be static?
>> drivers/net/ethernet/mellanox/mlx5/core/eq.c:978:6: sparse: sparse: symbol 'mlx5_eq_reap_irq_release' was not declared. Should it be static?

vim +/mlx5_reap_eq_irq_aff_change +25 drivers/net/ethernet/mellanox/mlx5/core/eq.c

    24	
  > 25	unsigned int mlx5_reap_eq_irq_aff_change;
    26	module_param(mlx5_reap_eq_irq_aff_change, int, 0644);
    27	MODULE_PARM_DESC(mlx5_reap_eq_irq_aff_change, "mlx5_reap_eq_irq_aff_change: 0 = Disable MLX5 EQ Reap upon IRQ affinity change, \
    28			 1 = Enable MLX5 EQ Reap upon IRQ affinity change. Default=0");
    29	enum {
    30		MLX5_EQE_OWNER_INIT_VAL	= 0x1,
    31	};
    32	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

RE: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[Praveen Kannoju]

Confidential - Oracle Restricted \Including External Recipients

Patch applied to the wrong git tree. Will resubmit.


Confidential - Oracle Restricted \Including External Recipients
> -----Original Message-----
> From: kernel test robot <lkp@intel.com>
> Sent: Thursday, March 5, 2026 4:47 PM
> To: Praveen Kannoju <praveen.kannoju@oracle.com>; saeedm@nvidia.com;
> leon@kernel.org; tariqt@nvidia.com; mbloch@nvidia.com;
> andrew+netdev@lunn.ch; davem@davemloft.net; edumazet@google.com;
> kuba@kernel.org; pabeni@redhat.com; netdev@vger.kernel.org; linux-
> rdma@vger.kernel.org; linux-kernel@vger.kernel.org
> Cc: oe-kbuild-all@lists.linux.dev; Rama Nichanamatlu
> <rama.nichanamatlu@oracle.com>; Manjunath Patil
> <manjunath.b.patil@oracle.com>; Anand Khoje
> <anand.a.khoje@oracle.com>; Praveen Kannoju
> <praveen.kannoju@oracle.com>
> Subject: Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration
>
> Hi Praveen,
>
> kernel test robot noticed the following build warnings:
>
> [auto build test WARNING on net-next/main] [also build test WARNING on
> net/main linus/master v7.0-rc2 next-20260304] [If your patch is applied to
> the wrong git tree, kindly drop us a note.
> And when submitting patch, we suggest to use '--base' as documented in
> https://git-scm.com/docs/git-format-patch#_base_tree_information]
>
> url:    https://github.com/intel-lab-lkp/linux/commits/Praveen-Kumar-
> Kannoju/net-mlx5-poll-mlx5-eq-during-irq-migration/20260305-003505
> base:   net-next/main
> patch link:    https://lore.kernel.org/r/20260304161704.910564-1-
> praveen.kannoju%40oracle.com
> patch subject: [PATCH] net/mlx5: poll mlx5 eq during irq migration
> config: loongarch-randconfig-r121-20260305
> (https://download.01.org/0day-
> ci/archive/20260305/202603051910.7oo8wCfc-lkp@intel.com/config)
> compiler: loongarch64-linux-gcc (GCC) 15.2.0
> sparse: v0.6.5-rc1
> reproduce (this is a W=1 build): (https://download.01.org/0day-
> ci/archive/20260305/202603051910.7oo8wCfc-lkp@intel.com/reproduce)
>
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <lkp@intel.com>
> | Closes:
> | https://lore.kernel.org/oe-kbuild-all/202603051910.7oo8wCfc-lkp@intel.
> | com/
>
> sparse warnings: (new ones prefixed by >>)
> >> drivers/net/ethernet/mellanox/mlx5/core/eq.c:25:14: sparse: sparse:
> symbol 'mlx5_reap_eq_irq_aff_change' was not declared. Should it be static?
> >> drivers/net/ethernet/mellanox/mlx5/core/eq.c:958:6: sparse: sparse:
> symbol 'mlx5_eq_reap_irq_notify' was not declared. Should it be static?
> >> drivers/net/ethernet/mellanox/mlx5/core/eq.c:978:6: sparse: sparse:
> symbol 'mlx5_eq_reap_irq_release' was not declared. Should it be static?
>
> vim +/mlx5_reap_eq_irq_aff_change +25
> drivers/net/ethernet/mellanox/mlx5/core/eq.c
>
>     24
>   > 25        unsigned int mlx5_reap_eq_irq_aff_change;
>     26        module_param(mlx5_reap_eq_irq_aff_change, int, 0644);
>     27        MODULE_PARM_DESC(mlx5_reap_eq_irq_aff_change,
> "mlx5_reap_eq_irq_aff_change: 0 = Disable MLX5 EQ Reap upon IRQ affinity
> change, \
>     28                         1 = Enable MLX5 EQ Reap upon IRQ affinity change.
> Default=0");
>     29        enum {
>     30                MLX5_EQE_OWNER_INIT_VAL = 0x1,
>     31        };
>     32
>
> --
> 0-DAY CI Kernel Test Service
> https://github.com/intel/lkp-tests/wiki

Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[Jason Gunthorpe]

On Thu, Mar 05, 2026 at 05:08:52PM +0000, Praveen Kannoju wrote:

>    Regardless of the underlying causes, which may include IRQ loss
>    or EQ re-arming failure, the TX queue becomes stuck, and the
>    timeout handler is only triggered once the queue is declared
>    full. In scenarios where only specialized packets, such as
>    heartbeat packets, are sent through the queue, it takes
>    significantly longer for the queue to fill and be identified as
>    stuck. A proven solution for this issue is polling the EQ
>    immediately after the corresponding IRQ migration, which allows
>    for earlier recovery and prevents the transmission queue from
>    becoming stuck.

I undersand all of this, but for upstreaming we want the root cause,
not bodges like this.

There is no reason to do what this patch does, the IRQ system is not
supposed to loose interrupts on migration, if that is happening on
your systems it is a serious bug that must be root caused.

Jason

RE: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[Praveen Kannoju]

Confidential - Oracle Restricted \Including External Recipients



Confidential - Oracle Restricted \Including External Recipients
> -----Original Message-----
> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Friday, March 6, 2026 6:02 AM
> To: Praveen Kannoju <praveen.kannoju@oracle.com>
> Cc: saeedm@nvidia.com; leon@kernel.org; tariqt@nvidia.com;
> mbloch@nvidia.com; andrew+netdev@lunn.ch; davem@davemloft.net;
> edumazet@google.com; kuba@kernel.org; pabeni@redhat.com;
> netdev@vger.kernel.org; linux-rdma@vger.kernel.org; linux-
> kernel@vger.kernel.org; Rama Nichanamatlu
> <rama.nichanamatlu@oracle.com>; Manjunath Patil
> <manjunath.b.patil@oracle.com>; Anand Khoje <anand.a.khoje@oracle.com>
> Subject: Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration
>
> On Thu, Mar 05, 2026 at 05:08:52PM +0000, Praveen Kannoju wrote:
>
> >    Regardless of the underlying causes, which may include IRQ loss
> >    or EQ re-arming failure, the TX queue becomes stuck, and the
> >    timeout handler is only triggered once the queue is declared
> >    full. In scenarios where only specialized packets, such as
> >    heartbeat packets, are sent through the queue, it takes
> >    significantly longer for the queue to fill and be identified as
> >    stuck. A proven solution for this issue is polling the EQ
> >    immediately after the corresponding IRQ migration, which allows
> >    for earlier recovery and prevents the transmission queue from
> >    becoming stuck.
>
> I undersand all of this, but for upstreaming we want the root cause, not
> bodges like this.
>
> There is no reason to do what this patch does, the IRQ system is not supposed
> to loose interrupts on migration, if that is happening on your systems it is a
> serious bug that must be root caused.

Thank you, Jason.
We'll evaluate more on it.

>
> Jason

Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[Jason Gunthorpe]

On Fri, Mar 06, 2026 at 02:19:09PM +0000, Praveen Kannoju wrote:
> 
> > On Thu, Mar 05, 2026 at 05:08:52PM +0000, Praveen Kannoju wrote:
> >
> > >    Regardless of the underlying causes, which may include IRQ loss
> > >    or EQ re-arming failure, the TX queue becomes stuck, and the
> > >    timeout handler is only triggered once the queue is declared
> > >    full. In scenarios where only specialized packets, such as
> > >    heartbeat packets, are sent through the queue, it takes
> > >    significantly longer for the queue to fill and be identified as
> > >    stuck. A proven solution for this issue is polling the EQ
> > >    immediately after the corresponding IRQ migration, which allows
> > >    for earlier recovery and prevents the transmission queue from
> > >    becoming stuck.
> >
> > I undersand all of this, but for upstreaming we want the root cause, not
> > bodges like this.
> >
> > There is no reason to do what this patch does, the IRQ system is not supposed
> > to loose interrupts on migration, if that is happening on your systems it is a
> > serious bug that must be root caused.
> 
> Thank you, Jason.
> We'll evaluate more on it.

If this is in a VM running under qemu - qemu does Lots Of Stuff
whenever a MSI-X is changed and that stuff has been buggy before and
resulted in lost things.

If it is bare metal, I'm shocked. Maybe an IOMMU driver bug in the
interrupt remapping?

Jason

RE: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[Praveen Kannoju]

Confidential - Oracle Restricted \Including External Recipients



Confidential - Oracle Restricted \Including External Recipients
> -----Original Message-----
> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Saturday, March 7, 2026 4:40 AM
> To: Praveen Kannoju <praveen.kannoju@oracle.com>
> Cc: saeedm@nvidia.com; leon@kernel.org; tariqt@nvidia.com;
> mbloch@nvidia.com; andrew+netdev@lunn.ch; davem@davemloft.net;
> edumazet@google.com; kuba@kernel.org; pabeni@redhat.com;
> netdev@vger.kernel.org; linux-rdma@vger.kernel.org; linux-
> kernel@vger.kernel.org; Rama Nichanamatlu
> <rama.nichanamatlu@oracle.com>; Manjunath Patil
> <manjunath.b.patil@oracle.com>; Anand Khoje <anand.a.khoje@oracle.com>
> Subject: Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration
>
> On Fri, Mar 06, 2026 at 02:19:09PM +0000, Praveen Kannoju wrote:
> >
> > > On Thu, Mar 05, 2026 at 05:08:52PM +0000, Praveen Kannoju wrote:
> > >
> > > >    Regardless of the underlying causes, which may include IRQ loss
> > > >    or EQ re-arming failure, the TX queue becomes stuck, and the
> > > >    timeout handler is only triggered once the queue is declared
> > > >    full. In scenarios where only specialized packets, such as
> > > >    heartbeat packets, are sent through the queue, it takes
> > > >    significantly longer for the queue to fill and be identified as
> > > >    stuck. A proven solution for this issue is polling the EQ
> > > >    immediately after the corresponding IRQ migration, which allows
> > > >    for earlier recovery and prevents the transmission queue from
> > > >    becoming stuck.
> > >
> > > I undersand all of this, but for upstreaming we want the root cause,
> > > not bodges like this.
> > >
> > > There is no reason to do what this patch does, the IRQ system is not
> > > supposed to loose interrupts on migration, if that is happening on
> > > your systems it is a serious bug that must be root caused.
> >
> > Thank you, Jason.
> > We'll evaluate more on it.
>
> If this is in a VM running under qemu - qemu does Lots Of Stuff whenever a
> MSI-X is changed and that stuff has been buggy before and resulted in lost
> things.
>
> If it is bare metal, I'm shocked. Maybe an IOMMU driver bug in the interrupt
> remapping?
>
> Jason


Hello Jason,

Yes, this is Qemu VM, on which we are having the issue.
In bare metal environment there is NO cpu scaling.
No issue seen on a bare metal so far. Maybe unlikely also.

We are having this issue as Qemu VM's go thru cpu scaling
based on business needs.

It had been very challenging to arrive at the cause.
we went thru many live debug sessions with Nvidia R&D team.
but we couldn't root cause. This tells why we eventually.
arrived at this mitigation as this issue is wide spread
and has been hurting many and many customers in cloud.

-
Praveen

Re: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[Jason Gunthorpe]

On Sat, Mar 07, 2026 at 05:43:56AM +0000, Praveen Kannoju wrote:

> It had been very challenging to arrive at the cause.
> we went thru many live debug sessions with Nvidia R&D team.
> but we couldn't root cause. This tells why we eventually.
> arrived at this mitigation as this issue is wide spread
> and has been hurting many and many customers in cloud.

It is almost certainly a qemu bug. If you cannot find it, then I
suggest you work around it by having qemu inject a spurious interrupt
around the migration situations.

But make sure you have the already known qemu and kernel bug fixes for
lost interrupts on MSI-X writes...

Jason

RE: [PATCH] net/mlx5: poll mlx5 eq during irq migration

[Praveen Kannoju]

Confidential - Oracle Restricted \Including External Recipients

> It is almost certainly a qemu bug. If you cannot find it, then I suggest you work
> around it by having qemu inject a spurious interrupt around the migration
> situations.
>
> But make sure you have the already known qemu and kernel bug fixes for lost
> interrupts on MSI-X writes...

Thank you, Jason.
We're working on it.

-
Praveen.

Confidential - Oracle Restricted \Including External Recipients