netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH net 0/4] Fix mixing atomic/non-atomic contexts in mlx5 IPsec code
@ 2023-06-05  8:09 Leon Romanovsky
  2023-06-05  8:09 ` [PATCH net 1/4] net/mlx5e: Don't delay release of hardware objects Leon Romanovsky
                   ` (3 more replies)
  0 siblings, 4 replies; 9+ messages in thread
From: Leon Romanovsky @ 2023-06-05  8:09 UTC (permalink / raw)
  To: Jakub Kicinski
  Cc: Leon Romanovsky, Eric Dumazet, netdev, Paolo Abeni,
	Patrisious Haddad, Raed Salem, Saeed Mahameed, Steffen Klassert

From: Leon Romanovsky <leonro@nvidia.com>

Hi,

This series fixes mistakes in mixing atomic/non-atomic contexts in
mlx5 IPsec code.

Thanks

Leon Romanovsky (3):
  net/mlx5e: Don't delay release of hardware objects
  net/mlx5e: Drop XFRM state lock when modifying flow steering
  net/mlx5e: Fix scheduling of IPsec ASO query while in atomic

Patrisious Haddad (1):
  net/mlx5e: Fix ESN update kernel panic

 .../mellanox/mlx5/core/en_accel/ipsec.c       | 22 ++++++++++++++-----
 .../mlx5/core/en_accel/ipsec_offload.c        | 17 +++++++++++---
 2 files changed, 30 insertions(+), 9 deletions(-)

-- 
2.40.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH net 1/4] net/mlx5e: Don't delay release of hardware objects
  2023-06-05  8:09 [PATCH net 0/4] Fix mixing atomic/non-atomic contexts in mlx5 IPsec code Leon Romanovsky
@ 2023-06-05  8:09 ` Leon Romanovsky
  2023-06-06  9:07   ` Simon Horman
  2023-06-05  8:09 ` [PATCH net 2/4] net/mlx5e: Fix ESN update kernel panic Leon Romanovsky
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 9+ messages in thread
From: Leon Romanovsky @ 2023-06-05  8:09 UTC (permalink / raw)
  To: Jakub Kicinski
  Cc: Leon Romanovsky, Eric Dumazet, netdev, Paolo Abeni,
	Patrisious Haddad, Raed Salem, Saeed Mahameed, Steffen Klassert

From: Leon Romanovsky <leonro@nvidia.com>

XFRM core provides two callbacks to release resources, one is .xdo_dev_policy_delete()
and another is .xdo_dev_policy_free(). This separation allows delayed release so
"ip xfrm policy free" commands won't starve. Unfortunately, mlx5 command interface
can't run in .xdo_dev_policy_free() callbacks as the latter runs in ATOMIC context.

 BUG: scheduling while atomic: swapper/7/0/0x00000100
 Modules linked in: act_mirred act_tunnel_key cls_flower sch_ingress vxlan mlx5_vdpa vringh vhost_iotlb vdpa rpcrdma rdma_ucm ib_iser libiscsi ib_umad scsi_transport_iscsi rdma_cm ib_ipoib iw_cm ib_cm mlx5_ib ib_uverbs ib_core xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat br_netfilter rpcsec_gss_krb5 auth_rpcgss oid_registry overlay mlx5_core zram zsmalloc fuse
 CPU: 7 PID: 0 Comm: swapper/7 Not tainted 6.3.0+ #1
 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
 Call Trace:
  <IRQ>
  dump_stack_lvl+0x33/0x50
  __schedule_bug+0x4e/0x60
  __schedule+0x5d5/0x780
  ? __mod_timer+0x286/0x3d0
  schedule+0x50/0x90
  schedule_timeout+0x7c/0xf0
  ? __bpf_trace_tick_stop+0x10/0x10
  __wait_for_common+0x88/0x190
  ? usleep_range_state+0x90/0x90
  cmd_exec+0x42e/0xb40 [mlx5_core]
  mlx5_cmd_do+0x1e/0x40 [mlx5_core]
  mlx5_cmd_exec+0x18/0x30 [mlx5_core]
  mlx5_cmd_delete_fte+0xa8/0xd0 [mlx5_core]
  del_hw_fte+0x60/0x120 [mlx5_core]
  mlx5_del_flow_rules+0xec/0x270 [mlx5_core]
  ? default_send_IPI_single_phys+0x26/0x30
  mlx5e_accel_ipsec_fs_del_pol+0x1a/0x60 [mlx5_core]
  mlx5e_xfrm_free_policy+0x15/0x20 [mlx5_core]
  xfrm_policy_destroy+0x5a/0xb0
  xfrm4_dst_destroy+0x7b/0x100
  dst_destroy+0x37/0x120
  rcu_core+0x2d6/0x540
  __do_softirq+0xcd/0x273
  irq_exit_rcu+0x82/0xb0
  sysvec_apic_timer_interrupt+0x72/0x90
  </IRQ>
  <TASK>
  asm_sysvec_apic_timer_interrupt+0x16/0x20
 RIP: 0010:default_idle+0x13/0x20
 Code: c0 08 00 00 00 4d 29 c8 4c 01 c7 4c 29 c2 e9 72 ff ff ff cc cc cc cc 8b 05 7a 4d ee 00 85 c0 7e 07 0f 00 2d 2f 98 2e 00 fb f4 <fa> c3 66 66 2e 0f 1f 84 00 00 00 00 00 65 48 8b 04 25 40 b4 02 00
 RSP: 0018:ffff888100843ee0 EFLAGS: 00000242
 RAX: 0000000000000001 RBX: ffff888100812b00 RCX: 4000000000000000
 RDX: 0000000000000001 RSI: 0000000000000083 RDI: 000000000002d2ec
 RBP: 0000000000000007 R08: 00000021daeded59 R09: 0000000000000001
 R10: 0000000000000000 R11: 000000000000000f R12: 0000000000000000
 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
  default_idle_call+0x30/0xb0
  do_idle+0x1c1/0x1d0
  cpu_startup_entry+0x19/0x20
  start_secondary+0xfe/0x120
  secondary_startup_64_no_verify+0xf3/0xfb
  </TASK>
 bad: scheduling from the idle thread!

Fixes: a5b8ca9471d3 ("net/mlx5e: Add XFRM policy offload logic")
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c
index 55b38544422f..d1c801723d35 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c
@@ -1040,11 +1040,17 @@ static int mlx5e_xfrm_add_policy(struct xfrm_policy *x,
 	return err;
 }
 
-static void mlx5e_xfrm_free_policy(struct xfrm_policy *x)
+static void mlx5e_xfrm_del_policy(struct xfrm_policy *x)
 {
 	struct mlx5e_ipsec_pol_entry *pol_entry = to_ipsec_pol_entry(x);
 
 	mlx5e_accel_ipsec_fs_del_pol(pol_entry);
+}
+
+static void mlx5e_xfrm_free_policy(struct xfrm_policy *x)
+{
+	struct mlx5e_ipsec_pol_entry *pol_entry = to_ipsec_pol_entry(x);
+
 	kfree(pol_entry);
 }
 
@@ -1065,6 +1071,7 @@ static const struct xfrmdev_ops mlx5e_ipsec_packet_xfrmdev_ops = {
 
 	.xdo_dev_state_update_curlft = mlx5e_xfrm_update_curlft,
 	.xdo_dev_policy_add = mlx5e_xfrm_add_policy,
+	.xdo_dev_policy_delete = mlx5e_xfrm_del_policy,
 	.xdo_dev_policy_free = mlx5e_xfrm_free_policy,
 };
 
-- 
2.40.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH net 2/4] net/mlx5e: Fix ESN update kernel panic
  2023-06-05  8:09 [PATCH net 0/4] Fix mixing atomic/non-atomic contexts in mlx5 IPsec code Leon Romanovsky
  2023-06-05  8:09 ` [PATCH net 1/4] net/mlx5e: Don't delay release of hardware objects Leon Romanovsky
@ 2023-06-05  8:09 ` Leon Romanovsky
  2023-06-06  9:08   ` Simon Horman
  2023-06-05  8:09 ` [PATCH net 3/4] net/mlx5e: Drop XFRM state lock when modifying flow steering Leon Romanovsky
  2023-06-05  8:09 ` [PATCH net 4/4] net/mlx5e: Fix scheduling of IPsec ASO query while in atomic Leon Romanovsky
  3 siblings, 1 reply; 9+ messages in thread
From: Leon Romanovsky @ 2023-06-05  8:09 UTC (permalink / raw)
  To: Jakub Kicinski
  Cc: Patrisious Haddad, Eric Dumazet, netdev, Paolo Abeni, Raed Salem,
	Saeed Mahameed, Steffen Klassert

From: Patrisious Haddad <phaddad@nvidia.com>

Previously during mlx5e_ipsec_handle_event the driver tried to execute
an operation that could sleep, while holding a spinlock, which caused
the kernel panic mentioned below.

Move the function call that can sleep outside of the spinlock context.

 Call Trace:
 <TASK>
 dump_stack_lvl+0x49/0x6c
 __schedule_bug.cold+0x42/0x4e
 schedule_debug.constprop.0+0xe0/0x118
 __schedule+0x59/0x58a
 ? __mod_timer+0x2a1/0x3ef
 schedule+0x5e/0xd4
 schedule_timeout+0x99/0x164
 ? __pfx_process_timeout+0x10/0x10
 __wait_for_common+0x90/0x1da
 ? __pfx_schedule_timeout+0x10/0x10
 wait_func+0x34/0x142 [mlx5_core]
 mlx5_cmd_invoke+0x1f3/0x313 [mlx5_core]
 cmd_exec+0x1fe/0x325 [mlx5_core]
 mlx5_cmd_do+0x22/0x50 [mlx5_core]
 mlx5_cmd_exec+0x1c/0x40 [mlx5_core]
 mlx5_modify_ipsec_obj+0xb2/0x17f [mlx5_core]
 mlx5e_ipsec_update_esn_state+0x69/0xf0 [mlx5_core]
 ? wake_affine+0x62/0x1f8
 mlx5e_ipsec_handle_event+0xb1/0xc0 [mlx5_core]
 process_one_work+0x1e2/0x3e6
 ? __pfx_worker_thread+0x10/0x10
 worker_thread+0x54/0x3ad
 ? __pfx_worker_thread+0x10/0x10
 kthread+0xda/0x101
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x29/0x37
 </TASK>
 BUG: workqueue leaked lock or atomic: kworker/u256:4/0x7fffffff/189754#012     last function: mlx5e_ipsec_handle_event [mlx5_core]
 CPU: 66 PID: 189754 Comm: kworker/u256:4 Kdump: loaded Tainted: G        W          6.2.0-2596.20230309201517_5.el8uek.rc1.x86_64 #2
 Hardware name: Oracle Corporation ORACLE SERVER X9-2/ASMMBX9-2, BIOS 61070300 08/17/2022
 Workqueue: mlx5e_ipsec: eth%d mlx5e_ipsec_handle_event [mlx5_core]
 Call Trace:
 <TASK>
 dump_stack_lvl+0x49/0x6c
 process_one_work.cold+0x2b/0x3c
 ? __pfx_worker_thread+0x10/0x10
 worker_thread+0x54/0x3ad
 ? __pfx_worker_thread+0x10/0x10
 kthread+0xda/0x101
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x29/0x37
 </TASK>
 BUG: scheduling while atomic: kworker/u256:4/189754/0x00000000

Fixes: cee137a63431 ("net/mlx5e: Handle ESN update events")
Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 .../mellanox/mlx5/core/en_accel/ipsec_offload.c    | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_offload.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_offload.c
index df90e19066bc..ca16cb9807ea 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_offload.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_offload.c
@@ -305,7 +305,17 @@ static void mlx5e_ipsec_update_esn_state(struct mlx5e_ipsec_sa_entry *sa_entry,
 	}
 
 	mlx5e_ipsec_build_accel_xfrm_attrs(sa_entry, &attrs);
+
+	/* It is safe to execute the modify below unlocked since the only flows
+	 * that could affect this HW object, are create, destroy and this work.
+	 *
+	 * Creation flow can't co-exist with this modify work, the destruction
+	 * flow would cancel this work, and this work is a single entity that
+	 * can't conflict with it self.
+	 */
+	spin_unlock_bh(&sa_entry->x->lock);
 	mlx5_accel_esp_modify_xfrm(sa_entry, &attrs);
+	spin_lock_bh(&sa_entry->x->lock);
 
 	data.data_offset_condition_operand =
 		MLX5_IPSEC_ASO_REMOVE_FLOW_PKT_CNT_OFFSET;
@@ -431,7 +441,7 @@ static void mlx5e_ipsec_handle_event(struct work_struct *_work)
 	aso = sa_entry->ipsec->aso;
 	attrs = &sa_entry->attrs;
 
-	spin_lock(&sa_entry->x->lock);
+	spin_lock_bh(&sa_entry->x->lock);
 	ret = mlx5e_ipsec_aso_query(sa_entry, NULL);
 	if (ret)
 		goto unlock;
@@ -447,7 +457,7 @@ static void mlx5e_ipsec_handle_event(struct work_struct *_work)
 		mlx5e_ipsec_handle_limits(sa_entry);
 
 unlock:
-	spin_unlock(&sa_entry->x->lock);
+	spin_unlock_bh(&sa_entry->x->lock);
 	kfree(work);
 }
 
-- 
2.40.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH net 3/4] net/mlx5e: Drop XFRM state lock when modifying flow steering
  2023-06-05  8:09 [PATCH net 0/4] Fix mixing atomic/non-atomic contexts in mlx5 IPsec code Leon Romanovsky
  2023-06-05  8:09 ` [PATCH net 1/4] net/mlx5e: Don't delay release of hardware objects Leon Romanovsky
  2023-06-05  8:09 ` [PATCH net 2/4] net/mlx5e: Fix ESN update kernel panic Leon Romanovsky
@ 2023-06-05  8:09 ` Leon Romanovsky
  2023-06-06  9:12   ` Simon Horman
  2023-06-05  8:09 ` [PATCH net 4/4] net/mlx5e: Fix scheduling of IPsec ASO query while in atomic Leon Romanovsky
  3 siblings, 1 reply; 9+ messages in thread
From: Leon Romanovsky @ 2023-06-05  8:09 UTC (permalink / raw)
  To: Jakub Kicinski
  Cc: Leon Romanovsky, Eric Dumazet, netdev, Paolo Abeni,
	Patrisious Haddad, Raed Salem, Saeed Mahameed, Steffen Klassert

From: Leon Romanovsky <leonro@nvidia.com>

XFRM state which is changed to be XFRM_STATE_EXPIRED doesn't really
need to hold lock while modifying flow steering rules to drop traffic.

That state can be deleted only and as such mlx5e_ipsec_handle_tx_limit()
work will be canceled anyway and won't run in parallel.

Fixes: b2f7b01d36a9 ("net/mlx5e: Simulate missing IPsec TX limits hardware functionality")
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 .../ethernet/mellanox/mlx5/core/en_accel/ipsec.c    | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c
index d1c801723d35..891d39b4bfd4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c
@@ -61,16 +61,19 @@ static void mlx5e_ipsec_handle_tx_limit(struct work_struct *_work)
 	struct mlx5e_ipsec_sa_entry *sa_entry = dwork->sa_entry;
 	struct xfrm_state *x = sa_entry->x;
 
-	spin_lock(&x->lock);
+	if (sa_entry->attrs.drop)
+		return;
+
+	spin_lock_bh(&x->lock);
 	xfrm_state_check_expire(x);
 	if (x->km.state == XFRM_STATE_EXPIRED) {
 		sa_entry->attrs.drop = true;
-		mlx5e_accel_ipsec_fs_modify(sa_entry);
-	}
-	spin_unlock(&x->lock);
+		spin_unlock_bh(&x->lock);
 
-	if (sa_entry->attrs.drop)
+		mlx5e_accel_ipsec_fs_modify(sa_entry);
 		return;
+	}
+	spin_unlock_bh(&x->lock);
 
 	queue_delayed_work(sa_entry->ipsec->wq, &dwork->dwork,
 			   MLX5_IPSEC_RESCHED);
-- 
2.40.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH net 4/4] net/mlx5e: Fix scheduling of IPsec ASO query while in atomic
  2023-06-05  8:09 [PATCH net 0/4] Fix mixing atomic/non-atomic contexts in mlx5 IPsec code Leon Romanovsky
                   ` (2 preceding siblings ...)
  2023-06-05  8:09 ` [PATCH net 3/4] net/mlx5e: Drop XFRM state lock when modifying flow steering Leon Romanovsky
@ 2023-06-05  8:09 ` Leon Romanovsky
  2023-06-06  9:13   ` Simon Horman
  3 siblings, 1 reply; 9+ messages in thread
From: Leon Romanovsky @ 2023-06-05  8:09 UTC (permalink / raw)
  To: Jakub Kicinski
  Cc: Leon Romanovsky, Eric Dumazet, netdev, Paolo Abeni,
	Patrisious Haddad, Raed Salem, Saeed Mahameed, Steffen Klassert

From: Leon Romanovsky <leonro@nvidia.com>

ASO query can be scheduled in atomic context as such it can't use usleep.
Use udelay as recommended in Documentation/timers/timers-howto.rst.

Fixes: 76e463f6508b ("net/mlx5e: Overcome slow response for first IPsec ASO WQE")
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 .../net/ethernet/mellanox/mlx5/core/en_accel/ipsec_offload.c   | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_offload.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_offload.c
index ca16cb9807ea..a3554bde3e07 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_offload.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_offload.c
@@ -606,7 +606,8 @@ int mlx5e_ipsec_aso_query(struct mlx5e_ipsec_sa_entry *sa_entry,
 	do {
 		ret = mlx5_aso_poll_cq(aso->aso, false);
 		if (ret)
-			usleep_range(2, 10);
+			/* We are in atomic context */
+			udelay(10);
 	} while (ret && time_is_after_jiffies(expires));
 	spin_unlock_bh(&aso->lock);
 	return ret;
-- 
2.40.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH net 1/4] net/mlx5e: Don't delay release of hardware objects
  2023-06-05  8:09 ` [PATCH net 1/4] net/mlx5e: Don't delay release of hardware objects Leon Romanovsky
@ 2023-06-06  9:07   ` Simon Horman
  0 siblings, 0 replies; 9+ messages in thread
From: Simon Horman @ 2023-06-06  9:07 UTC (permalink / raw)
  To: Leon Romanovsky
  Cc: Jakub Kicinski, Leon Romanovsky, Eric Dumazet, netdev,
	Paolo Abeni, Patrisious Haddad, Raed Salem, Saeed Mahameed,
	Steffen Klassert

On Mon, Jun 05, 2023 at 11:09:49AM +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
> 
> XFRM core provides two callbacks to release resources, one is .xdo_dev_policy_delete()
> and another is .xdo_dev_policy_free(). This separation allows delayed release so
> "ip xfrm policy free" commands won't starve. Unfortunately, mlx5 command interface
> can't run in .xdo_dev_policy_free() callbacks as the latter runs in ATOMIC context.
> 
>  BUG: scheduling while atomic: swapper/7/0/0x00000100
>  Modules linked in: act_mirred act_tunnel_key cls_flower sch_ingress vxlan mlx5_vdpa vringh vhost_iotlb vdpa rpcrdma rdma_ucm ib_iser libiscsi ib_umad scsi_transport_iscsi rdma_cm ib_ipoib iw_cm ib_cm mlx5_ib ib_uverbs ib_core xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat br_netfilter rpcsec_gss_krb5 auth_rpcgss oid_registry overlay mlx5_core zram zsmalloc fuse
>  CPU: 7 PID: 0 Comm: swapper/7 Not tainted 6.3.0+ #1
>  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
>  Call Trace:
>   <IRQ>
>   dump_stack_lvl+0x33/0x50
>   __schedule_bug+0x4e/0x60
>   __schedule+0x5d5/0x780
>   ? __mod_timer+0x286/0x3d0
>   schedule+0x50/0x90
>   schedule_timeout+0x7c/0xf0
>   ? __bpf_trace_tick_stop+0x10/0x10
>   __wait_for_common+0x88/0x190
>   ? usleep_range_state+0x90/0x90
>   cmd_exec+0x42e/0xb40 [mlx5_core]
>   mlx5_cmd_do+0x1e/0x40 [mlx5_core]
>   mlx5_cmd_exec+0x18/0x30 [mlx5_core]
>   mlx5_cmd_delete_fte+0xa8/0xd0 [mlx5_core]
>   del_hw_fte+0x60/0x120 [mlx5_core]
>   mlx5_del_flow_rules+0xec/0x270 [mlx5_core]
>   ? default_send_IPI_single_phys+0x26/0x30
>   mlx5e_accel_ipsec_fs_del_pol+0x1a/0x60 [mlx5_core]
>   mlx5e_xfrm_free_policy+0x15/0x20 [mlx5_core]
>   xfrm_policy_destroy+0x5a/0xb0
>   xfrm4_dst_destroy+0x7b/0x100
>   dst_destroy+0x37/0x120
>   rcu_core+0x2d6/0x540
>   __do_softirq+0xcd/0x273
>   irq_exit_rcu+0x82/0xb0
>   sysvec_apic_timer_interrupt+0x72/0x90
>   </IRQ>
>   <TASK>
>   asm_sysvec_apic_timer_interrupt+0x16/0x20
>  RIP: 0010:default_idle+0x13/0x20
>  Code: c0 08 00 00 00 4d 29 c8 4c 01 c7 4c 29 c2 e9 72 ff ff ff cc cc cc cc 8b 05 7a 4d ee 00 85 c0 7e 07 0f 00 2d 2f 98 2e 00 fb f4 <fa> c3 66 66 2e 0f 1f 84 00 00 00 00 00 65 48 8b 04 25 40 b4 02 00
>  RSP: 0018:ffff888100843ee0 EFLAGS: 00000242
>  RAX: 0000000000000001 RBX: ffff888100812b00 RCX: 4000000000000000
>  RDX: 0000000000000001 RSI: 0000000000000083 RDI: 000000000002d2ec
>  RBP: 0000000000000007 R08: 00000021daeded59 R09: 0000000000000001
>  R10: 0000000000000000 R11: 000000000000000f R12: 0000000000000000
>  R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>   default_idle_call+0x30/0xb0
>   do_idle+0x1c1/0x1d0
>   cpu_startup_entry+0x19/0x20
>   start_secondary+0xfe/0x120
>   secondary_startup_64_no_verify+0xf3/0xfb
>   </TASK>
>  bad: scheduling from the idle thread!
> 
> Fixes: a5b8ca9471d3 ("net/mlx5e: Add XFRM policy offload logic")
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>

Reviewed-by: Simon Horman <simon.horman@corigine.com>


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH net 2/4] net/mlx5e: Fix ESN update kernel panic
  2023-06-05  8:09 ` [PATCH net 2/4] net/mlx5e: Fix ESN update kernel panic Leon Romanovsky
@ 2023-06-06  9:08   ` Simon Horman
  0 siblings, 0 replies; 9+ messages in thread
From: Simon Horman @ 2023-06-06  9:08 UTC (permalink / raw)
  To: Leon Romanovsky
  Cc: Jakub Kicinski, Patrisious Haddad, Eric Dumazet, netdev,
	Paolo Abeni, Raed Salem, Saeed Mahameed, Steffen Klassert

On Mon, Jun 05, 2023 at 11:09:50AM +0300, Leon Romanovsky wrote:
> From: Patrisious Haddad <phaddad@nvidia.com>
> 
> Previously during mlx5e_ipsec_handle_event the driver tried to execute
> an operation that could sleep, while holding a spinlock, which caused
> the kernel panic mentioned below.
> 
> Move the function call that can sleep outside of the spinlock context.
> 
>  Call Trace:
>  <TASK>
>  dump_stack_lvl+0x49/0x6c
>  __schedule_bug.cold+0x42/0x4e
>  schedule_debug.constprop.0+0xe0/0x118
>  __schedule+0x59/0x58a
>  ? __mod_timer+0x2a1/0x3ef
>  schedule+0x5e/0xd4
>  schedule_timeout+0x99/0x164
>  ? __pfx_process_timeout+0x10/0x10
>  __wait_for_common+0x90/0x1da
>  ? __pfx_schedule_timeout+0x10/0x10
>  wait_func+0x34/0x142 [mlx5_core]
>  mlx5_cmd_invoke+0x1f3/0x313 [mlx5_core]
>  cmd_exec+0x1fe/0x325 [mlx5_core]
>  mlx5_cmd_do+0x22/0x50 [mlx5_core]
>  mlx5_cmd_exec+0x1c/0x40 [mlx5_core]
>  mlx5_modify_ipsec_obj+0xb2/0x17f [mlx5_core]
>  mlx5e_ipsec_update_esn_state+0x69/0xf0 [mlx5_core]
>  ? wake_affine+0x62/0x1f8
>  mlx5e_ipsec_handle_event+0xb1/0xc0 [mlx5_core]
>  process_one_work+0x1e2/0x3e6
>  ? __pfx_worker_thread+0x10/0x10
>  worker_thread+0x54/0x3ad
>  ? __pfx_worker_thread+0x10/0x10
>  kthread+0xda/0x101
>  ? __pfx_kthread+0x10/0x10
>  ret_from_fork+0x29/0x37
>  </TASK>
>  BUG: workqueue leaked lock or atomic: kworker/u256:4/0x7fffffff/189754#012     last function: mlx5e_ipsec_handle_event [mlx5_core]
>  CPU: 66 PID: 189754 Comm: kworker/u256:4 Kdump: loaded Tainted: G        W          6.2.0-2596.20230309201517_5.el8uek.rc1.x86_64 #2
>  Hardware name: Oracle Corporation ORACLE SERVER X9-2/ASMMBX9-2, BIOS 61070300 08/17/2022
>  Workqueue: mlx5e_ipsec: eth%d mlx5e_ipsec_handle_event [mlx5_core]
>  Call Trace:
>  <TASK>
>  dump_stack_lvl+0x49/0x6c
>  process_one_work.cold+0x2b/0x3c
>  ? __pfx_worker_thread+0x10/0x10
>  worker_thread+0x54/0x3ad
>  ? __pfx_worker_thread+0x10/0x10
>  kthread+0xda/0x101
>  ? __pfx_kthread+0x10/0x10
>  ret_from_fork+0x29/0x37
>  </TASK>
>  BUG: scheduling while atomic: kworker/u256:4/189754/0x00000000
> 
> Fixes: cee137a63431 ("net/mlx5e: Handle ESN update events")
> Signed-off-by: Patrisious Haddad <phaddad@nvidia.com>
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>

Reviewed-by: Simon Horman <simon.horman@corigine.com>


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH net 3/4] net/mlx5e: Drop XFRM state lock when modifying flow steering
  2023-06-05  8:09 ` [PATCH net 3/4] net/mlx5e: Drop XFRM state lock when modifying flow steering Leon Romanovsky
@ 2023-06-06  9:12   ` Simon Horman
  0 siblings, 0 replies; 9+ messages in thread
From: Simon Horman @ 2023-06-06  9:12 UTC (permalink / raw)
  To: Leon Romanovsky
  Cc: Jakub Kicinski, Leon Romanovsky, Eric Dumazet, netdev,
	Paolo Abeni, Patrisious Haddad, Raed Salem, Saeed Mahameed,
	Steffen Klassert

On Mon, Jun 05, 2023 at 11:09:51AM +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
> 
> XFRM state which is changed to be XFRM_STATE_EXPIRED doesn't really
> need to hold lock while modifying flow steering rules to drop traffic.
> 
> That state can be deleted only and as such mlx5e_ipsec_handle_tx_limit()
> work will be canceled anyway and won't run in parallel.
> 
> Fixes: b2f7b01d36a9 ("net/mlx5e: Simulate missing IPsec TX limits hardware functionality")
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>

Reviewed-by: Simon Horman <simon.horman@corigine.com>


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH net 4/4] net/mlx5e: Fix scheduling of IPsec ASO query while in atomic
  2023-06-05  8:09 ` [PATCH net 4/4] net/mlx5e: Fix scheduling of IPsec ASO query while in atomic Leon Romanovsky
@ 2023-06-06  9:13   ` Simon Horman
  0 siblings, 0 replies; 9+ messages in thread
From: Simon Horman @ 2023-06-06  9:13 UTC (permalink / raw)
  To: Leon Romanovsky
  Cc: Jakub Kicinski, Leon Romanovsky, Eric Dumazet, netdev,
	Paolo Abeni, Patrisious Haddad, Raed Salem, Saeed Mahameed,
	Steffen Klassert

On Mon, Jun 05, 2023 at 11:09:52AM +0300, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
> 
> ASO query can be scheduled in atomic context as such it can't use usleep.
> Use udelay as recommended in Documentation/timers/timers-howto.rst.
> 
> Fixes: 76e463f6508b ("net/mlx5e: Overcome slow response for first IPsec ASO WQE")
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>

Reviewed-by: Simon Horman <simon.horman@corigine.com>


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2023-06-06  9:13 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-06-05  8:09 [PATCH net 0/4] Fix mixing atomic/non-atomic contexts in mlx5 IPsec code Leon Romanovsky
2023-06-05  8:09 ` [PATCH net 1/4] net/mlx5e: Don't delay release of hardware objects Leon Romanovsky
2023-06-06  9:07   ` Simon Horman
2023-06-05  8:09 ` [PATCH net 2/4] net/mlx5e: Fix ESN update kernel panic Leon Romanovsky
2023-06-06  9:08   ` Simon Horman
2023-06-05  8:09 ` [PATCH net 3/4] net/mlx5e: Drop XFRM state lock when modifying flow steering Leon Romanovsky
2023-06-06  9:12   ` Simon Horman
2023-06-05  8:09 ` [PATCH net 4/4] net/mlx5e: Fix scheduling of IPsec ASO query while in atomic Leon Romanovsky
2023-06-06  9:13   ` Simon Horman

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).