[PATCH v3] net: ks8851: Fix deadlock with the SPI chip variant

[PATCH v3] net: ks8851: Fix deadlock with the SPI chip variant

[Ronald Wahl]

From: Ronald Wahl <ronald.wahl@raritan.com>

When SMP is enabled and spinlocks are actually functional then there is
a deadlock with the 'statelock' spinlock between ks8851_start_xmit_spi
and ks8851_irq:

    watchdog: BUG: soft lockup - CPU#0 stuck for 27s!
    call trace:
      queued_spin_lock_slowpath+0x100/0x284
      do_raw_spin_lock+0x34/0x44
      ks8851_start_xmit_spi+0x30/0xb8
      ks8851_start_xmit+0x14/0x20
      netdev_start_xmit+0x40/0x6c
      dev_hard_start_xmit+0x6c/0xbc
      sch_direct_xmit+0xa4/0x22c
      __qdisc_run+0x138/0x3fc
      qdisc_run+0x24/0x3c
      net_tx_action+0xf8/0x130
      handle_softirqs+0x1ac/0x1f0
      __do_softirq+0x14/0x20
      ____do_softirq+0x10/0x1c
      call_on_irq_stack+0x3c/0x58
      do_softirq_own_stack+0x1c/0x28
      __irq_exit_rcu+0x54/0x9c
      irq_exit_rcu+0x10/0x1c
      el1_interrupt+0x38/0x50
      el1h_64_irq_handler+0x18/0x24
      el1h_64_irq+0x64/0x68
      __netif_schedule+0x6c/0x80
      netif_tx_wake_queue+0x38/0x48
      ks8851_irq+0xb8/0x2c8
      irq_thread_fn+0x2c/0x74
      irq_thread+0x10c/0x1b0
      kthread+0xc8/0xd8
      ret_from_fork+0x10/0x20

This issue has not been identified earlier because tests were done on
a device with SMP disabled and so spinlocks were actually NOPs.

Now use spin_(un)lock_bh for TX queue related locking to avoid execution
of softirq work synchronously that would lead to a deadlock.

Fixes: 3dc5d4454545 ("net: ks8851: Fix TX stall caused by TX buffer overrun")
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Jakub Kicinski <kuba@kernel.org>
Cc: Paolo Abeni <pabeni@redhat.com>
Cc: Simon Horman <horms@kernel.org>
Cc: netdev@vger.kernel.org
Cc: stable@vger.kernel.org # 5.10+
Signed-off-by: Ronald Wahl <ronald.wahl@raritan.com>
---
V2: - use spin_lock_bh instead of moving netif_wake_queue outside of
      locked region (doing the same in the start_xmit function)
    - add missing net: tag

V3: - spin_lock_bh(ks->statelock) always except in xmit which is in BH
      already

 drivers/net/ethernet/micrel/ks8851_common.c | 8 ++++----
 drivers/net/ethernet/micrel/ks8851_spi.c    | 4 ++--
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/micrel/ks8851_common.c b/drivers/net/ethernet/micrel/ks8851_common.c
index 6453c92f0fa7..13462811eaae 100644
--- a/drivers/net/ethernet/micrel/ks8851_common.c
+++ b/drivers/net/ethernet/micrel/ks8851_common.c
@@ -352,11 +352,11 @@ static irqreturn_t ks8851_irq(int irq, void *_ks)
 		netif_dbg(ks, intr, ks->netdev,
 			  "%s: txspace %d\n", __func__, tx_space);

-		spin_lock(&ks->statelock);
+		spin_lock_bh(&ks->statelock);
 		ks->tx_space = tx_space;
 		if (netif_queue_stopped(ks->netdev))
 			netif_wake_queue(ks->netdev);
-		spin_unlock(&ks->statelock);
+		spin_unlock_bh(&ks->statelock);
 	}

 	if (status & IRQ_SPIBEI) {
@@ -635,14 +635,14 @@ static void ks8851_set_rx_mode(struct net_device *dev)

 	/* schedule work to do the actual set of the data if needed */

-	spin_lock(&ks->statelock);
+	spin_lock_bh(&ks->statelock);

 	if (memcmp(&rxctrl, &ks->rxctrl, sizeof(rxctrl)) != 0) {
 		memcpy(&ks->rxctrl, &rxctrl, sizeof(ks->rxctrl));
 		schedule_work(&ks->rxctrl_work);
 	}

-	spin_unlock(&ks->statelock);
+	spin_unlock_bh(&ks->statelock);
 }

 static int ks8851_set_mac_address(struct net_device *dev, void *addr)
diff --git a/drivers/net/ethernet/micrel/ks8851_spi.c b/drivers/net/ethernet/micrel/ks8851_spi.c
index 670c1de966db..3062cc0f9199 100644
--- a/drivers/net/ethernet/micrel/ks8851_spi.c
+++ b/drivers/net/ethernet/micrel/ks8851_spi.c
@@ -340,10 +340,10 @@ static void ks8851_tx_work(struct work_struct *work)

 	tx_space = ks8851_rdreg16_spi(ks, KS_TXMIR);

-	spin_lock(&ks->statelock);
+	spin_lock_bh(&ks->statelock);
 	ks->queued_len -= dequeued_len;
 	ks->tx_space = tx_space;
-	spin_unlock(&ks->statelock);
+	spin_unlock_bh(&ks->statelock);

 	ks8851_unlock_spi(ks, &flags);
 }
--
2.45.2

Re: [PATCH v3] net: ks8851: Fix deadlock with the SPI chip variant

[Simon Horman]

On Sat, Jul 06, 2024 at 12:13:37PM +0200, Ronald Wahl wrote:
> From: Ronald Wahl <ronald.wahl@raritan.com>
> 
> When SMP is enabled and spinlocks are actually functional then there is
> a deadlock with the 'statelock' spinlock between ks8851_start_xmit_spi
> and ks8851_irq:
> 
>     watchdog: BUG: soft lockup - CPU#0 stuck for 27s!
>     call trace:
>       queued_spin_lock_slowpath+0x100/0x284
>       do_raw_spin_lock+0x34/0x44
>       ks8851_start_xmit_spi+0x30/0xb8
>       ks8851_start_xmit+0x14/0x20
>       netdev_start_xmit+0x40/0x6c
>       dev_hard_start_xmit+0x6c/0xbc
>       sch_direct_xmit+0xa4/0x22c
>       __qdisc_run+0x138/0x3fc
>       qdisc_run+0x24/0x3c
>       net_tx_action+0xf8/0x130
>       handle_softirqs+0x1ac/0x1f0
>       __do_softirq+0x14/0x20
>       ____do_softirq+0x10/0x1c
>       call_on_irq_stack+0x3c/0x58
>       do_softirq_own_stack+0x1c/0x28
>       __irq_exit_rcu+0x54/0x9c
>       irq_exit_rcu+0x10/0x1c
>       el1_interrupt+0x38/0x50
>       el1h_64_irq_handler+0x18/0x24
>       el1h_64_irq+0x64/0x68
>       __netif_schedule+0x6c/0x80
>       netif_tx_wake_queue+0x38/0x48
>       ks8851_irq+0xb8/0x2c8
>       irq_thread_fn+0x2c/0x74
>       irq_thread+0x10c/0x1b0
>       kthread+0xc8/0xd8
>       ret_from_fork+0x10/0x20
> 
> This issue has not been identified earlier because tests were done on
> a device with SMP disabled and so spinlocks were actually NOPs.
> 
> Now use spin_(un)lock_bh for TX queue related locking to avoid execution
> of softirq work synchronously that would lead to a deadlock.
> 
> Fixes: 3dc5d4454545 ("net: ks8851: Fix TX stall caused by TX buffer overrun")
> Cc: "David S. Miller" <davem@davemloft.net>
> Cc: Eric Dumazet <edumazet@google.com>
> Cc: Jakub Kicinski <kuba@kernel.org>
> Cc: Paolo Abeni <pabeni@redhat.com>
> Cc: Simon Horman <horms@kernel.org>
> Cc: netdev@vger.kernel.org
> Cc: stable@vger.kernel.org # 5.10+
> Signed-off-by: Ronald Wahl <ronald.wahl@raritan.com>
> ---
> V2: - use spin_lock_bh instead of moving netif_wake_queue outside of
>       locked region (doing the same in the start_xmit function)
>     - add missing net: tag
> 
> V3: - spin_lock_bh(ks->statelock) always except in xmit which is in BH
>       already

I agree that this lock is now always either taken _bh,
or in the xmit path which is executed in BH context.

Reviewed-by: Simon Horman <horms@kernel.org>

Re: [PATCH v3] net: ks8851: Fix deadlock with the SPI chip variant

[Jakub Kicinski]

On Sat,  6 Jul 2024 12:13:37 +0200 Ronald Wahl wrote:
> From: Ronald Wahl <ronald.wahl@raritan.com>
> 
> When SMP is enabled and spinlocks are actually functional then there is
> a deadlock with the 'statelock' spinlock between ks8851_start_xmit_spi
> and ks8851_irq:

Applied, thanks!