From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Firo Yang <firo.yang@suse.com>,
"David S . Miller" <davem@davemloft.net>,
Sasha Levin <sashal@kernel.org>,
netdev@vger.kernel.org
Subject: [PATCH AUTOSEL 4.14 18/21] enic: prevent waking up stopped tx queues over watchdog reset
Date: Sat, 22 Feb 2020 21:24:08 -0500 [thread overview]
Message-ID: <20200223022411.2159-18-sashal@kernel.org> (raw)
In-Reply-To: <20200223022411.2159-1-sashal@kernel.org>
From: Firo Yang <firo.yang@suse.com>
[ Upstream commit 0f90522591fd09dd201065c53ebefdfe3c6b55cb ]
Recent months, our customer reported several kernel crashes all
preceding with following message:
NETDEV WATCHDOG: eth2 (enic): transmit queue 0 timed out
Error message of one of those crashes:
BUG: unable to handle kernel paging request at ffffffffa007e090
After analyzing severl vmcores, I found that most of crashes are
caused by memory corruption. And all the corrupted memory areas
are overwritten by data of network packets. Moreover, I also found
that the tx queues were enabled over watchdog reset.
After going through the source code, I found that in enic_stop(),
the tx queues stopped by netif_tx_disable() could be woken up over
a small time window between netif_tx_disable() and the
napi_disable() by the following code path:
napi_poll->
enic_poll_msix_wq->
vnic_cq_service->
enic_wq_service->
netif_wake_subqueue(enic->netdev, q_number)->
test_and_clear_bit(__QUEUE_STATE_DRV_XOFF, &txq->state)
In turn, upper netowrk stack could queue skb to ENIC NIC though
enic_hard_start_xmit(). And this might introduce some race condition.
Our customer comfirmed that this kind of kernel crash doesn't occur over
90 days since they applied this patch.
Signed-off-by: Firo Yang <firo.yang@suse.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/net/ethernet/cisco/enic/enic_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c
index 19f374b180fc1..52a3b32390a9c 100644
--- a/drivers/net/ethernet/cisco/enic/enic_main.c
+++ b/drivers/net/ethernet/cisco/enic/enic_main.c
@@ -1972,10 +1972,10 @@ static int enic_stop(struct net_device *netdev)
napi_disable(&enic->napi[i]);
netif_carrier_off(netdev);
- netif_tx_disable(netdev);
if (vnic_dev_get_intr_mode(enic->vdev) == VNIC_DEV_INTR_MODE_MSIX)
for (i = 0; i < enic->wq_count; i++)
napi_disable(&enic->napi[enic_cq_wq(enic, i)]);
+ netif_tx_disable(netdev);
if (!enic_is_dynamic(enic) && !enic_is_sriov_vf(enic))
enic_dev_del_station_addr(enic);
--
2.20.1
next prev parent reply other threads:[~2020-02-23 2:27 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-23 2:23 [PATCH AUTOSEL 4.14 01/21] ipmi:ssif: Handle a possible NULL pointer reference Sasha Levin
2020-02-23 2:23 ` [PATCH AUTOSEL 4.14 02/21] drm/msm: Set dma maximum segment size for mdss Sasha Levin
2020-02-23 2:23 ` [PATCH AUTOSEL 4.14 03/21] dax: pass NOWAIT flag to iomap_apply Sasha Levin
2020-02-23 2:23 ` [PATCH AUTOSEL 4.14 04/21] mac80211: consider more elements in parsing CRC Sasha Levin
2020-02-23 2:23 ` [PATCH AUTOSEL 4.14 05/21] cfg80211: check wiphy driver existence for drvinfo report Sasha Levin
2020-02-23 2:23 ` [PATCH AUTOSEL 4.14 06/21] qmi_wwan: re-add DW5821e pre-production variant Sasha Levin
2020-02-23 2:23 ` [PATCH AUTOSEL 4.14 07/21] qmi_wwan: unconditionally reject 2 ep interfaces Sasha Levin
2020-02-23 2:23 ` [PATCH AUTOSEL 4.14 08/21] arm/ftrace: Fix BE text poking Sasha Levin
2020-02-23 2:23 ` [PATCH AUTOSEL 4.14 09/21] net: ena: fix potential crash when rxfh key is NULL Sasha Levin
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 10/21] net: ena: fix uses of round_jiffies() Sasha Levin
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 11/21] net: ena: add missing ethtool TX timestamping indication Sasha Levin
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 12/21] net: ena: fix incorrect default RSS key Sasha Levin
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 13/21] net: ena: rss: fix failure to get indirection table Sasha Levin
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 14/21] net: ena: rss: store hash function as values and not bits Sasha Levin
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 15/21] net: ena: fix incorrectly saving queue numbers when setting RSS indirection table Sasha Levin
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 16/21] net: ena: ethtool: use correct value for crc32 hash Sasha Levin
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 17/21] net: ena: ena-com.c: prevent NULL pointer dereference Sasha Levin
2020-02-23 2:24 ` Sasha Levin [this message]
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 19/21] cifs: Fix mode output in debugging statements Sasha Levin
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 20/21] bcache: ignore pending signals when creating gc and allocator thread Sasha Levin
2020-02-23 2:24 ` [PATCH AUTOSEL 4.14 21/21] cfg80211: add missing policy for NL80211_ATTR_STATUS_CODE Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200223022411.2159-18-sashal@kernel.org \
--to=sashal@kernel.org \
--cc=davem@davemloft.net \
--cc=firo.yang@suse.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).