* [PATCH 0/1] qed: Yet another scheduling while atomic fix
@ 2023-07-26 17:19 Konstantin Khorenko
2023-07-26 17:19 ` [PATCH 1/1] qed: Fix scheduling in a tasklet while getting stats Konstantin Khorenko
0 siblings, 1 reply; 7+ messages in thread
From: Konstantin Khorenko @ 2023-07-26 17:19 UTC (permalink / raw)
To: netdev
Cc: Jakub Kicinski, Manish Chopra, Ariel Elior, David Miller,
Sudarsana Kalluru, Paolo Abeni, Konstantin Khorenko
Running an old RHEL7-based kernel we have got several cases of following
BUG_ON():
BUG: scheduling while atomic: swapper/24/0/0x00000100
[<ffffffffb41c6199>] schedule+0x29/0x70
[<ffffffffb41c5512>] schedule_hrtimeout_range_clock+0xb2/0x150
[<ffffffffb41c55c3>] schedule_hrtimeout_range+0x13/0x20
[<ffffffffb41c3bcf>] usleep_range+0x4f/0x70
[<ffffffffc08d3e58>] qed_ptt_acquire+0x38/0x100 [qed]
[<ffffffffc08eac48>] _qed_get_vport_stats+0x458/0x580 [qed]
[<ffffffffc08ead8c>] qed_get_vport_stats+0x1c/0xd0 [qed]
[<ffffffffc08dffd3>] qed_get_protocol_stats+0x93/0x100 [qed]
qed_mcp_send_protocol_stats
case MFW_DRV_MSG_GET_LAN_STATS:
case MFW_DRV_MSG_GET_FCOE_STATS:
case MFW_DRV_MSG_GET_ISCSI_STATS:
case MFW_DRV_MSG_GET_RDMA_STATS:
[<ffffffffc08e36d8>] qed_mcp_handle_events+0x2d8/0x890 [qed]
qed_int_assertion
qed_int_attentions
[<ffffffffc08d9490>] qed_int_sp_dpc+0xa50/0xdc0 [qed]
[<ffffffffb3aa7623>] tasklet_action+0x83/0x140
[<ffffffffb41d9125>] __do_softirq+0x125/0x2bb
[<ffffffffb41d560c>] call_softirq+0x1c/0x30
[<ffffffffb3a30645>] do_softirq+0x65/0xa0
[<ffffffffb3aa78d5>] irq_exit+0x105/0x110
[<ffffffffb41d8996>] do_IRQ+0x56/0xf0
The situation is clear - tasklet function called schedule, but the fix
is not so trivial.
Checking the mainstream code it seem the same calltrace is still
possible on the latest kernel as well, so here is the fix.
The was a similar case recently for QEDE driver (reading stats through
sysfs) which resulted in the commit:
42510dffd0e2 ("qed/qede: Fix scheduling while atomic")
i tried to implement the same logic as a fix for my case, but failed:
unfortunately it's not clear to me for this particular QED driver case
which statistic to collect in delay works for each particular device and
getting ALL possible stats for all devices, ignoring device type seems
incorrect.
Taking into account that i do not have access to the hardware at all,
the delay work approach is nearly impossible for me.
Thus i have taken the idea from patch v3 - just to provide the context
by the caller:
https://www.spinics.net/lists/netdev/msg901089.html
At least this solution is technically clear and hopefully i did not make
stupid mistakes here.
The patch is COMPILE TESTED ONLY.
i would appreciate if somebody can test the patch. :)
Konstantin Khorenko (1):
qed: Fix scheduling in a tasklet while getting stats
drivers/net/ethernet/qlogic/qed/qed_dev_api.h | 2 ++
drivers/net/ethernet/qlogic/qed/qed_fcoe.c | 19 ++++++++++----
drivers/net/ethernet/qlogic/qed/qed_fcoe.h | 6 +++--
drivers/net/ethernet/qlogic/qed/qed_hw.c | 26 ++++++++++++++++---
drivers/net/ethernet/qlogic/qed/qed_iscsi.c | 19 ++++++++++----
drivers/net/ethernet/qlogic/qed/qed_iscsi.h | 6 +++--
drivers/net/ethernet/qlogic/qed/qed_l2.c | 19 ++++++++++----
drivers/net/ethernet/qlogic/qed/qed_l2.h | 3 +++
drivers/net/ethernet/qlogic/qed/qed_main.c | 6 ++---
9 files changed, 80 insertions(+), 26 deletions(-)
--
2.31.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH 1/1] qed: Fix scheduling in a tasklet while getting stats 2023-07-26 17:19 [PATCH 0/1] qed: Yet another scheduling while atomic fix Konstantin Khorenko @ 2023-07-26 17:19 ` Konstantin Khorenko 2023-07-27 11:59 ` Simon Horman 0 siblings, 1 reply; 7+ messages in thread From: Konstantin Khorenko @ 2023-07-26 17:19 UTC (permalink / raw) To: netdev Cc: Jakub Kicinski, Manish Chopra, Ariel Elior, David Miller, Sudarsana Kalluru, Paolo Abeni, Konstantin Khorenko Here we've got to a situation when tasklet called usleep_range() in PTT acquire logic, thus welcome to the "scheduling while atomic" BUG(). BUG: scheduling while atomic: swapper/24/0/0x00000100 [<ffffffffb41c6199>] schedule+0x29/0x70 [<ffffffffb41c5512>] schedule_hrtimeout_range_clock+0xb2/0x150 [<ffffffffb41c55c3>] schedule_hrtimeout_range+0x13/0x20 [<ffffffffb41c3bcf>] usleep_range+0x4f/0x70 [<ffffffffc08d3e58>] qed_ptt_acquire+0x38/0x100 [qed] [<ffffffffc08eac48>] _qed_get_vport_stats+0x458/0x580 [qed] [<ffffffffc08ead8c>] qed_get_vport_stats+0x1c/0xd0 [qed] [<ffffffffc08dffd3>] qed_get_protocol_stats+0x93/0x100 [qed] qed_mcp_send_protocol_stats case MFW_DRV_MSG_GET_LAN_STATS: case MFW_DRV_MSG_GET_FCOE_STATS: case MFW_DRV_MSG_GET_ISCSI_STATS: case MFW_DRV_MSG_GET_RDMA_STATS: [<ffffffffc08e36d8>] qed_mcp_handle_events+0x2d8/0x890 [qed] qed_int_assertion qed_int_attentions [<ffffffffc08d9490>] qed_int_sp_dpc+0xa50/0xdc0 [qed] [<ffffffffb3aa7623>] tasklet_action+0x83/0x140 [<ffffffffb41d9125>] __do_softirq+0x125/0x2bb [<ffffffffb41d560c>] call_softirq+0x1c/0x30 [<ffffffffb3a30645>] do_softirq+0x65/0xa0 [<ffffffffb3aa78d5>] irq_exit+0x105/0x110 [<ffffffffb41d8996>] do_IRQ+0x56/0xf0 Fix this by making caller to provide the context whether it could be in atomic context flow or not when getting stats from QED driver. QED driver based on the context provided decide to schedule out or not when acquiring the PTT BAR window. We faced the BUG_ON() while getting vport stats, but according to the code same issue could happen for fcoe and iscsi statistics as well, so fixing them too. Fixes: 6c75424612a7 ("qed: Add support for NCSI statistics.") Fixes: 1e128c81290a ("qed: Add support for hardware offloaded FCoE.") Fixes: 2f2b2614e893 ("qed: Provide iSCSI statistics to management") Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com> --- drivers/net/ethernet/qlogic/qed/qed_dev_api.h | 2 ++ drivers/net/ethernet/qlogic/qed/qed_fcoe.c | 19 ++++++++++---- drivers/net/ethernet/qlogic/qed/qed_fcoe.h | 6 +++-- drivers/net/ethernet/qlogic/qed/qed_hw.c | 26 ++++++++++++++++--- drivers/net/ethernet/qlogic/qed/qed_iscsi.c | 19 ++++++++++---- drivers/net/ethernet/qlogic/qed/qed_iscsi.h | 6 +++-- drivers/net/ethernet/qlogic/qed/qed_l2.c | 19 ++++++++++---- drivers/net/ethernet/qlogic/qed/qed_l2.h | 3 +++ drivers/net/ethernet/qlogic/qed/qed_main.c | 6 ++--- 9 files changed, 80 insertions(+), 26 deletions(-) diff --git a/drivers/net/ethernet/qlogic/qed/qed_dev_api.h b/drivers/net/ethernet/qlogic/qed/qed_dev_api.h index f8682356d0cf..2ec8333363ae 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_dev_api.h +++ b/drivers/net/ethernet/qlogic/qed/qed_dev_api.h @@ -192,6 +192,8 @@ void qed_hw_remove(struct qed_dev *cdev); * exported function). */ struct qed_ptt *qed_ptt_acquire(struct qed_hwfn *p_hwfn); +struct qed_ptt *qed_ptt_acquire_context(struct qed_hwfn *p_hwfn, + bool is_atomic); /** * qed_ptt_release(): Release PTT Window. diff --git a/drivers/net/ethernet/qlogic/qed/qed_fcoe.c b/drivers/net/ethernet/qlogic/qed/qed_fcoe.c index 3764190b948e..04602ac94708 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_fcoe.c +++ b/drivers/net/ethernet/qlogic/qed/qed_fcoe.c @@ -693,13 +693,14 @@ static void _qed_fcoe_get_pstats(struct qed_hwfn *p_hwfn, } static int qed_fcoe_get_stats(struct qed_hwfn *p_hwfn, - struct qed_fcoe_stats *p_stats) + struct qed_fcoe_stats *p_stats, + bool is_atomic) { struct qed_ptt *p_ptt; memset(p_stats, 0, sizeof(*p_stats)); - p_ptt = qed_ptt_acquire(p_hwfn); + p_ptt = qed_ptt_acquire_context(p_hwfn, is_atomic); if (!p_ptt) { DP_ERR(p_hwfn, "Failed to acquire ptt\n"); @@ -973,19 +974,27 @@ static int qed_fcoe_destroy_conn(struct qed_dev *cdev, QED_SPQ_MODE_EBLOCK, NULL); } +static int qed_fcoe_stats_context(struct qed_dev *cdev, + struct qed_fcoe_stats *stats, + bool is_atomic) +{ + return qed_fcoe_get_stats(QED_AFFIN_HWFN(cdev), stats, is_atomic); +} + static int qed_fcoe_stats(struct qed_dev *cdev, struct qed_fcoe_stats *stats) { - return qed_fcoe_get_stats(QED_AFFIN_HWFN(cdev), stats); + return qed_fcoe_stats_context(cdev, stats, false); } void qed_get_protocol_stats_fcoe(struct qed_dev *cdev, - struct qed_mcp_fcoe_stats *stats) + struct qed_mcp_fcoe_stats *stats, + bool is_atomic) { struct qed_fcoe_stats proto_stats; /* Retrieve FW statistics */ memset(&proto_stats, 0, sizeof(proto_stats)); - if (qed_fcoe_stats(cdev, &proto_stats)) { + if (qed_fcoe_stats_context(cdev, &proto_stats, is_atomic)) { DP_VERBOSE(cdev, QED_MSG_STORAGE, "Failed to collect FCoE statistics\n"); return; diff --git a/drivers/net/ethernet/qlogic/qed/qed_fcoe.h b/drivers/net/ethernet/qlogic/qed/qed_fcoe.h index 19c85adf4ceb..883a3942a1af 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_fcoe.h +++ b/drivers/net/ethernet/qlogic/qed/qed_fcoe.h @@ -29,7 +29,8 @@ void qed_fcoe_setup(struct qed_hwfn *p_hwfn); void qed_fcoe_free(struct qed_hwfn *p_hwfn); void qed_get_protocol_stats_fcoe(struct qed_dev *cdev, - struct qed_mcp_fcoe_stats *stats); + struct qed_mcp_fcoe_stats *stats, + bool is_atomic); #else /* CONFIG_QED_FCOE */ static inline int qed_fcoe_alloc(struct qed_hwfn *p_hwfn) { @@ -40,7 +41,8 @@ static inline void qed_fcoe_setup(struct qed_hwfn *p_hwfn) {} static inline void qed_fcoe_free(struct qed_hwfn *p_hwfn) {} static inline void qed_get_protocol_stats_fcoe(struct qed_dev *cdev, - struct qed_mcp_fcoe_stats *stats) + struct qed_mcp_fcoe_stats *stats, + bool is_atomic) { } #endif /* CONFIG_QED_FCOE */ diff --git a/drivers/net/ethernet/qlogic/qed/qed_hw.c b/drivers/net/ethernet/qlogic/qed/qed_hw.c index 554f30b0cfd5..6263f847b6b9 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_hw.c +++ b/drivers/net/ethernet/qlogic/qed/qed_hw.c @@ -23,7 +23,10 @@ #include "qed_reg_addr.h" #include "qed_sriov.h" -#define QED_BAR_ACQUIRE_TIMEOUT 1000 +#define QED_BAR_ACQUIRE_TIMEOUT_USLEEP_CNT 1000 +#define QED_BAR_ACQUIRE_TIMEOUT_USLEEP 1000 +#define QED_BAR_ACQUIRE_TIMEOUT_UDELAY_CNT 100000 +#define QED_BAR_ACQUIRE_TIMEOUT_UDELAY 10 /* Invalid values */ #define QED_BAR_INVALID_OFFSET (cpu_to_le32(-1)) @@ -84,12 +87,22 @@ void qed_ptt_pool_free(struct qed_hwfn *p_hwfn) } struct qed_ptt *qed_ptt_acquire(struct qed_hwfn *p_hwfn) +{ + return qed_ptt_acquire_context(p_hwfn, false); +} + +struct qed_ptt *qed_ptt_acquire_context(struct qed_hwfn *p_hwfn, bool is_atomic) { struct qed_ptt *p_ptt; - unsigned int i; + unsigned int i, count; + + if (is_atomic) + count = QED_BAR_ACQUIRE_TIMEOUT_UDELAY_CNT; + else + count = QED_BAR_ACQUIRE_TIMEOUT_USLEEP_CNT; /* Take the free PTT from the list */ - for (i = 0; i < QED_BAR_ACQUIRE_TIMEOUT; i++) { + for (i = 0; i < count; i++) { spin_lock_bh(&p_hwfn->p_ptt_pool->lock); if (!list_empty(&p_hwfn->p_ptt_pool->free_list)) { @@ -105,7 +118,12 @@ struct qed_ptt *qed_ptt_acquire(struct qed_hwfn *p_hwfn) } spin_unlock_bh(&p_hwfn->p_ptt_pool->lock); - usleep_range(1000, 2000); + + if (is_atomic) + udelay(QED_BAR_ACQUIRE_TIMEOUT_UDELAY); + else + usleep_range(QED_BAR_ACQUIRE_TIMEOUT_USLEEP, + QED_BAR_ACQUIRE_TIMEOUT_USLEEP * 2); } DP_NOTICE(p_hwfn, "PTT acquire timeout - failed to allocate PTT\n"); diff --git a/drivers/net/ethernet/qlogic/qed/qed_iscsi.c b/drivers/net/ethernet/qlogic/qed/qed_iscsi.c index 511ab214eb9c..980e7289b481 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_iscsi.c +++ b/drivers/net/ethernet/qlogic/qed/qed_iscsi.c @@ -999,13 +999,14 @@ static void _qed_iscsi_get_pstats(struct qed_hwfn *p_hwfn, } static int qed_iscsi_get_stats(struct qed_hwfn *p_hwfn, - struct qed_iscsi_stats *stats) + struct qed_iscsi_stats *stats, + bool is_atomic) { struct qed_ptt *p_ptt; memset(stats, 0, sizeof(*stats)); - p_ptt = qed_ptt_acquire(p_hwfn); + p_ptt = qed_ptt_acquire_context(p_hwfn, is_atomic); if (!p_ptt) { DP_ERR(p_hwfn, "Failed to acquire ptt\n"); return -EAGAIN; @@ -1336,9 +1337,16 @@ static int qed_iscsi_destroy_conn(struct qed_dev *cdev, QED_SPQ_MODE_EBLOCK, NULL); } +static int qed_iscsi_stats_context(struct qed_dev *cdev, + struct qed_iscsi_stats *stats, + bool is_atomic) +{ + return qed_iscsi_get_stats(QED_AFFIN_HWFN(cdev), stats, is_atomic); +} + static int qed_iscsi_stats(struct qed_dev *cdev, struct qed_iscsi_stats *stats) { - return qed_iscsi_get_stats(QED_AFFIN_HWFN(cdev), stats); + return qed_iscsi_stats_context(cdev, stats, false); } static int qed_iscsi_change_mac(struct qed_dev *cdev, @@ -1358,13 +1366,14 @@ static int qed_iscsi_change_mac(struct qed_dev *cdev, } void qed_get_protocol_stats_iscsi(struct qed_dev *cdev, - struct qed_mcp_iscsi_stats *stats) + struct qed_mcp_iscsi_stats *stats, + bool is_atomic) { struct qed_iscsi_stats proto_stats; /* Retrieve FW statistics */ memset(&proto_stats, 0, sizeof(proto_stats)); - if (qed_iscsi_stats(cdev, &proto_stats)) { + if (qed_iscsi_stats_context(cdev, &proto_stats, is_atomic)) { DP_VERBOSE(cdev, QED_MSG_STORAGE, "Failed to collect ISCSI statistics\n"); return; diff --git a/drivers/net/ethernet/qlogic/qed/qed_iscsi.h b/drivers/net/ethernet/qlogic/qed/qed_iscsi.h index dec2b00259d4..7d8d6ad7faa9 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_iscsi.h +++ b/drivers/net/ethernet/qlogic/qed/qed_iscsi.h @@ -43,7 +43,8 @@ void qed_iscsi_free(struct qed_hwfn *p_hwfn); * Return: Void. */ void qed_get_protocol_stats_iscsi(struct qed_dev *cdev, - struct qed_mcp_iscsi_stats *stats); + struct qed_mcp_iscsi_stats *stats, + bool is_atomic); #else /* IS_ENABLED(CONFIG_QED_ISCSI) */ static inline int qed_iscsi_alloc(struct qed_hwfn *p_hwfn) { @@ -56,7 +57,8 @@ static inline void qed_iscsi_free(struct qed_hwfn *p_hwfn) {} static inline void qed_get_protocol_stats_iscsi(struct qed_dev *cdev, - struct qed_mcp_iscsi_stats *stats) {} + struct qed_mcp_iscsi_stats *stats, + bool is_atomic) {} #endif /* IS_ENABLED(CONFIG_QED_ISCSI) */ #endif diff --git a/drivers/net/ethernet/qlogic/qed/qed_l2.c b/drivers/net/ethernet/qlogic/qed/qed_l2.c index 7776d3bdd459..970b9aabbc3d 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_l2.c +++ b/drivers/net/ethernet/qlogic/qed/qed_l2.c @@ -1863,7 +1863,8 @@ static void __qed_get_vport_stats(struct qed_hwfn *p_hwfn, } static void _qed_get_vport_stats(struct qed_dev *cdev, - struct qed_eth_stats *stats) + struct qed_eth_stats *stats, + bool is_atomic) { u8 fw_vport = 0; int i; @@ -1872,10 +1873,11 @@ static void _qed_get_vport_stats(struct qed_dev *cdev, for_each_hwfn(cdev, i) { struct qed_hwfn *p_hwfn = &cdev->hwfns[i]; - struct qed_ptt *p_ptt = IS_PF(cdev) ? qed_ptt_acquire(p_hwfn) - : NULL; + struct qed_ptt *p_ptt; bool b_get_port_stats; + p_ptt = IS_PF(cdev) ? qed_ptt_acquire_context(p_hwfn, is_atomic) + : NULL; if (IS_PF(cdev)) { /* The main vport index is relative first */ if (qed_fw_vport(p_hwfn, 0, &fw_vport)) { @@ -1900,6 +1902,13 @@ static void _qed_get_vport_stats(struct qed_dev *cdev, } void qed_get_vport_stats(struct qed_dev *cdev, struct qed_eth_stats *stats) +{ + qed_get_vport_stats_context(cdev, stats, false); +} + +void qed_get_vport_stats_context(struct qed_dev *cdev, + struct qed_eth_stats *stats, + bool is_atomic) { u32 i; @@ -1908,7 +1917,7 @@ void qed_get_vport_stats(struct qed_dev *cdev, struct qed_eth_stats *stats) return; } - _qed_get_vport_stats(cdev, stats); + _qed_get_vport_stats(cdev, stats, is_atomic); if (!cdev->reset_stats) return; @@ -1960,7 +1969,7 @@ void qed_reset_vport_stats(struct qed_dev *cdev) if (!cdev->reset_stats) { DP_INFO(cdev, "Reset stats not allocated\n"); } else { - _qed_get_vport_stats(cdev, cdev->reset_stats); + _qed_get_vport_stats(cdev, cdev->reset_stats, false); cdev->reset_stats->common.link_change_count = 0; } } diff --git a/drivers/net/ethernet/qlogic/qed/qed_l2.h b/drivers/net/ethernet/qlogic/qed/qed_l2.h index a538cf478c14..8aa2cbc50ede 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_l2.h +++ b/drivers/net/ethernet/qlogic/qed/qed_l2.h @@ -250,6 +250,9 @@ qed_sp_eth_rx_queues_update(struct qed_hwfn *p_hwfn, struct qed_spq_comp_cb *p_comp_data); void qed_get_vport_stats(struct qed_dev *cdev, struct qed_eth_stats *stats); +void qed_get_vport_stats_context(struct qed_dev *cdev, + struct qed_eth_stats *stats, + bool is_atomic); void qed_reset_vport_stats(struct qed_dev *cdev); diff --git a/drivers/net/ethernet/qlogic/qed/qed_main.c b/drivers/net/ethernet/qlogic/qed/qed_main.c index f5af83342856..c278f8893042 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_main.c +++ b/drivers/net/ethernet/qlogic/qed/qed_main.c @@ -3092,7 +3092,7 @@ void qed_get_protocol_stats(struct qed_dev *cdev, switch (type) { case QED_MCP_LAN_STATS: - qed_get_vport_stats(cdev, ð_stats); + qed_get_vport_stats_context(cdev, ð_stats, true); stats->lan_stats.ucast_rx_pkts = eth_stats.common.rx_ucast_pkts; stats->lan_stats.ucast_tx_pkts = @@ -3100,10 +3100,10 @@ void qed_get_protocol_stats(struct qed_dev *cdev, stats->lan_stats.fcs_err = -1; break; case QED_MCP_FCOE_STATS: - qed_get_protocol_stats_fcoe(cdev, &stats->fcoe_stats); + qed_get_protocol_stats_fcoe(cdev, &stats->fcoe_stats, true); break; case QED_MCP_ISCSI_STATS: - qed_get_protocol_stats_iscsi(cdev, &stats->iscsi_stats); + qed_get_protocol_stats_iscsi(cdev, &stats->iscsi_stats, true); break; default: DP_VERBOSE(cdev, QED_MSG_SP, -- 2.31.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/1] qed: Fix scheduling in a tasklet while getting stats 2023-07-26 17:19 ` [PATCH 1/1] qed: Fix scheduling in a tasklet while getting stats Konstantin Khorenko @ 2023-07-27 11:59 ` Simon Horman 2023-07-27 15:26 ` [PATCH v2 0/1] qed: Yet another scheduling while atomic fix Konstantin Khorenko 0 siblings, 1 reply; 7+ messages in thread From: Simon Horman @ 2023-07-27 11:59 UTC (permalink / raw) To: Konstantin Khorenko Cc: netdev, Jakub Kicinski, Manish Chopra, Ariel Elior, David Miller, Sudarsana Kalluru, Paolo Abeni On Wed, Jul 26, 2023 at 08:19:30PM +0300, Konstantin Khorenko wrote: ... > diff --git a/drivers/net/ethernet/qlogic/qed/qed_iscsi.h b/drivers/net/ethernet/qlogic/qed/qed_iscsi.h > index dec2b00259d4..7d8d6ad7faa9 100644 > --- a/drivers/net/ethernet/qlogic/qed/qed_iscsi.h > +++ b/drivers/net/ethernet/qlogic/qed/qed_iscsi.h > @@ -43,7 +43,8 @@ void qed_iscsi_free(struct qed_hwfn *p_hwfn); > * Return: Void. > */ > void qed_get_protocol_stats_iscsi(struct qed_dev *cdev, > - struct qed_mcp_iscsi_stats *stats); > + struct qed_mcp_iscsi_stats *stats, > + bool is_atomic); Hi Konstantin, Please add is_atomic to the kernel doc for qed_get_protocol_stats_iscsi, which is immediately above this function declaration. > #else /* IS_ENABLED(CONFIG_QED_ISCSI) */ > static inline int qed_iscsi_alloc(struct qed_hwfn *p_hwfn) > { > @@ -56,7 +57,8 @@ static inline void qed_iscsi_free(struct qed_hwfn *p_hwfn) {} > > static inline void > qed_get_protocol_stats_iscsi(struct qed_dev *cdev, > - struct qed_mcp_iscsi_stats *stats) {} > + struct qed_mcp_iscsi_stats *stats, > + bool is_atomic) {} > #endif /* IS_ENABLED(CONFIG_QED_ISCSI) */ > > #endif -- pw-bot: changes-requested ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v2 0/1] qed: Yet another scheduling while atomic fix 2023-07-27 11:59 ` Simon Horman @ 2023-07-27 15:26 ` Konstantin Khorenko 2023-07-27 15:26 ` [PATCH v2 1/1] qed: Fix scheduling in a tasklet while getting stats Konstantin Khorenko 2023-07-29 16:20 ` [PATCH v2 0/1] qed: Yet another scheduling while atomic fix patchwork-bot+netdevbpf 0 siblings, 2 replies; 7+ messages in thread From: Konstantin Khorenko @ 2023-07-27 15:26 UTC (permalink / raw) To: Simon Horman Cc: Jakub Kicinski, Manish Chopra, Ariel Elior, David Miller, Sudarsana Kalluru, netdev, Paolo Abeni, Konstantin Khorenko Running an old RHEL7-based kernel we have got several cases of following BUG_ON(): BUG: scheduling while atomic: swapper/24/0/0x00000100 [<ffffffffb41c6199>] schedule+0x29/0x70 [<ffffffffb41c5512>] schedule_hrtimeout_range_clock+0xb2/0x150 [<ffffffffb41c55c3>] schedule_hrtimeout_range+0x13/0x20 [<ffffffffb41c3bcf>] usleep_range+0x4f/0x70 [<ffffffffc08d3e58>] qed_ptt_acquire+0x38/0x100 [qed] [<ffffffffc08eac48>] _qed_get_vport_stats+0x458/0x580 [qed] [<ffffffffc08ead8c>] qed_get_vport_stats+0x1c/0xd0 [qed] [<ffffffffc08dffd3>] qed_get_protocol_stats+0x93/0x100 [qed] qed_mcp_send_protocol_stats case MFW_DRV_MSG_GET_LAN_STATS: case MFW_DRV_MSG_GET_FCOE_STATS: case MFW_DRV_MSG_GET_ISCSI_STATS: case MFW_DRV_MSG_GET_RDMA_STATS: [<ffffffffc08e36d8>] qed_mcp_handle_events+0x2d8/0x890 [qed] qed_int_assertion qed_int_attentions [<ffffffffc08d9490>] qed_int_sp_dpc+0xa50/0xdc0 [qed] [<ffffffffb3aa7623>] tasklet_action+0x83/0x140 [<ffffffffb41d9125>] __do_softirq+0x125/0x2bb [<ffffffffb41d560c>] call_softirq+0x1c/0x30 [<ffffffffb3a30645>] do_softirq+0x65/0xa0 [<ffffffffb3aa78d5>] irq_exit+0x105/0x110 [<ffffffffb41d8996>] do_IRQ+0x56/0xf0 The situation is clear - tasklet function called schedule, but the fix is not so trivial. Checking the mainstream code it seem the same calltrace is still possible on the latest kernel as well, so here is the fix. The was a similar case recently for QEDE driver (reading stats through sysfs) which resulted in the commit: 42510dffd0e2 ("qed/qede: Fix scheduling while atomic") i tried to implement the same logic as a fix for my case, but failed: unfortunately it's not clear to me for this particular QED driver case which statistic to collect in delay works for each particular device and getting ALL possible stats for all devices, ignoring device type seems incorrect. Taking into account that i do not have access to the hardware at all, the delay work approach is nearly impossible for me. Thus i have taken the idea from patch v3 - just to provide the context by the caller: https://www.spinics.net/lists/netdev/msg901089.html At least this solution is technically clear and hopefully i did not make stupid mistakes here. The patch is COMPILE TESTED ONLY. i would appreciate if somebody can test the patch. :) Konstantin Khorenko (1): qed: Fix scheduling in a tasklet while getting stats --- v1->v2: - Fixed kdoc for qed_get_protocol_stats_iscsi() (added new arg description) - Added kdoc descriptions for qed_ptt_acquire_context(), qed_get_protocol_stats_fcoe(), qed_get_vport_stats() and qed_get_vport_stats_context() drivers/net/ethernet/qlogic/qed/qed_dev_api.h | 16 ++++++++++++ drivers/net/ethernet/qlogic/qed/qed_fcoe.c | 19 ++++++++++---- drivers/net/ethernet/qlogic/qed/qed_fcoe.h | 17 ++++++++++-- drivers/net/ethernet/qlogic/qed/qed_hw.c | 26 ++++++++++++++++--- drivers/net/ethernet/qlogic/qed/qed_iscsi.c | 19 ++++++++++---- drivers/net/ethernet/qlogic/qed/qed_iscsi.h | 8 ++++-- drivers/net/ethernet/qlogic/qed/qed_l2.c | 19 ++++++++++---- drivers/net/ethernet/qlogic/qed/qed_l2.h | 24 +++++++++++++++++ drivers/net/ethernet/qlogic/qed/qed_main.c | 6 ++--- 9 files changed, 128 insertions(+), 26 deletions(-) -- 2.31.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v2 1/1] qed: Fix scheduling in a tasklet while getting stats 2023-07-27 15:26 ` [PATCH v2 0/1] qed: Yet another scheduling while atomic fix Konstantin Khorenko @ 2023-07-27 15:26 ` Konstantin Khorenko 2023-07-29 11:07 ` Simon Horman 2023-07-29 16:20 ` [PATCH v2 0/1] qed: Yet another scheduling while atomic fix patchwork-bot+netdevbpf 1 sibling, 1 reply; 7+ messages in thread From: Konstantin Khorenko @ 2023-07-27 15:26 UTC (permalink / raw) To: Simon Horman Cc: Jakub Kicinski, Manish Chopra, Ariel Elior, David Miller, Sudarsana Kalluru, netdev, Paolo Abeni, Konstantin Khorenko Here we've got to a situation when tasklet called usleep_range() in PTT acquire logic, thus welcome to the "scheduling while atomic" BUG(). BUG: scheduling while atomic: swapper/24/0/0x00000100 [<ffffffffb41c6199>] schedule+0x29/0x70 [<ffffffffb41c5512>] schedule_hrtimeout_range_clock+0xb2/0x150 [<ffffffffb41c55c3>] schedule_hrtimeout_range+0x13/0x20 [<ffffffffb41c3bcf>] usleep_range+0x4f/0x70 [<ffffffffc08d3e58>] qed_ptt_acquire+0x38/0x100 [qed] [<ffffffffc08eac48>] _qed_get_vport_stats+0x458/0x580 [qed] [<ffffffffc08ead8c>] qed_get_vport_stats+0x1c/0xd0 [qed] [<ffffffffc08dffd3>] qed_get_protocol_stats+0x93/0x100 [qed] qed_mcp_send_protocol_stats case MFW_DRV_MSG_GET_LAN_STATS: case MFW_DRV_MSG_GET_FCOE_STATS: case MFW_DRV_MSG_GET_ISCSI_STATS: case MFW_DRV_MSG_GET_RDMA_STATS: [<ffffffffc08e36d8>] qed_mcp_handle_events+0x2d8/0x890 [qed] qed_int_assertion qed_int_attentions [<ffffffffc08d9490>] qed_int_sp_dpc+0xa50/0xdc0 [qed] [<ffffffffb3aa7623>] tasklet_action+0x83/0x140 [<ffffffffb41d9125>] __do_softirq+0x125/0x2bb [<ffffffffb41d560c>] call_softirq+0x1c/0x30 [<ffffffffb3a30645>] do_softirq+0x65/0xa0 [<ffffffffb3aa78d5>] irq_exit+0x105/0x110 [<ffffffffb41d8996>] do_IRQ+0x56/0xf0 Fix this by making caller to provide the context whether it could be in atomic context flow or not when getting stats from QED driver. QED driver based on the context provided decide to schedule out or not when acquiring the PTT BAR window. We faced the BUG_ON() while getting vport stats, but according to the code same issue could happen for fcoe and iscsi statistics as well, so fixing them too. Fixes: 6c75424612a7 ("qed: Add support for NCSI statistics.") Fixes: 1e128c81290a ("qed: Add support for hardware offloaded FCoE.") Fixes: 2f2b2614e893 ("qed: Provide iSCSI statistics to management") Cc: Sudarsana Kalluru <skalluru@marvell.com> Cc: David Miller <davem@davemloft.net> Cc: Manish Chopra <manishc@marvell.com> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com> --- v1->v2: - Fixed kdoc for qed_get_protocol_stats_iscsi() (added new arg description) - Added kdoc descriptions for qed_ptt_acquire_context(), qed_get_protocol_stats_fcoe(), qed_get_vport_stats() and qed_get_vport_stats_context() --- drivers/net/ethernet/qlogic/qed/qed_dev_api.h | 16 ++++++++++++ drivers/net/ethernet/qlogic/qed/qed_fcoe.c | 19 ++++++++++---- drivers/net/ethernet/qlogic/qed/qed_fcoe.h | 17 ++++++++++-- drivers/net/ethernet/qlogic/qed/qed_hw.c | 26 ++++++++++++++++--- drivers/net/ethernet/qlogic/qed/qed_iscsi.c | 19 ++++++++++---- drivers/net/ethernet/qlogic/qed/qed_iscsi.h | 8 ++++-- drivers/net/ethernet/qlogic/qed/qed_l2.c | 19 ++++++++++---- drivers/net/ethernet/qlogic/qed/qed_l2.h | 24 +++++++++++++++++ drivers/net/ethernet/qlogic/qed/qed_main.c | 6 ++--- 9 files changed, 128 insertions(+), 26 deletions(-) diff --git a/drivers/net/ethernet/qlogic/qed/qed_dev_api.h b/drivers/net/ethernet/qlogic/qed/qed_dev_api.h index f8682356d0cf..94d4f9413ab7 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_dev_api.h +++ b/drivers/net/ethernet/qlogic/qed/qed_dev_api.h @@ -193,6 +193,22 @@ void qed_hw_remove(struct qed_dev *cdev); */ struct qed_ptt *qed_ptt_acquire(struct qed_hwfn *p_hwfn); +/** + * qed_ptt_acquire_context(): Allocate a PTT window honoring the context + * atomicy. + * + * @p_hwfn: HW device data. + * @is_atomic: Hint from the caller - if the func can sleep or not. + * + * Context: The function should not sleep in case is_atomic == true. + * Return: struct qed_ptt. + * + * Should be called at the entry point to the driver + * (at the beginning of an exported function). + */ +struct qed_ptt *qed_ptt_acquire_context(struct qed_hwfn *p_hwfn, + bool is_atomic); + /** * qed_ptt_release(): Release PTT Window. * diff --git a/drivers/net/ethernet/qlogic/qed/qed_fcoe.c b/drivers/net/ethernet/qlogic/qed/qed_fcoe.c index 3764190b948e..04602ac94708 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_fcoe.c +++ b/drivers/net/ethernet/qlogic/qed/qed_fcoe.c @@ -693,13 +693,14 @@ static void _qed_fcoe_get_pstats(struct qed_hwfn *p_hwfn, } static int qed_fcoe_get_stats(struct qed_hwfn *p_hwfn, - struct qed_fcoe_stats *p_stats) + struct qed_fcoe_stats *p_stats, + bool is_atomic) { struct qed_ptt *p_ptt; memset(p_stats, 0, sizeof(*p_stats)); - p_ptt = qed_ptt_acquire(p_hwfn); + p_ptt = qed_ptt_acquire_context(p_hwfn, is_atomic); if (!p_ptt) { DP_ERR(p_hwfn, "Failed to acquire ptt\n"); @@ -973,19 +974,27 @@ static int qed_fcoe_destroy_conn(struct qed_dev *cdev, QED_SPQ_MODE_EBLOCK, NULL); } +static int qed_fcoe_stats_context(struct qed_dev *cdev, + struct qed_fcoe_stats *stats, + bool is_atomic) +{ + return qed_fcoe_get_stats(QED_AFFIN_HWFN(cdev), stats, is_atomic); +} + static int qed_fcoe_stats(struct qed_dev *cdev, struct qed_fcoe_stats *stats) { - return qed_fcoe_get_stats(QED_AFFIN_HWFN(cdev), stats); + return qed_fcoe_stats_context(cdev, stats, false); } void qed_get_protocol_stats_fcoe(struct qed_dev *cdev, - struct qed_mcp_fcoe_stats *stats) + struct qed_mcp_fcoe_stats *stats, + bool is_atomic) { struct qed_fcoe_stats proto_stats; /* Retrieve FW statistics */ memset(&proto_stats, 0, sizeof(proto_stats)); - if (qed_fcoe_stats(cdev, &proto_stats)) { + if (qed_fcoe_stats_context(cdev, &proto_stats, is_atomic)) { DP_VERBOSE(cdev, QED_MSG_STORAGE, "Failed to collect FCoE statistics\n"); return; diff --git a/drivers/net/ethernet/qlogic/qed/qed_fcoe.h b/drivers/net/ethernet/qlogic/qed/qed_fcoe.h index 19c85adf4ceb..214e8299ecb4 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_fcoe.h +++ b/drivers/net/ethernet/qlogic/qed/qed_fcoe.h @@ -28,8 +28,20 @@ int qed_fcoe_alloc(struct qed_hwfn *p_hwfn); void qed_fcoe_setup(struct qed_hwfn *p_hwfn); void qed_fcoe_free(struct qed_hwfn *p_hwfn); +/** + * qed_get_protocol_stats_fcoe(): Fills provided statistics + * struct with statistics. + * + * @cdev: Qed dev pointer. + * @stats: Points to struct that will be filled with statistics. + * @is_atomic: Hint from the caller - if the func can sleep or not. + * + * Context: The function should not sleep in case is_atomic == true. + * Return: Void. + */ void qed_get_protocol_stats_fcoe(struct qed_dev *cdev, - struct qed_mcp_fcoe_stats *stats); + struct qed_mcp_fcoe_stats *stats, + bool is_atomic); #else /* CONFIG_QED_FCOE */ static inline int qed_fcoe_alloc(struct qed_hwfn *p_hwfn) { @@ -40,7 +52,8 @@ static inline void qed_fcoe_setup(struct qed_hwfn *p_hwfn) {} static inline void qed_fcoe_free(struct qed_hwfn *p_hwfn) {} static inline void qed_get_protocol_stats_fcoe(struct qed_dev *cdev, - struct qed_mcp_fcoe_stats *stats) + struct qed_mcp_fcoe_stats *stats, + bool is_atomic) { } #endif /* CONFIG_QED_FCOE */ diff --git a/drivers/net/ethernet/qlogic/qed/qed_hw.c b/drivers/net/ethernet/qlogic/qed/qed_hw.c index 554f30b0cfd5..6263f847b6b9 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_hw.c +++ b/drivers/net/ethernet/qlogic/qed/qed_hw.c @@ -23,7 +23,10 @@ #include "qed_reg_addr.h" #include "qed_sriov.h" -#define QED_BAR_ACQUIRE_TIMEOUT 1000 +#define QED_BAR_ACQUIRE_TIMEOUT_USLEEP_CNT 1000 +#define QED_BAR_ACQUIRE_TIMEOUT_USLEEP 1000 +#define QED_BAR_ACQUIRE_TIMEOUT_UDELAY_CNT 100000 +#define QED_BAR_ACQUIRE_TIMEOUT_UDELAY 10 /* Invalid values */ #define QED_BAR_INVALID_OFFSET (cpu_to_le32(-1)) @@ -84,12 +87,22 @@ void qed_ptt_pool_free(struct qed_hwfn *p_hwfn) } struct qed_ptt *qed_ptt_acquire(struct qed_hwfn *p_hwfn) +{ + return qed_ptt_acquire_context(p_hwfn, false); +} + +struct qed_ptt *qed_ptt_acquire_context(struct qed_hwfn *p_hwfn, bool is_atomic) { struct qed_ptt *p_ptt; - unsigned int i; + unsigned int i, count; + + if (is_atomic) + count = QED_BAR_ACQUIRE_TIMEOUT_UDELAY_CNT; + else + count = QED_BAR_ACQUIRE_TIMEOUT_USLEEP_CNT; /* Take the free PTT from the list */ - for (i = 0; i < QED_BAR_ACQUIRE_TIMEOUT; i++) { + for (i = 0; i < count; i++) { spin_lock_bh(&p_hwfn->p_ptt_pool->lock); if (!list_empty(&p_hwfn->p_ptt_pool->free_list)) { @@ -105,7 +118,12 @@ struct qed_ptt *qed_ptt_acquire(struct qed_hwfn *p_hwfn) } spin_unlock_bh(&p_hwfn->p_ptt_pool->lock); - usleep_range(1000, 2000); + + if (is_atomic) + udelay(QED_BAR_ACQUIRE_TIMEOUT_UDELAY); + else + usleep_range(QED_BAR_ACQUIRE_TIMEOUT_USLEEP, + QED_BAR_ACQUIRE_TIMEOUT_USLEEP * 2); } DP_NOTICE(p_hwfn, "PTT acquire timeout - failed to allocate PTT\n"); diff --git a/drivers/net/ethernet/qlogic/qed/qed_iscsi.c b/drivers/net/ethernet/qlogic/qed/qed_iscsi.c index 511ab214eb9c..980e7289b481 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_iscsi.c +++ b/drivers/net/ethernet/qlogic/qed/qed_iscsi.c @@ -999,13 +999,14 @@ static void _qed_iscsi_get_pstats(struct qed_hwfn *p_hwfn, } static int qed_iscsi_get_stats(struct qed_hwfn *p_hwfn, - struct qed_iscsi_stats *stats) + struct qed_iscsi_stats *stats, + bool is_atomic) { struct qed_ptt *p_ptt; memset(stats, 0, sizeof(*stats)); - p_ptt = qed_ptt_acquire(p_hwfn); + p_ptt = qed_ptt_acquire_context(p_hwfn, is_atomic); if (!p_ptt) { DP_ERR(p_hwfn, "Failed to acquire ptt\n"); return -EAGAIN; @@ -1336,9 +1337,16 @@ static int qed_iscsi_destroy_conn(struct qed_dev *cdev, QED_SPQ_MODE_EBLOCK, NULL); } +static int qed_iscsi_stats_context(struct qed_dev *cdev, + struct qed_iscsi_stats *stats, + bool is_atomic) +{ + return qed_iscsi_get_stats(QED_AFFIN_HWFN(cdev), stats, is_atomic); +} + static int qed_iscsi_stats(struct qed_dev *cdev, struct qed_iscsi_stats *stats) { - return qed_iscsi_get_stats(QED_AFFIN_HWFN(cdev), stats); + return qed_iscsi_stats_context(cdev, stats, false); } static int qed_iscsi_change_mac(struct qed_dev *cdev, @@ -1358,13 +1366,14 @@ static int qed_iscsi_change_mac(struct qed_dev *cdev, } void qed_get_protocol_stats_iscsi(struct qed_dev *cdev, - struct qed_mcp_iscsi_stats *stats) + struct qed_mcp_iscsi_stats *stats, + bool is_atomic) { struct qed_iscsi_stats proto_stats; /* Retrieve FW statistics */ memset(&proto_stats, 0, sizeof(proto_stats)); - if (qed_iscsi_stats(cdev, &proto_stats)) { + if (qed_iscsi_stats_context(cdev, &proto_stats, is_atomic)) { DP_VERBOSE(cdev, QED_MSG_STORAGE, "Failed to collect ISCSI statistics\n"); return; diff --git a/drivers/net/ethernet/qlogic/qed/qed_iscsi.h b/drivers/net/ethernet/qlogic/qed/qed_iscsi.h index dec2b00259d4..974cb8d26608 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_iscsi.h +++ b/drivers/net/ethernet/qlogic/qed/qed_iscsi.h @@ -39,11 +39,14 @@ void qed_iscsi_free(struct qed_hwfn *p_hwfn); * * @cdev: Qed dev pointer. * @stats: Points to struct that will be filled with statistics. + * @is_atomic: Hint from the caller - if the func can sleep or not. * + * Context: The function should not sleep in case is_atomic == true. * Return: Void. */ void qed_get_protocol_stats_iscsi(struct qed_dev *cdev, - struct qed_mcp_iscsi_stats *stats); + struct qed_mcp_iscsi_stats *stats, + bool is_atomic); #else /* IS_ENABLED(CONFIG_QED_ISCSI) */ static inline int qed_iscsi_alloc(struct qed_hwfn *p_hwfn) { @@ -56,7 +59,8 @@ static inline void qed_iscsi_free(struct qed_hwfn *p_hwfn) {} static inline void qed_get_protocol_stats_iscsi(struct qed_dev *cdev, - struct qed_mcp_iscsi_stats *stats) {} + struct qed_mcp_iscsi_stats *stats, + bool is_atomic) {} #endif /* IS_ENABLED(CONFIG_QED_ISCSI) */ #endif diff --git a/drivers/net/ethernet/qlogic/qed/qed_l2.c b/drivers/net/ethernet/qlogic/qed/qed_l2.c index 7776d3bdd459..970b9aabbc3d 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_l2.c +++ b/drivers/net/ethernet/qlogic/qed/qed_l2.c @@ -1863,7 +1863,8 @@ static void __qed_get_vport_stats(struct qed_hwfn *p_hwfn, } static void _qed_get_vport_stats(struct qed_dev *cdev, - struct qed_eth_stats *stats) + struct qed_eth_stats *stats, + bool is_atomic) { u8 fw_vport = 0; int i; @@ -1872,10 +1873,11 @@ static void _qed_get_vport_stats(struct qed_dev *cdev, for_each_hwfn(cdev, i) { struct qed_hwfn *p_hwfn = &cdev->hwfns[i]; - struct qed_ptt *p_ptt = IS_PF(cdev) ? qed_ptt_acquire(p_hwfn) - : NULL; + struct qed_ptt *p_ptt; bool b_get_port_stats; + p_ptt = IS_PF(cdev) ? qed_ptt_acquire_context(p_hwfn, is_atomic) + : NULL; if (IS_PF(cdev)) { /* The main vport index is relative first */ if (qed_fw_vport(p_hwfn, 0, &fw_vport)) { @@ -1900,6 +1902,13 @@ static void _qed_get_vport_stats(struct qed_dev *cdev, } void qed_get_vport_stats(struct qed_dev *cdev, struct qed_eth_stats *stats) +{ + qed_get_vport_stats_context(cdev, stats, false); +} + +void qed_get_vport_stats_context(struct qed_dev *cdev, + struct qed_eth_stats *stats, + bool is_atomic) { u32 i; @@ -1908,7 +1917,7 @@ void qed_get_vport_stats(struct qed_dev *cdev, struct qed_eth_stats *stats) return; } - _qed_get_vport_stats(cdev, stats); + _qed_get_vport_stats(cdev, stats, is_atomic); if (!cdev->reset_stats) return; @@ -1960,7 +1969,7 @@ void qed_reset_vport_stats(struct qed_dev *cdev) if (!cdev->reset_stats) { DP_INFO(cdev, "Reset stats not allocated\n"); } else { - _qed_get_vport_stats(cdev, cdev->reset_stats); + _qed_get_vport_stats(cdev, cdev->reset_stats, false); cdev->reset_stats->common.link_change_count = 0; } } diff --git a/drivers/net/ethernet/qlogic/qed/qed_l2.h b/drivers/net/ethernet/qlogic/qed/qed_l2.h index a538cf478c14..2d2f82c785ad 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_l2.h +++ b/drivers/net/ethernet/qlogic/qed/qed_l2.h @@ -249,8 +249,32 @@ qed_sp_eth_rx_queues_update(struct qed_hwfn *p_hwfn, enum spq_mode comp_mode, struct qed_spq_comp_cb *p_comp_data); +/** + * qed_get_vport_stats(): Fills provided statistics + * struct with statistics. + * + * @cdev: Qed dev pointer. + * @stats: Points to struct that will be filled with statistics. + * + * Return: Void. + */ void qed_get_vport_stats(struct qed_dev *cdev, struct qed_eth_stats *stats); +/** + * qed_get_vport_stats_context(): Fills provided statistics + * struct with statistics. + * + * @cdev: Qed dev pointer. + * @stats: Points to struct that will be filled with statistics. + * @is_atomic: Hint from the caller - if the func can sleep or not. + * + * Context: The function should not sleep in case is_atomic == true. + * Return: Void. + */ +void qed_get_vport_stats_context(struct qed_dev *cdev, + struct qed_eth_stats *stats, + bool is_atomic); + void qed_reset_vport_stats(struct qed_dev *cdev); /** diff --git a/drivers/net/ethernet/qlogic/qed/qed_main.c b/drivers/net/ethernet/qlogic/qed/qed_main.c index f5af83342856..c278f8893042 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_main.c +++ b/drivers/net/ethernet/qlogic/qed/qed_main.c @@ -3092,7 +3092,7 @@ void qed_get_protocol_stats(struct qed_dev *cdev, switch (type) { case QED_MCP_LAN_STATS: - qed_get_vport_stats(cdev, ð_stats); + qed_get_vport_stats_context(cdev, ð_stats, true); stats->lan_stats.ucast_rx_pkts = eth_stats.common.rx_ucast_pkts; stats->lan_stats.ucast_tx_pkts = @@ -3100,10 +3100,10 @@ void qed_get_protocol_stats(struct qed_dev *cdev, stats->lan_stats.fcs_err = -1; break; case QED_MCP_FCOE_STATS: - qed_get_protocol_stats_fcoe(cdev, &stats->fcoe_stats); + qed_get_protocol_stats_fcoe(cdev, &stats->fcoe_stats, true); break; case QED_MCP_ISCSI_STATS: - qed_get_protocol_stats_iscsi(cdev, &stats->iscsi_stats); + qed_get_protocol_stats_iscsi(cdev, &stats->iscsi_stats, true); break; default: DP_VERBOSE(cdev, QED_MSG_SP, -- 2.31.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/1] qed: Fix scheduling in a tasklet while getting stats 2023-07-27 15:26 ` [PATCH v2 1/1] qed: Fix scheduling in a tasklet while getting stats Konstantin Khorenko @ 2023-07-29 11:07 ` Simon Horman 0 siblings, 0 replies; 7+ messages in thread From: Simon Horman @ 2023-07-29 11:07 UTC (permalink / raw) To: Konstantin Khorenko Cc: Simon Horman, Jakub Kicinski, Manish Chopra, Ariel Elior, David Miller, Sudarsana Kalluru, netdev, Paolo Abeni On Thu, Jul 27, 2023 at 06:26:09PM +0300, Konstantin Khorenko wrote: > Here we've got to a situation when tasklet called usleep_range() in PTT > acquire logic, thus welcome to the "scheduling while atomic" BUG(). > > BUG: scheduling while atomic: swapper/24/0/0x00000100 > > [<ffffffffb41c6199>] schedule+0x29/0x70 > [<ffffffffb41c5512>] schedule_hrtimeout_range_clock+0xb2/0x150 > [<ffffffffb41c55c3>] schedule_hrtimeout_range+0x13/0x20 > [<ffffffffb41c3bcf>] usleep_range+0x4f/0x70 > [<ffffffffc08d3e58>] qed_ptt_acquire+0x38/0x100 [qed] > [<ffffffffc08eac48>] _qed_get_vport_stats+0x458/0x580 [qed] > [<ffffffffc08ead8c>] qed_get_vport_stats+0x1c/0xd0 [qed] > [<ffffffffc08dffd3>] qed_get_protocol_stats+0x93/0x100 [qed] > qed_mcp_send_protocol_stats > case MFW_DRV_MSG_GET_LAN_STATS: > case MFW_DRV_MSG_GET_FCOE_STATS: > case MFW_DRV_MSG_GET_ISCSI_STATS: > case MFW_DRV_MSG_GET_RDMA_STATS: > [<ffffffffc08e36d8>] qed_mcp_handle_events+0x2d8/0x890 [qed] > qed_int_assertion > qed_int_attentions > [<ffffffffc08d9490>] qed_int_sp_dpc+0xa50/0xdc0 [qed] > [<ffffffffb3aa7623>] tasklet_action+0x83/0x140 > [<ffffffffb41d9125>] __do_softirq+0x125/0x2bb > [<ffffffffb41d560c>] call_softirq+0x1c/0x30 > [<ffffffffb3a30645>] do_softirq+0x65/0xa0 > [<ffffffffb3aa78d5>] irq_exit+0x105/0x110 > [<ffffffffb41d8996>] do_IRQ+0x56/0xf0 > > Fix this by making caller to provide the context whether it could be in > atomic context flow or not when getting stats from QED driver. > QED driver based on the context provided decide to schedule out or not > when acquiring the PTT BAR window. > > We faced the BUG_ON() while getting vport stats, but according to the > code same issue could happen for fcoe and iscsi statistics as well, so > fixing them too. > > Fixes: 6c75424612a7 ("qed: Add support for NCSI statistics.") > Fixes: 1e128c81290a ("qed: Add support for hardware offloaded FCoE.") > Fixes: 2f2b2614e893 ("qed: Provide iSCSI statistics to management") > Cc: Sudarsana Kalluru <skalluru@marvell.com> > Cc: David Miller <davem@davemloft.net> > Cc: Manish Chopra <manishc@marvell.com> > nit: no blank line here. > Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com> Reviewed-by: Simon Horman <horms@kernel.org> ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 0/1] qed: Yet another scheduling while atomic fix 2023-07-27 15:26 ` [PATCH v2 0/1] qed: Yet another scheduling while atomic fix Konstantin Khorenko 2023-07-27 15:26 ` [PATCH v2 1/1] qed: Fix scheduling in a tasklet while getting stats Konstantin Khorenko @ 2023-07-29 16:20 ` patchwork-bot+netdevbpf 1 sibling, 0 replies; 7+ messages in thread From: patchwork-bot+netdevbpf @ 2023-07-29 16:20 UTC (permalink / raw) To: Konstantin Khorenko Cc: simon.horman, kuba, manishc, aelior, davem, skalluru, netdev, pabeni Hello: This patch was applied to netdev/net.git (main) by David S. Miller <davem@davemloft.net>: On Thu, 27 Jul 2023 18:26:08 +0300 you wrote: > Running an old RHEL7-based kernel we have got several cases of following > BUG_ON(): > > BUG: scheduling while atomic: swapper/24/0/0x00000100 > > [<ffffffffb41c6199>] schedule+0x29/0x70 > [<ffffffffb41c5512>] schedule_hrtimeout_range_clock+0xb2/0x150 > [<ffffffffb41c55c3>] schedule_hrtimeout_range+0x13/0x20 > [<ffffffffb41c3bcf>] usleep_range+0x4f/0x70 > [<ffffffffc08d3e58>] qed_ptt_acquire+0x38/0x100 [qed] > [<ffffffffc08eac48>] _qed_get_vport_stats+0x458/0x580 [qed] > [<ffffffffc08ead8c>] qed_get_vport_stats+0x1c/0xd0 [qed] > [<ffffffffc08dffd3>] qed_get_protocol_stats+0x93/0x100 [qed] > qed_mcp_send_protocol_stats > case MFW_DRV_MSG_GET_LAN_STATS: > case MFW_DRV_MSG_GET_FCOE_STATS: > case MFW_DRV_MSG_GET_ISCSI_STATS: > case MFW_DRV_MSG_GET_RDMA_STATS: > [<ffffffffc08e36d8>] qed_mcp_handle_events+0x2d8/0x890 [qed] > qed_int_assertion > qed_int_attentions > [<ffffffffc08d9490>] qed_int_sp_dpc+0xa50/0xdc0 [qed] > [<ffffffffb3aa7623>] tasklet_action+0x83/0x140 > [<ffffffffb41d9125>] __do_softirq+0x125/0x2bb > [<ffffffffb41d560c>] call_softirq+0x1c/0x30 > [<ffffffffb3a30645>] do_softirq+0x65/0xa0 > [<ffffffffb3aa78d5>] irq_exit+0x105/0x110 > [<ffffffffb41d8996>] do_IRQ+0x56/0xf0 > > [...] Here is the summary with links: - [v2,1/1] qed: Fix scheduling in a tasklet while getting stats https://git.kernel.org/netdev/net/c/e346e231b42b You are awesome, thank you! -- Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-07-29 16:20 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-07-26 17:19 [PATCH 0/1] qed: Yet another scheduling while atomic fix Konstantin Khorenko 2023-07-26 17:19 ` [PATCH 1/1] qed: Fix scheduling in a tasklet while getting stats Konstantin Khorenko 2023-07-27 11:59 ` Simon Horman 2023-07-27 15:26 ` [PATCH v2 0/1] qed: Yet another scheduling while atomic fix Konstantin Khorenko 2023-07-27 15:26 ` [PATCH v2 1/1] qed: Fix scheduling in a tasklet while getting stats Konstantin Khorenko 2023-07-29 11:07 ` Simon Horman 2023-07-29 16:20 ` [PATCH v2 0/1] qed: Yet another scheduling while atomic fix patchwork-bot+netdevbpf
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).