From mboxrd@z Thu Jan 1 00:00:00 1970 From: Song Liu Subject: pegged softirq and NAPI race (?) Date: Tue, 18 Sep 2018 08:41:36 +0000 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Cc: "edumazet@google.com" , "jeffrey.t.kirsher@intel.com" , "alexander.h.duyck@intel.com" , Michael Chan , Kernel Team To: Networking Return-path: Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:46350 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729238AbeIRONU (ORCPT ); Tue, 18 Sep 2018 10:13:20 -0400 Content-Language: en-US Content-ID: <02EF574EEFC10844906860014DE94DB0@namprd15.prod.outlook.com> Sender: netdev-owner@vger.kernel.org List-ID: We are debugging this issue that netconsole message triggers pegged softirq (ksoftirqd taking 100% CPU for many seconds). We found this issue in production with both bnxt and ixgbe, on a 4.11 based kernel. This is easily reproducible with ixgbe on 4.11, and latest net/net-next (see [1] for more detail). After debugging for some time, we found that this issue is likely related to 39e6c8208d7b ("net: solve a NAPI race"). After reverting this commit, the steps described in [1] cannot reproduce the issue on ixgbe. Reverting this commit also reduces the chances we hit the issue with bnxt (it still happens with a lower rate). I tried to fix this issue with relaxed variant (or older version) of napi_schedule_prep() in netpoll, just like the one on napi_watchdog(). However, my tests do not always go as expected. Please share your comments/suggestions on which direction shall we try to fix this. Thanks in advance! Song [1] https://www.spinics.net/lists/netdev/msg522328.html