From mboxrd@z Thu Jan 1 00:00:00 1970 From: Song Liu Subject: Re: pegged softirq and NAPI race (?) Date: Tue, 18 Sep 2018 16:19:13 +0000 Message-ID: <6FA4008E-CEEB-4EAB-BAD8-267D41574248@fb.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Cc: netdev , Jeff Kirsher , Alexander Duyck , "michael.chan@broadcom.com" , Kernel Team To: Eric Dumazet Return-path: Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:48314 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729739AbeIRVw6 (ORCPT ); Tue, 18 Sep 2018 17:52:58 -0400 In-Reply-To: Content-Language: en-US Content-ID: <78C03B6D3C0570479FA907FE3B1204C5@namprd15.prod.outlook.com> Sender: netdev-owner@vger.kernel.org List-ID: > On Sep 18, 2018, at 6:45 AM, Eric Dumazet wrote: >=20 > On Tue, Sep 18, 2018 at 1:41 AM Song Liu wrote: >>=20 >> We are debugging this issue that netconsole message triggers pegged soft= irq >> (ksoftirqd taking 100% CPU for many seconds). We found this issue in >> production with both bnxt and ixgbe, on a 4.11 based kernel. This is eas= ily >> reproducible with ixgbe on 4.11, and latest net/net-next (see [1] for mo= re >> detail). >>=20 >> After debugging for some time, we found that this issue is likely relate= d >> to 39e6c8208d7b ("net: solve a NAPI race"). After reverting this commit, >> the steps described in [1] cannot reproduce the issue on ixgbe. Revertin= g >> this commit also reduces the chances we hit the issue with bnxt (it stil= l >> happens with a lower rate). >>=20 >> I tried to fix this issue with relaxed variant (or older version) of >> napi_schedule_prep() in netpoll, just like the one on napi_watchdog(). >> However, my tests do not always go as expected. >>=20 >> Please share your comments/suggestions on which direction shall we try >> to fix this. >>=20 >> Thanks in advance! >> Song >>=20 >>=20 >> [1] https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.spinics.n= et_lists_netdev_msg522328.html&d=3DDwIBaQ&c=3D5VD0RTtNlTh3ycd41b3MUw&r=3Di6= WobKxbeG3slzHSIOxTVtYIJw7qjCE6S0spDTKL-J4&m=3DiSaOapj1kxjhGYLgQr0Qd8mQCzVdo= bmgT1L4JwFvzxs&s=3DlCEhrz6wQJUUaJOkxFmtOszAgkf3Jh4reX_i1GbI5RI&e=3D >=20 > You have not traced ixgbe to understand why driver hits > "clean_complete=3Dfalse" all the time ? The trace showed that we got "clean_complete=3Dfalse" because=20 ixgbe_clean_rx_irq() used all budget (64). It feels like the driver is tricked to process old data on the rx_ring for one more time.=20 Have you seen similar issue? Thanks, Song=