From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: pegged softirq and NAPI race (?) Date: Tue, 18 Sep 2018 09:33:29 -0700 Message-ID: References: <6FA4008E-CEEB-4EAB-BAD8-267D41574248@fb.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Cc: netdev , Jeff Kirsher , Alexander Duyck , michael.chan@broadcom.com, kernel-team To: songliubraving@fb.com Return-path: Received: from mail-it0-f65.google.com ([209.85.214.65]:36869 "EHLO mail-it0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729819AbeIRWHF (ORCPT ); Tue, 18 Sep 2018 18:07:05 -0400 Received: by mail-it0-f65.google.com with SMTP id h20-v6so3836139itf.2 for ; Tue, 18 Sep 2018 09:33:43 -0700 (PDT) In-Reply-To: <6FA4008E-CEEB-4EAB-BAD8-267D41574248@fb.com> Sender: netdev-owner@vger.kernel.org List-ID: On Tue, Sep 18, 2018 at 9:19 AM Song Liu wrote: > > > > > On Sep 18, 2018, at 6:45 AM, Eric Dumazet wrote: > > > > On Tue, Sep 18, 2018 at 1:41 AM Song Liu wrote: > >> > >> We are debugging this issue that netconsole message triggers pegged so= ftirq > >> (ksoftirqd taking 100% CPU for many seconds). We found this issue in > >> production with both bnxt and ixgbe, on a 4.11 based kernel. This is e= asily > >> reproducible with ixgbe on 4.11, and latest net/net-next (see [1] for = more > >> detail). > >> > >> After debugging for some time, we found that this issue is likely rela= ted > >> to 39e6c8208d7b ("net: solve a NAPI race"). After reverting this commi= t, > >> the steps described in [1] cannot reproduce the issue on ixgbe. Revert= ing > >> this commit also reduces the chances we hit the issue with bnxt (it st= ill > >> happens with a lower rate). > >> > >> I tried to fix this issue with relaxed variant (or older version) of > >> napi_schedule_prep() in netpoll, just like the one on napi_watchdog(). > >> However, my tests do not always go as expected. > >> > >> Please share your comments/suggestions on which direction shall we try > >> to fix this. > >> > >> Thanks in advance! > >> Song > >> > >> > >> [1] https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.spinics= .net_lists_netdev_msg522328.html&d=3DDwIBaQ&c=3D5VD0RTtNlTh3ycd41b3MUw&r=3D= i6WobKxbeG3slzHSIOxTVtYIJw7qjCE6S0spDTKL-J4&m=3DiSaOapj1kxjhGYLgQr0Qd8mQCzV= dobmgT1L4JwFvzxs&s=3DlCEhrz6wQJUUaJOkxFmtOszAgkf3Jh4reX_i1GbI5RI&e=3D > > > > You have not traced ixgbe to understand why driver hits > > "clean_complete=3Dfalse" all the time ? > > The trace showed that we got "clean_complete=3Dfalse" because > ixgbe_clean_rx_irq() used all budget (64). It feels like the driver > is tricked to process old data on the rx_ring for one more time. Process old data ???? That would be quite an horrible bug ! Probably ASAN would help here, detecting use-after-free or things like that= .