From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: WARN_ON in TLP causing RT throttling Date: Wed, 26 Sep 2018 17:09:12 -0700 Message-ID: <3c112d25-1105-524f-e210-8a7cb63df1c7@gmail.com> References: <7aa9932a59aad7a21c7f8a8146dd0542@codeaurora.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Cc: "soheil@google.com" To: stranche@codeaurora.org, eric.dumazet@gmail.com Return-path: Received: from mail-lj1-f195.google.com ([209.85.208.195]:34315 "EHLO mail-lj1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726469AbeI0GYq (ORCPT ); Thu, 27 Sep 2018 02:24:46 -0400 Received: by mail-lj1-f195.google.com with SMTP id f8-v6so706443ljk.1 for ; Wed, 26 Sep 2018 17:09:17 -0700 (PDT) In-Reply-To: <7aa9932a59aad7a21c7f8a8146dd0542@codeaurora.org> Content-Language: en-US Sender: netdev-owner@vger.kernel.org List-ID: On 09/26/2018 04:46 PM, stranche@codeaurora.org wrote: > Hi Eric, > > Someone recently reported a crash to us on the 4.14.62 kernel where excessive > WARNING prints were spamming the logs and causing watchdog bites. The kernel > does have the following commit by Soheil: > bffd168c3fc5 "tcp: clear tp->packets_out when purging write queue" > > Before this bug we see over 1 second of continuous WARN_ON prints from > tcp_send_loss_probe() like so: > > 7795.530450:   <2>  tcp_send_loss_probe+0x194/0x1b8 > 7795.534833:   <2>  tcp_write_timer_handler+0xf8/0x1c4 > 7795.539492:   <2>  tcp_write_timer+0x4c/0x74 > 7795.543348:   <2>  call_timer_fn+0xc0/0x1b4 > 7795.547113:   <2>  run_timer_softirq+0x248/0x81c > > Specifically, the prints come from the following check: > >     /* Retransmit last segment. */ >     if (WARN_ON(!skb)) >         goto rearm_timer; > > Since skb is always NULL, we know there's nothing on the write queue or the > retransmit queue, so we just keep resetting the timer, waiting for more data > to be queued. However, we were able to determine that the TCP socket is in the > TCP_FIN_WAIT1 state, so we will no longer be sending any data and these queues > remain empty. > > Would it be appropriate to stop resetting the TLP timer if we detect that the > connection is starting to close and we have no more data to send the probe with, > or is there some way that this scenario should already be handled? > > Unfortunately, we don't have a reproducer for this crash. > Something is fishy. If there is no skb in the queues, then tp->packets_out should be 0, therefore tcp_rearm_rto() should simply call inet_csk_clear_xmit_timer(sk, ICSK_TIME_RETRANS); I have never seen this report before.