public inbox for stable@vger.kernel.org
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Eric Dumazet <edumazet@google.com>,
	"David S. Miller" <davem@davemloft.net>
Subject: [PATCH 3.18 07/36] Fix an intermittent pr_emerg warning about lo becoming free.
Date: Mon,  3 Jul 2017 15:34:04 +0200	[thread overview]
Message-ID: <20170703133256.608537557@linuxfoundation.org> (raw)
In-Reply-To: <20170703133256.260692013@linuxfoundation.org>

3.18-stable review patch.  If anyone has any objections, please let me know.

------------------

From: Krister Johansen <kjlx@templeofstupid.com>


[ Upstream commit f186ce61bb8235d80068c390dc2aad7ca427a4c2 ]

It looks like this:

Message from syslogd@flamingo at Apr 26 00:45:00 ...
 kernel:unregister_netdevice: waiting for lo to become free. Usage count = 4

They seem to coincide with net namespace teardown.

The message is emitted by netdev_wait_allrefs().

Forced a kdump in netdev_run_todo, but found that the refcount on the lo
device was already 0 at the time we got to the panic.

Used bcc to check the blocking in netdev_run_todo.  The only places
where we're off cpu there are in the rcu_barrier() and msleep() calls.
That behavior is expected.  The msleep time coincides with the amount of
time we spend waiting for the refcount to reach zero; the rcu_barrier()
wait times are not excessive.

After looking through the list of callbacks that the netdevice notifiers
invoke in this path, it appears that the dst_dev_event is the most
interesting.  The dst_ifdown path places a hold on the loopback_dev as
part of releasing the dev associated with the original dst cache entry.
Most of our notifier callbacks are straight-forward, but this one a)
looks complex, and b) places a hold on the network interface in
question.

I constructed a new bcc script that watches various events in the
liftime of a dst cache entry.  Note that dst_ifdown will take a hold on
the loopback device until the invalidated dst entry gets freed.

[      __dst_free] on DST: ffff883ccabb7900 IF tap1008300eth0 invoked at 1282115677036183
    __dst_free
    rcu_nocb_kthread
    kthread
    ret_from_fork
Acked-by: Eric Dumazet <edumazet@google.com>

Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
 net/core/dst.c |   14 ++++++++++++++
 1 file changed, 14 insertions(+)

--- a/net/core/dst.c
+++ b/net/core/dst.c
@@ -397,6 +397,20 @@ static int dst_dev_event(struct notifier
 		spin_lock_bh(&dst_garbage.lock);
 		dst = dst_garbage.list;
 		dst_garbage.list = NULL;
+		/* The code in dst_ifdown places a hold on the loopback device.
+		 * If the gc entry processing is set to expire after a lengthy
+		 * interval, this hold can cause netdev_wait_allrefs() to hang
+		 * out and wait for a long time -- until the the loopback
+		 * interface is released.  If we're really unlucky, it'll emit
+		 * pr_emerg messages to console too.  Reset the interval here,
+		 * so dst cleanups occur in a more timely fashion.
+		 */
+		if (dst_garbage.timer_inc > DST_GC_INC) {
+			dst_garbage.timer_inc = DST_GC_INC;
+			dst_garbage.timer_expires = DST_GC_MIN;
+			mod_delayed_work(system_wq, &dst_gc_work,
+					 dst_garbage.timer_expires);
+		}
 		spin_unlock_bh(&dst_garbage.lock);
 
 		if (last)

  parent reply	other threads:[~2017-07-03 13:36 UTC|newest]

Thread overview: 36+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-07-03 13:33 [PATCH 3.18 00/36] 3.18.60-stable review Greg Kroah-Hartman
2017-07-03 13:33 ` [PATCH 3.18 01/36] xhci: fix deadlock at host remove by running watchdog correctly Greg Kroah-Hartman
2017-07-03 13:33 ` [PATCH 3.18 02/36] ipv6: release dst on error in ip6_dst_lookup_tail Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 03/36] netfilter: xt_TCPMSS: add more sanity tests on tcph->doff Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 04/36] netfilter: synproxy: fix conntrackd interaction Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 05/36] net: dont call strlen on non-terminated string in dev_set_alias() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 06/36] decnet: dn_rtmsg: Improve input length sanitization in dnrmg_receive_user_skb Greg Kroah-Hartman
2017-07-03 13:34 ` Greg Kroah-Hartman [this message]
2017-07-03 13:34 ` [PATCH 3.18 08/36] net: caif: Fix a sleep-in-atomic bug in cfpkt_create_pfx Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 09/36] igmp: acquire pmc lock for ip_mc_clear_src() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 10/36] igmp: add a missing spin_lock_init() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 11/36] ipv6: fix calling in6_ifa_hold incorrectly for dad work Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 12/36] decnet: always not take dst->__refcnt when inserting dst into hash table Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 13/36] net: 8021q: Fix one possible panic caused by BUG_ON in free_netdev Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 14/36] NFSv4: fix a reference leak caused WARNING messages Greg Kroah-Hartman
2017-07-03 14:33   ` Trond Myklebust
2017-07-03 15:02     ` gregkh
2017-07-03 13:34 ` [PATCH 3.18 15/36] arm64: cpuinfo: Missing NULL terminator in compat_hwcap_str Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 16/36] MIPS: Avoid accidental raw backtrace Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 17/36] MIPS: pm-cps: Drop manual cache-line alignment of ready_count Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 18/36] MIPS: Fix IRQ tracing & lockdep when rescheduling Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 19/36] ALSA: hda - set input_path bitmap to zero after moving it to new place Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 20/36] drm/vmwgfx: Free hash table allocated by cmdbuf managed res mgr Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 21/36] usb: gadget: f_fs: Fix possibe deadlock Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 22/36] sysctl: enable strict writes Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 23/36] mm: numa: avoid waiting on freed migrated pages Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 25/36] net: korina: Fix NAPI versus resources freeing Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 27/36] xfrm: fix stack access out of bounds with CONFIG_XFRM_SUB_POLICY Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 28/36] xfrm: NULL dereference on allocation failure Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 29/36] xfrm: Oops on error in pfkey_msg2xfrm_state() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 30/36] watchdog: bcm281xx: Fix use of uninitialized spinlock Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 31/36] ARM: 8685/1: ensure memblock-limit is pmd-aligned Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 32/36] iommu/vt-d: Dont over-free page table directories Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 33/36] iommu/amd: Fix incorrect error handling in amd_iommu_bind_pasid() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 3.18 34/36] cpufreq: s3c2416: double free on driver init error path Greg Kroah-Hartman
2017-07-03 19:34 ` [PATCH 3.18 00/36] 3.18.60-stable review Guenter Roeck

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170703133256.608537557@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=davem@davemloft.net \
    --cc=edumazet@google.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox