All of lore.kernel.org
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Eric Dumazet <edumazet@google.com>,
	"David S. Miller" <davem@davemloft.net>
Subject: [PATCH 4.4 006/101] Fix an intermittent pr_emerg warning about lo becoming free.
Date: Mon,  3 Jul 2017 15:34:06 +0200	[thread overview]
Message-ID: <20170703133335.497480770@linuxfoundation.org> (raw)
In-Reply-To: <20170703133334.237346187@linuxfoundation.org>

4.4-stable review patch.  If anyone has any objections, please let me know.

------------------

From: Krister Johansen <kjlx@templeofstupid.com>


[ Upstream commit f186ce61bb8235d80068c390dc2aad7ca427a4c2 ]

It looks like this:

Message from syslogd@flamingo at Apr 26 00:45:00 ...
 kernel:unregister_netdevice: waiting for lo to become free. Usage count = 4

They seem to coincide with net namespace teardown.

The message is emitted by netdev_wait_allrefs().

Forced a kdump in netdev_run_todo, but found that the refcount on the lo
device was already 0 at the time we got to the panic.

Used bcc to check the blocking in netdev_run_todo.  The only places
where we're off cpu there are in the rcu_barrier() and msleep() calls.
That behavior is expected.  The msleep time coincides with the amount of
time we spend waiting for the refcount to reach zero; the rcu_barrier()
wait times are not excessive.

After looking through the list of callbacks that the netdevice notifiers
invoke in this path, it appears that the dst_dev_event is the most
interesting.  The dst_ifdown path places a hold on the loopback_dev as
part of releasing the dev associated with the original dst cache entry.
Most of our notifier callbacks are straight-forward, but this one a)
looks complex, and b) places a hold on the network interface in
question.

I constructed a new bcc script that watches various events in the
liftime of a dst cache entry.  Note that dst_ifdown will take a hold on
the loopback device until the invalidated dst entry gets freed.

[      __dst_free] on DST: ffff883ccabb7900 IF tap1008300eth0 invoked at 1282115677036183
    __dst_free
    rcu_nocb_kthread
    kthread
    ret_from_fork
Acked-by: Eric Dumazet <edumazet@google.com>

Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
 net/core/dst.c |   14 ++++++++++++++
 1 file changed, 14 insertions(+)

--- a/net/core/dst.c
+++ b/net/core/dst.c
@@ -462,6 +462,20 @@ static int dst_dev_event(struct notifier
 		spin_lock_bh(&dst_garbage.lock);
 		dst = dst_garbage.list;
 		dst_garbage.list = NULL;
+		/* The code in dst_ifdown places a hold on the loopback device.
+		 * If the gc entry processing is set to expire after a lengthy
+		 * interval, this hold can cause netdev_wait_allrefs() to hang
+		 * out and wait for a long time -- until the the loopback
+		 * interface is released.  If we're really unlucky, it'll emit
+		 * pr_emerg messages to console too.  Reset the interval here,
+		 * so dst cleanups occur in a more timely fashion.
+		 */
+		if (dst_garbage.timer_inc > DST_GC_INC) {
+			dst_garbage.timer_inc = DST_GC_INC;
+			dst_garbage.timer_expires = DST_GC_MIN;
+			mod_delayed_work(system_wq, &dst_gc_work,
+					 dst_garbage.timer_expires);
+		}
 		spin_unlock_bh(&dst_garbage.lock);
 
 		if (last)

  parent reply	other threads:[~2017-07-03 15:10 UTC|newest]

Thread overview: 106+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-07-03 13:34 [PATCH 4.4 000/101] 4.4.76-stable review Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 001/101] ipv6: release dst on error in ip6_dst_lookup_tail Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 002/101] net: dont call strlen on non-terminated string in dev_set_alias() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 003/101] decnet: dn_rtmsg: Improve input length sanitization in dnrmg_receive_user_skb Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 004/101] net: Zero ifla_vf_info in rtnl_fill_vfinfo() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 005/101] af_unix: Add sockaddr length checks before accessing sa_family in bind and connect handlers Greg Kroah-Hartman
2017-07-03 13:34 ` Greg Kroah-Hartman [this message]
2017-07-03 13:34 ` [PATCH 4.4 007/101] net: caif: Fix a sleep-in-atomic bug in cfpkt_create_pfx Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 008/101] igmp: acquire pmc lock for ip_mc_clear_src() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 009/101] igmp: add a missing spin_lock_init() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 010/101] ipv6: fix calling in6_ifa_hold incorrectly for dad work Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 011/101] net/mlx5: Wait for FW readiness before initializing command interface Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 012/101] decnet: always not take dst->__refcnt when inserting dst into hash table Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 013/101] net: 8021q: Fix one possible panic caused by BUG_ON in free_netdev Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 014/101] sfc: provide dummy definitions of vswitch functions Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 015/101] ipv6: Do not leak throw route references Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 016/101] rtnetlink: add IFLA_GROUP to ifla_policy Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 017/101] netfilter: xt_TCPMSS: add more sanity tests on tcph->doff Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 018/101] netfilter: synproxy: fix conntrackd interaction Greg Kroah-Hartman
2017-08-17  5:57   ` Stefan Bader
2017-08-17 16:47     ` Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 019/101] NFSv4: fix a reference leak caused WARNING messages Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 020/101] drm/ast: Handle configuration without P2A bridge Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 021/101] mm, swap_cgroup: reschedule when neeed in swap_cgroup_swapoff() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 022/101] MIPS: Avoid accidental raw backtrace Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 023/101] MIPS: pm-cps: Drop manual cache-line alignment of ready_count Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 024/101] MIPS: Fix IRQ tracing & lockdep when rescheduling Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 025/101] ALSA: hda - Fix endless loop of codec configure Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 026/101] ALSA: hda - set input_path bitmap to zero after moving it to new place Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 027/101] drm/vmwgfx: Free hash table allocated by cmdbuf managed res mgr Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 028/101] usb: gadget: f_fs: Fix possibe deadlock Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 029/101] sysctl: enable strict writes Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 030/101] block: fix module reference leak on put_disk() call for cgroups throttle Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 031/101] mm: numa: avoid waiting on freed migrated pages Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 033/101] scsi: sd: Fix wrong DPOFUA disable in sd_read_cache_type Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 034/101] scsi: lpfc: Set elsiocb contexts to NULL after freeing it Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 035/101] qla2xxx: Fix erroneous invalid handle message Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 037/101] net: mvneta: Fix for_each_present_cpu usage Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 038/101] MIPS: ath79: fix regression in PCI window initialization Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 039/101] net: korina: Fix NAPI versus resources freeing Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 040/101] MIPS: ralink: MT7688 pinmux fixes Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 041/101] MIPS: ralink: fix USB frequency scaling Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 042/101] MIPS: ralink: Fix invalid assignment of SoC type Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 043/101] MIPS: ralink: fix MT7628 pinmux typos Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 044/101] MIPS: ralink: fix MT7628 wled_an pinmux gpio Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 046/101] bgmac: fix a missing check for build_skb Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 048/101] bgmac: Fix reversed test of build_skb() return value Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 049/101] net: bgmac: Fix SOF bit checking Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 050/101] net: bgmac: Start transmit queue in bgmac_open Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 051/101] net: bgmac: Remove superflous netif_carrier_on() Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 052/101] powerpc/eeh: Enable IO path on permanent error Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 053/101] gianfar: Do not reuse pages from emergency reserve Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 054/101] Btrfs: fix truncate down when no_holes feature is enabled Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 055/101] virtio_console: fix a crash in config_work_handler Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 056/101] swiotlb-xen: update dev_addr after swapping pages Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 057/101] net: sctp: fix array overrun read on sctp_timer_tbl Greg Kroah-Hartman
2017-07-04 18:48   ` Ben Hutchings
2017-07-05 12:17     ` Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 058/101] xen-netfront: Fix Rx stall during network stress and OOM Greg Kroah-Hartman
2017-07-03 13:34 ` [PATCH 4.4 059/101] scsi: virtio_scsi: Reject commands when virtqueue is broken Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 060/101] platform/x86: ideapad-laptop: handle ACPI event 1 Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 061/101] amd-xgbe: Check xgbe_init() return code Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 062/101] net: dsa: Check return value of phy_connect_direct() Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 064/101] vfio/spapr: fail tce_iommu_attach_group() when iommu_data is null Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 065/101] virtio_net: fix PAGE_SIZE > 64k Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 066/101] vxlan: do not age static remote mac entries Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 067/101] ibmveth: Add a proper check for the availability of the checksum features Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 068/101] kernel/panic.c: add missing \n Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 069/101] HID: i2c-hid: Add sleep between POWER ON and RESET Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 070/101] scsi: lpfc: avoid double free of resource identifiers Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 071/101] spi: davinci: use dma_mapping_error() Greg Kroah-Hartman
2017-07-05 14:24   ` Ben Hutchings
2018-04-06  8:21     ` Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 072/101] arm64: assembler: make adr_l work in modules under KASLR Greg Kroah-Hartman
2017-07-04  9:24   ` Ard Biesheuvel
2017-07-04  9:29     ` Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 073/101] mac80211: initialize SMPS field in HT capabilities Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 074/101] x86/mpx: Use compatible types in comparison to fix sparse error Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 075/101] coredump: Ensure proper size of sparse core files Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 076/101] swiotlb: ensure that page-sized mappings are page-aligned Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 077/101] s390/ctl_reg: make __ctl_load a full memory barrier Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 078/101] be2net: fix status check in be_cmd_pmac_add() Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 079/101] perf probe: Fix to show correct locations for events on modules Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 080/101] net/mlx4_core: Eliminate warning messages for SRQ_LIMIT under SRIOV Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 081/101] sctp: check af before verify address in sctp_addr_id2transport Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 082/101] ravb: Fix use-after-free on `ifconfig eth0 down` Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 083/101] jump label: fix passing kbuild_cflags when checking for asm goto support Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 084/101] xfrm: fix stack access out of bounds with CONFIG_XFRM_SUB_POLICY Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 085/101] xfrm: NULL dereference on allocation failure Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 086/101] xfrm: Oops on error in pfkey_msg2xfrm_state() Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 087/101] watchdog: bcm281xx: Fix use of uninitialized spinlock Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 088/101] sched/loadavg: Avoid loadavg spikes caused by delayed NO_HZ accounting Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 089/101] ARM64/ACPI: Fix BAD_MADT_GICC_ENTRY() macro implementation Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 090/101] ARM: 8685/1: ensure memblock-limit is pmd-aligned Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 091/101] x86/mpx: Correctly report do_mpx_bt_fault() failures to user-space Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 092/101] x86/mm: Fix flush_tlb_page() on Xen Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 093/101] ocfs2: o2hb: revert hb threshold to keep compatible Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 094/101] iommu/vt-d: Dont over-free page table directories Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 095/101] iommu: Handle default domain attach failure Greg Kroah-Hartman
2017-07-05 18:52   ` Ben Hutchings
2017-07-03 13:35 ` [PATCH 4.4 096/101] iommu/amd: Fix incorrect error handling in amd_iommu_bind_pasid() Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 097/101] cpufreq: s3c2416: double free on driver init error path Greg Kroah-Hartman
2017-07-03 13:35 ` [PATCH 4.4 098/101] KVM: x86: fix emulation of RSM and IRET instructions Greg Kroah-Hartman
2017-07-03 19:37 ` [PATCH 4.4 000/101] 4.4.76-stable review Guenter Roeck
2017-07-04  8:00   ` Greg Kroah-Hartman
     [not found] ` <595aa84b.4eec1c0a.8c9e2.e137@mx.google.com>
2017-07-04  8:02   ` Greg Kroah-Hartman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170703133335.497480770@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=davem@davemloft.net \
    --cc=edumazet@google.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.