* lockdep problem? netlink_dump & rtnl_lock
@ 2015-11-18 10:11 Arkadiusz Miskiewicz
2015-11-18 17:58 ` Cong Wang
0 siblings, 1 reply; 2+ messages in thread
From: Arkadiusz Miskiewicz @ 2015-11-18 10:11 UTC (permalink / raw)
To: netdev
Hi.
I'm running 4.3.0 kernel (+ few backported mm patches). Machine is doing many rsnapshots
in parallel (so heavily loaded).
I'm seeing this from time to time (below). What would be the reason for so long hangs?
[ 2278.385668] INFO: task snmpd:1775 blocked for more than 120 seconds.
[ 2278.392089] Not tainted 4.3.0-0.1 #1
[ 2278.396226] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2278.404111] snmpd D ffff88081fc158c0 0 1775 1 0x00000000
[ 2278.411229] ffff8807e6a8bcf8 0000000000000082 ffffffff81c13500 ffff8807ebb88d80
[ 2278.418753] ffff8807e6a8c000 ffff8807ebb88d80 ffffffff81cc7c84 00000000ffffffff
[ 2278.426286] ffffffff81cc7c88 ffff8807e6a8bd10 ffffffff8161c203 ffffffff81cc7c80
[ 2278.433854] Call Trace:
[ 2278.436341] [<ffffffff8161c203>] schedule+0x33/0x80
[ 2278.441349] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
[ 2278.447733] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
[ 2278.453956] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
[ 2278.459133] [<ffffffff81523f15>] rtnl_lock+0x15/0x20
[ 2278.464212] [<ffffffff8158b902>] devinet_ioctl+0xd2/0x6e0
[ 2278.469757] [<ffffffff81208c14>] ? mntput+0x24/0x40
[ 2278.474769] [<ffffffff811ebe62>] ? __fput+0x142/0x200
[ 2278.479955] [<ffffffff8158d5a0>] inet_ioctl+0x50/0x70
[ 2278.485122] [<ffffffff814f6e05>] sock_do_ioctl+0x25/0x50
[ 2278.490578] [<ffffffff814f7322>] sock_ioctl+0x1d2/0x290
[ 2278.495920] [<ffffffff811fc3f5>] do_vfs_ioctl+0x295/0x470
[ 2278.501444] [<ffffffff814f6cd6>] ? sock_alloc_file+0xa6/0x130
[ 2278.507338] [<ffffffff81090eb4>] ? task_work_add+0x44/0x60
[ 2278.512942] [<ffffffff811ebff7>] ? fput+0x47/0x90
[ 2278.517771] [<ffffffff811fc649>] SyS_ioctl+0x79/0x90
[ 2278.522845] [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75
[ 2278.529341] INFO: task sudo:4534 blocked for more than 120 seconds.
[ 2278.535635] Not tainted 4.3.0-0.1 #1
[ 2278.539759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2278.547680] sudo D 0000000000000000 0 4534 1 0x00000004
[ 2278.554825] ffff880080677bb8 0000000000000086 ffff8807f3874380 ffff8807f3eb8000
[ 2278.562389] ffff880080678000 ffff8807f3eb8000 ffffffff81cc7c84 00000000ffffffff
[ 2278.569914] ffffffff81cc7c88 ffff880080677bd0 ffffffff8161c203 ffffffff81cc7c80
[ 2278.577436] Call Trace:
[ 2278.579911] [<ffffffff8161c203>] schedule+0x33/0x80
[ 2278.584918] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
[ 2278.591329] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
[ 2278.597571] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
[ 2278.602743] [<ffffffff8154746e>] __netlink_dump_start+0x5e/0x190
[ 2278.608890] [<ffffffff81526100>] ? rtnl_fill_ifinfo+0x13f0/0x13f0
[ 2278.615105] [<ffffffff81526eac>] rtnetlink_rcv_msg+0x1bc/0x230
[ 2278.621054] [<ffffffff81526100>] ? rtnl_fill_ifinfo+0x13f0/0x13f0
[ 2278.627262] [<ffffffff81526cf0>] ? rtnetlink_rcv+0x30/0x30
[ 2278.632861] [<ffffffff815498d4>] netlink_rcv_skb+0xa4/0xc0
[ 2278.638463] [<ffffffff81526ce8>] rtnetlink_rcv+0x28/0x30
[ 2278.643908] [<ffffffff81548fc7>] netlink_unicast+0x127/0x1a0
[ 2278.649692] [<ffffffff81549604>] netlink_sendmsg+0x4f4/0x5f0
[ 2278.655480] [<ffffffff814f8bd8>] sock_sendmsg+0x38/0x50
[ 2278.660830] [<ffffffff814fa2a1>] SyS_sendto+0x101/0x150
[ 2278.666189] [<ffffffff814f6cd6>] ? sock_alloc_file+0xa6/0x130
[ 2278.672064] [<ffffffff81206715>] ? fd_install+0x25/0x30
[ 2278.677407] [<ffffffff814f985f>] ? SyS_socket+0x9f/0xf0
[ 2278.682749] [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75
[ 7433.492510] INFO: task bird:1503 blocked for more than 120 seconds.
[ 7433.498819] Not tainted 4.3.0-0.1 #1
[ 7433.502954] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 7433.510840] bird D ffff88081fd158c0 0 1503 1 0x00000000
[ 7433.517966] ffff8807eb923bb8 0000000000000086 ffff8807fb711b00 ffff8807f38a4380
[ 7433.525493] ffff8807eb924000 ffff8807f38a4380 ffffffff81cc7c84 00000000ffffffff
[ 7433.533017] ffffffff81cc7c88 ffff8807eb923bd0 ffffffff8161c203 ffffffff81cc7c80
[ 7433.540561] Call Trace:
[ 7433.543050] [<ffffffff8161c203>] schedule+0x33/0x80
[ 7433.548041] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
[ 7433.554415] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
[ 7433.560624] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
[ 7433.565810] [<ffffffff81546be1>] netlink_dump+0x21/0x2c0
[ 7433.571236] [<ffffffff815470c9>] netlink_recvmsg+0x249/0x3d0
[ 7433.577005] [<ffffffff814f8eeb>] sock_recvmsg+0x3b/0x50
[ 7433.582347] [<ffffffff814f96ab>] ___sys_recvmsg+0xdb/0x1f0
[ 7433.587952] [<ffffffff810146ec>] ? __switch_to+0x25c/0x4a0
[ 7433.593579] [<ffffffff810a6f17>] ? update_curr+0xb7/0x130
[ 7433.599115] [<ffffffff810a354c>] ? __enqueue_entity+0x6c/0x70
[ 7433.604970] [<ffffffff810a44f1>] ? set_next_entity+0x71/0x840
[ 7433.610832] [<ffffffff810146ec>] ? __switch_to+0x25c/0x4a0
[ 7433.616431] [<ffffffff814fa9b1>] __sys_recvmsg+0x51/0x90
[ 7433.621873] [<ffffffff814faa02>] SyS_recvmsg+0x12/0x20
[ 7433.627125] [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75
[ 7433.633642] INFO: task kworker/3:4:12895 blocked for more than 120 seconds.
[ 7433.640651] Not tainted 4.3.0-0.1 #1
[ 7433.644797] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 7433.652682] kworker/3:4 D 0000000000000000 0 12895 2 0x00000000
[ 7433.659794] Workqueue: ipv6_addrconf addrconf_verify_work
[ 7433.665240] ffff8803b0073d58 0000000000000046 ffff880781adb600 ffff880736aa3600
[ 7433.672763] ffff8803b0074000 ffff880736aa3600 ffffffff81cc7c84 00000000ffffffff
[ 7433.680287] ffffffff81cc7c88 ffff8803b0073d70 ffffffff8161c203 ffffffff81cc7c80
[ 7433.687811] Call Trace:
[ 7433.690288] [<ffffffff8161c203>] schedule+0x33/0x80
[ 7433.695297] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
[ 7433.701688] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
[ 7433.707923] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
[ 7433.713093] [<ffffffff81523f15>] rtnl_lock+0x15/0x20
[ 7433.718168] [<ffffffff815c903e>] addrconf_verify_work+0xe/0x20
[ 7433.724114] [<ffffffff8108cc6f>] process_one_work+0x14f/0x430
[ 7433.730006] [<ffffffff8108cf9b>] worker_thread+0x4b/0x4b0
[ 7433.735517] [<ffffffff8161bb71>] ? __schedule+0x2a1/0x900
[ 7433.741031] [<ffffffff8108cf50>] ? process_one_work+0x430/0x430
[ 7433.747083] [<ffffffff8108cf50>] ? process_one_work+0x430/0x430
[ 7433.753131] [<ffffffff81092be8>] kthread+0xd8/0xf0
[ 7433.758063] [<ffffffff81092b10>] ? kthread_worker_fn+0x170/0x170
[ 7433.764204] [<ffffffff8161fc0f>] ret_from_fork+0x3f/0x70
[ 7433.769657] [<ffffffff81092b10>] ? kthread_worker_fn+0x170/0x170
config:
http://sprunge.us/BBLN
# ethtool -i eth0
driver: igb
version: 5.3.0-k
firmware-version: 3.16, 0x800004d7
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
# lspci -v
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
Subsystem: Super Micro Computer Inc Device 0804
Flags: bus master, fast devsel, latency 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
Kernel driver in use: ie31200_edac
Kernel modules: ie31200_edac
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Capabilities: [88] Subsystem: Super Micro Computer Inc Device 0804
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [a0] Express Root Port (Slot+), MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [140] Root Complex Link
Capabilities: [d94] #19
Kernel driver in use: pcieport
Kernel modules: shpchp
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: f7100000-f72fffff
Capabilities: [88] Subsystem: Super Micro Computer Inc Device 0804
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [a0] Express Root Port (Slot+), MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [140] Root Complex Link
Capabilities: [d94] #19
Kernel driver in use: pcieport
Kernel modules: shpchp
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) (prog-if 30 [XHCI])
Subsystem: Super Micro Computer Inc Device 0804
Flags: bus master, medium devsel, latency 0, IRQ 34
Memory at f7500000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) (prog-if 20 [EHCI])
Subsystem: Super Micro Computer Inc Device 0804
Flags: bus master, medium devsel, latency 0, IRQ 16
Memory at f7513000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Debug port: BAR=1 offset=00a0
Capabilities: [98] PCI Advanced Features
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=03, subordinate=04, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: f6000000-f70fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804
Capabilities: [a0] Power Management version 3
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: f7400000-f74fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804
Capabilities: [a0] Power Management version 3
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
I/O behind bridge: 0000b000-0000bfff
Memory behind bridge: f7300000-f73fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804
Capabilities: [a0] Power Management version 3
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) (prog-if 20 [EHCI])
Subsystem: Super Micro Computer Inc Device 0804
Flags: bus master, medium devsel, latency 0, IRQ 23
Memory at f7512000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Capabilities: [58] Debug port: BAR=1 offset=00a0
Capabilities: [98] PCI Advanced Features
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05)
Subsystem: Super Micro Computer Inc Device 0804
Flags: bus master, medium devsel, latency 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
Kernel driver in use: lpc_ich
Kernel modules: lpc_ich
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) (prog-if 01 [AHCI 1.0])
Subsystem: Super Micro Computer Inc Device 0804
Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 29
I/O ports at f050 [size=8]
I/O ports at f040 [size=4]
I/O ports at f030 [size=8]
I/O ports at f020 [size=4]
I/O ports at f000 [size=32]
Memory at f7511000 (32-bit, non-prefetchable) [size=2K]
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [70] Power Management version 3
Capabilities: [a8] SATA HBA v1.0
Kernel driver in use: ahci
Kernel modules: ahci
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
Subsystem: Super Micro Computer Inc Device 0804
Flags: medium devsel, IRQ 18
Memory at f7510000 (64-bit, non-prefetchable) [size=256]
I/O ports at 0580 [size=32]
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
Subsystem: Super Micro Computer Inc Device 0804
Flags: bus master, fast devsel, latency 0, IRQ 10
Memory at dfff00000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
Subsystem: Super Micro Computer Inc Device 0691
Flags: bus master, fast devsel, latency 0
I/O ports at e000 [size=256]
Memory at f7240000 (64-bit, non-prefetchable) [size=64K]
Memory at f7200000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at f7100000 [disabled] [size=1M]
Capabilities: [50] Power Management version 3
Capabilities: [68] Express Endpoint, MSI 00
Capabilities: [d0] Vital Product Data
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [c0] MSI-X: Enable+ Count=16 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [1e0] #19
Capabilities: [1c0] Power Budgeting <?>
Capabilities: [190] #16
Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
Kernel driver in use: mpt2sas
Kernel modules: mpt2sas
03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=03, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: f6000000-f70fffff
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
Capabilities: [a4] Subsystem: Super Micro Computer Inc Device 0804
Capabilities: [100] Virtual Channel
Kernel modules: shpchp
04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30) (prog-if 00 [VGA controller])
Subsystem: Super Micro Computer Inc Device 0804
Flags: bus master, medium devsel, latency 0, IRQ 11
Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
Memory at f7000000 (32-bit, non-prefetchable) [size=128K]
I/O ports at d000 [size=128]
Expansion ROM at <unassigned> [disabled]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
Subsystem: Super Micro Computer Inc Device 1533
Flags: bus master, fast devsel, latency 0
Memory at f7400000 (32-bit, non-prefetchable) [size=512K]
I/O ports at c000 [size=32]
Memory at f7480000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-25-90-ff-ff-87-5d-0e
Capabilities: [1a0] Transaction Processing Hints
Kernel driver in use: igb
Kernel modules: igb
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
Subsystem: Super Micro Computer Inc Device 1533
Flags: bus master, fast devsel, latency 0
Memory at f7300000 (32-bit, non-prefetchable) [size=512K]
I/O ports at b000 [size=32]
Memory at f7380000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-25-90-ff-ff-87-5d-0f
Capabilities: [1a0] Transaction Processing Hints
Kernel driver in use: igb
Kernel modules: igb
--
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: lockdep problem? netlink_dump & rtnl_lock
2015-11-18 10:11 lockdep problem? netlink_dump & rtnl_lock Arkadiusz Miskiewicz
@ 2015-11-18 17:58 ` Cong Wang
0 siblings, 0 replies; 2+ messages in thread
From: Cong Wang @ 2015-11-18 17:58 UTC (permalink / raw)
To: arekm; +Cc: netdev
On Wed, Nov 18, 2015 at 2:11 AM, Arkadiusz Miskiewicz
<a.miskiewicz@gmail.com> wrote:
>
> Hi.
>
> I'm running 4.3.0 kernel (+ few backported mm patches). Machine is doing many rsnapshots
> in parallel (so heavily loaded).
>
> I'm seeing this from time to time (below). What would be the reason for so long hangs?
It is not a lockdep problem, it is that someone was holding that mutex
for a long time or that mutex was not released correctly.
Can you do a sysrq-t when you reproduce this problem?
Thanks!
>
> [ 2278.385668] INFO: task snmpd:1775 blocked for more than 120 seconds.
> [ 2278.392089] Not tainted 4.3.0-0.1 #1
> [ 2278.396226] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 2278.404111] snmpd D ffff88081fc158c0 0 1775 1 0x00000000
> [ 2278.411229] ffff8807e6a8bcf8 0000000000000082 ffffffff81c13500 ffff8807ebb88d80
> [ 2278.418753] ffff8807e6a8c000 ffff8807ebb88d80 ffffffff81cc7c84 00000000ffffffff
> [ 2278.426286] ffffffff81cc7c88 ffff8807e6a8bd10 ffffffff8161c203 ffffffff81cc7c80
> [ 2278.433854] Call Trace:
> [ 2278.436341] [<ffffffff8161c203>] schedule+0x33/0x80
> [ 2278.441349] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
> [ 2278.447733] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
> [ 2278.453956] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
> [ 2278.459133] [<ffffffff81523f15>] rtnl_lock+0x15/0x20
> [ 2278.464212] [<ffffffff8158b902>] devinet_ioctl+0xd2/0x6e0
> [ 2278.469757] [<ffffffff81208c14>] ? mntput+0x24/0x40
> [ 2278.474769] [<ffffffff811ebe62>] ? __fput+0x142/0x200
> [ 2278.479955] [<ffffffff8158d5a0>] inet_ioctl+0x50/0x70
> [ 2278.485122] [<ffffffff814f6e05>] sock_do_ioctl+0x25/0x50
> [ 2278.490578] [<ffffffff814f7322>] sock_ioctl+0x1d2/0x290
> [ 2278.495920] [<ffffffff811fc3f5>] do_vfs_ioctl+0x295/0x470
> [ 2278.501444] [<ffffffff814f6cd6>] ? sock_alloc_file+0xa6/0x130
> [ 2278.507338] [<ffffffff81090eb4>] ? task_work_add+0x44/0x60
> [ 2278.512942] [<ffffffff811ebff7>] ? fput+0x47/0x90
> [ 2278.517771] [<ffffffff811fc649>] SyS_ioctl+0x79/0x90
> [ 2278.522845] [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75
> [ 2278.529341] INFO: task sudo:4534 blocked for more than 120 seconds.
> [ 2278.535635] Not tainted 4.3.0-0.1 #1
> [ 2278.539759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 2278.547680] sudo D 0000000000000000 0 4534 1 0x00000004
> [ 2278.554825] ffff880080677bb8 0000000000000086 ffff8807f3874380 ffff8807f3eb8000
> [ 2278.562389] ffff880080678000 ffff8807f3eb8000 ffffffff81cc7c84 00000000ffffffff
> [ 2278.569914] ffffffff81cc7c88 ffff880080677bd0 ffffffff8161c203 ffffffff81cc7c80
> [ 2278.577436] Call Trace:
> [ 2278.579911] [<ffffffff8161c203>] schedule+0x33/0x80
> [ 2278.584918] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
> [ 2278.591329] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
> [ 2278.597571] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
> [ 2278.602743] [<ffffffff8154746e>] __netlink_dump_start+0x5e/0x190
> [ 2278.608890] [<ffffffff81526100>] ? rtnl_fill_ifinfo+0x13f0/0x13f0
> [ 2278.615105] [<ffffffff81526eac>] rtnetlink_rcv_msg+0x1bc/0x230
> [ 2278.621054] [<ffffffff81526100>] ? rtnl_fill_ifinfo+0x13f0/0x13f0
> [ 2278.627262] [<ffffffff81526cf0>] ? rtnetlink_rcv+0x30/0x30
> [ 2278.632861] [<ffffffff815498d4>] netlink_rcv_skb+0xa4/0xc0
> [ 2278.638463] [<ffffffff81526ce8>] rtnetlink_rcv+0x28/0x30
> [ 2278.643908] [<ffffffff81548fc7>] netlink_unicast+0x127/0x1a0
> [ 2278.649692] [<ffffffff81549604>] netlink_sendmsg+0x4f4/0x5f0
> [ 2278.655480] [<ffffffff814f8bd8>] sock_sendmsg+0x38/0x50
> [ 2278.660830] [<ffffffff814fa2a1>] SyS_sendto+0x101/0x150
> [ 2278.666189] [<ffffffff814f6cd6>] ? sock_alloc_file+0xa6/0x130
> [ 2278.672064] [<ffffffff81206715>] ? fd_install+0x25/0x30
> [ 2278.677407] [<ffffffff814f985f>] ? SyS_socket+0x9f/0xf0
> [ 2278.682749] [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75
>
>
>
>
>
>
>
>
> [ 7433.492510] INFO: task bird:1503 blocked for more than 120 seconds.
> [ 7433.498819] Not tainted 4.3.0-0.1 #1
> [ 7433.502954] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 7433.510840] bird D ffff88081fd158c0 0 1503 1 0x00000000
> [ 7433.517966] ffff8807eb923bb8 0000000000000086 ffff8807fb711b00 ffff8807f38a4380
> [ 7433.525493] ffff8807eb924000 ffff8807f38a4380 ffffffff81cc7c84 00000000ffffffff
> [ 7433.533017] ffffffff81cc7c88 ffff8807eb923bd0 ffffffff8161c203 ffffffff81cc7c80
> [ 7433.540561] Call Trace:
> [ 7433.543050] [<ffffffff8161c203>] schedule+0x33/0x80
> [ 7433.548041] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
> [ 7433.554415] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
> [ 7433.560624] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
> [ 7433.565810] [<ffffffff81546be1>] netlink_dump+0x21/0x2c0
> [ 7433.571236] [<ffffffff815470c9>] netlink_recvmsg+0x249/0x3d0
> [ 7433.577005] [<ffffffff814f8eeb>] sock_recvmsg+0x3b/0x50
> [ 7433.582347] [<ffffffff814f96ab>] ___sys_recvmsg+0xdb/0x1f0
> [ 7433.587952] [<ffffffff810146ec>] ? __switch_to+0x25c/0x4a0
> [ 7433.593579] [<ffffffff810a6f17>] ? update_curr+0xb7/0x130
> [ 7433.599115] [<ffffffff810a354c>] ? __enqueue_entity+0x6c/0x70
> [ 7433.604970] [<ffffffff810a44f1>] ? set_next_entity+0x71/0x840
> [ 7433.610832] [<ffffffff810146ec>] ? __switch_to+0x25c/0x4a0
> [ 7433.616431] [<ffffffff814fa9b1>] __sys_recvmsg+0x51/0x90
> [ 7433.621873] [<ffffffff814faa02>] SyS_recvmsg+0x12/0x20
> [ 7433.627125] [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75
> [ 7433.633642] INFO: task kworker/3:4:12895 blocked for more than 120 seconds.
> [ 7433.640651] Not tainted 4.3.0-0.1 #1
> [ 7433.644797] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 7433.652682] kworker/3:4 D 0000000000000000 0 12895 2 0x00000000
> [ 7433.659794] Workqueue: ipv6_addrconf addrconf_verify_work
> [ 7433.665240] ffff8803b0073d58 0000000000000046 ffff880781adb600 ffff880736aa3600
> [ 7433.672763] ffff8803b0074000 ffff880736aa3600 ffffffff81cc7c84 00000000ffffffff
> [ 7433.680287] ffffffff81cc7c88 ffff8803b0073d70 ffffffff8161c203 ffffffff81cc7c80
> [ 7433.687811] Call Trace:
> [ 7433.690288] [<ffffffff8161c203>] schedule+0x33/0x80
> [ 7433.695297] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10
> [ 7433.701688] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120
> [ 7433.707923] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30
> [ 7433.713093] [<ffffffff81523f15>] rtnl_lock+0x15/0x20
> [ 7433.718168] [<ffffffff815c903e>] addrconf_verify_work+0xe/0x20
> [ 7433.724114] [<ffffffff8108cc6f>] process_one_work+0x14f/0x430
> [ 7433.730006] [<ffffffff8108cf9b>] worker_thread+0x4b/0x4b0
> [ 7433.735517] [<ffffffff8161bb71>] ? __schedule+0x2a1/0x900
> [ 7433.741031] [<ffffffff8108cf50>] ? process_one_work+0x430/0x430
> [ 7433.747083] [<ffffffff8108cf50>] ? process_one_work+0x430/0x430
> [ 7433.753131] [<ffffffff81092be8>] kthread+0xd8/0xf0
> [ 7433.758063] [<ffffffff81092b10>] ? kthread_worker_fn+0x170/0x170
> [ 7433.764204] [<ffffffff8161fc0f>] ret_from_fork+0x3f/0x70
> [ 7433.769657] [<ffffffff81092b10>] ? kthread_worker_fn+0x170/0x170
>
> config:
> http://sprunge.us/BBLN
>
> # ethtool -i eth0
> driver: igb
> version: 5.3.0-k
> firmware-version: 3.16, 0x800004d7
> bus-info: 0000:05:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> # lspci -v
> 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
> Subsystem: Super Micro Computer Inc Device 0804
> Flags: bus master, fast devsel, latency 0
> Capabilities: [e0] Vendor Specific Information: Len=0c <?>
> Kernel driver in use: ie31200_edac
> Kernel modules: ie31200_edac
>
> 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
> Capabilities: [88] Subsystem: Super Micro Computer Inc Device 0804
> Capabilities: [80] Power Management version 3
> Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
> Capabilities: [a0] Express Root Port (Slot+), MSI 00
> Capabilities: [100] Virtual Channel
> Capabilities: [140] Root Complex Link
> Capabilities: [d94] #19
> Kernel driver in use: pcieport
> Kernel modules: shpchp
>
> 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
> I/O behind bridge: 0000e000-0000efff
> Memory behind bridge: f7100000-f72fffff
> Capabilities: [88] Subsystem: Super Micro Computer Inc Device 0804
> Capabilities: [80] Power Management version 3
> Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
> Capabilities: [a0] Express Root Port (Slot+), MSI 00
> Capabilities: [100] Virtual Channel
> Capabilities: [140] Root Complex Link
> Capabilities: [d94] #19
> Kernel driver in use: pcieport
> Kernel modules: shpchp
>
> 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) (prog-if 30 [XHCI])
> Subsystem: Super Micro Computer Inc Device 0804
> Flags: bus master, medium devsel, latency 0, IRQ 34
> Memory at f7500000 (64-bit, non-prefetchable) [size=64K]
> Capabilities: [70] Power Management version 2
> Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
> Kernel driver in use: xhci_hcd
> Kernel modules: xhci_pci
>
> 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) (prog-if 20 [EHCI])
> Subsystem: Super Micro Computer Inc Device 0804
> Flags: bus master, medium devsel, latency 0, IRQ 16
> Memory at f7513000 (32-bit, non-prefetchable) [size=1K]
> Capabilities: [50] Power Management version 2
> Capabilities: [58] Debug port: BAR=1 offset=00a0
> Capabilities: [98] PCI Advanced Features
> Kernel driver in use: ehci-pci
> Kernel modules: ehci_pci
>
> 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=03, subordinate=04, sec-latency=0
> I/O behind bridge: 0000d000-0000dfff
> Memory behind bridge: f6000000-f70fffff
> Capabilities: [40] Express Root Port (Slot+), MSI 00
> Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
> Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804
> Capabilities: [a0] Power Management version 3
> Kernel driver in use: pcieport
> Kernel modules: shpchp
>
> 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
> I/O behind bridge: 0000c000-0000cfff
> Memory behind bridge: f7400000-f74fffff
> Capabilities: [40] Express Root Port (Slot+), MSI 00
> Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
> Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804
> Capabilities: [a0] Power Management version 3
> Kernel driver in use: pcieport
> Kernel modules: shpchp
>
> 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
> I/O behind bridge: 0000b000-0000bfff
> Memory behind bridge: f7300000-f73fffff
> Capabilities: [40] Express Root Port (Slot+), MSI 00
> Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
> Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804
> Capabilities: [a0] Power Management version 3
> Kernel driver in use: pcieport
> Kernel modules: shpchp
>
> 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) (prog-if 20 [EHCI])
> Subsystem: Super Micro Computer Inc Device 0804
> Flags: bus master, medium devsel, latency 0, IRQ 23
> Memory at f7512000 (32-bit, non-prefetchable) [size=1K]
> Capabilities: [50] Power Management version 2
> Capabilities: [58] Debug port: BAR=1 offset=00a0
> Capabilities: [98] PCI Advanced Features
> Kernel driver in use: ehci-pci
> Kernel modules: ehci_pci
>
> 00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05)
> Subsystem: Super Micro Computer Inc Device 0804
> Flags: bus master, medium devsel, latency 0
> Capabilities: [e0] Vendor Specific Information: Len=0c <?>
> Kernel driver in use: lpc_ich
> Kernel modules: lpc_ich
>
> 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) (prog-if 01 [AHCI 1.0])
> Subsystem: Super Micro Computer Inc Device 0804
> Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 29
> I/O ports at f050 [size=8]
> I/O ports at f040 [size=4]
> I/O ports at f030 [size=8]
> I/O ports at f020 [size=4]
> I/O ports at f000 [size=32]
> Memory at f7511000 (32-bit, non-prefetchable) [size=2K]
> Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
> Capabilities: [70] Power Management version 3
> Capabilities: [a8] SATA HBA v1.0
> Kernel driver in use: ahci
> Kernel modules: ahci
>
> 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
> Subsystem: Super Micro Computer Inc Device 0804
> Flags: medium devsel, IRQ 18
> Memory at f7510000 (64-bit, non-prefetchable) [size=256]
> I/O ports at 0580 [size=32]
> Kernel driver in use: i801_smbus
> Kernel modules: i2c_i801
>
> 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
> Subsystem: Super Micro Computer Inc Device 0804
> Flags: bus master, fast devsel, latency 0, IRQ 10
> Memory at dfff00000 (64-bit, non-prefetchable) [size=4K]
> Capabilities: [50] Power Management version 3
> Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
>
> 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
> Subsystem: Super Micro Computer Inc Device 0691
> Flags: bus master, fast devsel, latency 0
> I/O ports at e000 [size=256]
> Memory at f7240000 (64-bit, non-prefetchable) [size=64K]
> Memory at f7200000 (64-bit, non-prefetchable) [size=256K]
> Expansion ROM at f7100000 [disabled] [size=1M]
> Capabilities: [50] Power Management version 3
> Capabilities: [68] Express Endpoint, MSI 00
> Capabilities: [d0] Vital Product Data
> Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
> Capabilities: [c0] MSI-X: Enable+ Count=16 Masked-
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [1e0] #19
> Capabilities: [1c0] Power Budgeting <?>
> Capabilities: [190] #16
> Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
> Kernel driver in use: mpt2sas
> Kernel modules: mpt2sas
>
> 03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=03, secondary=04, subordinate=04, sec-latency=0
> I/O behind bridge: 0000d000-0000dfff
> Memory behind bridge: f6000000-f70fffff
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
> Capabilities: [78] Power Management version 3
> Capabilities: [80] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
> Capabilities: [a4] Subsystem: Super Micro Computer Inc Device 0804
> Capabilities: [100] Virtual Channel
> Kernel modules: shpchp
>
> 04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30) (prog-if 00 [VGA controller])
> Subsystem: Super Micro Computer Inc Device 0804
> Flags: bus master, medium devsel, latency 0, IRQ 11
> Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
> Memory at f7000000 (32-bit, non-prefetchable) [size=128K]
> I/O ports at d000 [size=128]
> Expansion ROM at <unassigned> [disabled]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
>
> 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
> Subsystem: Super Micro Computer Inc Device 1533
> Flags: bus master, fast devsel, latency 0
> Memory at f7400000 (32-bit, non-prefetchable) [size=512K]
> I/O ports at c000 [size=32]
> Memory at f7480000 (32-bit, non-prefetchable) [size=16K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
> Capabilities: [a0] Express Endpoint, MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Device Serial Number 00-25-90-ff-ff-87-5d-0e
> Capabilities: [1a0] Transaction Processing Hints
> Kernel driver in use: igb
> Kernel modules: igb
>
> 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
> Subsystem: Super Micro Computer Inc Device 1533
> Flags: bus master, fast devsel, latency 0
> Memory at f7300000 (32-bit, non-prefetchable) [size=512K]
> I/O ports at b000 [size=32]
> Memory at f7380000 (32-bit, non-prefetchable) [size=16K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
> Capabilities: [a0] Express Endpoint, MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Device Serial Number 00-25-90-ff-ff-87-5d-0f
> Capabilities: [1a0] Transaction Processing Hints
> Kernel driver in use: igb
> Kernel modules: igb
>
>
> --
> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2015-11-18 17:58 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-18 10:11 lockdep problem? netlink_dump & rtnl_lock Arkadiusz Miskiewicz
2015-11-18 17:58 ` Cong Wang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).